Infrared Thermal Image Analysis Device, Infrared Thermal Image Analysis Method, and Program

Abstract

Provided are an infrared thermal image analysis device, an infrared thermal image analysis method, and a program that can properly reduce a temperature gradient. An infrared thermal image analysis device includes a processor that acquires a first infrared thermal image of a structure surface obtained by imaging a structure to be inspected, acquires area information of at least one area for distinguishing an area of the structure surface corresponding to the first infrared thermal image, and estimates a temperature gradient in the at least one area on the basis of the area information and a second infrared thermal image, to reduce effects of the temperature gradient from the first infrared thermal image.

Description

Technical Field

[0001] The present invention relates to an infrared thermal image analysis device, an infrared thermal image analysis method, and a program.

Background Art

[0002] There is known a technique for discriminating damaged parts such as voids and cracks and healthy parts included in a structure by using an infrared thermal image obtained when photographing the structure such as concrete with an infrared camera. When there is a damaged part in the structure, a temperature difference occurs between the surface temperature of the damaged part and the surface temperature of the healthy part. Therefore, if there is a portion having a temperature different from that of the surroundings locally in the infrared thermal image, it can be determined that there is a damaged part inside that portion.

[0003] On the other hand, due to partial differences in the amount of heat received or radiated on the surface of the structure depending on the shape of the structure, the environment around the structure, etc., a temperature gradient may occur in the surface temperature of the structure. When such a temperature gradient exists on the surface of the structure, it becomes difficult to discriminate between healthy parts and damaged parts. In addition, the partial difference in the amount of heat received or radiated on the surface of the structure can also be caused by partial differences in the color, roughness, unevenness, thermal conductivity, and emissivity of the surface of the structure in addition to the shape of the structure and the environment around the structure.

[0004] Regarding this problem, Patent Document 1 discloses creating an average temperature distribution image obtained by moving averaging an infrared thermal image with a predetermined number of pixels, calculating the temperature difference between the same pixels of the infrared thermal image and the average temperature distribution image to create a temperature difference image, and displaying the temperature difference image.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] By the way, when a structure has a plurality of surfaces with different inclinations, on the surface of the structure, due to the influence of solar radiation or the like, the average temperature, the direction and inclination of the temperature gradient are different for each surface. If the infrared thermal image is smoothed and differentiated without distinguishing each surface with different average temperatures and temperature gradients, the boundary between different surfaces will be erroneously detected as a damaged part. In addition, since other surfaces are included in the pixel range used for smoothing, there is a problem that the temperature gradient cannot be correctly reduced.

[0007] The present invention has been made in view of such circumstances, and an object thereof is to provide an infrared thermal image analysis apparatus, an infrared thermal image analysis method, and a program capable of correctly reducing a temperature gradient.

Means for Solving the Problems

[0008] An infrared thermal image analysis apparatus according to a first aspect is an infrared thermal image analysis apparatus including a processor, wherein the processor acquires a first infrared thermal image of the surface of a structure obtained by photographing the structure to be inspected, acquires region information for distinguishing a region of the surface of the structure corresponding to the first infrared thermal image for at least one region, estimates a temperature gradient in at least one region based on the region information and a second infrared thermal image, and reduces the influence of the temperature gradient from the first infrared thermal image.

[0009] In the infrared thermal image analysis apparatus according to a second aspect, the processor acquires region information based on information regarding the structure including a visible image obtained by photographing the structure.

[0010] In the infrared thermal image analysis apparatus according to a third aspect, the processor acquires region information based on information regarding the structure including at least one of a first infrared thermal image and a second infrared thermal image obtained by photographing the structure.

[0011] In the infrared thermal image analysis apparatus according to a fourth aspect, the processor acquires region information based on information regarding the structure including data obtained by measuring the distance to the structure.

[0012] In the infrared thermal image analysis device according to the fifth aspect, the processor acquires region information based on information regarding the structure including the drawing data of the structure.

[0013] In the infrared thermal image analysis device according to the sixth aspect, the processor preferentially applies the second infrared thermal image in the region over the second infrared thermal images in other regions, and estimates the temperature gradient in the region.

[0014] In the infrared thermal image analysis device according to the seventh aspect, in the preferential application, the processor smoothes the second infrared thermal image with different weightings for the region and other regions.

[0015] In the infrared thermal image analysis device according to the eighth aspect, in the preferential application, the processor extends along the boundary of the region in a range not including other regions, and smoothes the second infrared thermal image.

[0016] In the infrared thermal image analysis device according to the ninth aspect, the processor preferentially applies the second infrared thermal image in the peripheral region of the region, and estimates the temperature gradient in the region.

[0017] In the infrared thermal image analysis device according to the tenth aspect, the peripheral region is a region that includes at least pixels whose distance to the boundary is 1 / 2 or less of the distance from the center to the boundary when the smallest distance among the distances from the pixels in the region to each pixel on the boundary of the region is taken as the distance from each pixel to the boundary, and the distance at the pixel with the largest distance to the boundary is taken as the distance from the center to the boundary.

[0018] In the infrared thermal image analysis device according to the eleventh aspect, the processor estimates the temperature gradient in the region by thermal simulation.

[0019] In the infrared thermal image analysis device according to the twelfth aspect, when reducing the influence of the temperature gradient, the processor subtracts the temperature gradient from the first infrared thermal image or divides the first infrared thermal image by the temperature gradient.

[0020] In the infrared thermal image analysis apparatus according to the 13th aspect, at least one of the first infrared thermal image, the second infrared thermal image, and the information on the structure based on the region information is acquired at different timings.

[0021] In the infrared thermal image analysis apparatus according to the 14th aspect, at least one of the first infrared thermal image, the second infrared thermal image, and the information on the structure based on the region information is an image or information obtained by integrating a plurality of images or information.

[0022] In the infrared thermal image analysis apparatus according to the 15th aspect, the processor acquires the first infrared thermal image and the second infrared thermal image at a timing when there is solar radiation.

[0023] In the infrared thermal image analysis apparatus according to the 16th aspect, the surface of the structure includes at least one of a plurality of surfaces with different inclinations or discontinuous surfaces.

[0024] In the infrared thermal image analysis apparatus according to the 17th aspect, the processor displays on the display device a temperature gradient reduction image with the influence of the temperature gradient reduced from the first infrared thermal image.

[0025] In the infrared thermal image analysis apparatus according to the 18th aspect, the processor displays on the display device the temperature gradient reduction image that has been subjected to image processing.

[0026] In the infrared thermal image analysis apparatus according to the 19th aspect, the first infrared thermal image and the second infrared thermal image are the same infrared thermal image.

[0027] The infrared thermal image analysis method according to the 20th aspect includes acquiring a first infrared thermal image of the surface of a structure that has photographed the structure to be inspected, acquiring region information for distinguishing regions of the surface of the structure corresponding to the first infrared thermal image for at least one region, estimating a temperature gradient in at least one region based on the region information and a second infrared thermal image, and reducing the influence of the temperature gradient from the first infrared thermal image.

[0028] A program for causing a computer to execute according to the 21st aspect acquires a first infrared thermal image of the surface of a structure obtained by photographing the structure to be inspected, acquires region information for distinguishing regions of the surface of the structure corresponding to the first infrared thermal image for at least one region, estimates a temperature gradient in at least one region based on the region information and a second infrared thermal image, and causes the computer to reduce the influence of the temperature gradient from the first infrared thermal image.

Advantages of the Invention

[0029] According to the infrared thermal image analysis apparatus, infrared thermal image analysis method, and program of the present invention, the temperature gradient can be correctly reduced.

Brief Description of the Drawings

[0030]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0031] Hereinafter, preferred embodiments of an infrared thermal image analysis apparatus, Infrared thermal image analysis a method, and a program according to the present invention will be described with reference to the accompanying drawings. In the specification, the temperature distribution is a temperature difference (temperature change) caused by a healthy part and a damaged part, and the temperature gradient is a temperature difference (temperature change) that is not a temperature distribution.

[0032] As described above, when an infrared thermal image is smoothed and differentiated, the boundary between different surfaces is erroneously detected as a damaged part, and since the pixel range used for smoothing includes other surfaces, there is a problem that the temperature gradient cannot be correctly reduced. The present inventor has found this problem and arrived at the present invention.

[0033] In addition, differences in the average temperature and temperature gradient on each surface of a structure can occur not only when the structure is directly exposed to solar radiation but also when it is indirectly exposed. Also, it can occur due to receiving reflected light or radiant light of any light including infrared rays from the periphery of the structure (that is, it can occur regardless of day or night). Further, when the surface of the structure has a plurality of regions with different colors, or roughness, or unevenness, or thermal conductivity or emissivity, differences in the amount of heat received or radiated in each region can similarly cause differences in the average temperature and temperature gradient in each region, and similar problems can occur. Also, when there are regions on the surface of the structure that are exposed to solar radiation or other light and regions that are not, differences in the amount of heat received in each region can similarly cause differences in the average temperature and temperature gradient in each region, and similar problems can occur. Moreover, even if the surface of the structure is uniform, if there are locally different amounts of heat received or radiated due to steps, dents, discontinuities, other members, etc. in a part of it, different temperature gradients in different directions can occur starting from there, and similar problems can occur. Also, when the structure has a plurality of surfaces with different inclinations, since the infrared thermal radiation from the three-dimensional structure surface is projected two-dimensionally onto the imaging direction of the infrared camera and the resulting image is an infrared thermal image, the direction and inclination of the temperature gradient on each surface of the structure surface in the infrared thermal image naturally differ according to the angle between the structure surface and the imaging direction, and the average temperature can also differ. Therefore, if the infrared thermal image is smoothed and differentiated without distinguishing those surfaces, the above problems will occur. Naturally, there are also surfaces that are in the shade of other surfaces. In that case, since three-dimensionally discontinuous surfaces are adjacent and continuous in the infrared thermal image, if the infrared thermal image is smoothed and differentiated without distinguishing those surfaces, the above problems will occur.

[0034] As described above, when the structure has surfaces with a plurality of different inclinations, discontinuous surfaces (such as surfaces that become shaded, steps, recesses, discontinuities, and each surface separated by a separate member, etc.), a plurality of different colors, roughness, unevenness, regions with different thermal conductivities or emissivities, regions irradiated by sunlight or other light and regions not irradiated, etc., differences in the average temperature and temperature gradient in each region may occur. Therefore, if the infrared thermal image is smoothed and differentiated without distinguishing these regions, the boundaries of each region may be misdetected as damaged parts, and the problem that the temperature gradient cannot be correctly reduced may occur. This problem can occur regardless of day or night. The inventor has found the above and has further reached each embodiment.

[0035] [Hardware Configuration of Infrared Thermal Image Analysis Device] FIG. 1 is a block diagram showing an example of the hardware configuration of an infrared thermal image analysis device according to an embodiment.

[0036] As the infrared thermal image analysis device 10 shown in FIG. 1, a computer or a workstation can be used. The infrared thermal image analysis device 10 of this example mainly includes an input / output interface 12, a storage unit 16, an operation unit 18, a CPU (Central Processing Unit) 20, a RAM (Random Access Memory) 22, a ROM (Read Only Memory) 24, and a display control unit 26. A display device 30 is connected to the infrared thermal image analysis device 10, and under the command of the CPU 20, a display is performed on the display device 30 under the control of the display control unit 26. The display device 30 is configured by, for example, a monitor.

[0037] The input / output interface 12 (input / output I / F in the figure) can input various data (information) to the infrared thermal image analysis device 10. For example, the data stored in the storage unit 16 is input via the input / output interface 12.

[0038] The CPU (processor) 20 reads various programs stored in the storage unit 16 or the ROM 24, etc., expands them in the RAM 22, and performs calculations to comprehensively control each unit. Also, the CPU 20 reads the programs stored in the storage unit 16 or the ROM 24, performs calculations using the RAM 22, and conducts various processes of the infrared thermal image analysis device 10.

[0039] The infrared camera 32 shown in FIG. 1 photographs the structure 36 to be inspected and acquires an infrared thermal image of the surface of the structure. The visible camera 34 photographs the structure 36 to be inspected and acquires a visible image of the structure 36.

[0040] The infrared thermal image analysis device 10 can acquire an infrared thermal image from the infrared camera 32 via the input / output interface 12. Also, the infrared thermal image analysis device 10 can acquire a visible image from the visible camera 34 via the input / output interface 12. The acquired infrared thermal image and visible image can be stored in the storage unit 16, for example.

[0041] FIG. 2 is a block diagram showing the processing functions realized by the CPU 20.

[0042] The CPU 20 functions as an information acquisition unit 51, a region acquisition unit 53, a temperature gradient estimation unit 55, a temperature gradient reduction unit 57, and an information display unit 59. The specific processing functions of each unit will be described later. Since the information acquisition unit 51, the region acquisition unit 53, the temperature gradient estimation unit 55, the temperature gradient reduction unit 57, and the information display unit 59 are part of the CPU 20, it can also be said that the CPU 20 executes the processing of each unit.

[0043] Returning to FIG. 1, the storage unit (memory) 16 is a memory composed of a hard disk drive, a flash memory, etc. The storage unit 16 stores data and programs for operating the infrared thermal image analysis device 10, such as an operating system and a program for executing the infrared thermal image analysis method. The storage unit 16 also stores information used in the embodiments described below. Note that the program for operating the infrared thermal image analysis device 10 may be recorded on an external recording medium (not shown), distributed, and installed from the recording medium by the CPU 20. Alternatively, the program for operating the infrared thermal image analysis device 10 may be stored in a server or the like connected to a network in a state accessible from the outside, and downloaded to the storage unit 16 by the CPU 20 in response to a request, and installed and executed.

[0044] FIG. 3 is a diagram showing information and the like stored in the storage unit 16. The storage unit 16 is composed of a non-temporary recording medium such as a CD (Compact Disk), a DVD (Digital Versatile Disk), a hard disk, various semiconductor memory, etc. and its control unit.

[0045] The storage unit 16 mainly stores an infrared thermal image 101 and a visible image 103.

[0046] The infrared thermal image 101 is an image taken by the infrared camera 32, which detects infrared radiation energy radiated from the structure 36, converts the infrared radiation energy into temperature, and shows the temperature distribution on the surface of the structure. The visible image 103 is an image taken by the visible camera 34, which shows the distribution of the reflection intensity of visible light from the surface of the structure 36. Usually, a visible image consists of an RGB image in which the reflection intensity distributions in three different wavelength ranges in the wavelength range of visible light are imaged respectively, that is, it has color information (RGB signal values) for each pixel. It is also assumed to have color information in this example. Also, in this example, it is assumed that there is no misalignment between the infrared thermal image 101 and the visible image 103.

[0047] 1 includes a keyboard and a mouse, and a user can use these devices to perform necessary processing on the infrared thermal image analysis device 10. By using a touch panel type device, the display device 30 can function as the operation unit.

[0048] The display device 30 is, for example, a device such as a liquid crystal display, and is capable of displaying the results obtained by the infrared thermal image analysis device 10.

[0049] FIG. 4 is a flow diagram showing an infrared thermal image analysis method using the infrared thermal image analysis device 10.

[0050] First, the information acquisition unit 51 acquires a first infrared thermal image of the structure surface captured by photographing the structure to be inspected (first infrared image acquisition step: step S1). In this example, the first infrared thermal image is an infrared thermal image 101 stored in the storage unit 16. The infrared thermal image 101 is acquired from the storage unit 16 by the information acquisition unit 51.

[0051] Next, the area acquisition unit 53 acquires a small amount of area information for distinguishing areas of the surface of the structure corresponding to the first infrared thermal image based on the information about the structure. and In this example, the information about the structure is a visible image 103 stored in the storage unit 16. The visible image 103 is acquired from the storage unit 16 by the information acquisition unit 51. Based on the visible image 103, the area acquisition unit 53 acquires at least area information that distinguishes the area of ​​the structure surface corresponding to the infrared thermal image 101, which is the first infrared thermal image. and Another area is obtained.

[0052] Next, the temperature gradient estimation unit 55 estimates the temperature gradient in at least one region based on the region information and the second infrared thermal image (temperature gradient estimation step: step S3). In this example, the second infrared thermal image is the infrared thermal image 101 stored in the storage unit 16. Therefore, the first infrared thermal image and the second infrared thermal image are the same infrared thermal image 101. The infrared thermal image 101, which is the second infrared thermal image, has already been acquired in the first infrared image acquisition step (step S1). The temperature gradient estimation unit 55 estimates the temperature gradient in at least one region based on the region information and the infrared thermal image 101. Here, the information on the estimated temperature gradient may be in any form as long as it can reduce the influence of the temperature gradient from the first infrared thermal image in the subsequent temperature gradient reduction step. For example, it may be an image of the temperature gradient, a mathematical formula from which the temperature gradient can be derived, or a set of processes and data from which the temperature gradient can be derived.

[0053] Note that hereinafter, there may be cases where the first infrared thermal image and the second infrared thermal image are not particularly distinguished and are simply referred to as the "infrared thermal image".

[0054] Next, the temperature gradient reduction unit 57 reduces the influence of the temperature gradient from the first infrared thermal image (temperature gradient reduction step: step S4). In this example, the influence of the temperature gradient is reduced from the infrared thermal image 101, which is the first infrared thermal image. The temperature gradient reduction unit 57 can acquire a temperature gradient reduction image.

[0055] Next, the information display unit 59 displays the temperature gradient reduction image on the display device 30 (information display step: step S5). The information display unit 59 can also display the temperature gradient reduction image that has been image-processed on the display device 30.

[0056] As described above, the infrared thermal image analysis apparatus 10 in this example estimates the temperature gradient from the region information acquired based on the visible image 103 and the infrared thermal image 101, and reduces the influence of the temperature gradient from the infrared thermal image 101.

[0057] In the infrared thermal image analysis method, the execution order of the first infrared image acquisition step (step S1) and the temperature gradient estimation step (step S3) does not matter as long as it is before the temperature gradient reduction step (step S4).

[0058] Regarding each step below, an example in which the present invention is applied to a test body simulating the structure 36 will be given as a specific example and described.

[0059] <First Infrared Image Acquisition Step> The first infrared image acquisition step (step S1) is executed by the information acquisition unit 51. The information acquisition unit 51 acquires the infrared thermal image 101 of the surface of the structure obtained by photographing the structure 36 to be inspected stored in the storage unit 16 as the first infrared thermal image. When the infrared thermal image 101 is not stored in the storage unit 16, the information acquisition unit 51 acquires the infrared thermal image 101 from the outside. For example, the information acquisition unit 51 can acquire the infrared thermal image 101 through the network via the input / output interface 12, and the information acquisition unit 51 can also acquire the infrared thermal image 101 from the infrared camera 32 via the input / output interface 12.

[0060] FIG. 5 is a perspective view for explaining a test body group 40 simulating the structure 36. The test body group 40 includes four test bodies 41, 42, 43, and 44. Each of the test bodies 41, 42, 43, and 44 is a rectangular parallelepiped block made of concrete and is arranged on the cardboard 45 with a space therebetween.

[0061] A simulated bulge (cavity) is formed at a position 1 cm deep from the upper surface of the test body 44, at a position 2 cm deep from the upper surface of the test body 41, and at a position 3 cm deep from the upper surface of the test body 43. The test body group 40 is arranged outdoors on a sunny day and is irradiated with sunlight from the upper left. An infrared thermal image of the test body group 40 is taken by the infrared camera 32.

[0062] FIG. 6 is an infrared thermal image 101 taken by the infrared camera 32 and acquired as the first infrared thermal image by the information acquisition unit 51. In FIG. 6, the temperature of each pixel is displayed in grayscale. FIG. 6(A) is the infrared thermal image 101 of the test body group 40 actually taken, and FIG. 6(B) is a diagram in which arrows indicating the direction of the temperature gradient are added to the infrared thermal image 101 of FIG. 6(A).

[0063] As shown in FIG. 6(B), it can be seen from the infrared thermal image 101 of FIG. 6(A) that a temperature gradient (arrow A) with a downward slope from left to right is generated on the upper surfaces of the four test bodies 41, 42, 43, and 44. This temperature gradient is caused by solar radiation from the upper left and is due to the heat inflow from the direction of the left side surface near the boundary with the left side surface of the upper surface.

[0064] On the other hand, as shown in FIG. 6(B), it can be seen from the infrared thermal image 101 of FIG. 6(A) that a temperature gradient (arrow B) with a downward slope from top to bottom is generated on the left side surfaces of the four test bodies 41, 42, 43, and 44. This temperature gradient is due to the heat inflow from the direction of the upper surface near the boundary with the upper surface of the left side surface. table upper table surface.

[0065] It can be understood that a temperature gradient is generated on any surface of the four test bodies 41, 42, 43, and 44, and all are due to the heat inflow from the direction of the other surface near the boundary with the other surface.

[0066] The direction of the temperature gradient is different for each surface of the test bodies 41, 42, 43, and 44. Also, since the amount of heat received by each surface of the test bodies 41, 42, 43, and 44 due to solar radiation is different, the average temperature of each surface is also different, and the amount of heat inflow from the direction of the other surface near the boundary with the other surface of each surface is also different. As a result, the slope of the temperature gradient is also in a different state.

[0067] <Region acquisition step> The area acquisition step (step S2) is executed by the area acquisition unit 53. First, in this example, the information acquisition unit 51 acquires the visible image 103 of the surface of the structure 36 obtained by photographing the structure to be inspected stored in the storage unit 16. If the visible image 103 is not stored in the storage unit 16, the information acquisition unit 51 acquires the visible image 103 from the outside. For example, the information acquisition unit 51 can acquire the visible image 103 through the network via the input / output interface 12, and the information acquisition unit 51 can also acquire the visible image 103 from the visible camera 34 via the input / output interface 12.

[0068] FIG. 7 is a visible image 103 obtained by photographing the test body group 40 with the visible camera 34. As described above, the visible image 103 has RGB signal values for each pixel.

[0069] The area acquisition unit 53 acquires area information for distinguishing each area on the surface of the structure based on the RGB signal values of the acquired visible image 103 and information on spatial features such as edges and textures.

[0070] The area acquisition unit 53 extracts, for example, a group of pixels that can be regarded as the concrete surface among the pixels of the visible image 103. When the RGB signal values of each pixel are included in a predetermined range that can be regarded as the RGB values of the concrete surface, each pixel is extracted as a group of pixels that can be regarded as the concrete surface.

[0071] After extracting the group of pixels, the area acquisition unit 53 further distinguishes the group of pixels more finely based on the RGB signal values and spatial features of each pixel, fills the holes in each group of pixels by morphological dilation operation, and expands them. Finally, for each expanded group of pixels, the dynamic contour method is used to contract them to the optimal area, and the area corresponding to each group of pixels is determined. As the dynamic contour method, Snakes, Level set method, etc. can be applied.

[0072] The area acquisition unit 53 can distinguish at least one of surfaces with different slopes or discontinuous surfaces as different areas on the surface of the structure based on the RGB signal values of the visible image 103 and the information on spatial features. In addition, the area acquisition unit 53 can distinguish areas where any one of the color, roughness, unevenness, or presence or absence of sunlight on the surface of the structure is different as different areas. Note that discontinuous surfaces include surfaces such as shaded surfaces, steps, recesses, breaks, and each surface separated by a separate member, etc.

[0073] By executing the above-described processing, the area acquisition unit 53 acquires area information for distinguishing each area on the surface of the structure.

[0074] Incidentally, in the visible image 103, there may be cases where the RGB signal values of each pixel in the area are close to the RGB signal values of each pixel around the area, making it difficult to determine the boundary of the area. For example, in FIG. 7, among the four test bodies 41, 42, 43, and 44, the RGB signal value of the test body 43 is close to the signal value of the surrounding cardboard 45, making it difficult to determine the right boundary of the test body 43.

[0075] On the other hand, in the infrared thermal image 101 shown in FIG. 6(A), the temperature difference (difference in signal values) between the test body 43 and the cardboard 45 is significantly large, and the boundary is clearly shown. In such a case, the area acquisition unit 53 can also determine the optimal area based on both the visible image 103 and the infrared thermal image 101 and acquire the area information.

[0076] In cases such as photographing shaded areas on the structure where sunlight does not hit or night-time photographing, it may be difficult to determine the area based only on the visible image. In such cases, it is preferable to determine the area based on both the visible image and the infrared thermal image.

[0077] If a structure includes regions with different thermal conductivities or infrared emissivities, such as repair materials (repair marks), on its surface, the average temperature and temperature gradient may differ in those regions because the heat absorption or heat dissipation amounts are different from those of the surrounding concrete regions. Regions with different thermal conductivities or infrared emissivities may be difficult to determine in visible images, and it is preferable to determine the regions based on infrared thermal images. When applying an infrared thermal image as information about the structure for obtaining region information, either the first infrared thermal image or the second infrared thermal image may be included.

[0078] There are many methods for distinguishing and determining regions, such as the MeanShift method and the Graph Cuts method. Any of these methods may be applied to determine the regions. Machine learning may also be applied to determine the regions. For example, methods such as CNN (Convolutional Neural Network), FCN (Fully convolution network), U-net (Convolutional Networks for Biomedical Image Segmentation), and SegNet (A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation) may be used to determine the regions. As long as it is a method based on the features of visible images and infrared thermal images, it is not particularly limited, and any of these methods may be applied.

[0079] The region acquisition unit 53 acquires the necessary number (region Ar(i); i = 1, 2, 3, ··· N) of region information for distinguishing the regions on the surface of the structure.

[0080] FIG. 8 is a diagram of region information 105 of the result obtained by the region acquisition unit 53 distinguishing and acquiring the upper surfaces of four test bodies 41, 42, 43, and 44 as four regions based on the features of the visible image 103 and the infrared thermal image 101. Regions Ar(1), Ar(2), Ar(3), and Ar(4) indicate the regions corresponding to the upper surfaces of the test bodies 41, 42, 43, and 44, respectively. In this example, for the four regions of Ar(1), Ar(2), Ar(3), and Ar(4), a temperature gradient estimation step and a temperature gradient reduction step are executed.

[0081] <Temperature gradient estimation step> The temperature gradient estimation step (step S3) is executed by the temperature gradient estimation unit 55. The temperature gradient estimation unit 55 estimates the temperature gradient of the acquired region Ar(i) from the infrared thermal image of the region Ar(i). In this example, for the four regions of Ar(1), Ar(2), Ar(3), and Ar(4), the temperature gradients are estimated respectively.

[0082] As a method for estimating the temperature gradient, several examples can be given. For example, the temperature gradient estimation unit 55 can apply a smoothing filter with a predetermined number of pixels to the infrared thermal image to derive an image of the global temperature change and estimate it as the temperature gradient.

[0083] At that time, the temperature gradient estimation unit 55 preferentially applies the infrared thermal image in the region Ar(i) to the region other than the region Ar(i), that is, the region having a different average temperature and temperature gradient (direction and / or inclination) from the region Ar(i), and estimates the temperature gradient.

[0084] FIG. 9 is a diagram showing a first form of processing in the temperature gradient estimation step. FIG. 9 shows an example of a smoothing filter of 11×11 pixels (121 pixels) centered on a target pixel (indicated by a black pixel). In the vicinity of the boundary of the region Ar(i), a smoothing filter within a range that does not cross the boundary is applied and the temperature gradient estimation unit 55 executes a smoothing process. The range beyond the boundary of the region Ar(i) is shown in gray. In this case, the filter coefficients are set so that the sum within the range of the region Ar(i) that does not cross the boundary is "1". In FIG. 9, a filter coefficient of "1 / 73" is set. Note that in the case of a 121-pixel filter, if all the pixels of the filter do not cross the boundary, the coefficient is set to "1 / 121".

[0085] Note that even when changing the filter coefficients according to the pixel positions like in the Gaussian filter, it can be dealt with by multiplying each filter coefficient by a constant so that the total value becomes 1. Here, the Gaussian filter means a filter that increases the filter coefficient as it gets closer to the target pixel according to the Gaussian distribution function.

[0086] FIG. 10 is a diagram showing a second form of processing in the temperature gradient estimation step. As shown in FIG. 10, in the vicinity of the boundary of the region Ar(i), the temperature gradient estimation unit 55 extends the smoothing filter along the boundary within the range of the region Ar(i) and performs a smoothing process. In this case, since the number of pixels of the filter does not change regardless of the position of the target pixel, the filter coefficient is constant. In this example, a smoothing filter of 11×11 pixels (121 pixels) is extended along the boundary of the region, and a filter coefficient of "1 / 121" is set. In FIG. 10, the filter coefficient of "1 / 121" is displayed only for some pixels.

[0087] Next, a third form of the process in the temperature gradient estimation step will be described. As described above, when the surface of the structure is composed of a plurality of regions with different amounts of heat reception or heat dissipation, in each region, a temperature gradient occurs near the boundary due to heat inflow or heat outflow from other adjacent regions. Note that the plurality of regions with different amounts of heat reception or heat dissipation include cases where they are composed of surfaces with a plurality of different slopes, cases where they are composed of regions with a plurality of different colors, roughnesses, unevennesses, presence or absence of solar radiation, thermal conductivities, and infrared emissivities.

[0088] This indicates that the vicinity of the boundary serves as the starting point of the temperature gradient. Similarly, in each discontinuous region separated by a step, a dent, a break, a separate member, etc., the vicinity of the boundary serves as the starting point of the temperature gradient. From this phenomenon, it can be understood that the temperature gradient may be estimated based on the infrared thermal image near the boundary in each region.

[0089] This method is preferable because it is not affected by the temperature distribution (temperature change rather than temperature gradient) caused by the damaged part within the region compared to the method of estimating the temperature gradient based on the infrared thermal image of the entire region.

[0090] The third form of the process in the temperature gradient estimation step will be described based on FIG. 11.

[0091] FIG. 11(A) is an infrared thermal image 106 of the region Ar(1) on the upper surface of the test body 41. The infrared thermal image 106 is an image obtained by extracting only the region Ar(1) on the upper surface of the test body 41 from the infrared thermal image 101 of the test body group 40 based on the region information 105. Here, since the infrared thermal image 106 is the original image of the temperature gradient image 107 to be derived later, it is preferably an image obtained by smoothing the infrared thermal image 101. However, smoothing is not essential.

[0092] FIG. 11(B) is an image obtained by extracting the periphery of the boundary with a predetermined width for the region Ar(1) of the region information 105 shown in FIG. 8. Therefore, in FIG. 11(B), only the peripheral region of the region Ar(1) is shown.

[0093] FIG. 11(C) is an image in which only the infrared thermal image of the peripheral region of region Ar(1) is extracted based on the infrared thermal image 106 in FIG. 11(A) and the region information obtained by extracting the periphery of FIG. 11(B).

[0094] FIG. 11(D) is a temperature gradient image 107 derived by interpolation calculation from the infrared thermal image of the peripheral region in FIG. 11(C). Here, spline interpolation is applied to continuously (smoothly) interpolate up to the second derivative from the infrared thermal image of the peripheral region. As described above, by estimating the temperature gradient from the peripheral region, it is possible to reduce the influence of the temperature distribution (temperature difference) between the healthy part and the damaged part existing in the region on the estimation of the temperature gradient. Note that the interpolation calculation is not particularly limited, and other than spline interpolation can also be applied. For example, linear interpolation may be used.

[0095] Next, the peripheral regions in FIGS. 11(B) and (C) will be described with reference to FIG. 12. FIG. 12(A) is a diagram for explaining the distance L from each pixel of the infrared thermal image of region Ar(i) to the boundary and the distance La from the center of region Ar(i) to the boundary, and FIG. 12(B) is a diagram for explaining the peripheral region of region Ar(i).

[0096] As shown in FIG. 12, among the distances from each pixel Px included in the infrared thermal image within the region Ar(i) to each pixel on the boundary of the region, the smallest distance is defined as the distance L from the pixel Px to the boundary. Among the distances L from each pixel Px to the boundary, the largest distance L is defined as the distance La from the center to the boundary. In the region Ar(i), a region composed of pixels Px whose distance L to the boundary is smaller than the distance La from the center to the boundary is defined as the peripheral region. In FIG. 12(B), the peripheral region ER composed of pixels Px whose distance L to the boundary is 1 / 2 or less of the distance La from the center to the boundary is displayed in a dot pattern. Also shown is the peripheral region composed of pixels Px that are 1 / 8 or less. Here, as the third form of the process in the temperature gradient estimation step, in the example of FIG. 11, a method of estimating the temperature gradient based only on the infrared thermal image (or the smoothed infrared thermal image) of the peripheral region was described. However, it is not necessarily estimated only from the infrared thermal image of the peripheral region. The temperature gradient may be estimated by preferentially applying the infrared thermal image of the peripheral region. For example, when deriving the temperature gradient of the region Ar(i) by interpolation calculation from the infrared thermal image of the peripheral region of the region Ar(i), in the region Ar(i), the temperature gradient may be derived by interpolation calculation from the infrared thermal image of a region including not only the peripheral region but also other regions (regions other than the peripheral region). In that case, the infrared thermal image of the peripheral region may be interpolated with a larger weight than other regions, or more pixels of the peripheral region may be applied than other regions, for example, by sparsely arranging the pixels of the region other than the peripheral region for interpolation. At least, it is preferable to preferentially apply the infrared thermal image of the peripheral region whose distance L to the boundary is 1 / 2 or less of the distance La from the center to the boundary to estimate the temperature gradient. Further, at least, it is preferable to preferentially apply the infrared thermal image of the peripheral region whose distance L to the boundary is 1 / 4 or less of the distance La from the center to the boundary to estimate the temperature gradient. Further, at least, it is preferable to preferentially apply the infrared thermal image of the peripheral region whose distance L to the boundary is 1 / 8 or less of the distance La from the center to the boundary to estimate the temperature gradient. Note that the smallest value of the distance L to the boundary corresponds to the distance of one pixel.

[0097] As described above, in the temperature gradient estimation step, the temperature gradient is estimated by the temperature gradient estimation unit 55.

[0098] Figure 13 shows the temperature gradient image 107 derived in the temperature gradient estimation step. As shown in Figure 13, the temperature gradient image 107 derived in the temperature gradient estimation step includes temperature gradient images 107A, 107B, 107C, and 107D corresponding to the regions Ar(1), Ar(2), Ar(3), and Ar(4) on the upper surfaces of the four specimens 41, 42, 43, and 44, respectively.

[0099] <Temperature gradient reduction step> The temperature gradient reduction step (step S4) is executed by the temperature gradient reduction unit 57. The temperature gradient reduction unit 57 reduces the influence of the temperature gradient from the infrared thermal image 101, which is the first infrared thermal image. For example, the temperature gradient reduction unit 57 reduces the influence of the temperature gradient from the infrared thermal image 101 based on the temperature gradient image 107.

[0100] When reducing the influence of the temperature gradient, the temperature gradient reduction unit 57 can subtract the temperature gradient from the infrared thermal image 101 or divide the infrared thermal image 101 by the temperature gradient.

[0101] When subtracting the temperature gradient from the infrared thermal image 101, the value of each pixel in the temperature gradient image 107 can be subtracted from the value of each pixel in the infrared thermal image 101. Also, when dividing the infrared thermal image 101 by the temperature gradient, the value of each pixel in the infrared thermal image 101 can be divided by the value of each pixel in the temperature gradient image 107. Here, for example, temperature can be applied as the value of each pixel.

[0102] In the temperature gradient reduction step, the temperature gradient reduction unit 57 subtracts, for example, the temperature gradient image 107 shown in Figure 13 from the infrared thermal image 101 shown in Figure 6 to derive the temperature gradient reduction image 109 shown in Figure 14.

[0103] <Information display step> The information display step (step S5) is executed by the information display unit 59. The information display unit 59 displays the temperature gradient reduction image 109 shown in Figure 14 on the display device 30 (see Figure 1).

[0104] As shown in FIG. 14, the temperature gradient reduction image 109 derived in the temperature gradient reduction step includes temperature gradient reduction images 109A, 109B, 109C, and 109D corresponding to the regions Ar(1), Ar(2), Ar(3), and Ar(4) on the upper surfaces of the four specimens 41, 42, 43, and 44, respectively.

[0105] In the temperature gradient reduction image 109A, a high-temperature portion caused by a 2-cm deep bulge is visualized, and in the temperature gradient reduction image 109C, a high-temperature portion caused by a 3-cm deep bulge is visualized. It can be understood that the influence of the temperature gradient on the temperature gradient reduction image 109 is reduced.

[0106] FIG. 15 shows an image in which the prior art is applied to the infrared thermal image 101 of the specimen group 40 to reduce the temperature gradient. As shown in FIG. 15, since the prior art smoothes the infrared thermal image and takes the difference without distinguishing the respective surfaces with different average temperatures and temperature gradients of the specimens, the boundaries of the respective surfaces are misdetected. Furthermore, it can be understood that even after reducing the temperature gradient, the high-temperature portions caused by the 2-cm deep bulge and the 3-cm deep bulge cannot be visualized. Further, although the surfaces of the four specimens 41, 42, 43, and 44 in the specimen group 40 are discontinuous respectively, they are adjacent and continuous in the infrared thermal image 101. Therefore, it can also be seen that the prior art misdetects the boundaries of the respective surfaces by smoothing the infrared thermal image and taking the difference without distinguishing those surfaces.

[0107] Next, another preferred embodiment will be described.

[0108] In the above-described region acquisition step (step S2), the case of acquiring region information based on the visible image 103 and the infrared thermal image 101 has been described.

[0109] As long as the data can acquire region information, the region acquisition unit 53 may acquire region information using any data. As the data, data acquired by measuring a structure, data created regarding a structure, etc. can be applied.

[0110] For example, as data obtained by measuring a structure, data measured by LIDAR (Light Detection And Ranging) or the like to measure distances can be used to obtain region information for distinguishing surfaces with different slopes and / or discontinuous surfaces. LIDAR measures the distance to each point on the surface of the structure by measuring the time from when the laser light irradiates each point on the surface of the structure until it is reflected and returns. The region acquisition unit 53 can acquire region information based on the data measuring the distance to each point on the surface of the structure. The method of measuring the distance is not limited to LIDAR. For example, the distance may be measured by a TOF (Time Of Flight) camera, a stereo camera, or the like. Based on the data measuring the distance to each point on the surface of the structure, region information for distinguishing each surface with a different slope and / or each discontinuous surface on the surface of the structure can be obtained. Also, since the relationship between the coordinate system of the distance measurement and the coordinate system of the infrared camera 32 is known, in the infrared thermal image, the region corresponding to each surface on the surface of the structure can also be specified. The laser light irradiates each point on the surface of the structure, and the distance to each point on the surface of the structure is measured by measuring the time until the reflected light returns. The region acquisition unit 53 can acquire region information based on the data measuring the distance to each point on the surface of the structure. The method of measuring the distance is not limited to LIDAR. For example, the distance may be measured by a TOF (Time Of Flight) camera, a stereo camera, or the like. Based on the data measuring the distance to each point on the surface of the structure, each surface with a different slope and / or each discontinuous surface on the surface of the structure can be distinguished to obtain the region information. Also, since the relationship between the coordinate system of the distance measurement and the coordinate system of the infrared camera 32 is known, in the infrared thermal image, the region corresponding to each surface on the surface of the structure can also be specified.

[0111] Also, as data created for the structure, drawing data of the structure to be inspected can be used to obtain region information. The drawing data includes drawings and CAD (computer-aided design) data. The region acquisition unit 53 can, by separately specifying the position and the shooting direction (posture) of the infrared camera 32 by another method, based on the drawing data of the structure, specify the regions of each surface with a different slope and / or each discontinuous surface of the structure in the infrared thermal image taken from the above-mentioned position in the above-mentioned direction. Note that various parameters of the infrared camera 32 are known.

[0112] For specifying the position of the infrared camera 32, Wi-Fi positioning, acoustic wave positioning, etc. can be applied. Also For specifying the shooting direction, known methods such as a gyro sensor can be applied. The position and shooting direction (orientation) of the infrared camera 32 with respect to the structure to be inspected can also be specified by applying the technology of self-position estimation based on various sensor measurements called SLAM (Simultaneous Localization and Mapping). By applying the technology, the position and shooting direction (orientation) of the infrared camera 32 with respect to the structure to be inspected can be specified.

[0113] In addition, in the above-described region acquisition step (step S2), the case where the visible image 103 has color information (RGB signal values), that is, consists of three types of images (RGB images) has been described. However, the type of the visible image may be one type, two types, or four or more types. Region information can be acquired based on the visible image 103 consisting of any type of image and the infrared thermal image 101.

[0114] In addition, in the above-described region acquisition step (step S2), the case where region information is acquired based on the visible image 103 and the infrared thermal image 101 as information about the structure has been described. However, region information may be directly acquired without using information about the structure. For example, the region information corresponding to the first infrared thermal image is stored in the storage unit 16. First, the information acquisition unit 51 may acquire the region information from the storage unit 16, and then the region acquisition unit 53 may receive the region information as it is. Or, when the region information is not stored in the storage unit 16, the region acquisition unit 53 may acquire the region information from the outside. For example, first, the information acquisition unit 51 may acquire the region information through the network via the input / output interface 12, and then the region acquisition unit 53 may receive the region information as it is.

[0115] In the above-described region acquisition step (step S2), as already explained, on the surface of the structure, at least one of each surface with a different inclination or each discontinuous surface is regarded as and distinguished from a different region. Also, on the surface of the structure, each region with a difference in any one of color, roughness, unevenness, presence or absence of solar radiation, thermal conductivity, and infrared emissivity is regarded as and distinguished from a different region. Note that each discontinuous surface includes each surface separated by a shaded surface, a step, a dent, a break, a separate member, etc.

[0116] In the above-described temperature gradient estimation step (step S3), when estimating the temperature gradient of the region Ar(i), only the infrared thermal image of the region Ar(i) is used. In FIG. 9, "0" is applied to the coefficient of the smoothing filter in the range beyond the boundary of the region Ar(i). In FIG. 10, the smoothing filter is extended along the boundary within the range of the region Ar(i).

[0117] However, when estimating the temperature gradient of the region Ar(i), the infrared thermal image used does not necessarily have to be strictly limited to only the region Ar(i). If the difference in temperature distribution between the region Ar(i) and the adjacent region across the boundary is not large near the boundary of the region Ar(i), even if some pixels (adjacent pixels) of the adjacent region beyond the boundary are included to estimate the temperature gradient of the region Ar(i), there is almost no influence on the estimated temperature gradient of the region Ar(i), while the noise of the temperature gradient can be reduced.

[0118] For example, before extracting the infrared thermal image near the boundary of the region Ar(1) in FIG. 11(C), it is preferable to smooth the infrared thermal image to reduce the noise of the infrared thermal image. At this time, some pixels of the adjacent region beyond the boundary of the region Ar(1) may be included. In short, the temperature gradient of the region Ar(i) may be estimated by preferentially applying the infrared thermal image of the region Ar(i). Preferentially applying the infrared thermal image of the region Ar(i) means applying the infrared thermal image of the region Ar(i) more than the infrared thermal images of other regions and / or applying the infrared thermal image of the region Ar(i) with a greater weight than the infrared thermal images of other regions.

[0119] Note that, as already described, the information on the estimated temperature gradient may be in any form as long as it can reduce the influence of the temperature gradient from the first infrared thermal image in the subsequent temperature gradient reduction step. Also, the method for estimating the temperature gradient is not limited to only smoothing and interpolation of the second infrared thermal image, but may also be a method of preliminarily determining a mathematical model for expressing the temperature gradient and optimizing the parameters of the model so that the temperature gradient expressed by the model matches the second infrared thermal image. As a simple method, a function representing the temperature gradient may be preliminarily determined, and the temperature gradient may be estimated by optimizing the parameters of the function so that the function best matches the second infrared thermal image. Alternatively, based on the region information, a heat simulation (simulation of heat conduction, radiation, and convection) may be performed, and the temperature gradient may be estimated by optimizing the parameters of the simulation so that the temperature gradient derived from the simulation best matches the second infrared thermal image. In this method, first, region information is obtained based on information about the structure, such as visible images, infrared thermal images, data on the distance measured to the structure by LIDAR, etc., and drawing data. Next, various parameters for the heat simulation are variously changed and set under the setting of the region information, and the heat simulation is performed. The temperature gradient in each region is reproduced by the heat simulation (simulation of heat conduction inside the structure by the finite element method (FEM) or the finite-difference time-domain method (FDTD), radiation and convective heat transfer on the surface of the structure, solar heating, etc.). Then, the temperature gradient in each region reproduced by the heat simulation is compared with the temperature change in each region in the second infrared thermal image, and the temperature gradient in the case of the best match may be adopted. Here, when comparing the temperature gradient by simulation with the temperature change in the second infrared thermal image, the temperature gradient by simulation may be scaled so that the average value of the temperatures of the simulation and the second infrared thermal image matches.

[0120] The various parameters to be optimized vary depending on the types of regions to be distinguished. For example, when distinguishing regions with different colors, roughness, and unevenness, the thermal conductivity and emissivity of each region are used as parameters. When distinguishing between areas irradiated by solar radiation or other light (e.g., reflected and / or radiated light from around the structure) and areas not irradiated, the amount of heat received by solar radiation or other light in each region is used as a parameter. Even for a uniform surface, when distinguishing each region separated by steps, dents, discontinuities, etc., geometric shapes such as the height, depth, and width of these steps, dents, and discontinuities, and the heat flux at the discontinuous locations are used as parameters.

[0121] The content of the region information also varies depending on the types of regions to be set. For example, when setting regions with different colors, roughness, and unevenness as different regions, as the region information for each, the information (size, shape, position) of the two-dimensional region on the surface of the structure may be set. When setting surfaces with different slopes or discontinuous surfaces as different regions, it is necessary to set the three-dimensional structure including these multiple surfaces as the region information. When setting each region separated by steps, dents, discontinuities, etc., if the three-dimensional structure can be obtained as the region information, the three-dimensional structure is set as the region information (not as the parameter to be optimized). When using a visible image or an infrared thermal image as information about the structure, it is possible to derive the three-dimensional structure of the surface of the structure from the image, but it may be difficult to derive the three-dimensional structure such as small steps, dents, and discontinuities. On the other hand, when using data measuring distances or drawing data as information about the structure, it is possible to obtain information on the three-dimensional structure including small steps, dents, and discontinuities. by When using data measuring distances or drawing data as information about the structure, it is possible to obtain information on the three-dimensional structure including small steps, dents, and discontinuities.

[0122] The region information is preferably set in as wide a range as possible so that the temperature gradient of each region on the surface of the structure corresponding to at least the second infrared thermal image is reproduced. When using drawing data as information about the structure, it is possible to set the structure of the entire structure to be inspected as the region information.

[0123] In performing thermal simulation, it is desirable to measure and identify in advance parameters that can be measured and identified by other methods, such as the thermal conductivity and emissivity of the structure itself, the outside air temperature, the direction of solar radiation, and the heating amount, and set them as fixed parameters. For example, the outside air temperature on the day when the structure to be inspected is photographed and the day before can be measured at predetermined time intervals (for example, one-hour intervals) and input as fixed parameters for the simulation. In addition, the thermal conductivity and emissivity of general concrete can be set as the thermal conductivity and emissivity of the structure itself.

[0124] In addition, in order to perform thermal simulation, of course, the region information must accurately reflect the size, shape, and position of each region on the surface of the structure. When setting surfaces with different slopes and discontinuous surfaces as different regions, the information must accurately reflect their three-dimensional structures. (However, in order to perform the simulation, it is not necessarily required to accurately know the size, shape, position, and three-dimensional structure of each region. It is sufficient if their relative relationships match the actual situation). That is, under the condition that the actual size, shape, position, and three-dimensional structure of each region on the surface of the structure to be inspected are accurately reflected, thermal simulation is performed to derive the temperature gradient. When comparing with the second infrared thermal image, each region on the surface of the structure is compared in association with each region in the second infrared thermal image. Similarly, when reducing the influence of the temperature gradient from the first infrared thermal image, each region on the surface of the structure is reduced in association with each region in the first infrared thermal image. That is, the region information obtained based on the information about the structure is information that accurately reflects the size, shape, position, and three-dimensional structure of each region on the surface of the structure, and is also information about the regions in the first infrared thermal image and the second infrared thermal image corresponding to those regions. Therefore, it is necessary to associate each region on the surface of the structure with each region in the first infrared thermal image and the second infrared thermal image. When obtaining region information based on the data measuring the distance to the structure, as already explained, it is possible to make the association. When obtaining region information based on the drawing data, as already explained, it is also possible to make the association by separately specifying the position and shooting direction of the infrared camera 32 by a method. When obtaining region information based on the visible image and the infrared thermal image (assuming there is no positional deviation between the visible image and the infrared thermal image), for example, based on the principle of triangulation from visible images taken from two or more different viewpoints, the distance from each visible camera 34 that took the visible images to each point on the surface of the structure can be measured, so the association is possible. The visible images from two or more different viewpoints can be obtained by shooting with two or more different viewpoints of visible cameras 34, or by using one visible camera 34 to shoot the same location on the surface of the structure with different viewpoints.However, when using one visible camera 34, it is necessary to separately specify the shooting position and shooting direction at each viewpoint by another method. Or, even when using one visible camera, by applying the SfM (Structure from Motion (multi-viewpoint stereo photogrammetry)) technology, shooting can be performed at each viewpoint. Based only on the images, it is possible to estimate the shooting position and shooting direction at each viewpoint, and it is also possible to estimate the distance to each point on the surface of the structure at each viewpoint, so they can be associated. Or, by separately specifying the actual size and shape of the entire surface of the structure by another method, such as drawing data or separate measurement, it is possible to make an association based on the overall size and shape of the surface of the structure in the visible image. That is, by comparing the actual size and shape of the entire surface of the structure with the size and shape in the visible image, it is possible to specify the distance from the visible camera 34 that captured the visible image to each point on the surface of the structure, so they can be associated. Similarly, it is also possible to make an association by separately specifying the actual size and shape of a plurality of characteristic locations on the surface of the structure by another method. Or, it is also possible to make an association by fixing the visible camera 34 at a specific position with respect to the surface of the structure, for example, a position facing directly.

[0125] The method of performing thermal simulation based on the region information described above to estimate the temperature gradient is effective in cases where the damaged part is large and spans multiple regions. Fig. 16 shows an example where the damaged part is large and spans two regions. Fig. 16(A) is a visible image, Fig. 16(B) shows the damaged part in the visible image with white on a black background. Fig. 16(C) shows an infrared thermal image. It can be seen that the damaged part is large and spans two regions, the upper surface and the side surface of the structure surface.

[0126] Next, a modified example of obtaining the first infrared thermal image, the second infrared thermal image, and information about the structure based on the region information (also referred to as the acquired information group), which is different from the above, will be described. The information about the structure based on the region information includes the first infrared thermal image, the second infrared thermal image, the visible image, and measurement data, etc.

[0127] A first modification of information acquisition is a case where different infrared thermal images are used as the first infrared thermal image and the second infrared thermal image. Such a modification of information acquisition is necessary, for example, when an infrared camera is mounted on a moving body such as a drone and the surface of a structure is photographed in close proximity. When photographing the surface of a structure with an infrared camera mounted on a moving body, the surface of the structure within the photographing range is uniform, while on the other hand, a temperature gradient exists, and the boundary with another region that is the starting point of the temperature gradient may be outside the photographing range.

[0128] Here, the surface of the structure being uniform means that the inclination of the surface is constant and there are no discontinuous surfaces. The absence of discontinuous surfaces means that there are no discontinuous surfaces due to shade, steps, dents, breaks, separate members, etc. It also means that the color, roughness, unevenness, presence or absence of solar radiation, thermal conductivity, and infrared emissivity of the surface are also constant.

[0129] In this case, the structure is photographed with an infrared camera from a distance to obtain a second infrared thermal image. Similarly, the structure is photographed with a visible camera from a distance, and region information is obtained from the visible image. Or, instead of a visible camera, LIDAR etc. by Region information can be obtained from the measured data. The temperature gradient of each region is estimated from these region information and the second infrared thermal image.

[0130] Each region with the estimated temperature gradient is photographed in close proximity with an infrared camera to obtain a first infrared thermal image. The range photographed in close proximity within the range where the temperature gradient is estimated (the range where the first infrared thermal image is photographed) can be extracted, and the influence of the extracted temperature gradient can be reduced from the first infrared thermal image.

[0131] Since the temperature gradient is a large-scale temperature change compared to the temperature distribution due to the damaged part, the temperature gradient can also be estimated based on the second infrared thermal image photographed from a distance in this way.

[0132] The second modification example of information acquisition is the case of integrating a plurality of infrared thermal images and a plurality of visible images. In the second modification example, a plurality of adjacent captured visible images are integrated until the boundary of at least one region is included to obtain region information. Based on the region information from the infrared thermal image obtained by integrating the plurality of infrared thermal images, the temperature gradient of at least one region is estimated. The temperature gradient is reduced from the integrated infrared thermal image. In the second modification example, the first infrared thermal image and the second infrared thermal image are the same infrared thermal image. Instead of the visible image, region information can also be obtained from data measured by LIDAR or the like.

[0133] The third modification example of information acquisition is the case of integrating a plurality of infrared thermal images and not integrating visible images. Region information is obtained based on a visible image captured by a visible camera from a distance. For each region, proximity shooting is performed with an infrared camera, and a plurality of infrared thermal images are integrated until the boundary is included. The temperature gradient is estimated from the integrated infrared thermal image and the region information. The temperature gradient is reduced from the integrated infrared thermal image. In the third modification example, the first infrared thermal image and the second infrared thermal image are the same infrared thermal image. Instead of the visible image, region information can also be obtained from data measured by LIDAR or the like.

[0134] As described above, using different first infrared thermal images, second infrared thermal images, and information about the structure based on region information, and information about the structure based on a plurality of first infrared thermal images, second infrared thermal images, and region information, region information can be acquired and / or the temperature gradient can be estimated. This can be done.

[0135] In the case of these methods, it is necessary to separately specify the position and shooting direction (posture) of the shooting system with respect to the structure to be inspected at the time of each shooting by a separate method. It can be specified by applying a known method using various sensors (such as an acceleration sensor, a gyro sensor, etc.) or the SLAM technology which is a self-position estimation technology.

[0136] The first infrared thermal image, the second infrared thermal image, and the information about the structure used in the first infrared image acquisition step, the region acquisition step, and the temperature gradient estimation step do not necessarily have to be images captured or data measured at exactly the same timing. Images captured at different timings (the first infrared thermal image, the second infrared thermal image, visible image) in each step, or data measured (data measured by LIDAR or the like) may be used, or images captured or data measured at multiple timings may be integrated and used. However, in any form, it is necessary to use the second infrared thermal image captured at a timing suitable for estimating the temperature gradient in the first infrared thermal image and reducing its influence. As already explained, when the surface of the structure has surfaces with a plurality of different slopes, discontinuous surfaces, a plurality of different colors, roughness, unevenness, regions with different thermal conductivities and emissivities, regions irradiated by solar radiation or other light (including infrared rays) and regions not irradiated, differences in the average temperature and temperature gradient in each region may occur, resulting in problems such as the inability to correctly reduce the temperature gradient and problems of false detection. To solve this problem, the present invention estimates the temperature gradient by distinguishing each region and reduces its influence. Here, differences in the average temperature and temperature gradient in each region can occur regardless of day or night, but vary depending on the time (timing). Therefore, it is necessary to use the second infrared thermal image captured at an appropriate timing so that the average temperature and temperature gradient in each region are close between the first infrared thermal image and the second infrared thermal image. Note that by using an image captured at a timing with daytime solar radiation as the second infrared thermal image, it becomes possible to estimate the temperature gradient due to daytime solar radiation.

[0137] When using a visible image as information about the structure and capturing the visible image at night, it is necessary to ensure the amount of visible light required for imaging by methods such as illuminating the structure to be inspected with visible light illumination or using the flash function of the visible camera.

[0138] Next, a preferred form in the information display step (step S5) will be described. As described above, the information display unit 59 displays the temperature gradient reduction image and / or the image-processed temperature gradient reduction image on the display device 30.

[0139] The information display unit 59 can display the temperature gradient reduction image derived by the temperature gradient reduction unit 57 alone. Also, the captured images (the first infrared thermal image, the second infrared thermal image, the visible image) applied when deriving the temperature gradient reduction image can be displayed in a form such as juxtaposed, superimposed, or embedded together with the temperature gradient reduction image.

[0140] The temperature gradient reduction image can also be processed (image-processed). In the case of daytime shooting, in the temperature gradient reduction image, only the pixels with values equal to or higher than a predetermined threshold can be regarded as damaged parts and displayed. In the case of nighttime shooting, in the temperature gradient reduction image, only the pixels with values equal to or lower than a predetermined threshold can be regarded as damaged parts and displayed.

[0141] It can also be binarized with a predetermined threshold. In the case of daytime shooting, a pixel group with values equal to or higher than the threshold, and in the case of nighttime shooting, a pixel group with values equal to or lower than the threshold can be regarded as damaged parts and displayed. By setting multi-stage thresholds, it can be processed and displayed in a ternary, quaternary, etc. manner so that the temperature distribution (temperature difference) of the damaged part can be understood to some extent. Furthermore, it can be processed so that the damage level can be understood from the temperature distribution. The processing (image processing) of the temperature gradient reduction image is executed by the CPU 20.

[0142] The temperature gradient reduction image can be stored and accumulated in the storage unit 16 or the like. By storing and accumulating the temperature gradient reduction image, changes over time can be investigated. Also, from the temperature gradient reduction image, in the case of daytime shooting, a quantitative value of damage such as the total value of values equal to or higher than a predetermined threshold can be obtained, displayed, and stored. In the case of nighttime shooting, a quantitative value of damage such as the total value of values equal to or lower than a predetermined threshold can be obtained, displayed, and stored.

[0143] In the infrared thermal image analysis apparatus of the above-described embodiment, necessary captured images or measurement data can be acquired separately by another device. Further, the infrared thermal image analysis apparatus may be incorporated in a device (such as an infrared camera or a visible camera) that captures a captured image.

[0144] The program of the above-described embodiment may be implemented by a dedicated analysis program, and the device for implementation is not limited. For example, it can be implemented on a personal computer. Also, the device or program for implementing each step may be integrated or separated.

[0145] As described above, in this embodiment, region information for distinguishing regions on the surface of a structure to be inspected is acquired, a temperature gradient is estimated based on the region information and the infrared thermal image, and the influence of the temperature gradient is reduced from the infrared thermal image. As a result, in the infrared thermal image of the surface of a structure composed of a plurality of regions having different average temperatures and temperature gradients, the influence of the temperature gradient of each region can be accurately reduced, and the discrimination accuracy between damaged parts and healthy parts is improved.

[0146] <Others> In the above description, the information acquisition unit 51 has been described in terms of a form of acquiring information stored in the storage unit 16, but it is not limited thereto. For example, when the necessary information is not stored in the storage unit 16, the information acquisition unit 51 may acquire information from the outside via the input / output interface 12. Specifically, the information acquisition unit 51 acquires information input via the input / output interface 12 from outside the infrared thermal image analysis apparatus 10.

[0147] In the above embodiment, the hardware structure of the processing unit that executes various processes is various processors as shown below. Various processes The processing unit includes a CPU (Central Processing Unit), which is a general-purpose processor that executes software (program) and functions as various processing units, a programmable logic device (PLD) such as an FPGA (Field Programmable Gate Array), which is a processor whose circuit configuration can be changed after manufacturing, and an application-specific integrated circuit (ASIC), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically to execute specific processing.

[0148] One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units can be composed of one processor. Examples of composing multiple processing units with one processor include, firstly, a form in which one processor is composed of a combination of one or more CPUs and software, as represented by a computer such as a client or a server, and this processor functions as multiple processing units. Secondly, there is a form in which a processor that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip, as represented by a system on chip (SoC). Thus, various processing units are configured using one or more of the above various processors as a hardware structure.

[0149] Furthermore, the hardware structure of these various processors is more specifically an electric circuit (circuitry) that combines circuit elements such as semiconductor elements.

[0150] Each of the above configurations and functions can be appropriately realized by any hardware, software, or a combination of both. For example, the present invention can also be applied to a program that causes a computer to execute the above-described processing steps (processing procedures), a computer-readable recording medium (non-transitory recording medium) that records such a program, or a computer to which such a program can be installed.

[0151] Although the examples of the present invention have been described above, it goes without saying that the present invention is not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present invention.

Explanation of Reference Numerals

[0152] 10 Infrared Thermal Image Analyzer 12 Input / Output Interface 16 Storage Unit 18 Operation Unit 20 CPU 22 RAM 24 ROM 26 Display Control Unit 30 Display Device 32 Infrared Camera 34 Visible Camera 36 Structure 40 Specimen Group 41 Specimen 42 Specimen 43 Specimen 44 Specimen 45 Cardboard 51 Information Acquisition Unit 53 Region Acquisition Unit 55 Temperature Gradient Estimation Unit 57 Temperature Gradient Reduction Unit 59 Information Display Unit 101 Infrared Thermal Image 103 Visible Image 105 Region Information 106 Infrared Thermal Image 107 Temperature Gradient Image 107A Temperature Gradient Image 107B Temperature Gradient Image 107C Temperature gradient image 109 Temperature gradient reduction image 109A Temperature gradient reduction image 109B Temperature gradient reduction image 109C Temperature gradient reduction image L Distance La Distance Px Pixel S1 Step S2 Step S3 Step S4 Step S5 Step

Claims

1. An infrared thermal image analysis device comprising a processor, wherein the processor acquires an infrared thermal image of the surface of a structure obtained by photographing the structure to be inspected as a first infrared thermal image, on the surface of the structure for which the first infrared thermal image was taken, regions with different slopes and / or discontinuous regions are regarded as different regions, or regions where any one of color, roughness, unevenness, or presence or absence of solar radiation is different are regarded as different regions, and region information for distinguishing each region is acquired for at least one region Ar(i), acquires an infrared thermal image of the surface of the structure obtained by photographing the structure with the region Ar(i) included in the photographing range as a second infrared thermal image, estimates a temperature gradient in the region Ar(i) based on the region information of the region Ar(i) and the second infrared thermal image, reduces the influence of the temperature gradient from the first infrared thermal image, an infrared thermal image analysis device.

2. The processor acquires the region information based on information regarding the structure including a visible image of the structure taken, The infrared thermal image analysis device according to Claim 1.

3. The processor acquires the region information based on information regarding the structure including at least one of the first infrared thermal image or the second infrared thermal image of the structure taken, The infrared thermal image analysis device according to Claim 1 or 2.

4. The processor acquires the region information based on information regarding the structure including data measuring the distance to the structure, The infrared thermal image analysis device according to any one of Claims 1 to 3.

5. The processor acquires the region information based on information regarding the structure including drawing data of the structure, The infrared thermal image analysis device according to any one of Claims 1 to 4.

6. The processor applies the second infrared thermal image in the region Ar(i) preferentially over the second infrared thermal images of other regions, and estimates the temperature gradient in the region Ar(i), The infrared thermal image analysis device according to any one of Claims 1 to 5.

7. The processor in the preferential application, smoothes the second infrared thermal image with different weightings for the region Ar(i) and the other regions, The infrared thermal image analysis device according to Claim 6.

8. The processor In the preferential application, along the boundary of the region Ar(i), extending in a range that does not include the other regions, and smoothing the second infrared thermal image. The infrared thermal image analysis device according to claim 6.

9. The processor Preferentially applies the second infrared thermal image in the peripheral region of the region Ar(i), and estimates the temperature gradient in the region Ar(i). The infrared thermal image analysis device according to any one of claims 1 to 8.

10. The peripheral region Among the distances from the pixels within the region Ar(i) to each pixel on the boundary of the region Ar(i), taking the smallest distance as the distance from each pixel to the boundary, and when taking the distance at the pixel with the largest distance to the boundary as the distance from the center to the boundary, The region that includes at least pixels whose distance to the boundary is 1 / 2 or less of the distance from the center to the boundary. The infrared thermal image analysis device according to claim 9.

11. The processor Estimates the temperature gradient in the region Ar(i) by thermal simulation. The infrared thermal image analysis device according to any one of claims 1 to 10.

12. The processor When reducing the influence of the temperature gradient, subtracting the temperature gradient from the first infrared thermal image, or dividing the first infrared thermal image by the temperature gradient. The infrared thermal image analysis device according to any one of claims 1 to 11.

13. At least one of the first infrared thermal image, the second infrared thermal image, and the information regarding the structure based on the region information is acquired at different timings. The infrared thermal image analysis device according to any one of claims 1 to 12.

14. At least one of the first infrared thermal image, the second infrared thermal image, and the information regarding the structure based on the region information is an image or information obtained by integrating a plurality of images or information. The infrared thermal image analysis device according to any one of claims 1 to 12.

15. The processor Acquires the first infrared thermal image and the second infrared thermal image at a timing with solar radiation. The infrared thermal image analysis device according to any one of claims 1 to 14.

16. The surface of the structure includes at least one of a plurality of surfaces with different inclinations or discontinuous surfaces. The infrared thermal image analysis device according to any one of claims 1 to 15.

17. The processor Display a temperature gradient reduction image with the influence of the temperature gradient reduced from the first infrared thermal image on a display device. An infrared thermal image analysis device according to any one of claims 1 to 16.

18. The processor Displays the temperature gradient reduction image subjected to image processing on a display device. An infrared thermal image analysis device according to claim 17.

19. The first infrared thermal image and the second infrared thermal image are the same infrared thermal image. An infrared thermal image analysis device according to any one of claims 1 to 18.

20. Acquire an infrared thermal image of the surface of a structure obtained by photographing the structure to be inspected as a first infrared thermal image. On the surface of the structure where the first infrared thermal image is taken, regions with different slopes and / or discontinuous regions are regarded as different regions, or regions where any one of color, roughness, unevenness, and presence or absence of solar radiation is different are regarded as different regions, and acquire region information for distinguishing each region for at least one region Ar(i). Acquire an infrared thermal image of the surface of the structure obtained by photographing the structure with the region Ar(i) included in the photographing range as a second infrared thermal image. Estimate the temperature gradient in the region Ar(i) based on the region information of the region Ar(i) and the second infrared thermal image. Reduce the influence of the temperature gradient from the first infrared thermal image. Infrared thermal image analysis method.

21. Acquire an infrared thermal image of the surface of a structure obtained by photographing the structure to be inspected as a first infrared thermal image. On the surface of the structure where the first infrared thermal image is taken, regions with different slopes and / or discontinuous regions are regarded as different regions, or regions where any one of color, roughness, unevenness, and presence or absence of solar radiation is different are regarded as different regions, and acquire region information for distinguishing each region for at least one region Ar(i). Acquire an infrared thermal image of the surface of the structure obtained by photographing the structure with the region Ar(i) included in the photographing range as a second infrared thermal image. Estimate the temperature gradient in the region Ar(i) based on the region information of the region Ar(i) and the second infrared thermal image. Reduce the influence of the temperature gradient from the first infrared thermal image. A program for causing a computer to execute the above.

Images