Panoramic thermal imaging method and apparatus

Abstract

To create a panoramic thermal image with less noise from a series of thermal images. The panoramic thermal image creation device 5 creates a rectangular or square matching area Rm that is a part of each thermal image. t Select the matching area Rm for each thermal image. t A plurality of feature points are identified in the thermal image, and feature point matching is performed for each pair of thermal images adjacent in time. t is determined based on the result of feature point matching, and the stitched region Rs t The panoramic thermal image is generated by synthesizing the matching area Rm t and the bonding area Rs t includes the center of the thermal image.

Description

[Technical Field]

[0001] The present invention relates to a method and apparatus for creating a panoramic thermal image using a series of thermal images of a subject taken by an infrared thermography camera mounted on an aircraft during flight of the aircraft. [Background technology]

[0002] In recent years, in order to inspect or check facilities and their equipment (hereinafter referred to as "facilities, etc."), drones equipped with cameras are flown over the target facilities, and abnormalities in the facilities, etc. are identified based on images of the facilities, etc. taken from the air by the cameras.

[0003] Identifying anomalies from a single still image of an entire facility is generally difficult due to factors such as the camera's pixel resolution and drone altitude limitations. Therefore, drones typically fly at a height where the camera's field of view includes part of the facility. Frames are identified from video captured by the camera during flight, and the anomalies are identified based on the still image capturing the portion of the facility. Linking a still image to the portion of the facility captured in that still image is achieved, for example, using GPS data. A series of still images identified from a video stream can also be stitched together to create a single composite image capturing the entire facility, i.e., a panoramic image. Creating such a panoramic image is valuable because, unlike an image capturing only a portion of the facility, the relative position of the anomaly and the entire facility or equipment can be instantly determined from the panoramic image.

[0004] When it is difficult to identify an abnormality from visible images, an infrared thermography camera is used as a camera mounted on a drone, and the abnormality is identified based on the captured thermal image of the facility, etc. Examples of targets for abnormality detection using a drone equipped with an infrared thermography camera include solar power generation facilities, particularly large-scale solar power generation facilities called megasolar power plants. JP 2016-186682 A discloses an aerial image processing system that creates a composite image from a series of infrared images of the solar panel array of a megasolar power plant taken by an infrared camera mounted on a drone. [Prior art documents] [Non-patent literature]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-186682 Summary of the Invention [Problem to be solved by the invention]

[0006] A known problem with infrared thermography cameras is that the greater the angle of incident light relative to the camera's imaging sensor, the more likely noise is generated in the thermal image due to the influence of infrared rays reflected from the subject. Feature point matching is commonly used to create panoramic images using a series of still images, but when creating a panoramic image from a series of thermal images, feature points can be extracted due to noise, leading to inappropriate feature point matching. This results in an inappropriate synthesis of the thermal images, resulting in a panoramic thermal image that does not accurately reflect the subject's true image. Furthermore, noise in the thermal images that form the basis of a panoramic thermal image can distort the actual temperature distribution, potentially resulting in the creation of an abnormal temperature distribution in an area where no abnormality or failure actually exists. This problem is particularly exacerbated when generating panoramic thermal images of solar panel arrays at a solar power generation facility based on a series of thermal images taken with an infrared thermography camera mounted on a drone, where significant noise in the thermal images is caused by infrared reflection from the glass protecting the solar panels.

[0007] The present invention solves the above problems and provides a method and apparatus for creating a panoramic thermal image using a series of thermal images of a subject taken from the air by an infrared thermography camera mounted on an aircraft such as a drone. [Means for solving the problem]

[0008] The method for generating a panoramic thermal image of the present invention is a method for generating a panoramic thermal image from a series of thermal images using computer means, comprising the steps of: Each of the series of thermal images is a rectangular still image corresponding to a frame of a moving image obtained by photographing a subject while the aircraft is flying using an infrared thermography camera mounted on the aircraft; the computer means selecting, based on input from a user, for each thermal image, a same rectangular or square first image region that is part of the thermal image; the computer means identifying a plurality of feature points within a first image region of each thermal image; the computer means performs feature point matching for each pair of temporally adjacent thermal images in the series of thermal images based on a plurality of feature points identified within a first image region of each thermal image; said computer means selecting a rectangular or square second image area that is part of each thermal image from at least a second thermal image to a last thermal image in said series of thermal images, said computer means selecting said second image area of ​​said thermal image based on input from a user and results of feature point matching performed on each pair of temporally adjacent thermal images in the thermal images from the first thermal image to said last thermal image; said computer means synthesizing a second image area of ​​at least a second thermal image through a last thermal image in said series of thermal images to generate a panoramic thermal image; It contains a first image area of ​​each thermal image in the series of thermal images includes a center of the thermal image, and a second image area of ​​each thermal image from at least a second thermal image to a last thermal image in the series of thermal images includes a center of the thermal image; It is a method.

[0009] The method for generating a panoramic thermal image of the present invention may further include the computer means setting a lower limit to the width of the first image region of each thermal image based on the photographing conditions of the subject.

[0010] In the method for generating a panoramic thermal image of the present invention, the width of the first image region of each thermal image may be limited to a predetermined value that is less than the width of each thermal image.

[0011] In the method for generating a panoramic thermal image of the present invention, the subject may include a group of solar panels.

[0012] In the method for generating a panoramic thermal image of the present invention, the flying object may be controlled to maintain its speed and direction of movement while the subject is being photographed.

[0013] The non-transitory computer readable medium of the present invention is a non-transitory computer readable medium storing program instructions executable on a computer means for creating a panoramic thermal image from a series of thermal images, the program instructions comprising: Each of the series of thermal images is a rectangular still image corresponding to a frame of a moving image obtained by photographing a subject while the aircraft is flying using an infrared thermography camera mounted on the aircraft; The program instructions cause the computer means to: selecting, for each thermal image, a same rectangular or square first image region that is part of the thermal image based on input from a user; identifying a plurality of feature points within a first image region of each thermal image; performing feature point matching for each pair of temporally adjacent thermal images in the series of thermal images based on a plurality of feature points identified within a first image region of each thermal image; selecting a rectangular or square second image area that is part of each thermal image from at least a second thermal image to a last thermal image in the series of thermal images, the second image area being selected based on input from a user and results of feature point matching performed on each pair of temporally adjacent thermal images from the first thermal image to the last thermal image; synthesizing a second image area of ​​at least a second thermal image from a last thermal image in the series of thermal images to generate a panoramic thermal image; and A first image area of ​​each thermal image in the series of thermal images includes a center of the thermal image, and a second image area of ​​each thermal image from at least the second thermal image to the last thermal image in the series of thermal images includes a center of the thermal image.

[0014] The panoramic thermal image creation device of the present invention is a panoramic thermal image creation device that creates a panoramic thermal image from a series of thermal images, Each of the series of thermal images is a rectangular still image corresponding to a frame of a moving image obtained by photographing a subject while the aircraft is flying using an infrared thermography camera mounted on the aircraft; The panoramic thermal imaging device includes: selecting, for each thermal image based on input from a user, the same rectangular or square first image region that is part of the thermal image; Identifying a plurality of feature points within a first image region of each thermal image; performing feature point matching for each pair of temporally adjacent thermal images in the series of thermal images based on the plurality of feature points identified within the first image region of each thermal image; selecting, for each thermal image from at least the second thermal image to the last thermal image of the series of thermal images, a rectangular or square second image area that is part of the thermal image; synthesizing a second image area from at least a second thermal image to a last thermal image in the series of thermal images to generate a panoramic thermal image; for each thermal image of at least a second thermal image through a last thermal image of the series of thermal images, a second image region of the thermal image is selected based on input from a user and results of feature point matching performed on each pair of temporally adjacent thermal images in the thermal images from the first thermal image to the last thermal image; A first image area of ​​each thermal image in the series of thermal images includes a center of the thermal image, and a second image area of ​​each thermal image from at least the second thermal image to the last thermal image in the series of thermal images includes a center of the thermal image. [Effects of the Invention]

[0015] According to the present invention, it is possible to identify feature points from a series of thermal images used as the basis for a panoramic thermal image by excluding areas near the edges of the images that are affected by noise or are easily affected by noise, and then perform feature point matching.Furthermore, it is possible to create a panoramic thermal image by stitching together the central parts of thermal images that are free of or have little noise.As a result, the temperature distribution shown in the panoramic thermal image created based on the present invention more accurately reflects the shape of the subject and suppresses the influence of noise. [Brief explanation of the drawings]

[0016] [Figure 1] Figure 1 is an explanatory diagram showing the process of taking aerial photographs of a solar panel array at a solar power generation facility using an infrared thermography camera mounted on a drone. [Figure 2] Figure 2 shows an example of a thermal image, a frame from a video sequence of a solar panel array taken from the air by an infrared thermography camera mounted on a drone. [Figure 3] FIG. 3 is a block diagram showing an embodiment of the panoramic thermal imaging device of the present invention. [Figure 4] FIG. 4 is a flowchart outlining one embodiment of the panoramic thermal image creation method of the present invention. [Figure 5] FIG. 5 is an explanatory diagram showing the process of converting a thermal image data file into data files of thermal images of individual frames that make up a dynamic thermal image in one embodiment of the panoramic thermal image creation method of the present invention. [Figure 6] FIG. 6 is an explanatory diagram for explaining the matching area and stitching area of ​​a thermal image according to an embodiment of the panoramic thermal image creating method of the present invention. [Figure 7] FIG. 7 shows an example of a setting screen displayed on the display device of an embodiment of the panoramic thermal image creating device of the present invention. [Figure 8] FIG. 8 is an explanatory diagram outlining the steps involved in one embodiment of the panoramic thermal imaging method of the present invention. [Figure 9]FIG. 9 is an explanatory diagram showing how a panoramic thermal image is created by executing an embodiment of the panoramic thermal image creating method of the present invention. [Figure 10] Figure 10 shows an example of a thermal image obtained from a video of a solar panel array taken from the air using an infrared thermography camera mounted on a drone, and shows an example of the matching and stitching areas set for the thermal image. DETAILED DESCRIPTION OF THE INVENTION

[0017] The present invention relates to a method and apparatus for creating a panoramic thermal image using a series of thermal images obtained from a video of a subject captured by an infrared thermography camera mounted on an aircraft during flight. For example, the subject may be a large facility or one or more of its components that are subject to inspection or review. An aircraft equipped with an infrared thermography camera is flown over the facility, and a panoramic thermal image is created using a series of still images corresponding to all or some of the frames of the video captured by the infrared thermography camera. The panoramic thermal image shows the temperature distribution of the facility and its background, allowing for the identification of abnormalities in the temperature distribution.

[0018] In the following explanation, the subject, i.e., the object of inspection or testing using panoramic thermal images, is a group of solar panels at a solar power generation facility installed on the ground, more specifically a group of solar panel arrays, and the aircraft on which the infrared thermography camera is mounted is a drone (an airplane, rotorcraft, glider, airship, etc. that can be used for aviation purposes but is structurally incapable of carrying humans, and is remotely controlled or automatically piloted).

[0019] The present invention will be described below with reference to the drawings. FIG. 1 shows a drone 1 equipped with an infrared thermography camera 11 taking aerial photographs of a solar panel array 3 at a solar power generation facility. The solar panel array 3 is composed of multiple rectangular solar panels 31 fixed to a common mount (not shown) installed on the ground. The solar panel array 3 (hereinafter simply referred to as "array") illustrated in FIG. 1 has multiple solar panels 31 arranged so that their upper surfaces are in the same plane, giving the overall appearance of a rectangle. In a typical solar power generation facility, multiple arrays 3 are arranged vertically and horizontally on the ground.

[0020] The infrared thermography camera 11 (hereinafter simply referred to as "camera") photographs the area on the ground below the drone 1 from the air as moving images at a predetermined frame rate. The camera 11 is equipped with an infrared imaging element (not shown) with a rectangular light-receiving surface, and each frame that makes up the moving thermal image obtained from the air is a rectangular still thermal image (hereinafter simply referred to as "thermal image"). The attitude of the camera 11 relative to the drone 1 or the main body of the drone 1 is adjustable, so that the camera 11 can photograph not only the area below the drone 1 but also the sides of the drone 1.

[0021] 1, like a typical solar power generation facility, multiple arrays (not shown) are arranged in a strip-like, linear fashion to the right of the illustrated array 3. For example, drone 1 flies along the longitudinal direction of the arrays 3 so as to take aerial photographs of the illustrated array 3 or a series of arrays (hereinafter referred to as the "array group") including the array 3 located at the right end of the strip.

[0022] During aerial photography, drone 1 is preferably controlled to maintain a constant speed and a constant direction of movement. The altitude of drone 1 and the attitude of drone 1 or camera 11 are preferably adjusted so that the angle of view of camera 11 (shown by the dashed line in FIG. 1 ) completely encompasses a portion of array 3 across the width W (i.e., the length along the short side) of array 3, and the optical axis of camera 11 is perpendicular or nearly perpendicular to the top surface of the array group. Furthermore, the speed of drone 1 and / or the frame rate of the dynamic thermal images are preferably set so that a significant portion of the images overlap in adjacent frames.

[0023] The drone 1 flies from one end of the array group to the other along the longitudinal direction of the array 3, and the camera 11 photographs the array group from the air while the drone 1 is flying. It is desirable that the camera 11 starts capturing thermal images before the array 3 at one end of the array group enters the field of view of the camera 11, and finishes capturing dynamic thermal images after the array 3 at the other end of the array group leaves the field of view of the camera 11.

[0024] The drone 1 is wirelessly controlled by a controller (not shown). The flight path, speed, altitude, camera 11 attitude, video frame rate, etc. of the drone 1 are input to the controller in advance, and the drone 1 flies automatically or autonomously as described above based on the various input aerial photography conditions, and photography is performed by the camera 11.

[0025] Figure 2 shows an example of a thermal image, which is one frame of a dynamic thermal image of an array group taken from the air by the camera 11 mounted on the drone 1 as described above. This example is included in a solar power generation facility where multiple arrays 3, each with 4 rows and 5 columns, are arranged vertically and horizontally on the ground. In the center of the thermal image in Figure 2, two adjacent arrays A n and Array A n+1 The temperature distribution of the two arrays A is shown partially covering the width W of the arrays. n and Array A n+1 Array B is located above (in Figure 2) n and Array B n+1 and array C placed belown and Array C n+1 The temperature distribution of the arrays is also partially shown, but the width W of the arrays is not fully shown.

[0026] When the present invention is applied in the case according to FIG. n and Array A n+1 A series of arrays arranged in a strip and in a straight line including the arrays are the subject, and a dynamic thermal image is obtained by aerial photography of the array group with the drone 1 as described above, and a panoramic thermal image of the array group is created from a series of thermal images obtained from the dynamic thermal image as described below. The movement direction of the drone 1 during aerial photography is maintained or set so as to be along or parallel to the longitudinal direction of the array group (if the movement direction is shown on the image, so as to be along or parallel to the longitudinal direction of the image).

[0027] In the thermal image example in Figure 2, array A is located near the left edge. n It can be seen that the temperature distribution of array A is affected by the above-mentioned noise caused by sunlight (a relatively dark area spreads near the left edge of Figure 2). By applying the present invention, it is possible to obtain a temperature distribution of array A without being affected by such noise. n and Array A n+1 A panoramic thermal image of the array group including the thermal image of FIG. 2 is created from a series of thermal images including the thermal image of FIG.

[0028] Data relating to dynamic thermal images of the array group taken by camera 11 from the air is stored in a storage medium (not shown) mounted on drone 1 and sent to computer means in real time after or during the aerial photography, and after the aerial photography, the computer means uses the dynamic thermal images to create a panoramic image based on the present invention.

[0029] 3 is a block diagram of a panoramic thermal image creating device 5 which constitutes the computer means in the panoramic thermal image creating method of the present invention or is one embodiment of the panoramic thermal image creating device of the present invention. As shown in Fig. 3, the panoramic thermal image creating device 5 includes a CPU (Central Processing Unit) 51, a GPU (Graphics Processing Unit) 52, a RAM (Random Access Memory) 53, a ROM (Read Only Memory) 54, an SSD (Solid State Drive) 55, a wireless communication device 56, an I / O interface 57, a display device 58, and an input device 59.

[0030] The CPU 51 controls the overall operation of the panoramic thermal image creation device 5 and executes commands from various programs. The RAM 53 is a work memory that stores programs executed by the CPU 51 and data in the middle of processing. The ROM 54 stores a UEFI (Unified Extensible Firmware Interface) or BIOS (Basic Input Output System) that controls the hardware constituting the panoramic thermal image creation device 5 and starts the operating system.

[0031] The SSD 55 stores data files of dynamic thermal images of the array group captured by the camera 11 from the air. The wireless communication device 56 enables wireless communication between the panoramic thermal image creation device 5 and external devices based on a short-range wireless communication standard such as Bluetooth (registered trademark). The storage medium included in the drone 1 is wirelessly connected to the panoramic thermal image creation device 5 via the wireless communication device 56, and the dynamic thermal image data is sent from the storage medium to the SSD 55. Alternatively, the storage medium included in the drone 1 is wired connected to the panoramic thermal image creation device 5 via the I / O interface 57 after the aerial photography is completed, and the dynamic thermal image data is sent from the storage medium to the SSD 55.

[0032] The SSD 55 stores not only the operating system of the panoramic thermal image creation device 5, but also a panoramic thermal image creation program that describes various processes required to create a panoramic thermal image. The panoramic thermal image creation program is loaded from the SSD 55 to the RAM 53 and executed by the CPU 51. The CPU 51 executes the panoramic thermal image creation program to create a panoramic thermal image using the data file of the dynamic thermal image stored in the SSD 55. The panoramic thermal image creation program may be stored in a non-transitory computer-readable medium such as a DVD (Digital Versatile Disc) and sent from the non-transitory computer-readable medium to the panoramic thermal image creation device 5 via the I / O interface 57 for execution, or may be stored in the SSD 55 of the panoramic thermal image creation device 5.

[0033] The display device 58 is, for example, an organic EL (Electroluminescence) display or a liquid crystal display, and the GPU 52 displays the panoramic thermal image created based on the present invention and images such as a setting screen 9 (described later) on the display device 58 in accordance with commands from the CPU 51. The input device 59 is, for example, a keyboard and / or a mouse, and various inputs related to the execution of the panoramic thermal image creation program are made by the user via the input device 59.

[0034] 4 is a flowchart showing an outline of an embodiment of the panoramic thermal image creating method of the present invention, which is executed using the panoramic thermal image creating device 5 as a computer means. The series of steps shown in FIG. 4 are written as instructions in the panoramic thermal image creating program stored in the SSD 55 or a non-transitory computer-readable medium, and are executed by the CPU 51 of the panoramic thermal image creating device 5.

[0035] First, the CPU 51 converts the data file 7 of the dynamic thermal image stored in the SSD 55 into rectangular thermal images of all frames that make up the dynamic thermal image, and stores the data files of the thermal images of each frame in the SSD 55 (step S1).

[0036] FIG. 5 is an explanatory diagram showing a schematic diagram of the process of step S1. In the following explanation, the symbol "t" indicates a time specific to each thermal image, and in the following explanation, the time is specified by a frame number, not a numerical value in seconds. In the thermal image of the first frame, t=0, and in the thermal image of the last frame, t=t end Here, t end is the total number of frames in the thermal image minus 1.

[0037] The thermal image data file 7 may be in the format of AVI (Audio Video Interleave), WMV (Windows Media Video), or MPEG4 (Moving Picture Experts Group 4). The thermal image data file may be in the format of JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), or TIFF (Tagged Image File Format). For example, the frame rate of the thermal image may be 60 fps, and the number of pixels per frame (horizontal and vertical) may be 640 x 480 pixels, 1280 x 960 pixels, or 1920 x 1200 pixels. Before step S1, some frames at the beginning and end of the thermal image may be deleted in advance from the thermal image, if they do not capture any of the array 3 constituting the subject.

[0038] 4, after step S1, the CPU 51 prompts the user to input aerial photography conditions for the moving images in the data file 7 via the input screen displayed on the display device 58, and the user inputs the aerial photography conditions to the panoramic thermal image creation device 5 via the input device 59 (step S3). The input numerical values ​​and the like for the aerial photography conditions are stored in the SSD 55 and / or RAM 53 and are referred to by the CPU 51.

[0039] The aerial photography conditions input in step S3 include the speed of the drone 1 during aerial photography, the actual distance corresponding to one pixel or the width of the thermal image, or information required from the user for the panoramic thermal image creation device 5 to determine the actual distance (the number of pixels in the thermal image can be obtained by referencing the data file). For example, the user can select an appropriate thermal image from the thermal images created in step S1, identify an object (e.g., the solar panel 31) shown in the thermal image, calculate the actual distance corresponding to the width of the thermal image based on the dimensions of the object, and input this information in step S3. Alternatively, in step S3, the altitude of the drone 1 and values ​​such as the dimensions and focal length of the infrared imaging element of the camera 11 may be input, and the actual distance corresponding to the width of the thermal image may be calculated in steps subsequent to step S3 based on these values. Note that in step S3, the controller of the drone 1 may be connected to the panoramic thermal image creation device 5 via a wired or wireless connection, and information related to the aerial photography conditions may be sent from the controller to the panoramic thermal image creation device 5.

[0040] After step S3, the CPU 51 prompts the user to input the parameter step via the input screen displayed on the display device 58, and the user inputs the value of step to the panoramic thermal image creation device 5 via the input device 59 (step S5). The input value of step is stored in the RAM 53 and is thereafter referred to by the CPU 51.

[0041] The parameter step is the time interval between two thermal images that are adjacent in time in the series of thermal images referenced in the processing after step S5 (i.e., the time interval between a pair of thermal images that are the subject of feature point matching, which will be described below). In this embodiment, step is an integer equal to or greater than 1, and when the value of step input in step S5 is 1, all frames of thermal images are used to create a panoramic thermal image. In FIG. 5, the thermal image (t=step*n, n is an integer equal to or greater than 1) that is specified when step=3 is set in step S5 is shown hatched.

[0042] For example, if the drone 1 moves at a fast speed, the step is set to a small value, and if the drone 1 moves at a slow speed, the step is set to a large value. The larger the step value, the faster the processing from step S5 onwards, but there is a limit to the step value that can be set when creating a panoramic thermal image. For example, before executing step S5, the user may identify several consecutive thermal image data files from the thermal image data files created in step S1, display those thermal images on the display device 58, and visually check the degree of overlap of the thermal images between adjacent frames, thereby appropriately selecting the step value to input in step S5. In addition, the upper limit of the step may be determined based on the aerial photography conditions and the frame rate of the dynamic thermal images input in step S3, and if the input step value exceeds the upper limit, the user may be prompted to input the step value again.

[0043] According to one feature of the present invention, the panoramic thermal image creation device 5 selects, based on user input, the same rectangular or square first image area that is part of each thermal image used to create the panoramic thermal image, and further identifies multiple feature points within the first image area of ​​each thermal image. The first image area of ​​each thermal image includes the center of the thermal image. The panoramic thermal image creation device 5 performs feature point matching for each pair of temporally adjacent thermal images based on the multiple feature points extracted from the first image area. By setting the first image area so that the left and right ends, as well as the top and bottom ends, of the thermal image that are likely to be affected by noise are excluded, a panoramic thermal image that properly reflects the shape of the subject can be obtained.

[0044] According to a further feature of the present invention, the panoramic thermal image creation device 5 individually selects a rectangular or square second image area that is part of each thermal image in a series of thermal images. Alternatively, the panoramic thermal image creation device 5 individually selects a rectangular or square second image area that is part of each thermal image except for the first thermal image. The second image area of ​​a thermal image includes the center of the thermal image. A panoramic thermal image is generated by combining all selected second image areas based on the results of feature point matching. For each thermal image except for the first thermal image, the selection of the second image area of ​​the thermal image is based on user input and the results of feature point matching performed on each pair of temporally adjacent thermal images from the first thermal image to the first thermal image. By combining the selected second image areas in the center of the images to create a panoramic thermal image, the accuracy of the temperature distribution shown in the panoramic thermal image is improved.

[0045] Referring again to FIG. 4, after step S5, the CPU 51 performs a process of prompting the user to input parameters necessary for selecting or determining the first image area and the second image area via the input screen displayed on the display device 58, and the user inputs those parameters into the panoramic thermal image creation device 5 via the input device 59 (step S7).

[0046] 6 is an explanatory diagram showing a first image region and a second image region of a thermal image at a certain time t. In this embodiment, the first image region of the thermal image at a certain time t is a rectangular or square matching region Rm t and includes the center Q of the thermal image. t is the matching region Rm t The upper and lower sides of the matching area Rm are parallel to the horizontal direction of the thermal image. t is defined so that the right and left sides are parallel to the vertical direction of the thermal image. t is the matching area Rm along the vertical direction of the thermal image. t The center line of the thermal image is defined to pass through the center Q of the thermal image.

[0047] Such a matching region Rm t is specified by determining the upper left position Pml of the matching area and the lower right position Pmr of the matching area. The upper left position Pml of the matching area and the lower right position Pmr of the matching area are specified by the x-coordinate and y-coordinate of the pixel coordinate system, with the upper left corner of the thermal image as the origin, the x-axis as the horizontal direction, and the y-axis as the vertical direction. The x-coordinate x of the upper left position Pml of the matching area is Pml and the x-coordinate of the bottom right position of the matching area Pmr Pmr What is that? x Pmr -x q =x q -x Pml where x q is the x-coordinate of the center Q.

[0048] In this embodiment, the matching area Rm t is the matching area width Wm(=x Pmr -x Pml ) is limited to a value equal to or greater than a predetermined lower limit. In this embodiment, this lower limit is an expected image deviation width determined based on the value of step input in step S3, the aerial photography conditions input in step S5 (drone speed and actual distance corresponding to the width of the thermal image), the frame rate of the video, and the number of pixels of the thermal image. The expected image deviation width is (1) Estimated image shift width = drone movement distance × actual distance per pixel Here, the drone travel distance is defined as: (2) Drone movement distance = drone speed × video frame rate (frames / second) × step The actual distance per pixel is calculated using the formula: (3) Actual distance per pixel = Actual distance corresponding to the width of the thermal image ÷ Width of the thermal image (pixels) The lower limit of the matching region width Wm may be set to an integer multiple (for example, two or three times) of the predicted image deviation width.

[0049] In this embodiment, the second image area of ​​the thermal image at a certain time t is a rectangular or square stitched area Rs t and includes the center Q of the thermal image. t is the bonding area Rs t The top and bottom sides of the thermal image are parallel to the horizontal direction of the thermal image, and the stitching area Rs t The right and left sides of the stitching region Rs are defined to be parallel to the vertical direction of the thermal image. t is the stitching area Rs along the vertical direction of the thermal image. t The center line of the thermal image is defined to pass through the center Q of the thermal image.

[0050] Such a bonding area Rs t is defined by the bonding region width Wp, the bonding region upper interval Hsu, and the bonding region lower interval Hsb. The bonding region upper interval Hsu and the bonding region lower interval Hsb are given in step S7, and the bonding region width Wp is calculated in a step to be described later. The bonding region upper interval Hsu is determined by the bonding region Rs t is the pixel distance between the top edge of the thermal image and the top edge of the stitched area, and the stitched area lower spacing Hsb is the pixel distance between the top edge of the stitched area Rs t The upper spacing Hsu of the stitched area is the pixel distance between the lower side of the stitched area Rs t The upper side of the matching area Rm t The lower spacing Hsb of the stitching area may be set to be above the upper side of the stitching area Rs t The bottom side of the matching area Rm t It may be set to be lower than the bottom side of the line, or may be set to zero.

[0051] In this embodiment, in step S7, the CPU 51 displays a setting screen 9 on the display device 58 as an input screen for the user to input the upper left position Pml of the matching area, the lower right position Pmr of the matching area, the upper spacing Hsu of the stitching area, and the lower spacing Hsb of the stitching area via a GUI (Graphical User Interface). Fig. 7 shows an example of the setting screen 9. The CPU 51 selects one data file from the data files of the thermal images created in step S1, sets the thermal image of that data file as the background of the setting screen 9 (in Fig. 7, the same thermal image as in Fig. 2 is used as the background), and displays a figure 91 indicating the center Q of the thermal image and a matching area Rm on the selected thermal image. t A frame 92 indicating the upper spacing Hsu of the stitching area, a horizontal line 93 indicating the lower spacing Hsb of the stitching area, and a pointer 95 for specifying the operation target are displayed.

[0052] For example, when the user moves a pointer 95 onto the frame 92, horizontal line 93, or horizontal line 94 via a mouse constituting the input device 59 and performs an operation such as drag-and-drop with the mouse, the frame 92 is deformed on the screen, or the horizontal line 93 or horizontal line 94 moves up or down. The user sets the shape of the frame 92 and the positions of the horizontal lines 93 and 94 based on the thermal image of the background. When the user operates the input device 59 to give an instruction to confirm, the pixel coordinates of the upper left position Pml and the lower right position Pmr of the matching area, and the values ​​of the upper spacing Hsu and the lower spacing Hsb of the stitching area are determined based on the shape of the frame 92 and the positions of the horizontal lines 93 and 94 at the time of confirmation, and these are stored in the RAM 53.

[0053] The setting screen 9 is preferably configured to allow the user to select and / or change the thermal image displayed as the background. For example, the setting screen 9 may play a dynamic thermal image and the user may select the thermal image displayed as the background by stopping the playback at a desired scene. The setting screen 9 is configured to prevent the setting of the frame 92, horizontal line 93, and horizontal line 94 that violates the restrictions. Specifically, the setting screen 9 is configured to prevent the frame 92 from being deformed so that the width of the frame 92 becomes less than the lower limit value (expected image deviation width) of the matching area width Wm.

[0054] In order to reduce the influence of noise in the final panoramic thermal image, the frame 92 is deformed so that the matching region width Wm is the same as or very close to the horizontal width of the thermal image, i.e., the matching region Rm t In one embodiment, it is not desirable to set the matching area Rm t To ensure that the left and right sides are separated, an upper limit is set for the matching area width Wm (for example, 60 to 90% of the horizontal width of the thermal image), and deformation of the frame 92 that would cause the matching area width Wm to exceed this upper limit is prohibited on the setting screen 9.

[0055] It is expected that the user will appropriately set the frame 92, horizontal line 93, and horizontal line 94 based on the thermal image displayed as the background on the setting screen 9. For example, in the case illustrated in FIG. 7, the influence of noise is seen at the left end of the image, so the user will need to adjust the frame 92 and the matching area Rm to avoid this. t In addition, array B, which does not constitute the array group of the object to be inspected, will be set. n , B n+1 , C n and C n+1 is the bonding area Rs t Array A of the array group to be inspected should not be included in the n and A n+1 The width W of the region Rs t The user will set horizontal lines 93 and 94 so that they are completely contained within the horizontal line 93.

[0056] 4 again, after step S7, the CPU 51 initializes a variable t, which is used to identify the time of a thermal image to be individually referenced in the following processing, to zero (step S9). After step S9, the CPU 51 reads out the data file of the thermal image at time t from the SSD 55 and sends it to the RAM 53, and the thermal image at time t is referenced (step S11). Note that, as shown in FIG. 4, in this embodiment, the variable t is incremented by the value of step in step S27 described later, and the processing returns to step S11, whereby step S11 and the subsequent steps are repeated.

[0057] After step S11, the CPU 51 calculates the matching region Rm of the thermal image at time t based on the parameters input in step S7. t After step S13, the CPU 51 reads the data file of the thermal image at time t+step from the SSD 55 and sends it to the RAM 53, and the thermal image at time t+step is referenced (step S15). After step S15, the CPU 51 selects or specifies the matching area Rm of the thermal image at time t+step based on the parameters input in step S7. t+step is selected or specified (step S17).

[0058] After step S17, the CPU 51 calculates the matching area Rm of the thermal image at time t. t and the matching area Rm of the thermal image at time t+step t+step A plurality of feature points in the image are calculated and identified (step S19).

[0059] After step S19, the CPU 51 calculates the matching area Rm of the thermal image at time t. t and the matching area Rm of the thermal image at time t+step t+stepIn step S21, feature point matching is performed to associate the feature points in the thermal image with the feature points in the thermal image (step S21). Note that in step S21, it is not necessary to identify correspondences for all of the feature points calculated in step S19. After step S21, the CPU 51 calculates the amount of deviation between the thermal image at time t and the thermal image at time t+step based on the result of feature point matching in step S21 (step S23).

[0060] Fig. 8 is an explanatory diagram outlining steps S19 to S23. The upper left of Fig. 8 shows the thermal image at time t referenced in step S11, and the upper right of Fig. 8 shows the thermal image at time t+step referenced in step S15. For the thermal image at time t, the matching area Rm t is selected, and for the thermal image at time t+step, the matching area Rm t+step By executing step S19, the matching area Rm t A number of feature points (shown as black circles in Fig. 8) within the matching region Rm t+step A number of feature points in are calculated.

[0061] Following step S19, feature point matching is performed in step S21, and a matching region Rm t feature points in the matching region Rm t+step 8, the correspondence between the feature points is indicated by dashed lines. For the calculation of the feature points and the matching therebetween in steps S19 and S21, known algorithms such as the ORB algorithm and the AKAZE algorithm can be used.

[0062] 8, the amount of deviation calculated by executing step S23 following step S21 is specified as the amount of deviation of the thermal image at time t+step from the thermal image at time t when the thermal image at time t+step is superimposed on the thermal image at time t (so that their edges overlap), and the thermal image at time t+step is shifted relative to the thermal image at time t based on the result of feature point matching so that corresponding feature points overlap. More specifically, the amount of deviation is the deviation Δx in the x direction, the deviation Δy in the y direction, and the angle deviation Δθ (rotation around the origin of the pixel coordinates of the thermal image at time t) based on the above-mentioned pixel coordinates of the thermal image at time t, and is calculated using a homography matrix calculation method such as the RANSAC algorithm, for example.

[0063] Referring again to FIG. 4, after step S23, the CPU 51 calculates the stitching region width Wp (see FIG. 6) for the thermal image at time t+step based on the correspondence between the feature points resulting from the feature point matching in step S21 and the amount of deviation obtained in step S23, and determines the stitching region Rs for the thermal image at time t+step based on the stitching region width Wp obtained by the calculation and the stitching region upper interval Hsu and the stitching region lower interval Hsb input in step S7. t+step Then, the CPU 51 selects or specifies the selected stitching region Rs t+step The intermediate panoramic thermal image is synthesized by stitching the thermal images from time 0 to time t together in the stitched region Rs0 to Rs t The intermediate panoramic thermal image and the stitched area Rs are combined to create a thermal image. t+step For example, pixel values ​​are averaged between the intermediate panoramic thermal image and the stitched region Rs t+step and are synthesized.

[0064] In this embodiment, in step S25, the stitching area width Wp is the average value of the lateral shift amounts of the multiple feature points at time t+step whose correspondence with the feature points of the thermal image at time t was identified in step S21, relative to the intermediate panoramic image. For each feature point of the thermal image at time t+step whose correspondence was identified, the lateral shift amount relative to the intermediate panoramic image is the lateral shift in the global pixel coordinate system (see FIG. 9) set for the intermediate panoramic image, and can be calculated using the series of shift amounts calculated in step S23 for each of the thermal images from the second thermal image (time is step) to the thermal image at time t+step. Note that the stitching area width Wp may be determined by adding an adjustment width δ (pixels) set by the user to the average value of the calculated lateral shift amounts of the feature points.

[0065] For the thermal image at time 0, the stitching area width Wp of the stitching area Rs0 may be specified by the user. Alternatively, the stitching area width Wp of the thermal image at time 0 may be determined by the panoramic thermal image creation device 5 based on the aerial photographing conditions, and may be set to, for example, the above-mentioned expected image shift width. Since no particular problem occurs even if part of the thermal image at time 0 is not included in the finally obtained panoramic thermal image, in one embodiment, for the thermal image at time 0, no stitching area is selected, and the stitching area width Wp is not specified or calculated.

[0066] After step S25, the CPU 51 increments the time variable t by step (step S27). After step S27, the CPU 51 determines whether the creation of the panoramic thermal image is complete (step S29). If it is determined in step S29 that the creation of the panoramic thermal image is complete, the CPU 51 ends the panoramic thermal image creation process, and stores the data of the panoramic thermal image stored in the RAM 53 in the SSD 55. If it is determined in step S29 that the creation of the panoramic thermal image is not complete, the CPU 51 executes step S11 and the subsequent steps again. In this embodiment, in step S29, the value obtained by adding step to the variable t is t endIf the value is equal to or greater than the value obtained by subtracting step from t (if there is no thermal image to be referenced again in step S17), it is determined that the creation of the panoramic thermal image is complete. end is the time of the last frame, which is the total number of frames that make up the dynamic thermal image minus 1.

[0067] 9 is an explanatory diagram showing how a panoramic thermal image is created in this embodiment. Step S25 is repeatedly executed, and in the first step S25, a global pixel coordinate system (X, Y) is set to identify each pixel constituting the panoramic thermal image or the intermediate panoramic thermal image. For example, the global pixel coordinate system (X, Y) may be the above-mentioned pixel coordinate system for the thermal image at time 0.

[0068] In the first step S25, the stitching region Rs0 of the thermal image at time 0 and the stitching region Rs of the thermal image at time step step are selected, and the stitching area Rs0 and the stitching area Rs step The stitching area Rs step The stitching area width Wp that identifies the stitching area Rs is determined as described above based on the shift amounts (Δx, Δy, Δθ) determined in the first step S23. Specifically, when the thermal image at time step is shifted based on the shift amount (so that corresponding feature points overlap) from the state where the thermal image at time step is superimposed on the thermal image at time 0, the average value of the lateral shift amount ΔX in the global pixel coordinate system is calculated for each of the multiple feature points of the thermal image at time step whose correspondence is identified by feature point matching. For example, when Δθ=0 for the thermal image at time step, the lateral shift amount ΔX of each feature point is Δx, so that the stitching area Rs step The width Wp of the pasted region is Δx (or Δx+δ).

[0069] The placement of the stitching region Rs0 on the global pixel coordinate system (X, Y) is determined arbitrarily (in some embodiments, the stitching region Rs0 is not selected or placed for the reasons described above). Based on the displacement amounts (Δx, Δy, Δθ) determined in step S23, the stitching region Rs step The position of each pixel in the global pixel coordinate system (X, Y) is determined. step The image is shown to be combined to create an intermediate panoramic thermal image.

[0070] In the center of Fig. 9, the overlapping area Rs0 of the thermal image at time 0 to the overlapping area Rs2 of the thermal image at time step *29 is shown. step*29 The lower part of Fig. 9 shows the intermediate panoramic thermal image created by combining the thermal image at time step *30 with the intermediate panoramic thermal image. step*30 The figure shows how the intermediate panoramic image is created by stitching together the thermal images at any time greater than step. t Regarding the bonding area Rs t The position of each pixel in the global pixel coordinate system (X, Y) that constitutes the stitched region Rs is determined using all the displacements (Δx, Δy, Δθ) calculated for the thermal image from the time step. In the example at the bottom of Figure 9, step*30 The position of each pixel in the global pixel coordinate system (X, Y) is determined based on the amount of deviation of a series of thermal images from time step to time step *30, which is determined by repeating step S23.

[0071] Bonding area Rs step*30The stitching area width Wp is calculated as the average value of the lateral shifts (in the global pixel coordinate system) of the multiple feature points at time step *30, for which correspondences with the multiple feature points in the thermal image at time step *29 have been identified, relative to the intermediate panoramic image (shown in the center of FIG. 10). This average value can be calculated using the series of shifts calculated in step S23 for each of the thermal images from time step *29 to time step *29. Specifically, when the thermal image at time step *29 is placed in the global pixel coordinate system and the thermal image at step *30 is superimposed on the thermal image at time step *29, the thermal image at time step *30 is shifted so that the corresponding feature points overlap. The average value of the lateral shifts ΔY in the global pixel coordinate system is calculated for each of the multiple feature points in the thermal image at time step *30, for which correspondences have been identified by feature point matching, and this average value becomes the stitching area width Wp. For images at any time other than time step *30 (excluding time 0), the stitching area Rs step*30 The width Wp of the pasted area can be calculated in the same way.

[0072] As described above, by maintaining the direction of movement of the drone constant during aerial photography, the absolute value of the angle deviation Δθ determined in step S23 is prevented from becoming too large, and therefore the panoramic thermal image created based on the present invention can be generated in a roughly band-like shape, like the intermediate panoramic thermal image shown schematically in the center of Figure 9.

[0073] Figure 10 shows an example of a thermal image obtained from a video of a solar panel array taken from the air by an infrared thermography camera mounted on a drone. t and the bonding area Rs t are shown as examples. The upper example is a thermal image at a certain time t (same as in FIG. 2), and the lower example is a thermal image at time t+step. In step S7 shown in FIG. 4, the matching region Rm t and the bonding area Rs t The parameters for the central array A n and Array A n+1It is desirable to set the width W of the matching area Rm so that it completely includes the width W of the matching area Rm. t is set so that both its left and right sides are sufficiently far from both the left and right edges of the thermal image.

[0074] Based on the above embodiment of the present invention, array A n and Array A n+1 A panoramic thermal image showing the temperature distribution of the arrays can be obtained from video images taken from the air by drone 1 of the arrays arranged in a strip and a line, including array B. n and Array B n+1 A group of arrays arranged in a strip and a line including array C n and Array C n+1 A single panoramic thermal image can be obtained based on the present invention for a group of arrays arranged in a strip and a line, including arrays A, B, C, D, E, F ... n and Array A n+1 , Array B n and Array B n+1 , Array C n and Array C n+1 A single panoramic image showing the temperature distribution of the above may be created.

[0075] In one embodiment of the present invention, as described above, a panoramic thermal image is created using a series of thermal images of the array group taken from the air by the camera 11 mounted on the drone 1, which makes it possible to identify feature points while suppressing the influence of noise that is specific to infrared thermography cameras.In addition, the panoramic thermal image is created by pasting together the central parts of the thermal images that are less affected by noise, thereby improving the accuracy of the temperature distribution of the array group shown in the panoramic thermal image.

[0076] In the above-described embodiment, the matching region Rm t and the bonding area Rs t The center line of the vertical direction of the images was set to pass through the center Q of the thermal image, but the matching area Rm t and / or bonding area Rs tThe vertical center line of the matching area Rm may be set so as to be laterally shifted from the vertical center line of the thermal image. The setting screen 9 shown in FIG. 7 allows the user to set such a matching area Rm t and / or bonding area Rs t The offset amount F (shown in FIG. 6) may be configured to be able to be specified.

[0077] The flowchart shown in Fig. 4 is an exemplary embodiment of the panoramic thermal image creating method of the present invention, and the order of steps may be changed, or steps may be integrated or separated, as long as the present invention is feasible. The block diagram shown in Fig. 3 is an exemplary embodiment of the panoramic thermal image creating device of the present invention, and the computer means related to the panoramic thermal image creating device of the present invention or the panoramic thermal image creating method of the present invention may be configured, for example, by multiple computers to which processing is shared, and is not limited to that shown in the block diagram of Fig. 3.

[0078] In the above description, the subject of the panoramic thermal image is a group of arrays installed in a solar power generation facility, but the subject of the present invention is not limited to a group of arrays or a group of solar panels, or a subject including them. For example, a group of pipes in a dam or various plants may be used as the subject, and a panoramic image showing the temperature distribution of the group of pipes may be created using the present invention.

[0079] In the above description, it is assumed that the drone 1 moves horizontally in the aerial thermal image. However, the panoramic thermal image creation method and apparatus of the present invention can also be applied when the drone 1 moves vertically in the thermal image, and the direction of movement of the aircraft is not particularly limited in the present invention. In the present invention, the direction of movement of the aircraft may be vertical, and the infrared thermography camera is adjusted to capture images of the side or horizontal direction of the aircraft, for example, a pipe installed vertically as the subject. If the subject is tilted, the flight path of the aircraft during aerial photography is set so that the altitude changes accordingly.

[0080] In the above description, the aircraft according to the present invention is a drone, but the aircraft on which the infrared thermography camera is mounted in the present invention is not limited to a drone, and may be a manned aircraft such as an airplane or helicopter.

[0081] The above description is for the purpose of explaining the present invention, and should not be construed as limiting the invention described in the claims or narrowing its scope. Furthermore, the configuration of each part of the present invention is not limited to the above embodiment, and various modifications are possible within the technical scope described in the claims. [Explanation of symbols]

[0082] 1. Drone 11. Infrared Thermography Camera 3. Solar panel array 31 Solar Panel 5. Panoramic thermal imaging device 51 CPU 58 Display device 59 Input Devices 7 Dynamic thermal image data file 9. Settings screen 92 slots 93 horizontal line 94 horizontal lines Rm t Matching Area Wm Matching area width Rs t Stitching area Ws Width of the bonding area

Claims

1. 1. A method for producing a panoramic thermal image from a series of thermal images using computer means, comprising: Each of the series of thermal images is a rectangular still image corresponding to a frame of a moving image obtained by photographing a subject during flight of the aircraft using an infrared thermography camera mounted on the aircraft; the computer means selecting, based on input from a user, for each thermal image, a same rectangular or square first image area that is part of the thermal image; the computer means identifying a plurality of feature points within a first image region of each thermal image; the computer means performs feature point matching for each pair of temporally adjacent thermal images in the series of thermal images based on a plurality of feature points identified within a first image region of each thermal image; the computer means selecting a rectangular or square second image area that is part of each thermal image from at least a second thermal image to a last thermal image in the series of thermal images, the computer means selecting the second image area of ​​the thermal image based on input from a user and results of feature point matching performed on each pair of temporally adjacent thermal images in the thermal images from the first thermal image to the last thermal image; said computer means synthesizing a second image area of ​​at least a second thermal image from a last thermal image in said series of thermal images to generate a panoramic thermal image; It contains A method wherein a first image area of ​​each thermal image in the series of thermal images includes a center of the thermal image, and a second image area of ​​each thermal image from at least a second thermal image to a last thermal image in the series of thermal images includes a center of the thermal image.

2. 2. The method of claim 1, wherein the computer means sets a lower limit on the width of the first image region of each thermal image based on the photographing conditions of the subject.

3. The method of claim 1 , wherein the width of the first image region of each thermal image is limited to no more than a predetermined value that is less than the width of each thermal image.

4. The method of claim 1 , wherein the object comprises a group of solar panels.

5. The method according to claim 1 , wherein the flying object is controlled to maintain its speed and direction of movement while the subject is being photographed.

6. 1. A non-transitory computer readable medium storing program instructions executable on a computer means for creating a panoramic thermal image from a series of thermal images, the program instructions comprising: Each of the series of thermal images is a rectangular still image corresponding to a frame of a moving image obtained by photographing a subject during flight of the aircraft using an infrared thermography camera mounted on the aircraft; The program instructions cause the computer means to: selecting, for each thermal image, a same rectangular or square first image region that is part of the thermal image based on input from a user; identifying a plurality of feature points within a first image region of each thermal image; performing feature point matching for each pair of temporally adjacent thermal images in the series of thermal images based on a plurality of feature points identified within a first image region of each thermal image; selecting a rectangular or square second image area that is part of each thermal image from at least a second thermal image to a last thermal image in the series of thermal images, the second image area being selected based on input from a user and results of feature point matching performed on each pair of temporally adjacent thermal images from the first thermal image to the last thermal image; synthesizing a second image area of ​​at least a second thermal image through a last thermal image in the series of thermal images to generate a panoramic thermal image; and a first image area of ​​each thermal image in the series of thermal images includes a center of the thermal image, and a second image area of ​​each thermal image from at least a second thermal image to a last thermal image in the series of thermal images includes a center of the thermal image.

7. A panoramic thermal image creation device for creating a panoramic thermal image from a series of thermal images, comprising: Each of the series of thermal images is a rectangular still image corresponding to a frame of a moving image obtained by photographing a subject during flight of the aircraft using an infrared thermography camera mounted on the aircraft; The panoramic thermal imaging device includes: selecting, for each thermal image based on input from a user, the same rectangular or square first image region that is part of the thermal image; Identifying a plurality of feature points within a first image region of each thermal image; performing feature point matching for each pair of temporally adjacent thermal images in the series of thermal images based on the plurality of feature points identified within the first image region of each thermal image; selecting, for each thermal image from at least the second thermal image to the last thermal image in the series of thermal images, a rectangular or square second image area that is part of the thermal image; synthesizing a second image region of at least a second thermal image through a last thermal image in the series of thermal images to generate a panoramic thermal image; for each thermal image from at least the second thermal image to the last thermal image in the series of thermal images, a second image region of the thermal image is selected based on input from a user and results of feature point matching performed on each pair of temporally adjacent thermal images in the thermal images from the first thermal image to the last thermal image; a first image area of ​​each thermal image in the series of thermal images includes a center of the thermal image, and a second image area of ​​each thermal image from at least a second thermal image through a last thermal image in the series of thermal images includes a center of the thermal image.

Images