Identification method for longitude and latitude of solar panel

Abstract

To provide an identification method for longitude and latitude of a solar panel that can identify longitude and latitude of a solar panel with high accuracy.SOLUTION: An identification method for longitude and latitude of a solar panel comprises steps of: acquiring pixel coordinate values of four vertices pt1 to pt4 of each of a plurality of solar panels 41 in a captured image 40; creating a first image 42 in which coordinate points in a rectangular coordinate system have distance conversion values; creating a third image 48 in which coordinate points of all pixels have data of real spatial distances in two coordinate axis directions in the rectangular coordinate system; creating a fourth image 50 in which all pixels have latitude and longitude information; and identifying, by comparing the fourth image 50 and the captured image 40, the latitude and longitude of a center of gravity of an abnormal panel.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a method for specifying the longitude and latitude of a solar panel based on a photographed image by a flying object.

Background Art

[0002] Conventionally, as an inspection device for a solar panel, the one described in Patent Document 1 is known. This inspection device is incorporated in a flying object. In this inspection device, while the flying object is located above the solar panel, inspection light is irradiated onto the solar panel, and damage to the solar panel is detected based on the difference between the irradiation optical axis and the reflected optical axis of the reflected light.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Generally, when damage to a solar panel is detected, it is necessary to specify the position such as the longitude and latitude of the solar panel in order to repair the solar panel. On the other hand, according to the above-described conventional detection device, although damage to the solar panel can be detected, there is a problem that it is impossible to specify the position of the damaged solar panel. This point is a problem not only in the case of a damaged solar panel but also when specifying the position of a solar panel that needs to be inspected for some reason.

[0005] The present invention has been made to solve the above problems, and an object thereof is to provide a method for specifying the longitude and latitude of a solar panel that can accurately specify the longitude and latitude of the solar panel.

Means for Solving the Problems

[0006] In order to achieve the above object, the invention according to claim 1 is a method for specifying the longitude and latitude of a predetermined part of a predetermined solar panel, which is at least one of a plurality of solar panels, based on a photographed image including a plurality of solar panels photographed from above by a photographing device and the longitude and latitude information of a positioning signal at the time of photographing. The method includes: a vertex coordinate value acquisition step of acquiring, by a predetermined image process, the coordinate values of pixels of four vertices of each of a plurality of solar panels including the predetermined solar panel in a first orthogonal coordinate system, which is an orthogonal coordinate system along the vertical and horizontal directions of the photographed image with a predetermined location of the photographed image as the origin; a first image creation step of extracting coordinate points of pixels in the first orthogonal coordinate system at predetermined positions of each of two opposite sides of each solar panel from the photographed image, and creating a first image configured such that the coordinate points have a distance conversion value, which is a conversion value of the real space distance per pixel in the two coordinate axis directions of the first orthogonal coordinate system; a second image creation process of creating a second image as an image configured such that the coordinate points of pixels in the first orthogonal coordinate system of the first image have a distance conversion value by performing a predetermined interpolation operation between the coordinate points of the pixels on the first image; a third image creation step of creating a third image as an image configured such that the coordinate points of pixels in the first orthogonal coordinate system have data of the real space distance in the two coordinate axis directions with the pixel at the center of the image as the origin based on the distance conversion value of the coordinate points of the pixels in the second image; a fourth image creation step of creating a fourth image as an image configured such that the pixel at the origin of the first orthogonal coordinate system, which is the center of the image, has the longitude and latitude information of the positioning signal and the pixels have the longitude and latitude information based on the longitude and latitude information of the positioning signal and the third image; and a longitude and latitude specification step of specifying the longitude and latitude of the predetermined part of the predetermined solar panel by comparing the fourth image and the photographed image, which are executed by an arithmetic processing device.

[0007] According to this method for specifying longitude and latitude, the coordinate values of the pixels at the four vertices of each of a plurality of solar panels including a predetermined solar panel in a first orthogonal coordinate system, which is an orthogonal coordinate system along the up, down, left, and right directions of the captured image with a predetermined location of the captured image as the origin, are obtained by a predetermined image process. While extracting the coordinate points of the pixels in the first orthogonal coordinate system at predetermined positions of each of the two opposite sides of each solar panel from the captured image, a first image is created such that the coordinate points have a distance conversion value that is a conversion value of the actual space distance per pixel in the two coordinate axis directions of the first orthogonal coordinate system.

[0008] Furthermore, by performing a predetermined interpolation operation between the coordinate points of the pixels on the first image, a second image is created as an image configured such that the coordinate points of the pixels in the first orthogonal coordinate system of the first image have a distance conversion value. Based on the distance conversion value of the coordinate points of the pixels in the second image, a third image is created as an image configured such that the coordinate points of the pixels in the first orthogonal coordinate system with the pixel at the center of the image as the origin have data of the actual space distance in the two coordinate axis directions. Then, based on the longitude and latitude information of the positioning signal and the third image, a fourth image is created as an image configured such that the pixel at the origin of the first orthogonal coordinate system, which is the center of the image, has the longitude and latitude information of the positioning signal and the pixels have longitude and latitude information. By comparing this fourth image and the captured image, the longitude and latitude of a predetermined part of the predetermined solar panel are specified. As described above, by creating the fourth image such that the pixels have longitude and latitude information, when such a fourth image and the captured image are compared, the longitude and latitude of the pixels in the captured image can be accurately specified. Therefore, according to this method for specifying longitude and latitude, the longitude and latitude of a predetermined part of a predetermined solar panel can be accurately specified. Note that "comparing the fourth image and the captured image" in this specification is not limited to comparing the fourth image and the captured image, but also includes overlapping and comparing the two.

[0009] In the present invention, it is preferable that the imaging device is mounted on an aircraft capable of receiving a positioning signal.

[0010] According to the method for identifying the latitude and longitude of this solar panel, since the imaging device is mounted on a flying object capable of receiving a positioning signal, a large number of solar panels can be photographed over a wide range. Thereby, the predetermined solar panel for which the latitude and longitude are to be identified can be searched over a wide range.

[0011] In the present invention, the flying object is equipped with, as an imaging device, a first imaging device for photographing a visible image and a second imaging device for photographing an infrared image. In the vertex coordinate value acquisition step, the coordinate values of the pixels at the four vertices of each solar panel in the visible image are acquired. In the first image creation step, the first image is created from the visible image. In the latitude and longitude identification step, when the imaging timings of the first imaging device and the second imaging device are simultaneous, or when the flying object is in a hovering state, the solar panels in the visible image and the infrared image are adjusted to have the same size, and the visible image and the infrared image are corrected for positional deviation caused by the installation states of the first imaging device and the second imaging device so that the centers of the fourth image and the infrared image are aligned. By superimposing the fourth image and the infrared image, it is preferable to identify the latitude and longitude of a predetermined part of a predetermined solar panel in the infrared image.

[0012] Here, when the first imaging device and the second imaging device are mounted on the flying object, even when the imaging timings of the first imaging device and the second imaging device are simultaneous or when the flying object is in a hovering state, a positional deviation of the captured images occurs due to the installation states of the two imaging devices. In contrast, according to the method for identifying the latitude and longitude of this solar panel, the solar panels in the visible image and the infrared image are adjusted to have the same size, and the visible image and the infrared image are corrected for positional deviation caused by the installation states of the first imaging device and the second imaging device so that the centers of the fourth image and the infrared image are aligned. By superimposing the fourth image and the infrared image, the latitude and longitude of a predetermined part of a predetermined solar panel in the infrared image are identified. Thereby, while correcting the positional deviation of the captured images of the first imaging device and the second imaging device, the latitude and longitude of a predetermined solar panel in the infrared image can be accurately identified.

[0013] In general, when detecting abnormalities in a solar panel based on a captured image, it is known that an infrared image is easier to detect compared to a visible image. Therefore, when a predetermined solar panel is regarded as a solar panel with an abnormality, the longitude and latitude of a predetermined part of the solar panel with the abnormality can be accurately specified. Further, the solar panel specified as having an abnormality in the infrared image can also be specified on the visible image to confirm the situation.

[0014] In the present invention, in the longitude and latitude specifying step, when there is a time difference in the imaging timings of the first imaging device and the second imaging device, when the imaging by the first imaging device and the second imaging device is performed during the flight of the aircraft, the solar panels in the visible image and the infrared image are adjusted to be the same size, and the positions of the fourth image and the infrared image are adjusted so that the centers of the fourth image and the infrared image are aligned. By superimposing the fourth image and the infrared image in a state where the displacement of the captured image due to the installation state of the first imaging device and the second imaging device and the displacement of the captured image due to the time difference in the imaging timings are corrected, it is preferable to specify the longitude and latitude of a predetermined part of a predetermined solar panel in the infrared image.

[0015] As in the present invention, when there is a time difference in the imaging timings of the first imaging device and the second imaging device, when the aircraft is flying, due to this time difference, a displacement of the captured images of the fourth image and the infrared image occurs. On the other hand, according to this method for specifying the longitude and latitude of the solar panel, the solar panels in the visible image and the infrared image are adjusted to be the same size, and the positions of the fourth image and the infrared image are adjusted so that the centers of the fourth image and the infrared image are aligned. By superimposing the fourth image and the infrared image in a state where the displacement of the captured image due to the installation state of the first imaging device and the second imaging device and the displacement of the captured image due to the time difference in the imaging timings are corrected, the longitude and latitude of a predetermined part of a predetermined solar panel in the infrared image are specified. Thereby, while correcting the displacement of the captured image due to the installation state and the imaging timings of the first imaging device and the second imaging device, the longitude and latitude of a predetermined solar panel in the infrared image can be accurately specified.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Mode for Carrying Out the Invention

[0017] Hereinafter, with reference to the drawings, a method for identifying the longitude and latitude of a solar panel according to the first embodiment of the present invention will be described. The method for identifying the longitude and latitude of the solar panel in this embodiment is executed by the arithmetic processing unit 1 shown in FIG. 1.

[0018] This arithmetic processing unit 1 is of a personal computer type and includes a display 1a, a device main body 1b, an input interface 1c, and the like. The display 1a as an output interface is of a liquid crystal display type, and various data are displayed. The input interface 1c is composed of a keyboard, a mouse, and the like for operating the arithmetic processing unit 1.

[0019] Further, the device main body 1b includes a storage such as an HDD, a processor, a memory (RAM, E2PROM, ROM, etc.) (none of which are shown). Application software for executing various arithmetic processes described later is installed in the storage of this device main body 1b, and the shooting data described below is stored.

[0020] Specifically, as the shooting data, a shooting image (visible image) of a large number of solar panels 14 in the solar farm 10 taken by the drone 20 shown in FIG. 2, the shooting date and time, and the longitude and latitude data at the time of image shooting are stored in the storage of the device main body 1b.

[0021] As shown in FIG. 2, a large number of solar arrays 12 are installed in the solar farm 10, and a large number of solar panels 14 are attached to the surface of each solar array 12 in a grid-like arrangement pattern. A passage 11 is secured between these solar arrays 12, 12, and this passage 11 is for workers and the like to pass through when inspecting and maintaining each solar array 12.

[0022] The drone 20 is equipped with a visible camera 21 and a GPS receiver (not shown). The drone 20 flies over the solar farm 10 and uses the visible camera 21 to photograph the solar arrays 12 in the solar farm 10 from above. At that time, in addition to the visible image, photographing data is created to include the longitude and latitude at the time of photographing and the data of the photographing date and time. In this case, the longitude and latitude at the time of photographing are determined from the GPS signal (positioning signal) received by the GPS receiver.

[0023] This photographing data is transmitted online to the arithmetic processing unit 1 and stored in the storage of the arithmetic processing unit 1. Note that the photographing data may be configured to be finally stored in the storage of the arithmetic processing unit 1 after being recovered from the memory in the drone 20. Also, in this embodiment, the drone 20 corresponds to the flying object, and the visible camera 21 corresponds to the photographing device.

[0024] Next, with reference to FIG. 3, the longitude and latitude identification process of the solar panel executed by the arithmetic processing unit 1 will be described. This longitude and latitude identification process is executed for each photographed image of the photographing data. Note that various data obtained, created, or identified in the following description are appropriately stored in the arithmetic processing unit 1. Also, as the photographed image, data photographed so that the upper side of the image is the north side, or data obtained by performing rotation correction processing on data photographed so that the upper side of the image is any of the east side, west side, and south side so that the upper side of the image is the north side is used.

[0025] As shown in the figure, first, vertex coordinate acquisition processing is executed (Fig. 3 / STEP1). In this vertex coordinate acquisition processing, the coordinate values of the pixels of the four vertices of each solar panel in the captured image are acquired.

[0026] Although a detailed explanation of this vertex coordinate acquisition processing is omitted here, in this vertex coordinate acquisition processing, the coordinate values of the pixels of the four vertices of the solar panel are acquired by the acquisition method (hereinafter referred to as the "proposed acquisition method") proposed by the applicant in Japanese Patent Application No. 2023-066434.

[0027] For example, by performing three types of filter processing and three types of binarization processing on the captured image 30 shown on the left side of Fig. 4, nine binarized images 31 as shown on the right side of Fig. 4 are created. Next, the four vertices of each solar panel 31a in the nine binarized images 31 are acquired by the following procedures (a1) to (a4).

[0028] (a1) In each of the nine binarized images, when the contour of the portion surrounded by a low luminance group (e.g., black) or a high luminance group (e.g., white) is detected as a polygon and the number of vertices of the polygon is four, a first set is generated by collecting all the sets of the four vertices. (a2) In the first set, the overlapping sets are combined into one to generate a second set. (a3) In the second set, a third set is generated by leaving only the sets of four vertices that are valid as the four vertices corresponding to the four vertices of the actual solar panel 14. (a4) The four vertices of each set in the third set are determined as the four vertices of each solar panel 31a.

[0029] Note that the method for acquiring the four vertices of the solar panel is not limited to the above acquisition method, and other acquisition methods in the proposed acquisition method may also be used.

[0030] In the following description, when the captured image is the captured image 40 shown in Fig. 5, it is described that the four vertices pt1 to pt4 (see the enlarged view on the right side of Fig. 5) of each solar panel 41 in the captured image 40 are acquired by the above method.

[0031] Furthermore, the coordinate values of the pixels of the four vertices pt1 to pt4 are set to the coordinate values of an orthogonal coordinate system (hereinafter referred to as the "first orthogonal coordinate system") along the vertical and horizontal directions in FIG. 5 with the pixel at the upper left corner of the captured image 40 as the origin.

[0032] Returning to FIG. 3, after the vertex coordinate acquisition process is executed as described above, the first image creation process is executed (FIG. 3 / STEP2). In this first image creation process, a first image 42 shown in FIG. 6 is created based on the coordinate values of the pixels of the four vertices pt1 to pt4 of each of the solar panels 41. This first image 42 is an image of the same size as the captured image 40 and is created as an image in which a large number of pixel coordinate points 42a are arranged. These coordinate points 42a are determined as follows.

[0033] That is, as shown in FIG. 7, from the coordinate values of the pixels of the four vertices pt1 to pt4 of each of the solar panels 41, the coordinate values of the pixels of the midpoints pm12 and pm34 of the upper and lower sides of each solar panel 41 are calculated. Then, for each of these midpoints pm12 and pm34, a large number of pixel coordinate points 42a are determined by adding two distance conversion values (unit: cm / pixel), which are the conversion values of the actual space distance per pixel.

[0034] Here, one distance conversion value of the coordinate point 42a corresponding to the midpoint pm12 is calculated by dividing the actual distance between the two points pt1 to pt2 by the Euclidean distance between the two points pt1 to pt2, with the two points pt1 and pt2 regarded as coordinates corresponding to the pixel positions in the image resolution. Furthermore, the other distance conversion value of the coordinate point 42a corresponding to the midpoint pm12 is calculated by dividing the actual distance between the two points pm12 to pm34 by the Euclidean distance between the two points pm12 to pm34, with the two points pm12 and pm34 regarded as coordinates corresponding to the pixel positions in the image resolution.

[0035] Also, one distance conversion value of the coordinate point 42a corresponding to the midpoint pm34 is calculated by dividing the actual distance between the two points pt3 to pt4 by the Euclidean distance between the two points pt3 to pt4, with the vertex pt3 and the vertex pt4 being the coordinates corresponding to the pixel positions in the image resolution, as described above. Further, the other distance conversion value of the coordinate point 42a corresponding to the midpoint pm34 is calculated by dividing the actual distance between the two points pm12 to pm34 by the Euclidean distance between the two points pm12 to pm34, with the midpoint pm12 and the midpoint pm34 being the coordinates corresponding to the pixel positions in the image resolution, as described above.

[0036] As described above, each of the coordinate points 42a of a large number of pixels is determined to have the coordinate values of the first orthogonal coordinate system and two distance conversion values. In this embodiment, the first image creation process corresponds to the first image creation step.

[0037] Returning to FIG. 3, after the first image creation process is executed as described above, the second image creation process is executed (FIG. 3 / STEP3). In this second image creation process, a second image is created by performing the following image interpolation process on the first image 42.

[0038] That is, as shown in FIG. 8, a cubic interpolation image 43 is created by performing cubic interpolation processing on the first image 42. At the same time, a nearest interpolation image 44 is created by performing nearest interpolation processing on the first image 42. Then, a second image 45 is created by performing a merging process (merged process) of combining these cubic interpolation image 43 and nearest interpolation image 44. In this case, in the overlapping portion between the cubic interpolation image 43 and the nearest interpolation image 44, the cubic interpolation image 43 is adopted.

[0039] By the above method, the second image 45 is configured such that the coordinate points of the first orthogonal coordinate system in all pixels have two distance conversion values. In addition, when creating the second image 45 from the first image 42, the reason for using both cubic interpolation processing and nearest interpolation processing is that with only cubic interpolation processing, it is not possible to configure the coordinate points of all pixels to have distance conversion values as in the second image 45. In this embodiment, the second image creation process corresponds to the second image creation step.

[0040] Returning to FIG. 3, after the second image creation process is executed as described above, the third image creation process is executed (FIG. 3 / STEP4). In this third image creation process, by performing various processes described below on the second image 45, the third image 48 shown on the right side of FIG. 11 is created. In the case of this third image 48, each pixel is created to have difference data of the distance from the origin with the center of the third image 48 as the origin of the first orthogonal coordinate system.

[0041] Specifically, first, in the second image 45 (distance density map) shown on the left side of FIG. 9, by calculating the cumulative sum of the distances in the NS direction (the vertical direction in FIG. 9) from the x-axis (i.e., the horizontal axis) of the first orthogonal coordinate system with the center of the second image 45 as the origin, the NS direction difference map 46 shown on the right side of FIG. 9 is created.

[0042] In this case, for pixels located north (the upper side in FIG. 9) of the x-axis of the first orthogonal coordinate system (hereinafter simply referred to as the "x-axis"), based on the distance conversion value that the pixel has, the distance from the x-axis is cumulatively added to calculate a positive distance value. On the other hand, for pixels located south (the lower side in FIG. 9) of the x-axis, based on the distance conversion value that the pixel has, the distance from the x-axis is cumulatively subtracted to calculate a negative distance value. As described above, each pixel of the NS direction difference map 46 is calculated to have difference data of the north-south direction distance from the x-axis.

[0043] Further, in the second image 45 (distance density map) shown on the left side of FIG. 10, an EW direction difference map 47 shown on the right side of FIG. 10 is created by calculating the cumulative sum of the distances in the east-west direction (the left-right direction in FIG. 10) from the y-axis (i.e., the vertical axis) of the first orthogonal coordinate system of the second image 45.

[0044] In this case, for pixels located on the east side (the right side of FIG. 10) of the y-axis of the first orthogonal coordinate system (hereinafter simply referred to as the "y-axis"), based on the distance conversion value of the pixel, the distance from the y-axis is cumulatively added so as to have a positive distance value. On the other hand, for pixels located on the west side (the left side of FIG. 10) of the y-axis, based on the distance conversion value of the pixel, the distance from the y-axis is cumulatively subtracted so as to have a negative distance value. As described above, each pixel of the EW direction difference map 47 is calculated to have difference data of the distance in the east-west direction from the y-axis.

[0045] Next, as shown in FIG. 11, a third image 48 is created by combining the distance difference data of the pixels in the NS direction difference map 46 and the distance difference data of the pixels in the EW direction difference map 47. Each pixel of this third image 48 is configured to have difference data of the distance from the origin with the center of the third image 48 as the origin of the first orthogonal coordinate system. In this embodiment, the third image creation process corresponds to the third image creation step.

[0046] Returning to FIG. 3, after the third image creation process is executed as described above, a fourth image creation process is executed (FIG. 3 / STEP5). In this fourth image creation process, a fourth image 50 shown on the left side of FIG. 12 is created based on the GPS signal received by the drone 20 at the time of shooting the shooting image 40 and the above third image 48.

[0047] This fourth image 50 is created by setting the latitude and longitude of the pixel at the center of the third image 48 to the latitude and longitude represented by the GPS signal, and determining the latitude and longitude of each pixel based on the difference data of the distances from the x-axis and y-axis at each pixel of the third image 48. In the following description, an image in which the latitude and longitude of each pixel is specified, such as the fourth image 50, is appropriately referred to as a "latitude and longitude specified image".

[0048] In the case of the fourth image 50 created as described above, all pixels will correspond to the pixels of the captured image 40 shown on the right side of FIG. 12. For example, the pixel at the center point p0, the pixel at the upper left corner point p1, and the pixel at the lower right corner point p2 will respectively correspond to the pixel at the midpoint px0, the pixel at the upper left corner point px1, and the pixel at the lower right corner point px2 of the captured image 40. In this embodiment, the fourth image creation process corresponds to the fourth image creation step.

[0049] Returning to FIG. 3, after the fourth image creation process is executed as described above, the centroid identification process is executed (FIG. 3 / STEP6). In this centroid identification process, by comparing the captured image 40 and the above-described fourth image 50, the latitude and longitude of the centroid of each solar panel 41 in the captured image 40 are identified. Specifically, it is determined which pixel in the fourth image 50 corresponds to the pixels at the four vertices of each solar panel 41 in the captured image 40, and by obtaining the latitude and longitude of the corresponding pixel, the latitude and longitude of the pixels at the four vertices of each solar panel 41 in the captured image 40 are identified. Then, based on the latitude and longitude of the pixels at the four vertices, the latitude and longitude of the centroid of each solar panel 41 in the captured image 40 are identified.

[0050] Subsequent to the centroid identification process, an abnormality determination process is executed (FIG. 3 / STEP7). This abnormality determination process determines whether an abnormality has occurred in any of the solar panels 41 within the captured image 40.

[0051] Although a detailed description of this abnormality determination process will be omitted here, in this abnormality determination process, an abnormality determination of the solar panel 41 is executed by the abnormality determination method (hereinafter referred to as the "proposed abnormality determination method") proposed by the applicant in Japanese Patent Application No. 2023-066434.

[0052] That is, based on the difference luminance between the output of the VAE (Variational Autoencoder) when the captured image of the solar panel 41 is input to the VAE for which the learning of the model parameters has been executed and the input captured image, an abnormality determination of the solar panel 41 is executed. In the following description, the solar panel 41 determined to be abnormal is referred to as an "abnormal panel".

[0053] Next, based on the result of the above abnormality determination process, it is determined whether there is an abnormal panel (Fig. 3 / STEP8). If this determination is negative (Fig. 3 / STEP8…NO) and there is no abnormal panel, this process ends as it is.

[0054] On the other hand, if this determination is positive (Fig. 3 / STEP8…YES) and there is an abnormal panel, an abnormal panel identification process is executed (Fig. 3 / STEP9). In this abnormal panel identification process, based on the third image 53, the longitude and latitude of the pixel at the center of gravity of the abnormal panel are identified (longitude and latitude identification step). Then, this process ends. In this embodiment, the longitude and latitude of the pixel at the center of gravity of the abnormal panel correspond to the longitude and latitude of a predetermined part of a predetermined solar panel.

[0055] Next, with reference to Fig. 13, the overall image creation process executed by the arithmetic processing unit 1 will be described. This overall image creation process creates an ortho image and, when there is an abnormal panel, creates an image showing the position of the abnormal panel within the ortho image, as described below.

[0056] In this overall image creation process, first, an ortho-image creation process is executed (Fig. 13 / STEP21). In this ortho-image creation process, ortho-correction is performed on a plurality of captured images captured so as to overlap each other, thereby creating an ortho-image 60 as shown in Fig. 14. In the case of this ortho-image 60, images of a large number of solar arrays 61 are included, and it is created such that the upward direction in Fig. 14 is the north side, the downward direction is the south side, the left direction is the west side, and the right direction is the east side.

[0057] Next, it is determined whether there is an abnormal panel (Fig. 13 / STEP22). In this case, in any of the solar panels in the ortho-image 60, when the longitude and latitude of the pixel at the center of gravity of the abnormal panel have been specified as described above, it is determined that there is an abnormal panel, and in other cases, it is determined that there is no abnormal panel.

[0058] If this determination is negative (Fig. 13 / STEP22…NO) and there is no abnormal panel, this process ends as it is. On the other hand, if this determination is positive (Fig. 13 / STEP22…YES) and there is an abnormal panel, an abnormal panel position display process is executed (Fig. 13 / STEP23).

[0059] In this abnormal panel position display process, as shown in Fig. 15, an image of a pin 63 indicating the position of the abnormal panel is displayed in the ortho-image 60. In this case, the tip of the pin 63 is displayed at the position of the pixel at the center of gravity of the abnormal panel. After executing the abnormal panel position display process as described above, this process ends.

[0060] As described above, according to the method for specifying the longitude and latitude of the solar panel of the first embodiment, for example, from the captured image 40, the third image 53 is created so that all pixels have longitude and latitude information. Then, by comparing such a third image 53 and the captured image 40, the longitude and latitude of all pixels in the captured image 40 can be accurately specified, and the longitude and latitude of the center of gravity of the abnormal panel can be accurately specified. Further, when the overall image creation process is executed, the position of the abnormal panel in the ortho-image 60 can be displayed by the pin 63.

[0061] Also, when the overall image creation process described above is executed, if there is an abnormal panel, the ortho image 60 is created with the pin 63 indicating the position of the abnormal panel being displayed. Therefore, an operator who performs the repair work on the abnormal panel can easily find the abnormal panel in the solar farm by carrying the printed matter of this ortho image 60.

[0062] In addition, according to the longitude and latitude identification method of the first embodiment, by comparing the captured image 40 and the fourth image 50, not only for the abnormal panel, but also the longitude and latitude of the center of gravity or the four vertices of all the solar panels 41 whose four vertices are acquired in the captured image 40 can be identified. Therefore, the predetermined solar panel may be any of all the solar panels 41 whose center of gravity or four vertices are acquired, not limited to the abnormal panel.

[0063] Also, in the first embodiment, an example using a personal computer type as the arithmetic processing device is shown. However, instead of this, as the arithmetic processing device, a server may be used, or a combination of a plurality of servers or a plurality of personal computers may be used, or a combination of a server and a personal computer may be used. For example, a personal computer and a server may be configured to be communicable, and various arithmetic processes may be executed on the server side when the user operates the personal computer.

[0064] Furthermore, in the first embodiment, when determining the coordinate point 42a in the first image 42, an example using the pixels of the midpoints pm12 and pm34 of the upper and lower sides of each solar panel 41 is shown. However, pixels at predetermined positions on the upper and lower sides may also be used. In this case, the predetermined positions on the upper and lower sides may be positions other than the midpoints.

[0065] Also, when determining the coordinate point 42a in the first image 42, instead of the pixels at the midpoints pm12 and pm34 of the upper and lower sides of each solar panel 41, the pixel at the midpoint on the left side (the part between the vertices pt1 and pt3) of each solar panel 41 and the pixel at the midpoint on the right side (the part between the vertices pt2 and pt4) may be used.

[0066] In that case, the distance conversion value of the coordinate point 42a corresponding to the midpoint on the left side is calculated by dividing the actual distance between the two points pt1 to pt3 by the Euclidean distance between the two points pt1 to pt3, with the vertices pt1 and pt3 regarded as coordinates corresponding to the pixel positions in the image resolution, as described above. Also, the distance conversion value of the coordinate point 42a corresponding to the midpoint on the right side is calculated by dividing the actual distance between the two points by the Euclidean distance between the two points pt2 to pt4, with the vertices pt2 and pt4 regarded as coordinates corresponding to the pixel positions in the image resolution.

[0067] Also, instead of the pixels at the midpoints of the left and right sides of each solar panel 41, pixels at predetermined positions on the left and right sides may be used. In this case, the predetermined positions on the left and right sides may be any positions other than the midpoints.

[0068] Furthermore, the first embodiment is an example using image interpolation processing as a method for creating the second image 45 from the first image 42, but the second image 45 may be created from the first image 42 using surface approximation by polynomial regression.

[0069] Also, the image indicating the position of the abnormal panel in the ortho-image 60 is not limited to the image of the pin 63 in the first embodiment, and any image that can show the position of the abnormal panel is acceptable. For example, the position of the abnormal panel may be configured to be indicated by a square image or a round image, or the part within the four vertices of the abnormal panel may be configured to be changed to a prominent color compared to the normal solar panel.

[0070] Furthermore, in the first embodiment, as the predetermined solar panel, an example is given in which the latitudes and longitudes of the four vertices and the centroid of the abnormal panel are specified. However, as the predetermined solar panel, it may be configured to specify the latitudes and longitudes of a predetermined part of the solar panel 14 without abnormality. For example, when comparing the third image 53 of the first embodiment with the captured image 40, it is possible to specify the latitudes and longitudes of all the pixels in the captured image 40. As a result, in all the solar panels 41 in which the coordinate values of the pixels at the four vertices in the captured image 40 are acquired, it is possible to specify the latitudes and longitudes of the predetermined part (for example, the four vertices or the centroid).

[0071] On the other hand, in the first embodiment, an example is given in which the solar panel 14 is photographed from above by the photographing device 21 mounted on the flying object 20. However, instead of this, it may be configured to photograph the solar panel 14 from above by the photographing device 21 attached to a crane or the like.

[0072] Furthermore, in the first embodiment, an example is given in which the drone 20 is used as the flying object. However, instead of this, a flying ship, a glider, an airplane, or the like may be used as the flying object.

[0073] Also, in the captured image, when the solar panel is photographed in a state where it is considerably inclined with respect to the left - right direction (or the up - down direction) of the captured image, an image for specifying the latitude and longitude (hereinafter referred to as "latitude - longitude specifying image") may be created by the following method, and the latitude and longitude of the centroid of the solar panel in the captured image may be specified.

[0074] That is, first, an image corresponding to the second image 45 (hereinafter referred to as "second - equivalent image") is created from the captured image by the method described above. The coordinate values of the pixels of this second - equivalent image are set to the coordinate values of the second orthogonal coordinate system along the upper and lower sides and the left and right sides of the inclined solar panel.

[0075] Next, the pixels of this second - equivalent image are rotation - transformed using the rotation matrix M, and an image (hereinafter referred to as "transformed image") in which the coordinate values of the pixels of the image are transformed into the coordinate values of the first orthogonal coordinate system along the up - down, left - right directions of the image is created.

[0076] Furthermore, from this converted image, two difference maps corresponding to the NS-direction difference map 46 and the EW-direction difference map are created by the method described above. By rotating and transforming the coordinate values of the pixels of these two difference maps into the coordinate values of the second orthogonal coordinate system using the inverse matrix of the rotation matrix M described above, two transformed difference maps are created.

[0077] Then, by combining the difference data of the distances at the pixels of these two transformed difference maps, an image corresponding to the third image 48 (hereinafter referred to as the "third-corresponding image") is created.

[0078] Next, based on the third-corresponding image and the GPS signal, an image corresponding to the fourth image 50, that is, a longitude and latitude identification image, is created. Then, by comparing this longitude and latitude identification image with the captured image, the longitude and latitude of the pixel at the center of gravity of each solar panel in the captured image are identified. Even when the above method is used, similar to the method of the first embodiment, the longitude and latitude of the pixel at the center of gravity of each solar panel in the captured image can be accurately identified.

[0079] Next, with reference to FIGS. 16 to 20, a method for identifying the longitude and latitude of a solar panel according to the second embodiment will be described.

[0080] In the case of this embodiment, an image of the solar panel is captured using the drone 17A shown in FIG. 16. In addition to the visible camera 18 and the GPS receiver described above, an infrared camera 19 is mounted on this drone 17A. When the drone 17A flies over the solar farm 10, visible and infrared images of the solar array 12 in the solar farm 10 are captured by the visible camera 18 and the infrared camera 19, respectively.

[0081] At that time, imaging data is created so as to include the moving speed of the drone 17A as information in addition to the visible image, the infrared image, the longitude and latitude at the time of imaging, and the imaging date and time data described above. In this embodiment, the drone 17A corresponds to the flying object, the visible camera 18 corresponds to the first imaging device, and the infrared camera 19 corresponds to the second imaging device.

[0082] In addition, when taking images with the visible camera 18 and the infrared camera 19, even when the drone 17A is in a hovering state, as shown in FIG. 16, due to the installation states of both, a deviation occurs between the imaging point of the visible image and the imaging point of the infrared image. Hereinafter, this deviation is referred to as "camera installation deviation".

[0083] Furthermore, in the case of the visible camera 18 and the infrared camera 19 of the present embodiment, even when a shooting command signal is input to both simultaneously, after the infrared image is shot first, the visible image is shot at a timing slightly delayed therefrom. That is, a deviation in shooting timing occurs between the infrared image and the visible image.

[0084] Therefore, as shown in FIG. 17, when shooting of the infrared image and the visible image by the visible camera 18 and the infrared camera 19 is respectively executed during the flight of the drone 17A, in addition to the camera installation deviation, due to the deviation in shooting timing between the infrared image and the visible image and the movement amount of the drone 17A, a deviation occurs between the imaging point of the visible image and the imaging point of the infrared image. Hereinafter, this deviation is referred to as "drone movement deviation". As shown in FIG. 17, when the drone 17A is flying from the left side to the right side in the figure, the imaging point of the visible image will be located on the right side of the imaging point of the infrared image.

[0085] In the case of the present embodiment, in the solar panel latitude and longitude identification process described below, for reasons described later, it is necessary to align the centers of the infrared image and the visible image. Therefore, as described below, when the drone 17A is hovering, camera installation deviation correction for correcting the above camera installation deviation is executed, and when the drone 17A is moving, in addition to the camera installation deviation correction, drone movement deviation correction for correcting the drone movement deviation is executed.

[0086] For example, as shown in FIG. 18, when the drone 17A is flying from the left side to the right side in the figure, when neither camera installation deviation correction nor drone movement deviation correction is performed, as in the image shown on the left side in FIG. 18, the solar array 65 in the infrared image is shifted to the left and upper sides with respect to the solar array 66 in the visible image.

[0087] On the other hand, when drone movement deviation correction is performed on the infrared image, as in the image shown in the center of FIG. 18, the solar array 65 in the infrared image is shifted only to the upper side with respect to the solar array 66 in the visible image.

[0088] Furthermore, when camera installation deviation correction is performed on the infrared image for which drone movement deviation correction has already been performed, as in the image shown on the right side of FIG. 18, the solar array 65 in the infrared image coincides with the solar array 66 in the visible image. That is, the alignment of the centers of the infrared image and the visible image is performed.

[0089] Next, with reference to FIG. 19, the longitude and latitude identification process of the solar panel of the present embodiment will be described. This longitude and latitude identification process is executed by the above-described arithmetic processing unit 1 based on the visible image and the infrared image respectively captured by the visible camera 18 and the infrared camera 19.

[0090] As shown in the figure, first, infrared image enlargement correction processing is executed (FIG. 19 / STEP51). In this infrared image enlargement correction processing, an infrared image 70 as shown in the upper left side of FIG. 20 is enlarged by a predetermined image enlargement correction amount, thereby creating an enlarged infrared image 71 as shown in the upper center of FIG. 20. This image enlargement correction amount is determined as follows, for example. That is, during the flight of the drone 17A, when capturing infrared images and visible images by the visible camera 18 and the infrared camera 19, hovering shooting (shooting in a stationary state) is performed so that the top of the image faces north, and the shooting data (pair data of the visible image and the infrared image) is used to determine the enlargement correction amount.

[0091] The above infrared image enlargement correction process is for adjusting the solar panel in the enlarged infrared image 71 obtained by enlarging and correcting the infrared image 70 so that it has the same size as the solar panel in the visible image 72, because the infrared image 70 captured by the infrared camera 19 is smaller in size than the visible image 72 shown on the lower left side of FIG. 20 captured by the visible camera 18.

[0092] Next, it is determined whether the drone 17A was hovering during shooting (FIG. 19 / STEP52). When this determination is affirmative (FIG. 19 / STEP52…YES), the process proceeds to the camera installation deviation correction process described later.

[0093] On the other hand, when this determination is negative (FIG. 19 / STEP52…NO) and the drone 17A was moving during shooting, the drone movement deviation correction process is executed (FIG. 19 / STEP53). In this drone movement deviation correction process, based on the movement speed of the drone 17A during shooting and the deviation in the shooting timing between the infrared image and the visible image, the drone movement deviation correction amount is calculated, and the enlarged infrared image 71 is corrected with respect to the visible image 72 by this drone movement deviation correction amount.

[0094] After the drone movement deviation correction process is executed as described above, or after it is determined that the drone 17A was hovering during shooting, the camera installation deviation correction process is executed (FIG. 19 / STEP54).

[0095] In this camera installation misalignment correction process, the enlarged infrared image 71 is corrected with respect to the visible image 72 by a predetermined camera installation misalignment correction amount. This camera installation misalignment correction amount is determined as follows, for example. That is, during the flight of the drone 17A, when shooting infrared images and visible images with the visible camera 18 and the infrared camera 19, hovering shooting (shooting in a stationary state) is performed so that the top of the image is facing north, and the shooting data (paired data of the visible image and the infrared image) is used to determine the camera installation misalignment correction amount. By the above correction process, as shown in the upper part of the center of FIG. 20, the enlarged infrared image 71 is aligned with the visible image 72 so that their centers coincide.

[0096] Note that the above infrared image enlargement process, drone movement misalignment correction process, and camera installation misalignment correction process may be configured to be executed simultaneously.

[0097] Next, the longitude and latitude identification image creation process is executed (FIG. 19 / STEP55). In this longitude and latitude identification image creation process, by executing the same processes as STEP1 to 5 of FIG. 3 described above, a longitude and latitude identification image 73 as shown in the lower part of the center of FIG. 20 is created from the visible image 72.

[0098] Subsequent to this longitude and latitude identification image creation process, an abnormality determination process is executed (FIG. 19 / STEP56). In this abnormality determination process, it is determined whether or not there is an abnormal panel in the enlarged infrared image 71 by using the enlarged infrared image 71 and an object detection model.

[0099] In this case, as the object detection model, for example, a model (e.g., yolov5) in which model parameters are learned by the stochastic gradient descent method using, as teacher data, an infrared image to which a rectangle (bounding box) indicating the position of the abnormal panel and the type of the abnormality are given as teacher data is used.

[0100] Next, based on the result of the above abnormality determination process, it is determined whether there is an abnormal panel (Fig. 19 / STEP57). If this determination is negative (Fig. 19 / STEP57…NO) and there is no abnormal panel, this process ends as it is.

[0101] On the other hand, if this determination is positive (Fig. 19 / STEP57…YES) and there is an abnormal panel, the abnormal panel identification process is executed (Fig. 19 / STEP58). In this abnormal panel identification process, when an abnormal panel is detected in the enlarged infrared image 71, as shown on the right side of Fig. 20, by comparing the positions of the longitude and latitude identification image 73 and the enlarged infrared image 71, the longitude and latitude of the pixel at the center of gravity of the abnormal panel are identified. Thereby, it becomes possible to identify the position (center of gravity position) of the abnormal panel in the visible image 72. Then, this process ends.

[0102] In the longitude and latitude identification process of Fig. 19, if there is no deviation in the shooting timing between the infrared image and the visible image in the visible camera 18 and the infrared camera 19 of the drone 17A, the processes of STEP52~53 in Fig. 19 are omitted.

[0103] As described above, according to the method for identifying the longitude and latitude of the solar panel of the second embodiment, the longitude and latitude of the center of gravity of the abnormal panel can be accurately identified using the infrared image and the visible image.

[0104] Furthermore, when detecting an abnormality of a solar panel based on a captured image, it is generally known that an infrared image is easier to detect than a visible image. Therefore, according to the method of the second embodiment, the identification accuracy of the longitude and latitude of the center of gravity of the abnormal panel can be improved.

Explanation of Signs

[0105] 1 arithmetic processing unit 14 solar panel 20 drone (flying object) 21 visible camera (imaging device) 14 solar panel 40 captured image 41 Solar panel pt1 to pt4 4 vertices 42 First image 42a Coordinate point 45 Second image 48 Third image 50 Fourth image 20A Drone (flying object) 21 Visible camera (first imaging device) 22 Infrared camera (second imaging device) 70 Infrared image 72 Visible image

Claims

1. A method for identifying the longitude and latitude of a predetermined portion of a predetermined solar panel, which is at least one of a plurality of solar panels, based on a captured image including the plurality of solar panels captured from above by an imaging device and latitude and latitude information of a positioning signal at the time of the capture, comprising: a vertex coordinate value acquisition step of acquiring, by a predetermined image processing, coordinate values ​​of pixels of four vertices of each of the plurality of solar panels including a predetermined solar panel in a first orthogonal coordinate system that is an orthogonal coordinate system along up, down, left and right directions of the captured image with a predetermined location of the captured image as an origin; a first image creation step of extracting coordinate points of pixels in the first orthogonal coordinate system at predetermined positions on each of two opposing sides of each of the solar panels from the captured image, and creating a first image configured so that the coordinate points have distance conversion values ​​that are conversion values ​​of real spatial distances per pixel in two coordinate axis directions of the first orthogonal coordinate system; a second image creation step of creating a second image as an image configured such that coordinate points of pixels of the first image in the first orthogonal coordinate system have distance conversion values ​​by performing a predetermined interpolation calculation between coordinate points of pixels of the first image; a third image creating step of creating a third image based on the distance conversion value of the coordinate point of the pixel in the second image, as an image configured such that the coordinate point of the pixel in the first orthogonal coordinate system has real space distance data in two coordinate axis directions, with a pixel at the center of the image as the origin; a fourth image creation step of creating a fourth image based on the longitude and latitude information of the positioning signal and the third image, such that a pixel at the origin of the first orthogonal coordinate system, which is the center of the image, has the longitude and latitude information of the positioning signal, and the pixel has the longitude and latitude information; a longitude and latitude determination step of determining the longitude and latitude of the predetermined portion of the predetermined solar panel by comparing the fourth image with the captured image; A method for determining the longitude and latitude of a solar panel, characterized in that the above-mentioned is executed by a calculation processing device.

2. The method for identifying the longitude and latitude of a solar panel according to claim 1, A method for determining the longitude and latitude of a solar panel, wherein the photographing device is mounted on an aircraft capable of receiving the positioning signal.

3. The method for identifying the longitude and latitude of a solar panel according to claim 2, The aircraft is equipped with a first photographing device that photographs visible images and a second photographing device that photographs infrared images as the photographing devices, In the vertex coordinate value acquisition step, coordinate values ​​of pixels of the four vertices of each of the solar panels in the visible image are acquired, In the first image creation step, the first image is created from the visible image; A method for determining the longitude and latitude of a solar panel, characterized in that, in the longitude and latitude determination step, when the photographing timing of the first photographing device and the second photographing device are simultaneous or when the flying object is in a hovering state, the solar panel in the visible image and the infrared image are adjusted to be the same size, and the positional deviation of the visible image and the infrared image caused by the installation state of the first photographing device and the second photographing device is corrected so that the centers of the fourth image and the infrared image are aligned, and the fourth image and the infrared image are superimposed to determine the longitude and latitude of the specified portion of the specified solar panel in the infrared image.

4. The method for identifying the longitude and latitude of a solar panel according to claim 3, In the latitude and longitude determination step, when there is a time difference between the photographing timing of the first photographing device and the second photographing device, when photographing by the first photographing device and the second photographing device is performed while the aircraft is flying, the solar panel in the visible image and the infrared image are adjusted to be the same size, and the positional deviation of the photographed image due to the installation state of the first photographing device and the second photographing device and the positional deviation of the photographed image due to the time difference in the photographing timing are corrected so that the centers of the fourth image and the infrared image are aligned, and the longitude and longitude of the specified portion of the specified solar panel in the infrared image are determined by overlaying the fourth image and the infrared image.

Images