Method for creating a learning model for inspecting solar panels, method for inspecting solar panels, device for creating a learning model for inspecting solar panels, and device for inspecting solar panels

Abstract

To provide a learning model creation method for solar panel inspection for appropriately creating a learning model for solar panel inspection for inspecting the existence / nonexistence of abnormality of a solar panel, and a solar panel inspection method.SOLUTION: In a solar farm, four vertices of solar panels are determined on the basis of a picked-up image of a solar array, a visible light image of an individual solar panel is segmented on the basis of the four vertices, and a normalized image of each solar panel is created. A learning model creation method for solar panel inspection uses a normalized image of the solar panel without abnormality as an input image, and executes machine learning to an unlearned VAE model 46 so as to make a difference between the input image and an output image small.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a method for creating a learning model for solar panel inspection using AI (Artificial Intelligence), a solar panel inspection method, a device for creating a learning model for solar panel inspection, and a solar panel inspection device. [Background technology]

[0002] Photovoltaic power generation has been attracting attention as a measure against global warming. In photovoltaic power generation, a solar array equipped with multiple solar panels is installed at a specified site. After installation, various abnormalities occur in the solar panels. Such abnormalities include, for example, scratches, cracks, dirt, fissures, and even shadows caused by utility poles or weeds. If these abnormalities are left unattended, they can lead not only to a decrease in the power generation capacity of the solar panels, but also to increased damage, so it is necessary to monitor for abnormalities and take prompt action when an abnormality is found.

[0003] Patent document 1 discloses a solar panel fault diagnosis system in which an aircraft equipped with an infrared camera (e.g., a radio-controlled helicopter) is flown above a solar panel installation site, photographs the solar panels on the ground with the infrared camera, and the presence or absence of hotspots on the solar panels is checked based on the infrared images.

[0004] Patent document 2 discloses an inspection device in which an aircraft (e.g., a drone) equipped with an irradiation unit, a receiving unit, and a calculation unit is flown above a solar panel installation site, inspection light is emitted from the irradiation unit toward the solar panels on the ground, the reflected light from the solar panels is received by the receiving unit, and the calculation device inspects whether the solar panels are damaged based on the difference between the optical axis of the irradiation light and the optical axis of the reflected light.

[0005] Patent Document 3 discloses a solar panel appearance monitoring device. In this device, a classifier is constructed based on a group of solar panel images taken for a certain period of time by a surveillance camera installed on the ground when there are no foreign objects (e.g. flying objects or vines) on the solar panel, and a group of foreign object images taken for a certain period of time, and during monitoring, the images taken by the surveillance camera are inspected by the classifier to determine whether or not there is a foreign object on the solar panel. For example, Random Forests is selected as the classifier. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2015-146371 A [Patent Document 2] JP 2019-100958 A [Patent Document 3] JP 2016-205910 A Summary of the Invention [Problem to be solved by the invention]

[0007] The solar panel fault diagnosis system in Patent Document 1 can check for hot spots on solar panels from infrared images, but requires visual inspection to check for physical damage and shadows caused by vegetation growth.

[0008] Although the inspection device in Patent Document 2 can inspect solar panels for physical damage and shadows caused by vegetation growth, it must target multiple solar panels one by one and shine inspection light on them in sequence, which is time-consuming and increases the working time.

[0009] The solar panel appearance monitoring device of Patent Document 3 requires images of both solar panels and foreign objects taken over a certain period of time in order to construct a classifier. In addition, since there are many types of foreign objects, it takes time to collect images of each type of foreign object, and it is difficult to detect abnormalities other than foreign objects, such as cracks or cloudiness in the solar panel itself.

[0010] On the other hand, when a machine learning learning model is used to check whether a solar panel is abnormal, it is necessary to collect images of the solar panel for training. Since solar panels are generally incorporated in a solar array, it goes without saying that it is efficient to take an aerial photograph of the solar array and extract multiple solar panels from the aerial photograph. However, Patent Documents 1-3 do not disclose or suggest a specific method for extracting individual solar panels from an original image such as a solar array image in which multiple solar panels are photographed together.

[0011] In view of the above problems, an object of the present invention is to provide a method for creating a learning model for solar panel inspection, a solar panel inspection method, a learning model creation device for solar panel inspection, and a solar panel inspection device, which create an accurate learning model for solar panel inspection to inspect for abnormalities in solar panels. [Means for solving the problem]

[0012] The method for creating a learning model for solar panel inspection of the present invention includes the steps of: A method for creating a learning model for solar panel inspection that generates an output image based on an input image of a visible light image of a solar panel to be inspected, comprising: determining four vertices of each solar panel based on an original visible light image including a plurality of solar panels, extracting a plurality of panel visible light images for each solar panel from the original visible light image based on the four vertices, and creating a plurality of normalized images by normalizing each panel visible light image; a step of inputting a normalized image of a solar panel without an abnormality among the plurality of normalized images as an input image to the learning model for inspecting a solar panel, and performing machine learning so as to reduce a difference between the input image and an output image, thereby adjusting weight parameters that define the learning model for inspecting a solar panel; Equipped with.

[0013] The solar panel inspection method of the present invention includes the steps of: The method for creating a learning model for solar panel inspection; A step of inputting an input image of a solar panel to be inspected into the learning model for solar panel inspection and outputting an output image; recognizing an abnormal state of the inspection target solar panel based on a difference between the input image and the output image; Equipped with.

[0014] The solar panel inspection learning model creation device of the present invention comprises: A device for creating a learning model for solar panel inspection that generates an output image based on an input image of a visible light image of a solar panel to be inspected, a normalized image creation unit that determines four vertices of each solar panel based on an original visible light image including a plurality of solar panels, cuts out a plurality of panel visible light images of each solar panel from the original visible light image based on the four vertices, and creates a plurality of normalized images by normalizing each panel visible light image; a parameter adjustment unit that inputs a normalized image of a solar panel without an abnormality among the plurality of normalized images as an input image to the learning model for solar panel inspection, and performs machine learning so as to reduce a difference between the input image and an output image, thereby adjusting weight parameters that define the learning model for solar panel inspection; Equipped with.

[0015] The solar panel inspection device of the present invention is The solar panel inspection model of the solar panel inspection learning model creation device; and a model control unit that inputs an input image of a solar panel to be inspected and outputs an output image to the learning model for solar panel inspection; a recognition unit that recognizes an abnormal state of the inspection target solar panel based on a difference between the input image and the output image; Equipped with. Effect of the Invention

[0016] According to the solar panel inspection learning model creation method and device, and the solar panel inspection method and device of the present invention, the four vertices of each solar panel are determined based on an original visible light image including multiple solar panels, and multiple panel visible light images are cut out from the original visible light image based on the four vertices to create a normalized image. This makes it possible to efficiently collect multiple training images from one original image without any problems. Then, a normalized image of a solar panel without an abnormality among the multiple normalized images is input as an input image to the solar panel inspection learning model, machine learning is performed so that the difference between the input image and the output image is reduced, and a weight parameter that defines the solar panel inspection learning model is adjusted to create an appropriate solar panel inspection learning model. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram showing an aerial photograph of a solar panel. [Diagram 2] 1 is an overall schematic diagram of an embodiment of a method for creating a learning model for solar panel inspection and a method for inspecting a solar panel; [Diagram 3] FIG. 3 is a detailed explanatory diagram of the visible light panel detection process of FIG. 2. [Figure 4A] 4A and 4B are diagrams showing the relationship between images photographed from the air at a low altitude and their binarized images for the individual panel detection in FIG. 3. [Figure 4B] 4A to 4C are diagrams showing the relationship between images photographed from the air at a high altitude and their binarized images for the individual panel detection in FIG. 3. [Diagram 5]FIG. 11 is an explanatory diagram of the process of cutting out a panel based on the four determined vertices. [Figure 6] FIG. 3 is a detailed illustration of the learning process of FIG. 2. [Figure 7] FIG. 1 is a detailed diagram of an unlearned VAE model and processing units before and after it. [Figure 8] FIG. 1 is an explanatory diagram of the learning progress of a trained VAE model. [Figure 9] FIG. 1 is an explanatory diagram of an inspection process. [Figure 10] FIG. 1 is a configuration diagram of a re-learning system. [Figure 11] 1A to 1C are diagrams illustrating the occurrence of erroneous detection in a solar panel inspection method. [Figure 12] FIG. 4 is an explanatory diagram of a corner detection filter. [Figure 13] 13 is an enlarged view of the upper left corner, the upper right corner, the lower left corner, and the lower right corner which are set as the corner search range in the corner detection image of FIG. 12. [Figure 14] FIG. 14 is an explanatory diagram of a process subsequent to the processes of FIGS. 12 and 13. [Figure 15] FIG. 15 is a comparative diagram of an uncorrected panel image in which the noise removal processing of FIGS. 11 to 14 has not been performed and a corrected panel image in which the noise removal processing has been performed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present invention will be described. It goes without saying that the present invention is not limited to the embodiment. Throughout the drawings, the same reference numerals are used to refer to common elements.

[0019] Although the input, output, and intermediate output of each image processing element are explained using images, it goes without saying that the specific processing target of each image processing element is not the image itself but the data associated with the image. Also, although the learning models are distinguished into three categories, unlearned, trained, and re-learned, these are merely names for the same machine learning model, distinguishing between the learning stage, execution stage, and re-learning stage.

[0020] (Photo of solar panels) 1 shows an aerial photograph of a solar panel 14. A large number of solar arrays 12 are installed in a solar farm 10, and passageways 11 are provided between the solar arrays 12 for workers to walk through when inspecting and maintaining each solar array 12. A drone 24 is equipped with a visible light camera 26, flies around in the sky above the solar farm 10, and photographs the solar arrays 12 in the solar farm 10 from above.

[0021] Each solar array 12 is typically arranged extending horizontally with its front surface facing the southern sky at a predetermined inclination angle with respect to the horizontal plane. A large number of solar panels 14 are attached to the front surface of each solar array 12 in a grid-like arrangement pattern to generate solar power. Each solar panel 14 is typically square or rectangular in shape. Gaps 16 between adjacent solar panels 14 in the solar array 12 expose the board panel substrate of the solar array 12. Each solar panel 14 has a plurality of cells 18 arranged in a grid pattern.

[0022] While the drone 24 is flying, the photographed image 30 (FIG. 3) of the solar array 12 captured by the visible light camera 26 may be transmitted online to, for example, a remote controller of a remote operator on the ground, or may be retrieved from the memory of the visible light camera 26 after the drone 24 has landed. Note that the drone 24 or the visible light camera 26 is equipped with a GPS, and the meta information of the captured image 30 includes, in addition to the photographed date and time, information on the photographed position obtained from the GPS.

[0023] (Method / Outline) 2 is an overall schematic diagram of an embodiment of a learning model creation method for solar panel inspection and a solar panel inspection method. STEP 100 is a process of detecting a visible light panel (solar panel) and is a process commonly applied to both the learning model creation method for solar panel inspection and the solar panel inspection method. STEP 200 is a process of AI anomaly detection and is composed of STEP 210 and STEP 220. STEP 210 is applied to the learning model creation method for solar panel inspection, and STEP 220 is applied to the solar panel inspection method.

[0024] (visible light panel detection) Figure 3 is a detailed illustration of the visible light panel detection process of Figure 2. First, a captured image 30 is prepared.

[0025] The captured image 30 in the embodiment is a visible light captured image (≠ infrared image) captured by the visible light camera 26 from a predetermined altitude of the solar farm 10. The predetermined altitude is, for example, in the range of 20 to 50 m from the ground, and a desirable height is a height that allows smooth extraction processing for extracting panel images as visible light images of each solar panel 14 from the captured image 30. The desirable height is affected by factors other than altitude, such as weather, time of capture, topography, and so on.

[0026] STEP 100 is made up of two sub-steps: STEP 110 (individual panel detection) and the next STEP 120 (panel cut-out). In STEP 110, the four vertices of a cut-out rectangle 36 that separates the multiple solar panels 14 on the captured image 30 are set in correspondence with the four vertices of the solar panel 14. Details of how the four vertices of the cut-out rectangle 36 in the captured image 30 are determined will be described later with reference to FIG.

[0027] STEP 120 is a panel cut-out process, and includes two sub-processes, STEP 121 (cut-out) and STEP 122 (projective transformation). In STEP 121, an individual panel visible light image 38 is cut out from the captured image 30 based on the cut-out rectangle 36 (more specifically, the four vertices of the cut-out rectangle 36) determined in STEP 110.

[0028] 3, only the peripheral outline of the individual panel visible light image 38 is shown, and the specific image of the contents is omitted. Although the solar panel 14 itself is rectangular (including square), the captured image 30 is taken from above the solar panel 14 at an angle, so the raw (unprocessed) individual panel visible light image 38 is not rectangular but quadrangular.

[0029] In STEP 122, normalization is performed to create a square projectively transformed image 40 of a predetermined size (e.g., 256 x 256 pixels per color) from the individual panel visible light image 38 by projective transformation. In the normalized projectively transformed image (also referred to as a "normalized image" as appropriate) 40, the length and width of the solar panel 14 when viewed from the front are also aligned. That is, the top, bottom, left and right sides of the solar panel 14 become the top, bottom, left and right sides of the projectively transformed image 40, respectively. The multiple projectively transformed images 40 that have been created are saved in the save folder 32.

[0030] The projectively transformed images 40 stored in the storage folder 32 are also input images for the untrained VAE model 46 (FIG. 6) and the trained VAE model 56 (FIG. 9). Since the input images for the untrained VAE model 46 are used as training images, it is necessary to prepare a large number of projectively transformed images 40. For this reason, there are usually multiple solar farms 10 at the collection site.

[0031] Although solar panels 14 from all manufacturers are rectangular (including square), the horizontal and vertical dimensions and aspect ratios as well as the color and internal structure (e.g., cells 18) differ from manufacturer to manufacturer. However, by normalizing the projectively transformed image 40, it can be applied as a training image and an inspection image (individual image for the solar panel 14 to be inspected) for the untrained VAE model 46 (FIG. 6) and the trained VAE model 56 (FIG. 9) regardless of the manufacturer and model.

[0032] In the embodiment, the individual panel visible light image 38 is cut out from the captured image 30 taken by the visible light camera 26 of the drone 24. The individual panel visible light image 38 may also be cut out from a captured image taken by a worker on the ground of the solar array 12. The individual panel visible light image 38 may be used as a training image or an inspection image.

[0033] 4A and 4B show the relationship between aerial images 31a and 31b taken from a low altitude (eg, 20 m) and a high altitude (eg, 30 m) and their binarized images for STEP 110 (individual panel detection) in FIG. 3.

[0034] The binary image is generated by setting a predetermined threshold value for the brightness of pixels constituting the color captured image 30, converting pixels whose brightness is equal to or greater than the predetermined value to one of the two values ​​(e.g., white), and converting pixels whose brightness is less than the predetermined value to the other of the two values ​​(e.g., black). Meanwhile, the color captured image 30 has a total of three pieces of information for the brightness at the same position, divided into the colors R (red), G (green), and B (blue). In the embodiment, the binary image from the captured image 30 is generated based only on the brightness of G, taking into account the compatibility of the color with the solar panel 14.

[0035] Specifically, the cutout in STEP 121 is processed using an averaging filter and a threshold value (0 to 255). In this embodiment, three averaging filters are used, 0, 3×3, and 5×5 in terms of pixel (picture element) size. An averaging filter of size 0 means that no averaging filter processing is performed. The larger the size of the averaging filter applied, the smoother the image is converted (the lower the resolution).

[0036] The method for binarizing each pixel to white (0) and black (255) is to use a threshold of 127. Value andIn addition to binarization based on an adaptive binarization method with a surrounding pixel count of 11 x 11 and binarization with a threshold value of 127, an opening process using a filter of size 5 x 5 is used. Therefore, a total of nine binarized images, 3 x 3, are generated from each of the aerial photographs 31a and 31b.

[0037] In Fig. 4A and Fig. 4B, only two binarized images of the nine binarized images of the aerial photographed images 31a and 31b are shown: (a) an averaging filter size = 0 and a threshold value = 127 (luminance equal to or greater than the threshold is converted to white, and brightness less than the threshold is converted to black), and (b) an averaging filter size = 5 x 5 and a threshold value = 127. The entire binarized images 33a1 and 33a2 are binarized images generated by the processes (a) and (b) of the aerial photographed image 31a, respectively. The entire binarized images 33b1 and 33b2 are binarized images generated by the processes (a) and (b) of the aerial photographed image 31b, respectively. The individual panel binarized images 35a1, 35a2, 35b1, and 35b2 on the far right are enlarged images of the entire binarized images 33a1, 33a2, 33b1, and 33b2 on the left side.

[0038] In the binarized image, the outline of the solar panel 14 (white stripes corresponding to the gaps 16) and the dividing lines of the cells 18 (white stripes) appear. The cells 18 are blackish in color. The cutout rectangle 36 needs to be determined from the outline of the solar panel 14, not from the dividing lines of the cells 18. In the aerial photographed image 31a photographed at a low altitude in FIG. 4A, it is easier to determine the cutout rectangle 36 from the individual panel binarized image 35a2 in the lower row in which the white stripes of the cells 18 have disappeared, so the cutout rectangle 36 is determined from the individual panel binarized image 35a2. In contrast, in the aerial photographed image 31b photographed at a high altitude in FIG. 4B, the cutout rectangle 36 is determined from the individual panel binarized image 35b1 in the upper row. This is because the outline of the solar panel 14 has disappeared in the individual panel binarized image 35b2 in the lower row, and the cutout rectangle 36 cannot be detected.

[0039] The cut-out rectangle 36 is determined based on a set of four vertices in each binarized image. The method for determining the set of four vertices that form the basis of the cut-out rectangle 36 is as follows.

[0040] STEP a1: In each of the binary images generated by processing in the 9 manner, the contour of the area surrounded by low-luminance groups (e.g., black) or high-luminance groups (e.g., white) is detected as a polygon, and if the polygon has four vertices, a first set is generated that includes all sets of four vertices. STEP a2: Generate a second set by combining duplicate pairs in the first set.

[0041] STEP a3: A third set is generated by leaving only the sets of four vertices from the second set that are valid as four vertices corresponding to the four vertices of the actual solar panel 14. Note that the criterion for determining validity may be, for example, whether the length between two of the four vertices on the binarized image or the aspect ratio matches those of the actual solar panel 14. STEP Pa4: The four vertices of each pair in the third set are determined as the four vertices of the cut-out rectangle 36.

[0042] An example other than the above example (hereinafter referred to as "Example 1") regarding how to determine the set of four vertices that form the basis of the cut-out rectangle 36 is as follows.

[0043] Example 2: In Example 1, the four vertices of the cut-out rectangle 36 are determined from a binary image (e.g., original binary images 33a1 and 33a2 in FIG. 4A). In contrast, in Example 2, the four vertices of the cut-out rectangle 36 are determined from a grayscale image (e.g., an image obtained by converting the aerial photograph image 31a, which is a color visible light image, into a monochrome image). That is, in the solar array 12, there is a certain degree of color difference or luminance difference between the inside (made up of a plurality of cells 18) and the outside (gaps 16 separating adjacent solar arrays 12) of the solar panel 14. Therefore, it is possible to identify an area in the grayscale image that is included in the luminance range to which the solar panel 14 belongs, and to identify the four vertices of the cut-out rectangle 36 based on the identified area.

[0044] Example 3: In the binary image of Example 1, a black and white binary image is generated by setting a predetermined threshold value for the brightness of the aerial image of the visible light image. In contrast, in Example 3, a binary image is created by edge generation through differentiation. The edges correspond to the contours of the solar panel 14 and the cells 18.

[0045] 5 is an explanatory diagram of the process of STEP 120 (panel extraction) based on the determined four vertices. In FIG. 5, the individual panel visible light image 38 to be extracted from the captured image 30 is selected from the solar panel 14 having an anomaly 39.

[0046] First, in STEP 110, a cut-out rectangle 36 is determined based on the determined four vertices Va, Vb, Vc, and Vd (hereinafter abbreviated as "Va-Vd"). Next, in STEP 121, an individual panel visible light image 38 is cut out from the captured image 30 as an image inside the cut-out rectangle 36. Note that one individual panel visible light image 38 means a collection of three images, one for each color, RGB.

[0047] Finally, in the projective transformation of STEP 122, the individual panel visible light image 38 is transformed so that the four vertices Va-Vd become the four vertices of a square, thereby obtaining a projectively transformed image 40 as a normalized visible light image of the solar panel 14. The top, bottom, left, and right sides of the projectively transformed image 40 correspond to the actual top, bottom, left, and right sides of the solar panel 14.

[0048] (Learning Process) Fig. 6 is a detailed explanatory diagram of STEP 210 (learning process) in Fig. 2. STEP 211 is composed of two sub-steps: STEP 212 (picking up) and STEP 213 (creating a data set).

[0049] 2, a large number of projectively transformed images 40 are stored in the storage folder 32. Most of the projectively transformed images 40 are normal panel images 40a, but a small number of abnormal panel images 40b are also included. The operator visually checks each projectively transformed image 40 in the storage folder 32 and picks out only the normal panel images 40a as training images (STEP 212), and constructs a dataset of training data, which is the data format of the training images (STEP 213).

[0050] It is possible to partially automate the visual inspection work when picking out only normal panel images 40a from among the projection-transformed images 40 in the storage folder 32 as training images. For example, first, the projection-transformed images 40 in the storage folder 32 are visually judged to be normal panel images 40a or not, and the projection-transformed images 40 judged to be normal panel images 40a are stored as pass images. Then, after the number of pass images reaches a predetermined number, an automatic machine compares the total value of the difference in brightness for each pixel between the projection-transformed images 40 automatically picked up from the storage folder 32 and the pass images with a predetermined threshold value, and if the difference is less than the threshold value, the projection-transformed image 40 is automatically judged to be a normal panel image 40a and is added to the pass images.

[0051] In this way, the machine judges the pass / fail status of the projective transformed images 40 in the storage folder 32 one after another while increasing the number of pass images. Then, the visual inspection is limited to the projective transformed images 40 that the machine has failed. At this time, the projective transformed images 40 that the visual inspection has determined to be normal panel images 40a are added to the pass images of the automatic machine. As the number of stored pass images increases, the number of projective transformed images 40 for which the automatic machine can automatically judge pass / fail also increases.

[0052] In the data set (N, 256, 256, 3) of STEP 213, N means the number of training data, 256, 256 means the pixel-converted size of each normal panel image 40a, and 3 means the number of RGB colors (= number of colors) of each normal panel image 40a.

[0053] STEP215 consists of two sub-processes: STEP216 (preprocessing) and STEP217 (main processing). In STEP216, the preprocessing unit 44 divides N pieces of training data into epochs, and trains the untrained VAE model 46 using the mini-batch method (batch size is, for example, 32 or 64).

[0054] In STEP 217, the untrained VAE model 46 performs unsupervised learning using the training data from the pre-processing unit 44. The pre-processing unit 44 and the loss calculation unit 48 are also the input unit and the output unit of the untrained VAE model 46, respectively.

[0055] 7 is a detailed diagram of the untrained VAE model 46 and its preceding and following processing units. The untrained VAE model 46 is composed of a VAE (Variational Autoencoder) and includes an encoder, a Z-space (parameter generating unit), and a decoder. Training data 52, which is the data form of the normal panel image 40a, is input to the untrained VAE model 46. The generated image 54 is an image corresponding to the output data generated by the untrained VAE model 46 from the input data of the normal panel image 40a.

[0056] The loss calculation unit 48 trains the unlearned VAE model 46 so as to reduce the loss (e.g., mean square error) of a predetermined loss function by using the normal panel image 40a as the correct value of the output of the unlearned VAE model 46. In this manner, each parameter of the unlearned VAE model 46 is updated by the gradient descent method. Note that reducing the loss also means reducing the difference between an input image 56i and an output image 56o, which will be described later.

[0057] FIG. 8 is a diagram showing the results of confirming the reconstructed images (output images) obtained stepwise for each learning epoch to the untrained VAE model 46 in the middle of learning and inputting the same input image. The horizontal axis indicates the number of learning times (epoch number). It can be seen that as the learning of the untrained VAE model 46 progresses, the generated image 54 gradually approaches the normal panel image 40a as the correct answer. From this, it can be visually understood that it is possible to create a model that generates panel images using the above learning method.

[0058] (Inspection process) FIG. 9 is an explanatory diagram of STEP220 (inspection process). STEP220 is composed of two sub-processes, STEP221 (image generation) and STEP222 (post-processing). The processes of STEP221 and 222 are performed by the trained VAE model 56 and the post-processing unit 58, respectively. The trained VAE model 56 and the post-processing unit 58 constitute the inspection unit 53. Note that, for convenience of explanation, the figure shows the images of the input image 56i, the output image 56o, the difference image 60d, and the judgment image 62e, but the actual output of the inspection unit 53 is data corresponding to these images.

[0059] The trained VAE model 56 outputs an output image 56o in response to an input image 56i. The difference analysis unit 60 outputs a difference image 60d which is an absolute value of the difference in brightness between the input image 56i and the output image 56o. The anomaly detection unit 62 outputs a determination image 62e which is a determination result regarding an anomaly in the solar panel 14 being inspected based on the difference image 60d.

[0060] The trained VAE model 56 infers an image (normal image) for an input image 56i when the input image 56i is of a normal solar panel 14. Therefore, a properly trained learning model outputs an output image 56o that is very similar to the input image 56i when the input is a normal image, and outputs an output image 56o that is very similar to an image in which the abnormal part is removed from the input image 56i when the input is an abnormal image. In this case, the difference in brightness (strictly speaking, absolute value) between the input image 56i and the output image 56o at each pixel becomes 0 (black) or a value close to 0, and no abnormality 39e appears in the judgment image 62e.

[0061] On the other hand, if the solar panel 14 to be inspected has an anomaly 39 (FIG. 5), the input image 56i includes the anomaly 39i. However, the trained VAE model 56 outputs an output image 56o of a normal image (an image of a normal solar panel 14) from the input image 56i, regardless of the presence or absence of the anomaly 39i. As a result, in the difference image 60d, a pixel region whose absolute value of the difference brightness is greater than a predetermined value appears as an anomaly 39d, and when the anomaly 39d satisfies a predetermined condition, it becomes an abnormal unit section 39e in the judgment image 62e.

[0062] (Basic example of how to determine abnormality) A basic example of the method by which the abnormality detection unit 62 determines whether or not the solar panel 14 is ultimately abnormal with respect to the abnormality 39e is as follows. STEPb1: The anomaly detection unit 62 divides the difference image 60d into a plurality of unit blocks (unit cells) of equal shape (e.g., square) and size (size) by using equally spaced grid lines in both the vertical and horizontal directions. Here, the total number of unit blocks is Na.

[0063] STEP b2: For each unit block in the difference image 60d, it is determined whether the total number Ma of pixels whose differential luminance value (difference luminance) is equal to or greater than the first threshold is equal to or greater than the second threshold. Note that the luminance of each pixel in the difference image 60d is the absolute value of the difference in luminance between corresponding pixels in the input image 56i and the output image 56o.

[0064] STEPb3: If the result of the determination in STEPb2 is positive, the unit section is determined to be an abnormal section, and if the result is negative, the unit section is determined to be a normal section. In the determination image 62e in FIG. 9, black indicates a normal section, and white indicates an abnormal section, which is indicated by an abnormality 39e.

[0065] STEPb4: The number Nb of abnormal compartments is counted. STEP b5: If Nb is equal to or greater than the third threshold, it is determined that the solar panel 14 being inspected is abnormal. If Nb is less than the third threshold, it is determined that the solar panel 14 being inspected is normal. Note that instead of Nb, Nb / Na can be compared with the third threshold to determine whether the solar panel 14 being inspected is normal or abnormal.

[0066] Rather than determining whether the solar panel 14 being inspected is abnormal or not based on whether the total number Ma of pixels whose differential brightness values ​​are equal to or greater than the first threshold value in the entire differential image 60 is equal to or greater than the second threshold value, the significance of following the steps STEPb1-STEPb5 as described above is as follows.

[0067] (a) The degree of abnormality of the solar panels 14 can be counted as the number of unit sections. That is, when placing a pin on the position of an abnormal solar panel 14 on an orthoimage (on the inspection result display application screen) showing the solar array 12 including many solar panels 14, it is possible to perform filtering of the display according to the degree of abnormality, such as "displaying only solar panels 14 with 5 or more abnormal unit sections" (in this case, "5" corresponds to the third threshold value, and the third threshold value is made adjustable as appropriate).

[0068] (b) Abnormalities in the solar panel 14 can be abstracted. In other words, by expressing an abnormality in a simpler way, it becomes easier to estimate the similarity of abnormal shapes between different panels.

[0069] (Method of determining abnormality / Modification) The following are examples of modified methods for determining an abnormality.

[0070] (e1) In the basic example, for each unit partition, if the total number Ma of pixels whose differential brightness value (differential brightness) is equal to or greater than the first threshold value is equal to or greater than the second threshold value, the unit partition is determined to be an abnormal partition. In the modified example, for each unit partition, if the nth (n is an integer equal to or greater than 1) differential brightness value from the maximum value is equal to or greater than the first threshold value, the unit partition is determined to be an abnormal partition. Furthermore, for each unit partition, if the average value of the differential brightness value (differential brightness) is equal to or greater than the first threshold value, the unit partition may be determined to be an abnormal partition.

[0071] (e2) In the basic example, the difference image 60d is equally divided into unit blocks by equally spaced grid lines in both the vertical and horizontal directions, but the unit blocks are divided into a number of groups (each group may contain any number of unit blocks). Then, for each group, if the ratio Ma / Mb of the total number of pixels Ma whose differential luminance values ​​(differential luminance) are equal to or greater than a first threshold value to the total number of pixels Mb of the group is equal to or greater than a second threshold value, the group is determined to be an abnormal group. Finally, if the number Nb of abnormal groups is equal to or greater than a third threshold value, the solar panel 14 to be inspected is determined to be abnormal. If Nb is less than the third threshold value, the solar panel 14 to be inspected is determined to be normal. Ma can also be changed for each group.

[0072] (e3) In the determination image 62e, a region where pixels having a differential brightness value equal to or greater than the first threshold value are adjacently arranged, or a region where the pixels are discontinuous but the distance between the pixels is within a predetermined value m (m is an integer equal to or greater than 2) in terms of the number of pixels, is determined to be an abnormal region, and if the number Nb of such abnormal regions in the entire solar panel 14 to be inspected is equal to or greater than a third threshold value, the solar panel 14 to be inspected is determined to be abnormal. If Nb is less than the third threshold value, the solar panel 14 to be inspected is determined to be normal.

[0073] 9, an abnormal state as an inspection result of the solar panel 14 being inspected is recognized by a judgment image 62e as an output of the abnormality detection unit 62, that is, an image display showing specific abnormal pixels of the solar panel 14. Other examples of the recognition of an abnormal state include the generation of an alert by voice or the display of characters or symbols, or storing an abnormality record in a database associated with the ID of the solar panel 14 in an abnormal state and the inspection date and time.

[0074] (Relearn) 10 is a configuration diagram of the re-learning system 64. The re-learning system 64 includes the inspection unit 53, an aerial image storage unit 66, a panel detector 68, a panel image former 70, a uniform shape image storage unit 72, a data combination and display unit 74, and a re-learning unit 80.

[0075] The aerial image storage unit 66 stores aerial images taken from above the solar farm 10 by the drone 24 using the visible light camera 26 as the drone 24 flies around above the solar farm 10. Each solar array 12 in the solar farm 10 is included in at least one aerial image, and an image of each solar panel 14 in the solar farm 10 is included in at least one aerial image stored in the aerial image storage unit 66. Note that the aerial images in the aerial image storage unit 66 are not limited to those of a single solar farm 10, and may be aerial images of multiple solar farms 10.

[0076] The panel detector 68 is a device that performs the individual panel detection of STEP 110 (FIG. 2) based on the four vertices Va-Vd described above. The panel image shaper 70 is a device that performs the panel cutout of STEP 120 (FIG. 2). The uniform shape image storage unit 72 refers to the save folder 32. The data combination and display unit 74 includes an additional training data acquisition unit 76 with a false detection button 78.

[0077] In response to the abnormality determination from the post-processing unit 58, the inspector retrieves the input image 56i on which the abnormality determination is based from the uniform shape image storage unit 72, and also retrieves information on the four vertices Va-Vd that define the cut-out rectangle 36 from the panel detector 68. Furthermore, the inspector displays these on the data combination and display unit 74 to verify the abnormality determination of the post-processing unit 58.

[0078] The advantages of the inspector knowing not only the input image 56i but also the information on the four vertices Va-Vd are as follows, for example. (a) An abnormality may be detected due to inappropriate determination of the four vertices Va-Vd (described later in Figures 11 to 15). (b) The four vertices Va-Vd indicate coordinates in the captured image 30, so that the solar panel 14 to be detected can be displayed on an application screen (such as the entire captured image 30 in FIG. 3). (c) Furthermore, by color-coding the results of the anomaly determination (e.g., abnormalities are framed in red and normal in green), it is possible to easily check which panel is abnormal on the app screen.

[0079] When the inspector finds no abnormality in the input image 56i through visual inspection of the image and determines that there is no problem with the determination of the four vertices Va-Vd, the inspector determines that the trained VAE model 56 has made a false detection on the input image 56i and presses the false detection button 78. This causes the input image 56i to be sent to the training data 82 of the re-learning unit 80 as a re-training image (strictly speaking, re-training image data). In this way, the training data 82 stores training images for re-training.

[0080] It is possible to improve the inspection efficiency by partially using machines instead of a complete visual inspection by an inspector. For example, even if the first or second threshold value in STEP b2 or the third threshold value in STEP b4 is increased by a predetermined amount, the input images 56i judged to be abnormal can be excluded from the visual inspection, thereby reducing the number of visual inspections. On the other hand, if at least one of these threshold values ​​is decreased by a predetermined amount, the input images 56i judged to be normal can be judged to require reinspection, and only such input images 56i can be subject to visual inspection, thereby reducing the number of visual inspections.

[0081] In order to retrain the trained VAE model 56, the re-training unit 80 copies the trained VAE model 56 to the re-training unit 80 to generate a re-trained VAE model 84, and then re-trains the re-trained VAE model 84 using training images for re-training. The re-trained VAE model 84 after re-training is transferred to the inspection unit 53 as the new trained VAE model 56. In this way, the old trained VAE model 56 before re-training is replaced with the new trained VAE model 56.

[0082] Generally, the performance of all statistical models declines over time after they are put into operation. For this reason, the trained VAE model 56 is periodically retrained using the latest data, and the trained VAE model 56 is replaced with the retrained VAE model 84 to prevent performance degradation.

[0083] (False positive) Fig. 11 is an explanatory diagram of the occurrence of false detection 90 in the solar panel inspection method. A cut-out image 81 in Fig. 11 is shown as an example of a cut-out image with noise that causes false detection 90. Note that the cut-out image 81 is a visible light image cut out from the captured image 30 as an individual panel image of the solar panel 14 to be inspected in the captured image 30 in the solar panel inspection method, and corresponds to the individual panel visible light image 38 in the solar panel inspection learning model creation method.

[0084] When a margin edge 87 exists on the periphery of the cut-out image 81, a false detection 90 occurs in the determination image 62e. For convenience of explanation, the margin edge 87 is called "margin", but its color in the actual color cut-out image 81 is a dark color (a low-luminance area) reflecting the color of the ground. The margin edge 87 is the ground that appears above the upper side of the solar array 12 in the captured image 30. The inventor has found that the margin edge 87 on the periphery of the cut-out image 81 is the cause of the false detection 90 in the determination image 62e.

[0085] In cut-out image 81, a blank edge 87 exists between a panel perimeter 86 of a panel image 85 of the solar panel 14 to be inspected and a perimeter 88 of cut-out image 81. Despite the existence of blank edge 87, output image 56o becomes an image in which input image 56i derived from cut-out image 81 corresponds to input image 56i without blank edge 87 (output image of normal solar panel 14). As a result, white horizontal lines 89 appear in difference image 60d, and white horizontal lines 89 cause erroneous detection 90 in difference analysis section 60e.

[0086] (Panel vertex adjustment) 12 is an explanatory diagram of a corner detection filter 96 (a collective term for corner detection filters 96a, 96b, 96c, and 96d) to be applied to the corner detection image 83a. The corner detection image 83a is set as a rectangular visible light image that includes the panel image 85 of the solar panel to be inspected in the captured image 30 including the visible light panel image 85 and is slightly wider than the panel image 85. The method of finding the four vertices of the panel image 85 before adjustment is as described above with reference to FIGS. 4A and 4B.

[0087] Each of the corner detection filters 96a, 96b, 96c, and 96d is associated with vertex detection of the upper left corner 101a, the upper right corner 101b, the lower left corner 101c, and the lower right corner 101d, which are the four corresponding corners of the corner detection image 83a. The corner detection filters 96a, 96b, 96c, and 96d each have a square region (white region in the figure) with a positive weight (e.g., "1"), and side regions (grayish regions in the figure) with a negative weight (e.g., "-1", "-2") extending along two sides of the square region at the upper left corner 101a, the upper right corner 101b, the lower left corner 101c, and the lower right corner 101d sides.

[0088] In the illustrated corner detection filter 96, the size in pixels is s×s (7×7), the size of the square area is w×w (6×6), and the size of the side area is set to width×length 1×7.

[0089] It should be noted that the corner detection image 83a is not rectangular but is a quadrangle, whereas the corner detection filter 96 is a square. Therefore, when the corner detection filter 96 is applied (spatial filtering) to the corner detection image 83a, a shape mismatch occurs at the corners. However, the mismatch is at most 7 pixels, and the difference in shape at the corners is unlikely to be a problem unless the solar panel 14 on the captured image 30 is extremely tilted. Conversely, if this causes the vertex correction to not function properly, it can be resolved by taking an image with the panel on the image not tilted as much.

[0090] When the corner detection image 83a is spatially filtered using the corner detection filters 96a, 96b, 96c, and 96d as image filters, filtered images 83aa, 83ab, 83ac, and 83ad are generated as shown in a horizontal row at the bottom of FIG.

[0091] It should be noted that, in applying the corner detection filter 96 to the corner detection image 83a, when the vertices of the corner detection image 83a have a predetermined brightness or higher (are bright), that is, for corners where no dark margin edge 87 exists (the lower left corner 101c and the lower right corner 101d in the illustrated example), the corner detection filter 96 is not applied, and therefore the vertices of the panel image 85 can be immediately determined as the true vertices of the solar panel 14, not from the corner detection image 83a.

[0092] Fig. 13 is an enlarged view of upper left corner 101a, upper right corner 101b, lower left corner 101c, and lower right corner 101d, which are set as the corner search range in corner detection image 83a in Fig. 12. In corner detection image 83a, upper left corner 101a and upper right corner 101b partially include margin edge 87 (this margin edge 87 corresponds to the captured portion of the ground above the upper side of solar array 12) and one adjacent adjacent panel image 92 adjacent in the left-right direction. Lower left corner 101c and lower right corner 101d do not include margin edge 87 (corresponding to the ground), but partially include a total of three adjacent adjacent panel images 92 adjacent in the left-right and up-down directions.

[0093] In Figure 12, when searching for corners in the upper left corner 101a, the upper right corner 101b, the lower left corner 101c, and the lower right corner 101d as the corner search range, the image portions of the upper left corner 101a, the upper right corner 101b, the lower left corner 101c, and the lower right corner 101d in the filtered images 83aa, 83ab, 83ac, and 83ad in Figure 11 are used, respectively.

[0094] Fig. 14 is an explanatory diagram of the vertex search process. The corner detection image 83a has a total of four corner search ranges, the upper left corner 101a, the upper right corner 101b, the lower left corner 101c, and the lower right corner 101d, and Fig. 14 will explain the upper left corner 101a. The corner searches in the other three search ranges are similar to that in the upper corner 101a.

[0095] 14 shows a comparison of the corner detection image 83a and the filter-processed image 83aa as a visible light image for the upper left corner 101a. The vertical line 105 and the horizontal line 107 are merely auxiliary lines for clarifying the correspondence between the corrected vertex 103p in the corner detection image 83a and the corrected vertex 103p in the filter-processed image 83aa.

[0096] The uncorrected vertex 103o in the corner detection image 83a is a vertex found from the cropped image 81 without vertex correction in Fig. 11. On the other hand, a filtered image 83aa is generated by applying the corner detection filter 96a for the upper left corner, and then a rectangular upper left corner 101a is set as a vertex search area at a corner of the filtered image 83aa. Then, the position of the pixel with the highest brightness in the vertex search area is set as a corrected vertex 103p.

[0097] The uncorrected vertex 103o is a vertex directly found from the panel image 85, without performing vertex correction such as generating the corner detection image 83a and applying the corner detection filter 96. If a margin edge 87 exists at a corner, a vertex that may cause a false detection 90 is detected. By performing vertex correction, the uncorrected vertex 103o can be changed to a corrected vertex 103p, thereby preventing the occurrence of the false detection 90.

[0098] Fig. 15 is a comparison diagram between an uncorrected panel image 38ua that was not subjected to the vertex correction process of Fig. 11 to Fig. 14 and a corrected panel image 38va that was subjected to the vertex correction process. For convenience, Fig. 15 shows a monochrome image, but in reality, both the uncorrected panel image 38ua and the corrected panel image 38va are color images. Both the uncorrected panel image 38ua and the corrected panel image 38va are images cut out from the captured image 30 based on the cut-out rectangle 36 determined by the vertices Va-Vd.

[0099] In the vertex correction process of Fig. 12, corner detection filters 96a, 96b, 96c, 96d are provided corresponding to the four corresponding corners of corner detection image 83a, namely upper left corner 101a, upper right corner 101b, lower left corner 101c, and lower right corner 101d, respectively, and have a square area with a positive weight (e.g. "1") and a side area that extends along the two sides of the square area on the corresponding corner side and has a weight of 0 or negative (e.g. "-1"). The aim of the corner detection filters 96a, 96b, 96c, 96d is to avoid the error of mechanically determining the four vertices of cropped image 81 as the vertices of corner detection image 83a. Therefore, as long as each of the corner detection filters 96a, 96b, 96c, and 96d reduces the brightness of the edge region along the corresponding side of the associated upper left corner 101a, upper right corner 101b, lower left corner 101c, and lower right corner 101d and increases the brightness of the inner region adjacent to the inside of the edge region, the weight distribution and the size of the edge region and inner region in each corner detection filter are arbitrary and can also be different for each corner detection filter.

[0100] (Supplements and Variations) In the embodiment, a VAE sequence is used as the machine learning technique. The machine learning technique of the present invention is not limited to this, and for example, a GAN sequence (AnoGAN), a diffusion model, a normalizing flow, and the like can be used.

[0101] In the embodiment, STEPb1-STEPb4 have been described as a method in which the anomaly detection unit 62 determines that the solar panel 14 is abnormal. The present invention is not limited to this method. For example, instead of STEPb2-STEPb3, it is also possible to determine for each unit block whether the average value of the differential luminance (for each unit block in the determination image 62e, the total luminance of all pixels / total number of pixels) is equal to or greater than another threshold, and if the determination result is positive, the unit block can be determined to be an abnormal block, and if the determination result is negative, the unit block can be determined to be a normal block.

[0102] Furthermore, the comparison of the average value of the difference luminance with another threshold value may be performed for the entire difference image 60d, rather than for each unit block in the difference image 60d. In other words, the average value of the difference luminance for the entire difference image 60d is calculated, and when the average value is equal to or greater than the other threshold value, the solar panel 14 under inspection may be recognized as being in an abnormal state. [Explanation of symbols]

[0103] 10 ···Solar farm, 12 ···Solar array, 14 ···Solar panel, 16 ···Gap, 18 ···Cell, 26 ···Visible light camera, 30 ···Captured image, 35 ···Individual panel binarized image, 36 ···Cropping frame, 38 ···Individual panel visible light image, 39 ···Anomaly, 40 ···Projection transformation image, 40a ···Normal panel image, 40b ···Abnormal panel image, 46 ···Untrained VAE model, 48 ···Loss calculation unit, 52 ···Training data, 53 ···Inspection unit, 54 ···Generated image, 56 ···Trained VAE model, 56i ···Input image, 56o···Output image, 56ia···Original input image, 57···Abnormal location, 58···Post-processing unit, 60···Difference analysis unit, 60d···Difference image, 62···Abnormality detection unit, 62e···Judgment image, 64···Re-learning system, 76···Additional training data acquisition unit, 78···False detection button, 80···Re-learning unit, 82···Training data, 84···Re-learning VAE model, 85···Target image, 86···Panel periphery, 87···Margin edge, 88···Periphery, 89···White horizontal line, 90···False detection, 96···Corner detection filter.

Claims

1. A method for creating a learning model for solar panel inspection that generates an output image based on an input image of a visible light image of a solar panel to be inspected, comprising: determining four vertices of each solar panel based on an original visible light image as a visible light image obtained by photographing a solar array including a plurality of solar panels, extracting a plurality of panel visible light images for each solar panel from the original visible light image based on the four vertices, and creating a plurality of normalized images by normalizing each panel visible light image; a step of inputting a normalized image of a solar panel without an abnormality from among the plurality of normalized images as an input image to the learning model for inspecting a solar panel, and performing machine learning so as to reduce a difference between the input image and an output image, thereby adjusting weight parameters that define the learning model for inspecting a solar panel; A method for creating a learning model for solar panel inspection, comprising:

2. The method for creating a learning model for solar panel inspection according to claim 1 , wherein the four vertices are identified based on a binarized image of the original visible light image.

3. The four vertices thus determined are: generating a plurality of averaged images by performing an averaging process on the original visible light image using a plurality of averaging filters each having a different size; Next, a plurality of binary images are generated by performing a binary process on each of the averaged images using a plurality of threshold values ​​for pixel values; Next, in each of the plurality of binarized images, a contour of a portion surrounded by a group of low-luminance pixels or a group of high-luminance pixels that are arranged continuously or quasi-continuously is detected as a polygon, and a set of four vertices of the polygon is selected; The method for creating a learning model for solar panel inspection according to claim 2 , wherein the determination is based on four vertices of the selected set.

4. A method for creating a learning model for solar panel inspection according to any one of claims 1 to 3; A step of inputting an input image of a solar panel to be inspected into the learning model for solar panel inspection and outputting an output image; recognizing an abnormal state of the inspection target solar panel based on a difference between the input image and the output image; A solar panel inspection method comprising:

5. 5. The solar panel inspection method according to claim 4, wherein the abnormal state is recognized based on a difference in luminance of each pixel in a difference image between the input image and the output image.

6. A step of setting a rectangular corner detection image that includes a panel image of the solar panel to be inspected in visible light and is wider than the panel image and includes the panel image therein in a captured image; a step of applying a total of four corner detection filters to the corner detection image, each of which is provided to correspond to four corresponding corners of the corner detection image, and which reduces the brightness of a side region along a corresponding side of the corner detection image and increases the brightness of an inner region adjacent to the inside of the side region, to the corner detection image to generate four filtered images; a step of setting a vertex search area at a corner corresponding to the corner detection filter applied to each filtered image, and detecting a vertex for cutting out the panel image from the captured image based on the position of the pixel with the highest brightness in the vertex search area; The method for inspecting a solar panel according to claim 4 , further comprising:

7. A device for creating a learning model for solar panel inspection that generates an output image based on an input image of a visible light image of a solar panel to be inspected, a normalized image creation unit that determines four vertices of each solar panel based on an original visible light image that is a visible light image obtained by photographing a solar array including a plurality of solar panels, cuts out a plurality of panel visible light images of each solar panel from the original visible light image based on the four vertices, and creates a plurality of normalized images by normalizing each panel visible light image; a parameter adjustment unit that inputs a normalized image of a solar panel without an abnormality among the plurality of normalized images as an input image to the learning model for solar panel inspection, and performs machine learning so as to reduce a difference between the input image and an output image, thereby adjusting weight parameters that define the learning model for solar panel inspection; A learning model creation device for solar panel inspection comprising:

8. The solar panel inspection learning model creation device according to claim 7, a model control unit that inputs an input image of a solar panel to be inspected and outputs an output image to the learning model for solar panel inspection; a recognition unit that recognizes an abnormal state of the inspection target solar panel based on a difference between the input image and the output image; A solar panel inspection device comprising:

Images