Method and device for solar panel inspection

Abstract

To provide a method and device for solar panel inspection to improve accuracy of abnormality determination when determining whether or not there is an abnormality in a solar panel to be inspected based on the difference image between two images by inputting an input image of the solar panel to be inspected into a machine learning model generated by unsupervised machine learning using only input images related to visible light images of normal solar panels as training images to output the output images to the model.SOLUTION: A parallel line removal filter is created based on an original difference image by using a difference image as the original difference image, a corrected difference image is generated by removing the parallel lines from the original difference image by using the parallel line removal filter, and the existence of an abnormality in the solar panel to be inspected is determined based on the corrected difference image.SELECTED DRAWING: Figure 12

Description

Technical Field

[0001] The present invention relates to a method and apparatus for inspecting a solar panel using AI (Artificial Intelligence).

Background Art

[0002] Solar power generation has attracted attention as a measure against global warming. In a solar farm for solar power generation, a large number of solar arrays equipped with a plurality of solar panels are installed. After installation, various abnormalities occur in the solar panels. Such abnormalities include, for example, scratches, cracks, dirt, fractures, and furthermore, shadows caused by utility poles and weeds. If these abnormalities are left unattended, they may not only lead to a decrease in the power generation of the solar panel but also an increase in damage. Therefore, it is necessary to monitor the abnormalities and take prompt action when an abnormality is detected.

[0003] Patent Document 1 discloses a solar panel failure diagnosis system in which a flying object (e.g., a radio-controlled helicopter) equipped with an infrared camera is flown over a solar farm of solar panels, the ground solar panels are photographed by the infrared camera, and the presence or absence of hot spots on the solar panels is examined based on the infrared photographed image.

[0004] Patent Document 2 discloses an inspection apparatus in which a flying object (e.g., a drone) equipped with an irradiation unit, a reception unit, and a calculation unit is flown over a solar farm, inspection light is emitted from the irradiation unit toward the ground solar panels, the reflected light from the solar panels is received by the reception unit, and the presence or absence of damage to the solar panels is inspected by the calculation device based on the difference between the optical axes of the irradiation light and the reflected light.

[0005] Patent Document 3 discloses a solar panel appearance monitoring device. In this device, an identifier is constructed based on a group of solar panel images taken by a monitoring camera installed on the ground for a certain period of time when there is no foreign matter (e.g., flying objects or vines) on the solar panel, and a group of foreign matter images taken for a certain period of time with foreign matter. During monitoring, the captured image of the monitoring camera is inspected by the identifier to determine the presence or absence of foreign matter on the solar panel. For example, Random Forests is selected as the identifier.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0007] Although the solar panel failure diagnosis system of Patent Document 1 can inspect the presence or absence of abnormalities in the hot spots of the solar panel from the infrared image, visual inspection is required for physical damage, shadows caused by the growth of vegetation, etc.

[0008] Even if the inspection device of Patent Document 2 can inspect physical damage, shadows caused by the growth of vegetation, etc. on the solar panel, it is necessary to aim at each of the plurality of solar panels one by one and irradiate inspection light in order, which is time-consuming and lengthens the working time.

[0009] The solar panel appearance monitoring device of Patent Document 3 requires images taken of the solar panel and foreign objects over a certain period for the construction of the discriminator. Also, since there are many types of foreign objects, it is time-consuming to collect images for each type of foreign object, and it is difficult to detect abnormalities other than foreign objects, such as cracks and cloudiness in the solar panel itself.

[0010] Therefore, in the previous Japanese Patent Application No. 2023-066434, the inventor adjusted the parameters of a machine learning model by using only the input images related to the panel visible light images of normal solar panels as training images and setting the correct values of the output images as the input images through unsupervised machine learning. Then, the input image of the solar panel to be inspected was input into the machine learning model to output the output image, and a solar panel inspection method was disclosed for determining the presence or absence of abnormalities in the solar panel to be inspected based on the difference image between the two images.

[0011] The inventor has found the following. First, vertical and horizontal lines exist in the difference image that forms the basis for determining the presence or absence of abnormalities, and this leads to misjudgments regarding the presence or absence of abnormalities. Second, the solar panel is composed of a plurality of cells arranged in a grid pattern vertically and horizontally, and the vertical and horizontal lines in the difference image are, for example, derived from the vertical and horizontal frame lines of the cells, and in the difference image, they tend to appear at equal intervals in the vertical and horizontal directions. Third, if the vertical and horizontal lines that appear at equal intervals in the difference image are removed as noise, it can lead to the prevention or reduction of misjudgments and improve the accuracy of abnormality determination. Furthermore, as a fourth point, by removing the vertical and horizontal lines at equal intervals, the accuracy of the line position in the image generated by the machine learning model can be reduced, so it becomes possible to reduce the required amount of training data and suppress the costs related to data collection and learning.

[0012] An object of the present invention is to provide a solar panel inspection method and apparatus that improve the accuracy of abnormality determination of a machine learning model based on the above findings.

Means for Solving the Problems

[0013] The solar panel inspection method of the present invention is A solar panel inspection method in which a solar panel includes a plurality of cells arranged in a grid pattern vertically and horizontally, and only a plurality of input images, each of which is a panel visible light image of a single normal solar panel, are used as training images, and the correct value of the output image is the input image. The input image of the solar panel to be inspected is input to a machine learning model whose parameters are adjusted by unsupervised machine learning, and the output image is output, and the presence or absence of an abnormality in the solar panel to be inspected is determined based on the difference image between the two images. A filter creation step of using the difference image as an original difference image and creating a parallel line removal filter based on the original difference image; A corrected difference image generation step of generating a corrected difference image in which parallel lines existing at equal distance intervals with a frequency equal to or higher than a predetermined threshold are removed from the original difference image using the parallel line removal filter; A determination step of determining the presence or absence of an abnormality in the solar panel to be inspected based on the corrected difference image; and includes.

[0014] The solar panel inspection apparatus of the present invention is A solar panel inspection apparatus in which a solar panel includes a plurality of cells arranged in a grid pattern vertically and horizontally, and only a plurality of input images, each of which is a panel visible light image of a single normal solar panel, are used as training images, and the correct value of the output image is the input image. The input image of the solar panel to be inspected is input to a machine learning model whose parameters are adjusted by unsupervised machine learning, and the output image is output, and the presence or absence of an abnormality in the solar panel to be inspected is determined based on the difference image between the two images. A filter creation unit that uses the difference image as an original difference image and creates a parallel line removal filter based on the original difference image; A corrected difference image generation step of generating a corrected difference image in which parallel lines existing at equal distance intervals with a frequency equal to or higher than a predetermined threshold are removed from the original difference image using the parallel line removal filter; A determination unit that determines the presence or absence of an abnormality in the solar panel to be inspected based on the corrected difference image; and includes.

Advantages of the Invention

[0015] According to the solar panel inspection method and apparatus of the present invention, a parallel line removal filter is created based on a difference image, and using the created parallel line removal filter, parallel lines existing at equal distance intervals with a frequency equal to or higher than a predetermined threshold are removed from the difference image to generate a corrected difference image. Thus, by performing an abnormality determination of the solar panel based on the corrected difference image, the determination accuracy can be significantly improved.

Brief Description of the Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Embodiments for Carrying Out the Invention

[0017] Hereinafter, embodiments of the present invention will be described. Needless to say, the present invention is not limited to the embodiments. Throughout the drawings, the same or common elements are denoted by the same reference numerals.

[0018] First, with reference to FIGS. 1-4, a solar panel inspection device 53 (FIG. 4) to which an embodiment of the present invention is applied will be described.

[0019] (Captured image) FIG. 1 shows the shooting mode of the solar panel 14 by aerial photography. In the solar farm 10, a large number of solar arrays 12 are installed, and a passage 11 for workers and the like to pass through when inspecting and maintaining each solar array 12 is secured between the solar arrays 12. The drone 24 is equipped with a visible light camera 26, flies around above the solar farm 10, and shoots the solar arrays 12 in the solar farm 10 from above.

[0020] Each solar array 12 is typically arranged to extend horizontally with the surface side facing southward at a predetermined inclination angle with respect to the horizontal plane. On the surface of each solar array 12, a large number of solar panels 14 are attached in a grid-like arrangement pattern and perform photovoltaic power generation. The typical shape of the solar panel 14 is square or rectangular. The gap 16 between adjacent solar panels 14 in the solar array 12 is where the board panel substrate of the solar array 12 is exposed. Each solar panel 14 has a plurality of cells 18 arranged in a vertical and horizontal grid arrangement. Note that the exposed portion of the board panel substrate in the solar array 12 is used as the frame line of the solar panel 14 when cutting out the image portion of the solar panel 14 from the captured image of the solar array 12.

[0021] During the flight of the drone 24, the shooting data of the captured image of the solar array 12 taken by the visible light camera 26 may be transmitted online, for example, to the remote controller of a ground-based remote operator, or may be retrieved from the memory of the visible light camera 26 after the drone 24 lands. Note that the drone 24 or the visible light camera 26 is equipped with GPS, and the meta information of the captured image includes, in addition to the shooting date and time, information on the shooting position obtained from GPS.

[0022] In FIG. 2, the leftmost of the three figures is an aerial image 31 taken by the drone 24 from above the solar farm 10 and includes a plurality of solar arrays. The middle figure is a binarized image 33 of the aerial image 31 of the visible light image (color image). The rightmost figure is an enlarged view of a binarized image 35 within the binarized image 33. The binarized image 35 is an image of a single solar panel 14, and the portions corresponding to the frame lines of the cells 18 appear as white vertical and horizontal lines.

[0023] Each of the training image and the inspection image described later is a visible light image (color image) of a single solar panel 14. Such a visible light image of a single solar panel 14 is created, for example, by cutting out based on the four vertices of the rectangular solar panel 14 from a visible light image such as the aerial image 31 including a plurality of solar panels 14. The positions of the four vertices are extracted, for example, from a binarized image obtained by binarizing a visible light image including a plurality of solar panels 14.

[0024] (Unsupervised learning) FIG. 3 is an explanatory diagram of unsupervised learning for the VAE (Variational Autoencoder) model 46 during training. Note that the VAE model 46 during training in FIG. 3 and the trained VAE model 56 in FIG. 4 described later are just different names for the same machine learning model for convenience of explanation. The VAE model 46 during training is composed of a VAE and includes an encoder, a Z space (latent parameter generation unit), and a decoder. Note that from FIG. 3 onward, for convenience of explanation, images are shown in the embodiments, but it goes without saying that the processing by the arithmetic device is performed on the data corresponding to the images, not on the images themselves. Images are input and output via interfaces such as displays and scanners. Therefore, the processing performed on the images is used in the sense that it naturally includes the processing on the data corresponding to the images.

[0025] During unsupervised learning for the VAE model 46 during training, the normal panel images 40a are input one by one as training data 52 a sufficient number of times. The normal panel images 40a themselves are those visually judged by a person to be normal solar panel images. The correct value of the generated image 54 output by the VAE model 46 during training is the normal panel image 40a.

[0026] The loss calculation unit 48 adjusts each parameter of the VAE model 46 during training by the gradient descent method so that the loss value (hereinafter simply referred to as "loss"; e.g., mean squared error) calculated by a predetermined loss function as the difference between the generated image 54 and the normal panel image 40a decreases. Reducing the loss means that the generated image 54 approaches the normal panel image 40a as the correct value. As a result, in the trained VAE model 56, when an input image to be inspected is given, even if the input image 56i has an abnormality, an output image 56o in a normal state will be output.

[0027] Note that the sizes and aspect ratios of the solar panels 14 vary depending on the manufacturer and model number. In the detection logic for cutting out the image portion of the solar panel 14 from the captured image of the solar array 12, all panels are normalized to a unified size, and by using the normalized image as a training image, the differences in the sizes and aspect ratios of the solar panels are eliminated from affecting the learning of the model. Specifically, normalization is performed, for example, by projective transformation so that the four vertices of the panel correspond to the four vertices of a square of a fixed size. The input image 56i at the time of inspection (inference) is also normalized in the same manner as the normal panel image 40a at the time of learning.

[0028] (Solar Panel Inspection Device) Figure 4 is a configuration diagram of the solar panel inspection device 53. The solar panel inspection device 53 includes a trained VAE model 56 and a post-processing unit 58. The trained VAE model 56 is composed of the training VAE model 46 in FIG. 3 that has been trained with the training data 52 and is determined to have completed a predetermined machine learning when the loss calculated by the loss calculation unit 48 reaches below a predetermined standard. The trained VAE model 56 outputs an output image 56o for the input of the input image 56i. The difference analysis unit 60 outputs a difference image 60d in which the luminance (pixel value) of each pixel is the difference in luminance between the corresponding pixels of the input image 56i and the output image 56o. The abnormality detection unit 62 outputs a determination image 62e that is a determination result regarding the abnormality of the solar panel 14 to be inspected based on the difference image 60d.

[0029] The trained VAE model 56 infers an image (normal image) when the input image 56i is that of a normal solar panel 14. Therefore, when the input is a normal image, the trained VAE model 56 outputs an output image 56o that is very close to the input image 56i, while when the input is an abnormal image, it outputs an output image 56o that is very close to an image with the abnormal portion removed from the input image 56i.

[0030] (Method for Determining Abnormality / Basic Example) A basic example of the method for finally determining whether the solar panel 14 is abnormal based on the abnormal unit section 39e by the abnormality detection unit 62 is as follows.

[0031] STEP1: The abnormality detection unit 62 divides the difference image 60d into a plurality of unit sections (unit cells) having the same shape (e.g., square) and size (size) with equally spaced grid lines both vertically and horizontally. Here, the total number of unit sections is designated as Na.

[0032] STEP2: For each unit section in the difference image 60d, it is determined whether the total number Ma of pixels whose difference luminance value (difference luminance) is equal to or greater than the first threshold is equal to or greater than the second threshold. Note that the luminance of each pixel in the difference image 60d is the absolute value of the difference in luminance between corresponding pixels in the input image 56i and the output image 56o.

[0033] STEP3: If the determination result in STEP2 is affirmative, the unit section is recognized as an abnormal section, and if it is negative, it is recognized as a normal section. In the determination image 62e of FIG. 4, black represents a normal unit section, and white represents the abnormal unit section 39e.

[0034] STEP4: Count the number Nb of abnormal unit sections.

[0035] STEP5: If Nb is equal to or greater than the third threshold, it is determined that the solar panel 14 to be inspected is abnormal. If Nb is less than the third threshold, it is determined that the solar panel 14 to be inspected is normal. Note that instead of Nb, Nb / Na can also be compared with the third threshold to determine whether the solar panel 14 to be inspected is normal or abnormal.

[0036] Rather than determining whether the solar panel 14 to be inspected is abnormal based on whether the total number Ma of pixels whose difference luminance value is equal to or greater than the first threshold in the difference image 60d is equal to or greater than the second threshold, the significance of following the procedures of STEP1 - STEP5 is as follows.

[0037] (a) The degree of abnormality of the solar panel 14 can be counted as the number of unit sections. That is, filtering of the display can be performed according to the degree of abnormality, such as "display only solar panels 14 with 5 or more abnormal unit sections", when placing a pin at the position of an abnormal solar panel 14 on the orthoimage showing the solar array 12 including a large number of solar panels 14. (In this case, "5" corresponds to the third threshold value, and the third threshold value can be adjusted as appropriate.)

[0038] (b) The abnormality of the solar panel 14 can be abstracted. That is, by expressing the abnormality in a simpler way, it becomes easier to estimate the similarity of abnormal shapes between different panels.

[0039] (False determination) FIG. 5 is an explanatory diagram of the false determination of the abnormality detection unit 62. In FIG. 5, the original, VAE output, difference, and detection result correspond to the input image 56i, output image 56o, difference image 60d, and determination image 62e, respectively. The input image 56i in FIG. 5 is an input image of a normal solar panel 14, and the output image 56o in FIG. 5 shows an image generated and output by the learned VAE model 56 from the normal input image 56i. However, in the difference image 60d of FIG. 5 output by the difference analysis unit 60, horizontal line-like differences occur throughout the image. As a result, abnormal unit section 29e appears in the determination image 62e output by the abnormality detection unit 62.

[0040] Regarding the reason why abnormal unit section 39e appears in the determination image 62e even though the input image 56i is an input image of a normal solar panel 14, the inventor estimates as follows. (a) Even if the learned VAE model 56 has become able to generate a normal panel image corresponding to the input image 56i, it is very difficult to reproduce without any deviation even the positions and color intensities of all the grid lines of the cells 18 in the grid array within the solar panel 14. (b) On the other hand, since the learned VAE model 56 that has learned the properties of normal panels generates panel images in which the lattice structure is arranged regularly (at equal intervals), when a line misalignment occurs with the input image, the corresponding lines are misaligned at equal intervals, resulting in equally spaced linear differences. (c) Also, when the color intensity of the lattice lines is different from that of the input image, equally spaced linear differences along the lattice lines will occur.

[0041] (Creation of parallel line removal filter) With reference to FIGS. 6 to 14, a method for removing vertical and horizontal parallel lines that are noise for abnormality determination from the difference image 60d will be described. Note that the removal of vertical parallel lines and horizontal parallel lines from the difference image 60d is performed separately, and after removal, the images are integrated to generate a corrected difference image 102 (FIG. 14).

[0042] (Width processing) FIG. 6 is an explanatory diagram of width processing for vertical and horizontal lines. A vertical line width processed image 84a (an example of the "first image" of the present invention) and a horizontal line width processed image 84b (an example of the "second image" of the present invention) are generated from the original difference image 80. Note that the original difference image 80 is an image output by the difference analysis unit 60, and the difference image 60d in FIG. 5 is the original difference image 80. Since the pixel value of the difference image 60d is the absolute value of the pixel value obtained by subtracting the pixel value (0 to 255) of the input image 56i from the pixel value (= luminance) (0 to 255) of the output image 56o, it is positive.

[0043] The sizes of the exclusive vertical line filter 82a and the exclusive horizontal line filter 82b will be expressed in terms of the number of vertical and horizontal cells. The sizes of the exclusive vertical line filter 82a and the exclusive horizontal line filter 82b are both plural in both vertical and horizontal directions. The larger size will be called the length, and the smaller size will be called the width. That is, the width of the exclusive vertical line filter 82a is horizontal, and the width of the exclusive horizontal line filter 82b is vertical. When explaining the value of a cell by specifying the width position within the filter for the cell, it shall apply to all cells in the entire length belonging to the width position.

[0044] Note that, in this example, the exclusive vertical line filter 82a has an odd horizontal size, with the central cell being 1, the two cells on each side of the central cell being 0, and the cells three or more cells to the left and right of the central cell being -1.

[0045] The "allowable width for recognition as a line" can be determined by the number of cells into which the value 0 enters. If an object is recognized by the central cell with a value of 1 and is within the width range of the value 0, the object will not become a negative value in the vertical line width processed image 84a. On the other hand, an object that reaches the cell with a value of -1 beyond the cell with a value of 0 will have its value become negative by the amount of erosion of the cell with a value of -1 and will no longer be recognized as a line.

[0046] In the exclusive vertical line filter 82a of the embodiment, since there are two cells with a value of 0 on each side of the central cell with a value of 1, an object with a width of up to 5 pixels is recognized as a line. On the other hand, when the width of the object becomes 6 pixels or more, it will step on the value -1 outside the cell with a value of 0, so the filtering value will become small and it will no longer be recognized as a line.

[0047] In this way, in the original difference image 80, vertical lines (an example of an object) with a horizontal width of 5 pixels or less (an example of the "first line width" of the present invention) remain in the vertical line width processed image 84a. On the other hand, vertical lines (an example of an object) with a horizontal width of 6 pixels or more disappear from the vertical line width processed image 84a.

[0048] In the exclusive vertical line filter 82a, the number of cells with a value of -1 is set to 3 on each side. In the exclusive vertical line filter 82a, the vertical size is, for example, 11, which is the same as the horizontal size. When applying the exclusive vertical line filter 82a to the left and right end ranges of the original difference image 80, since the left or right cell in the original difference image 80 is missing with respect to the central cell of the exclusive vertical line filter 82a, well-known zero padding is performed for the missing cells.

[0049] The exclusive horizontal line filter 82b is simply the vertical and horizontal directions of the exclusive vertical line filter 82a reversed, so its description will be omitted. When the exclusive horizontal line filter 82b is applied to the original difference image 80, horizontal lines (an example of an object) with a vertical width of 5 pixels or less (an example of the "second line width" of the present invention) will remain in the horizontal line width processed image 84b. The second line width does not have to be equal to the aforementioned first line width.

[0050] (Length processing) FIG. 7 is an explanatory diagram of the length processing of vertical and horizontal lines. The vertical line combined image 85a and the horizontal line combined image 85b are obtained by performing binarization processing on the vertical line width processed image 84a and the horizontal line width processed image 84b in FIG. 6, with a value of 0 as the threshold, replacing values greater than or equal to the threshold with 1 and values less than the threshold with 0. For the vertical line combined image 85a and the horizontal line combined image 85b, first an opening is performed, and then a closing is performed.

[0051] The size of the opening filter is width 1 × length 5. As a result, vertical and horizontal lines with a length of 5 or less are removed as noise, and the vertical line combined image 86a and the horizontal line combined image 86b are generated from the vertical line combined image 85a and the horizontal line combined image 85b (both binarized images).

[0052] The size of the next closing filter is width 1 × length 25. As a result, the vertical line combined binarized image 88a and the horizontal line combined binarized image 88b in which vertical and horizontal lines 25 pixels apart in the same pixel column and the same row are combined are generated from the vertical line combined image 86a and the horizontal line combined image 86b, respectively. Note that well-known morphological transformations are used for the opening and closing processes.

[0053] (Adjacent line integration processing) FIG. 8 is an image explanatory diagram for integrating adjacent vertical lines. The partially enlarged image 89a is an enlarged image of a part of the vertical line combined binarized image 88a. There is a short vertical line adjacent to a long vertical line. The long vertical line and the short vertical line exist in adjacent pixel columns in the horizontal direction.

[0054] In the adjacent line integration process, a vertical line integrated image 90a (an example of the "third image" of the present invention) is generated from the vertically combined binarized image 88a. The partially enlarged image 91a is an image obtained by enlarging a part of the vertically integrated image 90a. The partially enlarged image 89a and the partially enlarged image 91a show the same range. In the partially enlarged image 89a, there were a long vertical line and a short vertical line in adjacent columns respectively, but in the partially enlarged image 91a, the shorter vertical line disappeared and was combined (integrated) into the longer vertical line.

[0055] FIG. 9 is an image explanatory diagram of the adjacent line integration process for horizontal lines. The horizontally combined binarized image 88b is converted into a horizontally integrated image 90b (an example of the "fourth image" of the present invention). The horizontally combined binarized image 88b and the horizontally integrated image 90b respectively correspond to the vertically combined binarized image 88a and the vertically integrated image 90a. In the horizontally combined binarized image 88b and the horizontally integrated image 90b, in the horizontally adjacent pixel columns in the vertical direction, the shorter horizontal line disappears and is combined into the longer vertical line.

[0056] FIG. 10 shows a specific procedure of the adjacent line integration process. Each tabular image in FIG. 10 is a virtual image in which, for the sake of simplicity of explanation, the pixel values of each pixel are displayed numerically by extracting the pixel range of 7 in the vertical direction and 4 in the horizontal direction of the corresponding image. In each tabular image, a blank pixel means that its pixel value is 0.

[0057] The tabular image of STEP102 is extracted from the vertically combined binarized image 88a. In STEP102, weighting is performed on each pixel of the tabular image of STEP102. The weight for the pixels in each pixel column is determined by the number of 1s in each pixel column of the tabular image of STEP102. That is, in the tabular image of STEP102, the number of 1 pixels is 2, 4, 3, 0 in order from the left pixel column to the right. Therefore, when weighting is performed on each pixel of the tabular image of STEP102, the tabular image of STEP104 is generated.

[0058] The tabular image of STEP106 is generated by performing maximum value filtering on the tabular image of STEP104 using a horizontally long filter. That is, when a 1×3 horizontally long filter is applied to the tabular image of STEP104, a tabular image of maximum value filtering is generated in which the maximum value of three horizontally consecutive pixels including the pixel at the center of the horizontally long filter is changed to the pixel value of each pixel, centered around the pixel at the center of the horizontally long filter.

[0059] In STEP108, the tabular image of STEP106 is binarized. This binarization is performed by converting each pixel in the tabular image of STEP106 such that if the pixel value ≠ 0, it is set to 1, and if the pixel value = 0, it is maintained as 0, to generate the tabular image of STEP108.

[0060] In STEP110, the tabular image of STEP108 is weighted again. The method of weighting is almost the same as that in the case of STEP104. Specifically, the weights applied to each column at the time of STEP104 are recorded, and in STEP110, the recorded weights are reused. However, for columns without a record (weight 0), 1 is assigned for convenience.

[0061] The tabular image of STEP112 is generated by performing maximum value filtering (the same horizontally long filter as the 1×3 horizontally long filter of STEP106) on the tabular image of STEP110.

[0062] In STEP114, first, the consistency in each pixel column is checked between the tabular image of STEP112 and the tabular image of STEP114. In the left display format image of STEP114, for the cells where the corresponding pixels match between the tabular image of STEP110 and the tabular image of STEP112, 1 is entered, and for the cells where they do not match, 0 (blank) is entered.

[0063] After that, in STEP114, the tabular image of the matching part and the binarized tabular image of STEP108 are multiplied by the numerical values of corresponding pixels. As a result, the tabular image of STEP116 is generated. The pixel of 1 in STEP116 becomes the pixel constituting the vertical line. That is, in STEP102, when a long line and a short line are in contact between horizontally adjacent pixel columns, they are integrated into the longer one. Also, when there are a plurality of vertical lines that are separated vertically between horizontally adjacent pixel columns, the vertical line with a smaller weight moves to the pixel column of the vertical line with a larger weight given in the first weighting.

[0064] In the weighting method of STEP104 in FIG. 10, when the number of 1s is the same between horizontally adjacent pixel columns, the weights are equal values. For example, in the tabular image of STEP102, when the number of 1s in the third image column from the left is the same as the number of 1s in the second pixel column from the left, which is 4, the weights of the second and third image columns from the left are both 4. This hinders the determination of the distance between the vertical lines in FIG. 12.

[0065] FIG. 11 shows an improved example of weighting. In the weighting of FIG. 11, each weight is composed of an integer part and a decimal part. The integer part is set to the number of 1s in each pixel column. On the other hand, the decimal part is, for example, Un / Ut when the total number of pixel columns is Ut and the number of each pixel column is Un. As a result, even if pixel columns with the same number of 1s are continuous, the weight of the right pixel column becomes larger than the weight of the left pixel column, making it easier to uniquely determine the matching pixel columns in STEP114.

[0066] The description of the integration process of adjacent horizontal lines is omitted, but it is the same as the integration process of adjacent vertical lines in FIGS. 10 and 11. In the integration process of adjacent horizontal lines, only the vertical and horizontal directions of each tabular image in FIGS. 10 and 11 are reversed.

[0067] (Distance frequency) FIG. 12 is an explanatory diagram for completing the vertical parallel line removal filter 94a and the horizontal parallel line removal filter 94b based on the distance frequency. The vertical parallel line removal filter 94a removes vertical parallel lines that appear at a certain horizontal distance among a plurality of mutually parallel vertical lines included in the vertical line integrated image 90a from the difference image 60d. The horizontal parallel line removal filter 94b removes horizontal parallel lines that appear at a certain vertical distance among a plurality of mutually parallel horizontal lines included in the horizontal line integrated image 90b from the difference image 60d.

[0068] First, the method of determining the horizontal distance of the parallel vertical lines removed by the vertical parallel line removal filter 94a will be described. In the vertical line integrated image 90a, short vertical lines with a length less than 1 / 10 of the vertical size of the vertical line integrated image 90a are not regarded as straight lines and are excluded. That is, only vertical lines with a length of 1 / 10 or more of the vertical size of the vertical line integrated image 90a are the detection targets for the horizontal distance. Also, for the horizontal distance, an upper limit is set according to the horizontal size of the solar panel 14 and the cell 18, and horizontal distances exceeding the upper limit are not detected.

[0069] In this way, for all combinations of two target vertical lines existing in the vertical line integrated image 90a, the horizontal distance is detected. This combination is selected regardless of whether the horizontal ranges of the two target vertical lines overlap or not.

[0070] An example of the distance list of the detected horizontal distances is arranged in the detection order as [25, 50, 25] in FIG. 12. In this list example, three detection distances are described. This is because, as described above, short vertical lines with a length less than 1 / 10 of the vertical size of the vertical line integrated image 90a are not regarded as straight lines and are excluded in advance. Therefore, there are only three pixel columns of the vertical lines that remain without being excluded, and there are only three combinations of the detection distances.

[0071] Next, a distance candidate list arranged in descending order of distance (descending order of appearance frequency) based on the distance list and a frequency list are created. In this example, the distance candidate list is [25, 50], and the frequency list is [2, 1]. That is, in the detection list, there are two 25s, and the frequency is 2. There is one 50, and the frequency is 1. Note that this 1 corresponds to the "predetermined threshold" in the "equidistant interval with a frequency equal to or higher than the predetermined threshold" of the present invention.

[0072] Next, the adopted distance is determined starting from the distance candidate with the highest frequency. Initially, the adopted distance list is empty. For each distance candidate, it is checked whether its multiple or divisor already exists in the adopted distance list. When the distance candidate with the highest frequency is 25, since the adopted distance list is empty, 25 is directly adopted into the adopted distance list. Note that when checking whether a multiple or divisor of a distance candidate already exists in the adopted distance list, it is not limited to exact multiples and divisors with the same numerical value, and a difference of ±1 or a certain degree can be regarded as the same.

[0073] Next, for the distance candidate 50 with the second highest frequency, since its divisor 25 exists in the adopted distance list, the adoption into the adopted distance list is rejected. In this way, the adoptability of each distance candidate into the adopted distance list is determined until the last distance of the distance candidates.

[0074] The vertical parallel line removal filter 94a has a center cell value of 1, cells 25 to the left and right of the center cell, and cells in the columns 25 away from the center cell have a value of -0.1. This 25 is set based on the adopted distance in the adopted distance list related to the vertical lines. Also, this 0.1 in "-0.1" is used as the value of the removal intensity parameter.

[0075] The removal intensity parameter is defined as a parameter related to the strength of removal when removing vertical or horizontal lines as noise. The larger the value of the removal intensity parameter, the greater the removal effect is set to be. For example, to determine the value, set the initial value to "-1 / (width of the filter × 2)" and observe the situation. Here, the "width of the filter" is the vertical width in the case of the vertical parallel line removal filter 94a, and the horizontal width in the case of the horizontal parallel line removal filter 94b. When the setter determines that the removal effect is weak, the value is decreased (since the value is negative, the absolute value increases), and when it is determined that the removal effect is too strong, the value is increased (however, it is set to be less than 0 (negative value)).

[0076] The vertical parallel line removal filter 94a is applied to the original difference image 80. If the vertical size of the vertical parallel line removal filter 94a is 5 or more, when the central mass is at the position of the vertical line that appears every 25 masses in the horizontal direction, the pixel value of the pixels overlapping the central mass becomes 1 - 0.1×5×2, which becomes 0. Therefore, on the image of the drawing, the color changes from white to black.

[0077] Two types of vertical parallel line removal filters 94a are prepared: one with both ends being -0.1 as shown in the figure and another (not shown) with both ends being -0.2. And the vertical parallel line removal filter 94a with both ends being -0.2 is used only when the central mass is in the range from the 1st to the 24th in the left and right end columns of the difference image 60d.

[0078] The method of determining the vertical distance of the parallel horizontal lines removed by the horizontal parallel line removal filter 94b is also performed in the same way as the vertical parallel line removal filter 94a. In the case of the horizontal parallel line removal filter 94b, the detection distance list is [13, 27, 34, 13, 21, ~]. The distance candidate list is [13, 21, 27..~], and the adopted distance list is [13, 21]. The reason why 27 is not included in the adopted distance list is that when determining the admissibility of 27 to the adopted distance list, 13 has already been adopted in the adopted distance list, and 27 is a numerical value within the range of ±1 of 26 which is a multiple of 13.

[0079] In the horizontal parallel line removal filter 94b, since the adoption distances are two values, 13 and 21, the rows of -0.1 (similarly to the case of the vertical parallel line removal filter 94a, 0.1 is used as the value of the removal intensity parameter) will be set two each above and below the central cell with a value of 1 in the horizontal parallel line removal filter 94b. The application of the horizontal parallel line removal filter 94b to the original difference image 80 is also applied to the original difference image 80. The application of the horizontal parallel line removal filter 94b to the horizontal line extraction image is the same as the application of the vertical parallel line removal filter 94a to the vertical line extraction image.

[0080] (Generation of corrected difference image) Figure 13 is a comparison diagram of the original difference image 80, the vertical line removal image 100a, the horizontal line removal image 100b, the vertical line removal application area image 98a, and the horizontal line removal application area image 98b. The procedure for generating the corrected difference image 102 (Figure 14) from the original difference image 80 will be described in order. In the following description, the generation of the vertical corrected difference image is described prior to the description of the generation of the horizontal corrected difference image, but the order of generation may be reversed.

[0081] First, the vertical parallel line removal filter 94a is used to process the original difference image 80 to create the vertical line removal image 100a. Note that in the thus-generated vertical line removal image 100a, areas other than the vertical lines may also be excessively removed. The masking process using the vertical line removal application area image 98a, which will be described later, as a vertical line mask is for restoring the excessively removed vertical lines.

[0082] Each pixel value of the original difference image 80 is calculated by the difference analysis unit 60 based on the luminance of the corresponding pixels of the input image 56i and the output image 56o. For example, (a) taking the absolute value of the luminance difference between the corresponding pixels of the input image 56i and the output image 56o, or (b) subtracting the luminance value of the output image 56o from the luminance value of the input image 56i for each corresponding pixel, or (c) subtracting the luminance value of the input image 56i from the luminance value of the output image 56o for each corresponding pixel, etc. can be selected. (b) is effective in a situation where the luminance of the foreign object is greater than the surrounding area, and (c) is effective conversely in a situation where the luminance of the foreign object is smaller than the surrounding area. By adding a process of replacing negative values with 0 to set the lower limit of each pixel value to 0, a more emphasized original difference image 80 of the characteristics of the foreign object can be generated for both (b) and (c).

[0083] The vertical line mask as the vertical line removal application area image 98a is created by performing a dilation process (Erosion / Morphological transformation) on the vertical line integration image 90a (Fig. 8) in the width direction (the width direction in the vertical line removal application area image 98a is the horizontal direction). Note that in the vertical line removal application area image 98a due to the dilation process, the lines are thicker than those in the vertical line integration image 90a.

[0084] The vertical line mask is a binary image (0 or 1). In the vertical line area which is the vertical line inflated by the dilation process in the vertical line mask, the pixel value is assigned the "1" as the first value of the binary, and in the area outside the vertical line area, the pixel value is assigned the "0" as the second value of the binary. Applying the vertical line mask to the original difference image 80 generates a vertically corrected difference image (not shown).

[0085] In the vertically corrected difference image, the pixels in the original difference image 80 where the corresponding pixels of the vertical line mask are the first value are replaced with the pixel values of the vertical line removal image 100a. On the other hand, for the pixels in the original difference image 80 where the corresponding pixels of the vertical line mask are the second value, the pixel values of the original difference image 80 are maintained in the vertically corrected difference image.

[0086] In this embodiment, the value "1" as the first value is assigned to the vertical line region, and the value "0" as the second value is assigned to the region outside the vertical line region. However, specifically what numerical values are set as the first value and the second value is at the discretion of the designer. For example, in addition to setting the first value and the second value as 1 and 0 respectively, different numerical values such as 255 and 0 may be assigned.

[0087] The horizontal correction difference image (not shown) is also generated in the same procedure as the vertical correction difference image. Specifically, the horizontal line removal application region image 98b is created by performing a dilation process on the horizontal line integration image 90b (FIG. 9) in the width direction (the width direction in the horizontal line removal application region image 98b is the vertical direction). Next, using the horizontal line removal application region image 98b as a horizontal line mask and applying it to the original difference image 80, a horizontal correction difference image is generated.

[0088] In the specific masking process using the horizontal line mask, the value "1" as the third value is assigned as the pixel value to the horizontal line region which is the horizontal line inflated by the dilation process in the horizontal line mask, and the value "0" as the fourth value is assigned as the pixel value to the region outside the horizontal line region. In the horizontal correction difference image generated by applying the horizontal line mask to the original difference image 80, the pixels in the original difference image 80 where the corresponding pixel of the horizontal line mask is 1 are replaced with the pixel values of the horizontal line removal image 100b, and the pixels in the original difference image 80 where the corresponding pixel of the horizontal line mask is 0 maintain the pixel values of the original difference image 80 as they are.

[0089] However, also in the horizontal line mask, similar to the case of the vertical line mask, specifically what numerical values are set as the third value and the fourth value is at the discretion of the designer. For example, in addition to setting the third value and the fourth value as 1 and 0 respectively, different numerical values such as 255 and 0 may be assigned.

[0090] FIG. 14 is a diagram showing a comparison between the original difference image 80 and the corrected difference image 102. The original difference image 80 is the same as the difference image 60d. The corrected difference image 102 is generated by synthesizing the vertical line corrected difference image and the horizontal line corrected difference image described in FIG. 13. In this synthesis, for each pixel of the corrected difference image 102, when the pixel values are the same in the vertical line corrected difference image and the horizontal line corrected difference image, the same value is selected, but when they are different, the smaller pixel value (the pixel value with a greater removal effect) is selected.

[0091] FIG. 15 is a comparison diagram between the original difference image 80 and the corrected difference image 102 when the original difference image 80 includes an abnormality 39d. It can be seen that in the corrected difference image 102, the abnormality 39d of the original difference image 80 remains without being removed. Thus, it can be understood that by replacing the original difference image 80 with the corrected difference image 102, the abnormality detection unit 62 can determine abnormalities without problems.

[0092] (Solar panel inspection device) The solar panel inspection device 53 includes a filter creation unit (for example, a processing unit that performs the processing of FIG. 12) that uses the difference image 60d as the original difference image 80 and creates parallel line removal filters (for example, the vertical parallel line removal filter 94a and the horizontal parallel line removal filter 94b) based on the original difference image 80, a corrected difference image generation unit (for example, a processing unit that performs the processing corresponding to FIGS. 13 and 14) that generates a corrected difference image 102 by removing the parallel lines corresponding to the frame lines of the cell 18 from the original difference image 80 using the parallel line removal filters, and a determination unit (for example, the abnormality detection unit 62) that determines the presence or absence of an abnormality in the solar panel to be inspected based on the corrected difference image 102.

Explanation of reference numerals

[0093] 14... solar panel, 18... cell, 29e... abnormal unit section, 39d... abnormality, 40... normal panel image, 46... unlearned VAE model, 48... loss calculation unit, 52... training data, 53... solar panel inspection device, 54... generated image, 56... learned VAE model, 56i... input image, 56o... output image, 58... post-processing unit, 60... difference analysis unit, 60d... difference image, 62... abnormality detection unit, 62e... determination image, 80... original difference image, 82a... exclusive vertical line filter, 82b... exclusive horizontal line filter, 84a... vertical line width processing image, 84b... horizontal line width processing image, 86a... unnecessary vertical line removal image, 86b... unnecessary horizontal line removal image, 88a... vertical line combination image, 88b... horizontal line combination image, 90a... vertical line integration image, 90b... horizontal line integration image, 94a... vertical parallel line removal filter, 94b... horizontal parallel line removal filter, 100a... vertical line removal image, 100b... horizontal line removal image, 102... corrected difference image.

Claims

1. A solar panel inspection method, comprising a plurality of cells arranged in a grid pattern vertically and horizontally, using only a plurality of input images, each of which is a panel visible light image of a single normal solar panel, as training images, adjusting parameters by unsupervised machine learning with the correct value of the output image being the input image, inputting an input image of a solar panel to be inspected into a machine learning model, outputting an output image thereof, and determining the presence or absence of an abnormality in the solar panel to be inspected based on a difference image between the two images, wherein: A filter creation step of using the difference image as an original difference image and creating a parallel line removal filter based on the original difference image; A corrected difference image generation step of using the parallel line removal filter to remove parallel lines existing at equal distance intervals with a frequency equal to or higher than a predetermined threshold from the original difference image to generate a corrected difference image; A determination step of determining the presence or absence of an abnormality in the solar panel to be inspected based on the corrected difference image; A solar panel inspection method comprising the above steps.

2. In the solar panel inspection method according to Claim 1, the parallel line removal filter is created in the filter creation step by calculating the frequency of each distance between vertical lines and between horizontal lines in the original difference image (80), and removing vertical lines and horizontal lines existing in the original difference image at distances where the frequency is equal to or higher than the threshold, as a filter.

3. In the solar panel inspection method according to Claim 2, in the filter creation step, the corrected difference image is generated based on a first image and a second image in which only vertical lines with a first line width or less and only horizontal lines with a second line width or less are respectively left in the original difference image.

4. In the solar panel inspection method according to Claim 3, the corrected difference image is generated based on a third image, and the generation step of the third image includes: a step of determining weights for each pixel column of a vertically combined binary image generated by morphological transformation of the binary image of the first image based on the number of 1s included in the pixel column; When a plurality of vertical lines exist in a plurality of horizontally continuous pixel columns in the vertical line combined binary image, a single integrated pixel column is determined from among the plurality of pixel columns based on the weights of each pixel column, and the plurality of vertical lines of the plurality of pixel columns are transferred to the integrated pixel column and integrated as the vertical lines of the integrated pixel column to generate a third image as a vertical line integrated image. A solar panel inspection method including the above.

5. In the solar panel inspection method according to claim 4, In the step of generating the third image, When the number of 1s included in the horizontally continuous pixel columns is equal, the weight of the pixel column on one side in the horizontal direction is set to be greater than the weight of the pixel column on the other side. A solar panel inspection method.

6. In the solar panel inspection method according to claim 5, In the filter creation step, a vertical parallel line removal filter is created based on the horizontal distance between the vertical lines detected based on the third image and the frequency of each horizontal distance. In the corrected difference image generation step, the corrected difference image is generated based on the vertical line removal image generated by applying the vertical parallel line removal filter to the original difference image. A solar panel inspection method.

7. In the solar panel inspection method according to claim 6, A vertical line removal application area image including a vertical line area obtained by expanding each vertical line of the third image in the horizontal direction is generated, and the vertical line removal application area image is used as a vertical line mask composed of two values, a first value assigned to the vertical line area and a second value assigned to an area other than the vertical line area, for the original difference image. A vertical correction difference image is generated by replacing only the pixels in the original difference image whose corresponding pixels in the vertical line mask are the first value with the values of the vertical line removal image, and the corrected difference image is generated based on the vertical correction difference image. A solar panel inspection method.

8. In the solar panel inspection method according to any one of claims 3 to 7, The corrected difference image is generated based on a fourth image. The step of generating the fourth image For each pixel row of the horizontal line combined binary image generated by morphological transformation on the binary image of the second image, a step of determining a weight based on the number of 1s included in the pixel row. When a plurality of horizontal lines exist in a plurality of pixel rows continuous in the vertical direction in the horizontal line combined binary image, a single integrated pixel row is determined from among the plurality of pixel rows based on the weights of each pixel row, and the plurality of horizontal lines of the plurality of pixel rows are transferred to the integrated pixel row and integrated as horizontal lines of the integrated pixel row to generate the fourth image as a horizontal line integrated image. A solar panel inspection method including the above.

9. In the solar panel inspection method according to claim 8, In the step of generating the fourth image, When the number of 1s included in pixel rows continuous in the vertical direction is equal, the weight of the pixel row on one side in the vertical direction is set to be greater than the weight of the pixel row on the other side. A solar panel inspection method.

10. In the solar panel inspection method according to claim 9, In the filter creation step, a horizontal parallel line removal filter is created based on the vertical distance between the horizontal lines detected based on the fourth image and the frequency of each vertical distance. In the corrected difference image generation step, the corrected difference image is generated based on the horizontal line removal image generated by applying the horizontal parallel line removal filter to the original difference image. A solar panel inspection method.

11. In the solar panel inspection method according to claim 10, A horizontal line removal application area image including a horizontal line area obtained by expanding each horizontal line of the fourth image in the vertical direction is generated, and the horizontal line removal application area image is used for the original difference image as a horizontal line mask composed of a third value assigned to the horizontal line area and a fourth value assigned to an area other than the horizontal line area. A horizontal correction difference image is generated by replacing only the pixels in the original difference image whose corresponding pixels in the horizontal line mask are the third value with the values of the horizontal line removal image, and the corrected difference image is generated based on the horizontal correction difference image. A solar panel inspection method.

12. In the solar panel inspection method according to claim 11, The pixel value of each pixel of the corrected difference image becomes the same pixel value when the pixel value of the pixel corresponding to the pixel in the vertical correction difference image and the pixel value of the pixel in the horizontal correction difference image are the same, and becomes the smaller pixel value of the two pixel values when they are different. A solar panel inspection method for generating the corrected difference image.

13. A solar panel inspection apparatus includes a plurality of cells arranged in a grid pattern vertically and horizontally. Only a plurality of input images, each of which is a panel visible light image of a single normal solar panel, are used as training images, and the correct values of the output images are the input images. A machine learning model with parameters adjusted by unsupervised machine learning is used. An input image of the solar panel to be inspected is input to the machine learning model to output an output image, and the presence or absence of an abnormality in the solar panel to be inspected is determined based on the difference image between the two images. A filter creation unit that uses the difference image as an original difference image and creates a parallel line removal filter based on the original difference image. A corrected difference image generation step that uses the parallel line removal filter to generate a corrected difference image by removing parallel lines existing at equal distance intervals with a frequency equal to or higher than a predetermined threshold from the original difference image. A determination unit that determines the presence or absence of an abnormality in the solar panel to be inspected based on the corrected difference image. A solar panel inspection apparatus comprising the above components.

Images