A method and system for augmenting and identifying photovoltaic aerial infrared image data

Abstract

The present invention discloses a method and system for augmenting and identifying photovoltaic aerial infrared image data. The method includes: S1, correcting the distortion of the aerial image; S2, using Hough detection to detect the contour of the corrected aerial image to extract the complete photovoltaic module; S3, performing preliminary augmentation on the image to form an image data set; S4, inputting the image into a GAN model to obtain a photovoltaic module image set; S5, using MMD to screen the augmented image; S6, inputting the screened image as a test set into a classification model for identification and classification to obtain a classification result. The present invention takes into account the characteristics of photovoltaic aerial infrared images in the model design, making the model more suitable for augmentation and detection of photovoltaic infrared images. In addition, the model used in the present invention fully considers the requirements of different network structures for computing hardware, further reducing the model's dependence on hardware and improving the versatility of the invention.

Description

Technical Field

[0001] The present invention relates to the intersection of artificial intelligence and photovoltaic infrared inspection, and in particular to data augmentation and defect diagnosis technology for photovoltaic aerial infrared images. Background Art

[0002] With the continuous increase in photovoltaic installed capacity, photovoltaic power generation accounts for an increasingly larger proportion in the entire power system. The safe, stable, efficient and stable operation of photovoltaic power stations is of great significance to the entire power system and even the energy system. Timely identification of defects in photovoltaic power stations plays an important role in the status assessment of the power station. As the scale of photovoltaic power stations continues to expand, the traditional photovoltaic module defect diagnosis method that relies on manual labor will become difficult to apply, and an efficient and fast automatic defect diagnosis method is urgently needed. At present, the use of infrared imaging technology to inspect photovoltaic power stations is an effective technical means, but the current infrared detection only compares and observes the acquired images. The images can show the hot spot areas in the photovoltaic modules, but there are many interference factors in the images themselves, which affect the final judgment result, making the obtained defect judgment inaccurate and unable to accurately determine the type of defect, affecting the efficiency of subsequent maintenance work. There are also deep learning methods for image processing, but there are currently two problems in the photovoltaic inspection industry. The first is that the captured infrared images are easily distorted before input, which affects the subsequent processing process and easily leads to errors in the judgment of photovoltaic module defects. The second is that current deep learning models require strong hardware configuration, which brings great difficulties to the deployment of the algorithm. It is highly dependent on hardware and has low versatility. In addition, most of them are directly tested on images after training, and the detection results are not ideal and are prone to large errors. It is not suitable for defect judgment of photovoltaic infrared images. Summary of the Invention

[0003] The present invention provides a method and system for augmenting and identifying photovoltaic aerial infrared image data. The present invention takes into account the characteristics of photovoltaic aerial infrared images, can effectively process images to obtain accurate classification results, and can effectively determine defects in photovoltaic modules. The present invention also reduces the model's dependence on hardware and improves versatility.

[0004] The technical solutions of the present invention are as follows:

[0005] A method for augmenting and recognizing photovoltaic aerial infrared image data includes the following steps:

[0006] S1. Correct the perspective distortion of the original aerial infrared image;

[0007] S2. Detect the photovoltaic module outline in the rectified image using the Hough line detection method to extract a single photovoltaic module image;

[0008] S3, preliminarily augmenting the acquired photovoltaic module images to form an image dataset;

[0009] S4. Input the image dataset into the trained GAN (Generative Adversarial Networks) model for data augmentation to obtain the photovoltaic module image set augmented by the GAN model;

[0010] S5. Evaluate and filter each image in the photovoltaic module image set using MMD (Maximum Mean Discrepancy), discard images with values above a set threshold, and retain only images with values below the set threshold;

[0011] S6. Use the screened and retained photovoltaic module images as a test set, input them into the trained CNN-based classification model for recognition and classification, and output the classification results.

[0012] Furthermore, the original infrared image mentioned in step S1 refers to the infrared image of the photovoltaic components obtained by the drone during the inspection of the power station.

[0013] Furthermore, the specific process of performing perspective distortion correction in step S1 is as follows:

[0014] Based on the mathematical model of camera imaging distortion, a distortion correction mathematical model is constructed. This model obtains the four intersection points of the photovoltaic module area in the original infrared image through image binarization, and then performs distortion correction according to the camera parameter matrix.

[0015] For radial distortion, the distortion correction equation of the model is as follows:

[0016]

[0017] For tangential distortion, the distortion correction equation of the model is as follows:

[0018]

[0019] Where, x 、 y are the coordinates in the image, , and is the corrected value. k 1 ~ k 3 is the radial distortion parameter, P 1 ~ P 2 is the tangential distortion parameter.

[0020] Furthermore, step S2 is based on the strong linear features of the photovoltaic module contour in the infrared image, and then adopts the Hough line detection method, which determines the four endpoints of the photovoltaic module according to the intersection of the straight lines in the image, and then extracts a single photovoltaic module image from the image.

[0021] Furthermore, the preliminary augmentation in step S3 refers to performing a mirroring operation, a flipping operation, or a rotation operation on the photovoltaic component image.

[0022] Furthermore, the GAN model in step S4 includes a generator and a discriminator;

[0023] In the generator, three deconvolution layers are used, where the deconvolution kernels are in the form of 3×3×256×128, 5×5×128×64, and 5×5×64×32 respectively;

[0024] In the discriminator, three convolutional layers are used, which are in the form of 3×3×64×32, 5×5×128×64, and 5×5×256×128.

[0025] Furthermore, in step S5, the formula for evaluating and screening each image in the photovoltaic module image set using MMD is as follows:

[0026]

[0027] Where, P data is the distribution of the training dataset, P g is the generated dataset distribution, is the Gaussian kernel function, and Respectively indicate two P data A random variable, and Express obedience respectively P g A random variable, E Represents the expectation in the distribution of the dataset;

[0028] The quality of the generated images is evaluated using the above formula, images above the set threshold are discarded, and only images below the set threshold are retained.

[0029] Furthermore, the CNN-based classification model in step S6 includes a total of three convolutional modules and a two-layer fully connected layer; the first convolutional module includes two convolutional layers and one pooling layer; the second convolutional module includes three convolutional layers and one pooling layer; and the third convolutional module includes three convolutional layers and one pooling layer; in the convolutional modules, the convolution kernels all adopt a 3×3 form; the first layer of the fully connected layer contains 256 neurons, and the second layer contains 6 neurons; finally, softmax is used to identify and classify the photovoltaic module images, and the classification results are output;

[0030] The model classifies photovoltaic module images into six categories, including normal, diode short circuit, broken glass, connection point overheating, junction box overheating, and foreign object obstruction.

[0031] Furthermore, during the testing process, the classification model uses the stochastic gradient descent method, which is specifically described as follows:

[0032]

[0033] Where, is the calculated weight in the t+1th iteration of the neural network, is the calculated weight in the t-th iteration of the neural network, It is t+ Impulse parameter in 1 iteration, It is t The impulse parameter in the iteration, is the learning rate, is the impulse parameter, is the impulse coefficient, is the gradient value of the convolution kernel weight, F is the loss function value.

[0034] The present invention also provides a photovoltaic aerial infrared image data augmentation and recognition system, comprising:

[0035] Distortion correction module, used to correct perspective distortion of original aerial infrared images;

[0036] An extraction module is used to detect the outline of the photovoltaic module in the rectified image by using the Hough line detection method to extract a single photovoltaic module image;

[0037] A preliminary augmentation module, used to perform preliminary augmentation on the photovoltaic module image obtained from the extraction module to form an image dataset;

[0038] The GAN module is used to input the image dataset obtained from the preliminary augmentation module into the trained GAN model for data augmentation, and obtain the photovoltaic module image set after the GAN model augmentation;

[0039] The data screening module is used to evaluate and screen each image in the photovoltaic module image set obtained from the GAN module through MMD, discarding images above a set threshold and retaining only images below the set threshold;

[0040] The classification module is used to use the photovoltaic module images screened and retained by the data screening module as a test set, input the trained CNN-based classification model for recognition and classification, and output the classification results.

[0041] The beneficial effects of the present invention are:

[0042] This invention uses image correction technology to reduce image distortion during aerial photography. It then uses Hough line detection to accurately extract each PV module image. These images are then initially augmented and fed into a GAN model for further enhancement. The images are then filtered using MMD. Finally, the remaining image set is fed into a CNN-based classification model for identification and classification, outputting accurate classification results. This allows personnel to identify defects in the PV modules and perform quick and effective repairs. Furthermore, the model used in this invention fully considers the computing hardware requirements of different network structures, further reducing the model's hardware dependency and enhancing the invention's versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of the method of the present invention;

[0044] Figure 2 It is the original infrared image taken by aerial photography;

[0045] Figure 3 is the original infrared image after distortion correction;

[0046] Figure 4 This is the straight line detection effect diagram;

[0047] Figure 5 is the extracted image of a single photovoltaic module;

[0048] Figure 6 This is the image effect after being augmented by the GAN model;

[0049] Figure 7 This is the network structure diagram of the GAN model;

[0050] Figure 8 This is the network structure diagram of the CNN-based classification model. DETAILED DESCRIPTION

[0051] The drawings are for illustrative purposes only and should not be construed as limiting this patent. To better illustrate the embodiments, some components in the drawings may be omitted, enlarged, or reduced in size, and do not represent actual product dimensions. Those skilled in the art will understand that some well-known structures and their descriptions may be omitted from the drawings. The positional relationships depicted in the drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0052] Example 1:

[0053] like Figure 1 As shown, a method for augmenting and recognizing photovoltaic aerial infrared image data includes the following steps:

[0054] S1. Correct the perspective distortion of the original aerial infrared image;

[0055] S2. Detect the photovoltaic module outline in the rectified image using the Hough line detection method to extract a single photovoltaic module image;

[0056] S3, preliminarily augmenting the acquired photovoltaic module images to form an image dataset;

[0057] S4. Input the image dataset into the trained GAN (Generative Adversarial Networks) model for data augmentation to obtain the photovoltaic module image set augmented by the GAN model;

[0058] S5. Evaluate and filter each image in the photovoltaic module image set using MMD (Maximum Mean Discrepancy), discard images with values above a set threshold, and retain only images with values below the set threshold;

[0059] S6. Use the screened and retained photovoltaic module images as a test set, input them into the trained CNN-based classification model for recognition and classification, and output the classification results.

[0060] In this embodiment, the original infrared image in step S1 refers to an infrared image of a photovoltaic module obtained by a drone during a power station inspection. The specific process of correcting the perspective distortion of the image is as follows:

[0061] Based on the mathematical model of camera imaging distortion, a distortion correction mathematical model is constructed. This model obtains the four intersection points of the photovoltaic module area in the original infrared image through image binarization, and then performs distortion correction according to the camera parameter matrix, which can effectively reduce the impact of image distortion caused during aerial photography.

[0062] Among them, for radial distortion, the distortion correction equation of the model is as follows:

[0063]

[0064] For tangential distortion, the distortion correction equation of the model is as follows:

[0065]

[0066] Where, x 、 y are the coordinates in the image, , and is the corrected value. k 1 ~ k 3 is the radial distortion parameter, P 1 ~ P 2 is the tangential distortion parameter.

[0067] In this embodiment, step S2 utilizes the Hough line detection method, based on the strong linear characteristics of the PV module outlines in the infrared image. This method determines the four endpoints of the PV module based on the intersection of the lines in the image, and then extracts the image of a single PV module from the image. This method determines the four corner points of each module using the intersection of horizontal and vertical lines, and then extracts each module based on the coordinate endpoints of the four corners. This method effectively extracts the image corresponding to a single PV module, enabling augmented detection of each module in the aerial image for accurate classification.

[0068] In this embodiment, the preliminary augmentation in step S3 refers to performing operations such as mirroring, flipping, and rotating the photovoltaic module image to expand the number of images, form an image data set, and maintain a sufficient number of defect samples for augmented detection to improve accuracy and breadth.

[0069] In this embodiment, the GAN model in step S4 includes a generator and a discriminator, and the structural parameters are shown in the following table:

[0070]

[0071] The generator uses three deconvolution layers, with kernels of 3×3×256×128, 5×5×128×64, and 5×5×64×32, respectively. The discriminator uses three convolution layers, with kernels of 3×3×64×32, 5×5×128×64, and 5×5×256×128, respectively. After augmentation using the GAN model, the image of a single photovoltaic module becomes significantly clearer.

[0072] In this embodiment, the formula for evaluating and screening each image in the photovoltaic module image set by MMD in step S5 is as follows:

[0073]

[0074] Where, P data is the distribution of the training dataset, P g is the generated dataset distribution, is the Gaussian kernel function, and Respectively represent two P data A random variable, and Express obedience respectively P g A random variable, E Represents the expectation in the distribution of the dataset;

[0075] The quality of the generated images is evaluated using the above formula, images above the set threshold are discarded, and only images below the set threshold are retained.

[0076] The above formula is the maximum mean difference formula, which is used to judge the difference in the numerical distribution of two data samples. Therefore, it is used to judge the difference in the numerical distribution between the images generated by GAN and the real images. It is generally believed that the smaller the difference, the better.

[0077] Maximum Mean Discrepancy (MMD) is widely used in transfer learning. It is a measure of the difference between two distributions in Hilbert space. Therefore, MMD can be used to measure the distribution of training datasets. P data and generate dataset distribution P g The distance is then used as the evaluation index of GAN. If the MMD distance is smaller, it means P data and P g The closer they are, the better the performance of the GAN.

[0078] Evaluating the generated images through Maximum Mean Discrepancy (MMD) can effectively block interference and retain valid images. The threshold value is set according to the screening criteria set by the staff. If you want to screen a large range, you can set the threshold higher. On the contrary, if you want to screen a smaller range, you can set the threshold lower to improve the filtering effect and speed.

[0079] In this embodiment, the CNN-based classification model in step S6 includes a total of three convolution modules and a two-layer fully connected layer; the first convolution module includes two convolution layers and one pooling layer; the second convolution module includes three convolution layers and one pooling layer; the third convolution module includes three convolution layers and one pooling layer; in the convolution modules, the convolution kernels are all in the form of 3×3; the first layer of the fully connected layer contains 256 neurons, and the second layer contains 6 neurons; finally, softmax is used to identify and classify the photovoltaic module images, and the classification results are output.

[0080] The model classifies photovoltaic module images into six categories, including normal, diode short circuit, broken glass, connection point overheating, junction box overheating, and foreign object obstruction.

[0081] In this embodiment, during the test, the classification model adopts the stochastic gradient descent method, which is specifically described as follows:

[0082]

[0083] Where, is the calculated weight in the t+1th iteration of the neural network, is the calculated weight in the t-th iteration of the neural network, It is t+ Impulse parameter in 1 iteration, It is t The impulse parameter in the iteration, is the learning rate, is the impulse parameter, is the impulse coefficient, is the gradient value of the convolution kernel weight, F is the loss function value.

[0084] This invention uses image correction technology to reduce image distortion during aerial photography. It then uses Hough line detection to accurately extract each PV module image. These images are then initially augmented and fed into a GAN model for further enhancement. The images are then filtered using MMD. Finally, the remaining image set is fed into a CNN-based classification model for identification and classification, outputting accurate classification results. This allows personnel to identify defects in the PV modules and perform quick and effective repairs. Furthermore, the model used in this invention fully considers the computing hardware requirements of different network structures, further reducing the model's hardware dependency and enhancing the invention's versatility.

[0085] Example 2:

[0086] This example uses an aerial photography dataset (10MW) collected at a photovoltaic power station in Hainan Province as an example. The specific steps are as follows:

[0087] 1) Collect original infrared images of the photovoltaic components of the photovoltaic power station taken by drones, such as Figure 2 As shown;

[0088] 2) Correct the perspective distortion of the collected original infrared image. The corrected effect is shown in the figure below. Figure 3 As shown;

[0089] 3) Perform contour detection on the photovoltaic modules in the rectified image according to the Hough line detection method, and obtain the four endpoints of the photovoltaic modules according to the intersection of different lines (contours), and then extract a single photovoltaic module. The effect of line detection is shown in the figure below. Figure 4 As shown, the effect diagram of component extraction is shown in Figure 5;

[0090] 4) Perform preliminary augmentation operations such as mirroring, flipping, and rotating the photovoltaic module images to expand the images to form an image dataset;

[0091] 5) The image dataset after preliminary augmentation is input into the trained GAN model for data augmentation, and finally the GAN-augmented photovoltaic module image set is obtained; among them, the defect dataset for training GAN is 200 images per category, a total of 1000 defect images, and finally augmented to 8000 defect images. The effect of the augmented image is shown in the figure below. Figure 6 As shown, the network structure diagram of the GAN model is as follows Figure 7 As shown;

[0092] 6) Use MMD to evaluate and filter each image in the augmented PV module image set, discard images above a set threshold, and only retain images below the set threshold;

[0093] 7) Use the filtered images as the test set, input the trained CNN-based classification model for recognition and classification, and output the classification results. The network structure diagram of the classification model is as follows Figure 8 shown.

[0094] Through the above steps, the PV modules in the aerial photography dataset (10MW) collected from a photovoltaic power station in Hainan Province can be accurately identified and classified, allowing staff to quickly repair defective PV modules in a timely manner, effectively improving work efficiency and shortening maintenance time.

[0095] Example 3:

[0096] This embodiment provides a system for augmenting and recognizing photovoltaic aerial infrared image data. The system applies the method of the above-mentioned embodiment 1 and includes:

[0097] Distortion correction module, used to correct perspective distortion of original aerial infrared images;

[0098] An extraction module is used to detect the outline of the photovoltaic module in the rectified image by using the Hough line detection method to extract a single photovoltaic module image;

[0099] A preliminary augmentation module, used to perform preliminary augmentation on the photovoltaic module image obtained from the extraction module to form an image dataset;

[0100] The GAN module is used to input the image dataset obtained from the preliminary augmentation module into the trained GAN model for data augmentation, and obtain the photovoltaic module image set after the GAN model augmentation;

[0101] The data screening module is used to evaluate and screen each image in the photovoltaic module image set obtained from the GAN module through MMD, discarding images above a set threshold and retaining only images below the set threshold;

[0102] The classification module is used to use the photovoltaic module images screened and retained by the data screening module as a test set, input the trained CNN-based classification model for recognition and classification, and output the classification results.

[0103] The above system can be integrated into one device and the final classification results can be displayed through a display module, which helps the back-end staff to quickly locate defective photovoltaic modules and promptly notify the maintenance personnel in the corresponding area to repair the modules.

[0104] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A method for augmenting and recognizing photovoltaic aerial infrared image data, characterized in that: The steps include: S1. Correct the perspective distortion of the original aerial infrared image; S2. Detect the photovoltaic module outline in the rectified image using the Hough line detection method to extract a single photovoltaic module image; S3, preliminarily augmenting the acquired photovoltaic module images to form an image dataset; S4. Input the image dataset into the trained GAN model for data augmentation to obtain a photovoltaic module image set augmented by the GAN model; S5, evaluating and screening each image in the photovoltaic module image set by MMD, discarding images with an image value above a set threshold, and retaining only images with an image value below the set threshold; S6. Use the screened and retained PV module images as a test set, input them into the trained CNN-based classification model for recognition and classification, and output the classification results; The specific process of perspective distortion correction in step S1 is as follows: Based on the mathematical model of camera imaging distortion, a distortion correction mathematical model is constructed. This model obtains the four intersection points of the photovoltaic module area in the original infrared image through image binarization, and then performs distortion correction according to the camera parameter matrix. For radial distortion, the distortion correction equation of the model is as follows: For tangential distortion, the distortion correction equation of the model is as follows: Where x and y are the coordinates in the image, x corrected and y corrected is the corrected value, k1~k3 are radial distortion parameters, and P1~P2 are tangential distortion parameters.

2. The method for augmenting and recognizing photovoltaic aerial infrared image data according to claim 1, characterized in that: The original infrared image mentioned in step S1 refers to the infrared image of the photovoltaic components obtained by the drone during the inspection of the power station.

3. The method for augmenting and recognizing photovoltaic aerial infrared image data according to claim 1, characterized in that: Step S2 is based on the strong linear features of the photovoltaic module outline in the infrared image, and then uses the Hough line detection method. This method determines the four endpoints of the photovoltaic module according to the intersection of the straight lines in the image, and then extracts a single photovoltaic module image from the image.

4. The method for augmenting and recognizing photovoltaic aerial infrared image data according to claim 1, characterized in that: The preliminary augmentation in step S3 refers to performing a mirroring operation, a flipping operation, or a rotation operation on the photovoltaic component image.

5. The method for augmenting and recognizing photovoltaic aerial infrared image data according to claim 1, characterized in that: The GAN model in step S4 includes a generator and a discriminator; In the generator, three deconvolution layers are used, where the deconvolution kernels are in the form of 3×3×256×128, 5×5×128×64, and 5×5×64×32 respectively; In the discriminator, three convolutional layers are used, which are in the form of 3×3×64×32, 5×5×128×64, and 5×5×256×128.

6. The method for augmenting and recognizing photovoltaic aerial infrared image data according to claim 1, characterized in that: The formula for evaluating and screening each image in the photovoltaic module image set using MMD in step S5 is as follows: Where, P data is the distribution of the training dataset, P g is the distribution of the generated data set, k(a,b) is the Gaussian kernel function, a and a' represent two distributions that obey P data The random variables b and b' respectively represent the random variables that obey P g A random variable, E represents the expectation in the distribution of the data set; The above formula is used to evaluate the quality of the generated image, discarding images above the set threshold and retaining only images below the set threshold.

7. The method for augmenting and recognizing photovoltaic aerial infrared image data according to claim 1, characterized in that: The CNN-based classification model in step S6 consists of three convolutional modules and a two-layer fully connected layer. The first convolutional module consists of two convolutional layers and one pooling layer. The second convolutional module consists of three convolutional layers and one pooling layer. The third convolutional module consists of three convolutional layers and one pooling layer. In the convolutional modules, the convolution kernels are all 3×3. The first layer of the fully connected layer contains 256 neurons, and the second layer contains 6 neurons. Finally, softmax is used to identify and classify the photovoltaic module images, and the classification results are output. The model classifies photovoltaic module images into six categories, including normal, diode short circuit, broken glass, connection point overheating, junction box overheating, and foreign object obstruction.

8. The method for augmenting and recognizing photovoltaic aerial infrared image data according to claim 7, characterized in that: During the testing process, the classification model uses the stochastic gradient descent method, which is described as follows: Where W t+1 is the calculated weight in the t+1th iteration of the neural network, W t is the calculated weight in the tth iteration of the neural network, v t+1 is the impulse parameter in the t+1th iteration, v t is the impulse parameter in the tth iteration, η is the learning rate, v is the impulse parameter, γ is the impulse coefficient, is the gradient value of the convolution kernel weight, and F is the loss function value.

9. A photovoltaic aerial infrared image data augmentation and recognition system, characterized in that: include: Distortion correction module, used to correct perspective distortion of original aerial infrared images; An extraction module is used to detect the outline of the photovoltaic module in the rectified image by using the Hough line detection method to extract a single photovoltaic module image; A preliminary augmentation module, used to perform preliminary augmentation on the photovoltaic module image obtained from the extraction module to form an image dataset; The GAN module is used to input the image dataset obtained from the preliminary augmentation module into the trained GAN model for data augmentation, and obtain the photovoltaic module image set after the GAN model augmentation; The data screening module is used to evaluate and screen each image in the photovoltaic module image set obtained from the GAN module through MMD, discarding images above a set threshold and retaining only images below the set threshold; The classification module is used to use the photovoltaic module images retained by the data screening module as a test set, input the trained CNN-based classification model for recognition and classification, and output the classification results; Among them, for radial distortion, the distortion correction equation of the distortion correction module is as follows: For tangential distortion, the distortion correction equation of the distortion correction module is as follows: Where x and y are the coordinates in the image, x corrected and y corrected is the corrected value, k1~k3 are radial distortion parameters, and P1~P2 are tangential distortion parameters.

Images