Method for Locating and Defect Detecting Photovoltaic Modules in Infrared Images Based on Deep Learning

Abstract

The present invention relates to a method and system for positioning and defect detection of photovoltaic modules in infrared images based on deep learning. The method includes: S1: Annotating the positions and categories of photovoltaic modules in the collected infrared images of photovoltaic modules; constructing an infrared image dataset of photovoltaic modules and dividing it into a training set and a test set according to a preset ratio; S2: Inputting the training set into an edge detection network to obtain an edge mask map of the photovoltaic module. Among them, the edge detection network is constructed based on the Bottleneck of MobileNetV3 fused with pixel difference convolution and combined with a top-down feature pyramid network; S3: Inputting the edge mask map into a contour screening module and using a quadrilateral modeling algorithm to obtain a candidate position box of the photovoltaic module; S4: Inputting the single photovoltaic module image selected according to the position box into a trained improved ResNet network for classification; and outputting the position box of the defective photovoltaic module. The method provided by the present invention is easier to detect the edge information of photovoltaic modules, with fast detection speed and high accuracy.

Description

Technical Field

[0001] The present invention relates to the fields of image recognition and deep learning, and particularly relates to a method and system for positioning and defect detection of photovoltaic modules in infrared images based on deep learning. Background Art

[0002] According to NASA data, the carbon dioxide concentration has increased by 47% in the past 170 years. Such drastic changes will lead to various hazards such as global warming and reduced crop yields. Against this background, the goal of "carbon neutrality" pursuing energy conservation and emission reduction is proposed. To achieve this goal, clean energy with renewable and low-pollution characteristics is a key link, which can replace fossil energy and achieve "carbon reduction" at the source, such as wind power, solar energy, nuclear energy, etc. As one of the most promising clean energies, the utilization of solar energy has undergone substantial improvements in recent decades. After half a century of development, China's photovoltaic industry has gradually become mature, and its scale, production capacity, market share, and total installed capacity all rank first in the world.

[0003] However, with the rapid development of the photovoltaic industry, a large number of photovoltaic (PV) solar power plants are facing challenges in operation and maintenance. Photovoltaic modules are characterized by wide distribution, large quantity, and high failure rate, and their defects are difficult to detect and locate. Thousands of solar panels are usually located in remote corners such as mountains, and mainly rely on inspectors to perform inspection tasks according to the specified routes, shifts, and designated projects, with high costs and low efficiency. Since defects usually appear as local overheating or local temperature anomalies, infrared (IR) imaging is widely used to identify damaged solar panels on the surface temperature of photovoltaic modules, also known as thermal imaging. Compared with visible images, thermal imaging can enhance the recognition of defects and weaken the interference of the background area. At the same time, with the wide application of unmanned aerial vehicles (UAVs), more and more photovoltaic power plants have begun to attempt to use UAVs equipped with infrared cameras to collect images of photovoltaic modules and use image recognition technology to autonomously identify defects. Therefore, there is an urgent need for a method that can automatically identify photovoltaic modules based on the collected images and perform defect recognition on them, so as to assist the efficient operation and maintenance of future photovoltaic power plants, thereby promoting the stable development of China's photovoltaic industry and contributing to the optimization of the energy industry structure and sustainable development.

[0004] Currently, the defect diagnosis task of photovoltaic modules can be divided into two subtasks: positioning and classification, and can be roughly divided into three categories according to the technologies and means adopted:

[0005] Statistical methods. This type of method assumes that the temperature data of photovoltaic modules follows a certain data distribution, and reflects the fault points through abnormal data that does not satisfy this distribution. For example, Dotenco et al. proposed an automatic detection and analysis method for photovoltaic modules in infrared images under drones. First, the temperature of the infrared images of photovoltaic modules was modeled using the Gaussian distribution in statistics, effectively detecting the area of photovoltaic modules. Then, the overheated areas in the photovoltaic modules were identified through the temperature median, histogram, and Q - value test method of the photovoltaic module area.

[0006] Machine vision - based methods. This type of method uses manually designed feature descriptors to distinguish photovoltaic modules from the background area, and then uses machine learning methods such as support vector machines to distinguish between faulty and normal categories for the located photovoltaic modules. For example, Vega Díaz et al. proposed an algorithm based on classical machine learning for infrared images of photovoltaic modules with low contrast in complex backgrounds. First, the image contrast was enhanced through pre - processing, then an improved Canny algorithm based on multiple parallel Gaussian kernels was used to locate the photovoltaic modules, and finally, SVM was used to distinguish faulty modules for the located photovoltaic modules.

[0007] Deep learning - based methods. This type of method utilizes the powerful multi - level feature extraction ability of convolutional neural networks, and does not rely on manually extracted features. It uses the achievements in various fields of computer vision such as edge detection and image classification, and can perform more accurate defect identification of photovoltaic modules and has stronger robustness compared to the previous two methods.

[0008] In addition, during the flight of the drone, due to changes in flight angle and altitude, the size and arrangement angle of photovoltaic modules in the collected different images vary greatly, and the shape of the photovoltaic modules is distorted to a certain extent and is no longer the original rectangle; coupled with the interference of background factors, it has caused no small interference to the positioning of photovoltaic modules. Therefore, corresponding solutions need to be made for these problems. Summary of the Invention

[0009] To solve the above - mentioned technical problems, the present invention provides a method and system for locating and defect - detecting photovoltaic modules in infrared images based on deep learning.

[0010] The technical solution of the present invention is: A method for locating and defect - detecting photovoltaic modules in infrared images based on deep learning, including:

[0011] Step S1: Label the position and category of the photovoltaic modules in the collected infrared images of photovoltaic modules; construct an infrared image dataset of photovoltaic modules, and divide it into a training set and a test set according to a preset ratio;

[0012] Step S2: Input the training set into the edge detection network to obtain the edge mask map of the photovoltaic module. Among them, the edge detection network is constructed based on the Bottleneck of MobileNetV3 integrated with pixel differential convolution and combines a top-down feature pyramid network;

[0013] Step S3: Input the edge mask map into the contour screening module, and use the quadrilateral modeling algorithm to obtain the candidate position box of the photovoltaic module;

[0014] Step S4: Input the single photovoltaic module image selected according to the position box into the trained improved ResNet network for classification; output the position box of the defective photovoltaic module.

[0015] Compared with the prior art, the present invention has the following advantages:

[0016] 1. The present invention discloses a method for positioning and defect detection of photovoltaic modules in infrared images based on deep learning. After inputting the collected infrared image of the photovoltaic module, the diagnosed image can be directly output end-to-end, marking the defective photovoltaic modules.

[0017] 2. The present invention replaces the depth convolution in the Bottleneck structure of MobileNetV3 with pixel differential convolution. On the basis of retaining the lightweight characteristics of MobileNetV3, integrating pixel differential convolution is more conducive to detecting edge information.

[0018] 3. Since the shooting angle is not always perpendicular to the surface of the photovoltaic module during the flight of the drone, most of the photovoltaic modules in the collected images are not the original rectangles and have been distorted into quadrilaterals. The present invention uses quadrilaterals to model the recognized contours of the photovoltaic modules, which can more accurately extract the candidate position boxes of the photovoltaic modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of a method for positioning and defect detection of photovoltaic modules in infrared images based on deep learning in an embodiment of the present invention;

[0020] Figure 2 It is a schematic diagram of the edge detection network structure in an embodiment of the present invention;

[0021] Figure 3 It is a schematic diagram of obtaining the minimum circumscribed quadrilateral in an embodiment of the present invention;

[0022] Figure 4 It is a schematic diagram of the photovoltaic module defect detection result in an embodiment of the present invention;

[0023] Figure 5This is a structural block diagram of a photovoltaic module positioning and defect detection system in an infrared image based on deep learning in an embodiment of the present invention. Specific embodiments

[0024] The present invention provides a method for positioning and defect detection of photovoltaic modules in infrared images based on deep learning, which can more easily detect the edge information of photovoltaic modules, with fast detection speed and high accuracy.

[0025] In order to make the objectives, technical solutions and advantages of the present invention clearer, the following further elaborates on the present invention through specific embodiments and in conjunction with the accompanying drawings.

[0026] Embodiment 1

[0027] As Figure 1 shown, a method for positioning and defect detection of photovoltaic modules in infrared images based on deep learning provided by an embodiment of the present invention includes the following steps:

[0028] Step S1: Label the positions and categories of photovoltaic modules in the collected infrared images of photovoltaic modules; construct an infrared image dataset of photovoltaic modules, and divide it into a training set and a test set according to a preset ratio;

[0029] Step S2: Input the training set into an edge detection network to obtain an edge mask image of the photovoltaic module. Among them, the edge detection network is constructed based on the BottleNeck of MobileNetV3 fused with pixel difference convolution and combines a top-down feature pyramid network;

[0030] Step S3: Input the edge mask image into a contour screening module to obtain a candidate position box of the photovoltaic module using a quadrilateral modeling algorithm;

[0031] Step S4: Input the single photovoltaic module image selected according to the position box into a trained improved ResNet network for classification; output the position box of the defective photovoltaic module.

[0032] In one embodiment, the above step S1: Label the positions and categories of photovoltaic modules in the collected infrared images of photovoltaic modules; construct an infrared image dataset of photovoltaic modules, and divide it into a training set and a test set according to a preset ratio, specifically including:

[0033] During the flight of the unmanned aerial vehicle, collect the infrared images of photovoltaic modules, manually label the positions and categories of photovoltaic modules in the infrared images, construct an infrared image dataset of photovoltaic modules, including an edge dataset and a category dataset of single photovoltaic modules, and divide it into a training set and a test set according to a preset ratio.

[0034] In one embodiment, the above step S2: Input the training set into the edge detection network to obtain the edge mask map of the photovoltaic module. Among them, the edge detection network is constructed based on the Bottleneck of MobileNetV3 integrated with pixel difference convolution and combines a top-down feature pyramid network, specifically including:

[0035] Replace the depth convolution in the Bottleneck of MobileNetV3 with pixel difference convolution; the input image passes through the four stages of the backbone network of the edge detection network and sequentially outputs four feature maps C1, C2, C3, and C4 with different scales, and then passes through the top-down feature pyramid network to sequentially obtain feature maps F4, F3, F2, and F1; finally, the feature map F1 passes through a convolutional layer to obtain the edge mask map of the photovoltaic module.

[0036] As Figure 2 shown, the present invention uses the basic component Bottleneck of the lightweight MobileNetV3 network to construct the backbone network of the edge detection network. At the same time, in order to better apply to the edge detection task, the Bottleneck structure is modified, and the depth convolution between two pointwise convolutions is replaced with PDC (pixel difference convolution) to enable the network to learn the features of pixel differences, making the network more sensitive to edge information. Before each stage of the backbone network, maxpool is used for downsampling, which is beneficial to suppressing noise information. Except for the first stage, three improved Bottlenecks are used in each stage. Three types of PDCs, namely CPDC, APDC, and RPDC, are used in the first three Bottlenecks to replace the original depth convolution to extract edge information in different directions, and the fourth Bottleneck still uses ordinary convolution. The input infrared image of the photovoltaic module passes through the four stages of the backbone network and sequentially outputs four feature maps C1, C2, C3, and C4 with different scales.

[0037] The neck network adopts a top-down feature fusion method. The output of each stage passes through a 1×1 convolution to change the number of channels to 8, and then the high-level features are added to the low-level features after upsampling in turn, and the feature maps F4, F3, F2, and F1 are output in turn. Finally, the fused feature map F1 passes through a 3×3 convolution to output the final result: the edge mask map of the photovoltaic module. This top-down fusion method can transfer rich semantic features of high levels to low-level features with more detailed position information, which is more conducive to edge detection.

[0038] In one embodiment, the above-described method for locating and defect detecting photovoltaic modules in an infrared image based on deep learning is characterized in that, step S3: input the edge mask image into a contour screening module, and use a quadrilateral modeling algorithm to obtain the candidate position boxes of the photovoltaic modules, specifically including:

[0039] Step S31: Use the contour finding algorithm to obtain all candidate contour point sets in the edge mask image of the photovoltaic module; then use the minimum bounding rectangle algorithm to obtain the set of candidate bounding rectangles R = {r i , i = 1, 2,..., N} of the candidate contour point sets of each photovoltaic module, where N is the number of candidate bounding rectangles; calculate the area A i and aspect ratio R i of each set of candidate bounding rectangles of the photovoltaic module; determine the value ranges of A i and R i , and screen out the bounding rectangles that meet the conditions;

[0040] First, use the contour finding algorithm to obtain all candidate contour point sets in the edge mask image of the photovoltaic module, and then use the minimum bounding rectangle algorithm to obtain the set of minimum bounding rectangles R = {r i , i = 1, 2,..., N} of each candidate contour point set. Then, calculate the area A i and aspect ratio R i of each candidate rectangle, and then define the maximum threshold and minimum threshold of the area and aspect ratio: Obtain the median area A between the minimum area threshold and the maximum area threshold med , and then retain the rectangles that meet the conditions A med -500 < A i < A med +1000 and the condition . The above thresholds of the area and aspect ratio are selected based on the prior knowledge of the fixed range of the UAV flight altitude and the known geometric shape of the photovoltaic module. In addition, for the photovoltaic modules that are not fully displayed in the image, it is meaningless to perform defect detection on them, so they are screened out by judging whether all the vertices of the bounding rectangle are located in the image. The above method can adaptively adjust the area screening threshold in each image according to the characteristics of the photovoltaic modules in each image.

[0041] Step S32: Use the minimum circumscribed quadrilateral algorithm to extract the minimum circumscribed quadrilateral of the contour point set corresponding to the bounding rectangle that meets the conditions as the candidate position box of the photovoltaic module.

[0042] For the bounding rectangles that meet the conditions screened according to step S31, use a more accurate quadrilateral modeling to extract the candidate position boxes. For exampleFigure 3 As shown in Figure 3 , for the original contour point set corresponding to the selected rectangle, find its clockwise convex polygon to obtain the convex polygon set C = {c i , i = 1, 2,..., M}; then for each side of each convex polygon, calculate its direction angle, and map it to between 0 and 360°. If the direction angle difference between adjacent sides is less than the threshold t eps , then delete the latter side of the two; for the remaining sides, select four sides in the clockwise direction each time to determine the circumscribed quadrilateral. If the direction angle difference between the fourth side and the first side is not greater than 180°, then the quadrilateral obtained according to these four sides must not be a circumscribed quadrilateral, and this combination is discarded; if the direction angle difference between adjacent sides is greater than 180°, then the quadrilateral obtained according to these four sides is not necessarily a convex quadrilateral and does not meet the requirements, so it is discarded; if the adjacent sides are also adjacent sides in the original convex polygon, then the common point of the adjacent sides is one of the vertices of the quadrilateral. If the two sides are not adjacent on the original convex polygon, then calculate the intersection point of the two sides as the vertex. Finally, the quadrilateral that meets the requirements for each combination can be obtained, and the quadrilateral with the smallest area is recorded, which is the minimum circumscribed quadrilateral of the candidate contour point set. The vertex coordinates of the finally obtained minimum circumscribed quadrilateral reflect the candidate position box of the photovoltaic module.

[0043] Since the algorithm complexity of calculating the circumscribed quadrilateral is higher than that of the circumscribed rectangle, the contour screening module in the embodiment of the present invention first screens the circumscribed rectangles that meet the conditions, and then finds the circumscribed quadrilateral for the contour point set corresponding to the circumscribed rectangle, so as to avoid finding the circumscribed quadrilateral for all candidate point sets, which can save computing resources and improve computing speed.

[0044] In one embodiment, the above step S4: Input the single photovoltaic module image selected according to the position box into the trained improved ResNet network for classification; output the position box of the defective photovoltaic module, which specifically includes:

[0045] For the photovoltaic module obtained according to the candidate position box, use affine transformation to convert it into a rectangle and standardize the size to a unified size of 128×64, and then build a classification network based on Resnet18. Considering that the input image size is not large, the original Resnet18 network is simplified according to this task. As shown in Table 1, it is simplified to a classification network with only 7 convolutional layers and 2 fully connected layers.

[0046] Table 1: Structure of the improved ResNet network

[0047] Type Step size Output Conv, 3×3 2 64×32×64 Maxpool 2 32×16×64 Basicblock, 3×3 1 32×16×64 Basicblock, 3×3 1 32×16×64 Conv, 1×1 1 32×16×128 Conv, 3×3 1 32×16×256 Adaptive Maxpool Global 256 Connected 128 Connected 2

[0048] Table 2: Comparison of experimental results between the improved ResNet and Resnet18 of the present invention

[0049]

[0050]

[0051] It can also be seen from the results in Table 2 that in this classification task, the improved ResNet proposed in the present invention and the marco F1 score of ResNet18 are very close, both exceeding 0.95, indicating that both can be competent for the defect classification task of photovoltaic modules; however, the improved ResNet has significantly higher computing speed due to its fewer parameters, and the PFS has been improved by nearly twice, indicating the rationality of this simplification in the present invention.

[0052] As Figure 4 shown, the classification results of photovoltaic modules are presented. The photovoltaic modules marked with white frames represent defective photovoltaic modules, and the photovoltaic modules marked with black frames represent normal photovoltaic modules.

[0053] The present invention discloses a method for locating and defect detecting photovoltaic modules in infrared images based on deep learning. After inputting the collected infrared images of photovoltaic modules, the diagnosed images can be directly output end-to-end, marking the defective photovoltaic modules. The present invention replaces the depth convolution in the BottleNeck structure of MobileNetV3 with pixel difference convolution. On the basis of retaining the lightweight characteristics of MobileNetV3, using pixel difference convolution is more conducive to detecting edge information. Since during the flight of the drone, the shooting angle is not always perpendicular to the surface of the photovoltaic module, most of the photovoltaic modules in the collected images are not the original rectangles and have been distorted into quadrilaterals. The present invention uses quadrilaterals to model the recognized contours of photovoltaic modules, which can more accurately extract the candidate position frames of photovoltaic modules.

[0054] Example Two

[0055] As Figure 5 shown, the embodiment of the present invention provides a system for locating and defect detecting photovoltaic modules in infrared images based on deep learning, including the following modules:

[0056] A dataset construction module 51, which is used to label the positions and categories of photovoltaic modules in the collected infrared images of photovoltaic modules; construct an infrared image dataset of photovoltaic modules, and divide it into a training set and a test set according to a preset ratio;

[0057] A module 52 for obtaining an edge mask image of a photovoltaic module, which is used to input the training set into an edge detection network to obtain an edge mask image of the photovoltaic module. Among them, the edge detection network is constructed based on the BottleNeck of MobileNetV3 integrated with pixel difference convolution and combines a top-down feature pyramid network;

[0058] The photovoltaic module candidate position box obtaining module 53 is configured to input an edge mask image into the contour screening module, and obtain a candidate position box of the photovoltaic module by using a quadrilateral modeling algorithm;

[0059] The photovoltaic module classification module 54 is configured to input the single photovoltaic module image selected according to the position box into the trained improved ResNet network for classification; and output the position box of the defective photovoltaic module.

[0060] The above embodiments are provided only for the purpose of describing the present invention, rather than limiting the scope of the present invention. The scope of the present invention is defined by the appended claims. All equivalent substitutions and modifications made without departing from the spirit and principles of the present invention shall be covered within the scope of the present invention.

Claims

1. A method for positioning and defect detection of photovoltaic modules in infrared images based on deep learning, characterized in that, Including: Step S1: Label the positions and categories of photovoltaic modules in the collected infrared images of photovoltaic modules; construct an infrared image dataset of photovoltaic modules and divide it into a training set and a test set according to a preset ratio. Step S2: Input the training set into an edge detection network to obtain an edge mask map of the photovoltaic module. Among them, the edge detection network is constructed based on the Bottleneck of MobileNetV3 integrated with pixel difference convolution and combines a top-down feature pyramid network. Step S3: Input the edge mask map into a contour screening module and use a quadrilateral modeling algorithm to obtain the candidate position box of the photovoltaic module, specifically including: Step S31: Use the contour finding algorithm to obtain all candidate contour point sets in the edge mask map of the photovoltaic module; then use the minimum bounding rectangle algorithm to obtain the candidate bounding rectangle sets of the candidate contour point sets of each photovoltaic module , is the number of candidate bounding rectangles; calculate the area of the candidate bounding rectangle set of each photovoltaic module and the aspect ratio ; determine and value ranges, and filter out the bounding rectangles that meet the conditions; Step S32: Use the circumscribed quadrilateral algorithm to extract the minimum circumscribed quadrilateral of the contour point set corresponding to the qualified circumscribed rectangle as the candidate position box of the photovoltaic module. Step S4: Input the single photovoltaic module image selected according to the position box into a trained improved ResNet network for classification; output the position box of the defective photovoltaic module.

2. The method for positioning and defect detection of photovoltaic modules in infrared images based on deep learning according to claim 1, characterized in that, In the said Step S2: Input the training set into an edge detection network to obtain an edge mask map of the photovoltaic module. Among them, the edge detection network is constructed based on the Bottleneck of MobileNetV3 integrated with pixel difference convolution and combines a top-down feature pyramid network, specifically including: Replace the depth convolution in the Bottleneck of MobileNetV3 with pixel difference convolution; the input image passes through the four stages of the backbone network of the edge detection network and outputs four feature maps of different scales in sequence. , , and , and then through the top-down feature pyramid network, feature maps , , and are obtained in sequence; finally, the feature map passes through a convolutional layer to obtain the edge mask map of the photovoltaic module.

3. A system for positioning and defect detection of photovoltaic modules in infrared images based on deep learning, characterized in that, Including the following modules: A dataset construction module for labeling the positions and categories of photovoltaic modules in the collected infrared images of photovoltaic modules; constructing an infrared image dataset of photovoltaic modules and dividing it into a training set and a test set according to a preset ratio. A module for obtaining the edge mask map of the photovoltaic module, which is used to input the training set into an edge detection network to obtain an edge mask map of the photovoltaic module. Among them, the edge detection network is constructed based on the Bottleneck of MobileNetV3 integrated with pixel difference convolution and combines a top-down feature pyramid network. A module for obtaining the candidate position box of the photovoltaic module, which is used to input the edge mask map into a contour screening module and use a quadrilateral modeling algorithm to obtain the candidate position box of the photovoltaic module, specifically including: Step S31: Use the contour finding algorithm to obtain all candidate contour point sets in the edge mask map of the photovoltaic module; then use the minimum bounding rectangle algorithm to obtain the candidate bounding rectangle sets of the candidate contour point sets of each photovoltaic module , is the number of candidate bounding rectangles; calculate the area of the candidate bounding rectangle sets of each photovoltaic module and the aspect ratio ; determine and value ranges, and filter out the bounding rectangles that meet the conditions; Step S32: Use the circumscribed quadrilateral algorithm to extract the minimum circumscribed quadrilateral of the contour point set corresponding to the qualified circumscribed rectangle as the candidate position box of the photovoltaic module. A photovoltaic module classification module for inputting the single photovoltaic module image selected according to the candidate position box into a trained improved ResNet network for classification; outputting the position box of the defective photovoltaic module.

Images