A photovoltaic fault automatic detection method based on multispectral fusion

Abstract

Embodiments of the present application propose a photovoltaic fault automatic detection method based on multispectral fusion. Hot spot and diode conduction fault are one of the most common faults in photovoltaic modules. Due to dust accumulation, tree shade and foreign matter obstruction, etc. can also cause uneven heat of photovoltaic panel. Directly using infrared image to identify photovoltaic fault has a large error. This brings great threat to the safety and economic operation of photovoltaic power station. This paper proposes a photovoltaic fault detection algorithm based on infrared-visible light fusion. The photovoltaic area of infrared and visible light images is segmented. Then considering the obvious corner features of the photovoltaic panel, the SIFT feature point extraction algorithm is used to register the infrared-visible light image. Then the infrared and visible light images are fused and the features are extracted by using the Transformer multi-head attention mechanism. Then the fused image after feature extraction and enhancement is sent into the deep learning network for defect recognition.

Description

Technical Field

[0001] This invention relates to defect detection based on computer vision, specifically to an automatic photovoltaic fault detection method based on multispectral fusion. Background Technology

[0002] With the continuous increase in photovoltaic installed capacity, the photovoltaic operation and maintenance market continues to expand. However, due to the increase in the operating years of photovoltaic power plants, the failure of photovoltaic modules is gradually increasing, and the operation and maintenance pressure of photovoltaic power plants continues to expand. Traditional operation and maintenance technologies can no longer meet the current operation and maintenance requirements of photovoltaic power plants, which seriously endangers the safety and stable operation of photovoltaic power plants. There is an urgent need for an automatic inspection technology with high accuracy.

[0003] Traditional photovoltaic (PV) power plant operation and maintenance methods rely primarily on the electrical characteristics of the equipment. However, these electrical parameters are significantly affected by environmental factors such as weather, making accurate fault detection difficult. While handheld infrared detectors can pinpoint faults down to the individual modules, their efficiency is low and safety hazards are significant due to limitations imposed by terrain and module installation locations. Centralized PV power plants are mostly located in deserts, hills, or over water, covering large areas and heavily influenced by topography. Distributed PV power plants are often built on building rooftops, resulting in a more dispersed distribution and narrow inspection access routes. Therefore, traditional manual inspections of PV power plants are inefficient, costly, and prone to safety issues, necessitating a safe and efficient inspection technology. Drone-based PV inspection technology has emerged to address this need. Compared to traditional PV inspection technologies, drone-based PV inspection offers significant advantages in safety, efficiency, and flexibility.

[0004] By mounting dual-light cameras on drones to capture image data, image processing methods can be used for offline and even real-time analysis without affecting the operation of the photovoltaic power station. By fusing the images captured by the dual-light cameras and utilizing the image fusion features, fault conditions in the photovoltaic area can be detected. The multi-modal synergy allows for more accurate identification of defect types, providing strong assurance for detection results and data support for subsequent defect analysis. Summary of the Invention

[0005] The purpose of this invention is to provide an automatic photovoltaic fault detection method based on multispectral fusion. This method uses a series of digital image processing methods to register infrared-visible light images, fuse and enhance multimodal features, thereby improving the accuracy of deep learning models.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows:

[0007] An automatic photovoltaic fault detection method based on multispectral fusion includes the following steps:

[0008] Step 1: Segment the photovoltaic region of the image to obtain a photovoltaic region mask image;

[0009] Step 2: Based on the SIFT image feature point extraction method, image feature points are obtained, and corner features in the image are extracted to achieve accurate registration of infrared and visible light images;

[0010] Step 3: Fusion of infrared and visible light image features based on convolution and Transformer models;

[0011] Step 4: Improve the YOLOv4 photovoltaic fault detection model based on Transformer to identify photovoltaic panel faults.

[0012] Further, in step one, the image photovoltaic region segmentation and extraction method based on the OTSU threshold segmentation algorithm divides the image into foreground and background by obtaining a suitable threshold through calculating the maximum variance. For each grayscale value i, a σ is calculated. i Find the largest σ i The gray level i at this point is used as the separation threshold, as follows:

[0013] σ i =P min *(Avg total -Avg min ) 2 +P max *(Avg max -Avg total ) 2

[0014] Among them, P min This indicates the probability of a pixel with a grayscale value smaller than this value appearing; Avg total This represents the overall grayscale mean of the image.

[0015] Furthermore, after segmenting the photovoltaic images of the power plant using OTSU thresholding, the obtained images are first processed by erosion to remove environmental interference and pixel adhesion areas, with a kernel size of 3*3, repeated 5 times. Then, dilation is used to eliminate holes and restore the photovoltaic region, with a kernel size of 3*3, repeated 6 times. Finally, a contour detection algorithm is used to filter contour sizes and remove interference. This yields a mask image of the photovoltaic region in the image.

[0016] Furthermore, considering the distinct rectangular characteristics of the photovoltaic region, a SIFT-based image feature point extraction method is proposed to extract corner features from the image, thereby achieving registration between the photovoltaic and visible light images. This method first extracts feature points from the infrared and visible light images of the photovoltaic array;

[0017] First, feature points are extracted from the infrared and visible light images of the photovoltaic array to construct a Gaussian scale space. Difference-of-Gaussian (DIGaussian) method is then used to detect feature points in the images. Considering the rectangular characteristics of photovoltaic images, their obvious extremum properties, and the fact that photovoltaic photos are all overhead views with fixed feature directions, the mathematical description of using DIGaussian to detect image feature points is as follows:

[0018] D(x,y,σ)=[G(x,y,ησ)-G(x,y,σ)]*I(x,y)=L(x,y,ησ)-L(x,y,σ)

[0019] Where η is the scaling factor between two adjacent Gaussian scale spaces; G(x,y,σ) is a scale-variable Gaussian function, and its formula is:

[0020]

[0021] Where, σ 2 σ is the scale space factor, which is the standard deviation of the Gaussian normal distribution. 2 The size of σ is positively correlated with the smoothness of the image. The larger the value of σ, the smoother the image, that is, the more blurred the image. L(x,y,σ) represents the Gaussian scale of the image.

[0022] Then, the optimal image registration result is obtained based on the minimum error function. Registration is performed on the photovoltaic array infrared image and the natural light image, where the photovoltaic array infrared image is represented by I1 and the photovoltaic array natural light image is represented by I2. The feature point sets of I1 and I2 extracted using the Harris feature extraction method are represented as follows:

[0023] posI1={(x1, y1), (x2, y2), ..., (x i y i ), ..., (x h y h )},1≤i≤h

[0024] posI2={(x′1, y′1), (x′2, y′2),…, (x′ i y′ i ), …, (x′ k y′ k )},1≤i≤k

[0025] Where h and k are the number of feature points extracted from the photovoltaic array infrared image I1 and the natural light image I2, respectively. The photovoltaic array infrared image I1 and the natural light image I2 are placed sequentially on the same coordinate axis to form a new image of m×2n. The slope between any two feature points in I1 and I2 can be expressed as l=(y′+ny) / (x′-x). Based on prior knowledge, it is known that the slopes of the lines connecting all correctly matched point pairs should be approximately the same. The method for registering the infrared image and the natural light image using the slope constraint of feature point pairing between the two images is as follows:

[0026] 1) For any feature point in posI1, calculate its Euclidean distance to all feature points in posI2, and select the point with the smallest Euclidean distance as the registration point pair for coarse matching.

[0027] 2) Sort all coarse-matched registration point pairs by Euclidean distance from smallest to largest, and delete coarse-matched registration point pairs in posI1 where multiple feature points are registered to the same feature point in posI2. The remaining feature points are then represented by posI3 and posI4.

[0028] 3) Select the first 20 registration point pairs from posI3 and posI4, and calculate the slope between the two points, i.e.

[0029]

[0030] 4) For the new slope set l new Sort by frequency of occurrence in descending order and find the slope value l that appears most frequently. i .

[0031] 5) Traverse and calculate all pairs of values ​​in posI3 and posI4 that satisfy [l i -0.5, l i All feature point pairs within [+0.5] form a new set posk.

[0032] 6) Finally, count the number of matching point pairs obtained from different matrices, select the case with the most matching point pairs, and obtain the optimal matching parameters.

[0033] Furthermore, considering that the input image of the model contains both infrared and visible light modalities, it is necessary to fuse the different modalities to effectively extract intra-modal and inter-modal features, described as follows:

[0034]

[0035] I R and I T These represent the input RGB image and the input infrared image, respectively. F R Represents the RGB feature map, FT Representing infrared feature maps using network feature extraction functions and Feature mapping is performed on input images of different modalities, F 12sed This is the feature map for fusion. Modifications are made to the basic YOLOv5 network framework. The input consists of images from different modalities. The traditional single-input-stream YOLO backbone is replaced with a dual-stream network, a multimodal fusion backbone, and a Cross-Modality Fusion Transformer (CFT) is embedded within it. The feature maps from the two convolutional images are flattened and concatenated, denoted as I, with positional encoding added to form the CFT input. The Transformer mechanism enables interaction between features from different modalities. By concatenating multimodal features into a sequence, the Transformer can automatically perform intra-modal and inter-modal information fusion simultaneously, robustly capturing the potential interaction information between infrared and visible light. I is copied three times and multiplied by the corresponding linear transformation weight matrix W to obtain Q, K, and V, described as follows:

[0036] Q = IW Q

[0037] K = IW K

[0038] V = IW V

[0039]

[0040] The formula calculates the inner product of each row vector of matrices Q and K. To prevent the inner product from becoming too large, it is divided by .

[0041] Furthermore, the photovoltaic fault detection model structure based on Transformer-improved YOLOv4 includes: a CFB (Cross-Modal Fusion Skeleton) module, composed of convolution and CFT (Cross-Modal Fusion Transformer); a Neck module, composed of SPP and PAN; and a Prediction module. The input first passes through the CFB module, then through the SPP and PAN, and finally through the Prediction module for detection.

[0042] The beneficial effects of this invention are as follows: The image photovoltaic region segmentation and extraction method based on the OTSU threshold segmentation algorithm can accurately segment the photovoltaic region in the image, achieving accurate extraction of the photovoltaic panel image; the image feature point extraction method based on SIFT can extract a large number of matching features from infrared and visible light images, achieving accurate multispectral image registration; by fusing infrared and visible light image features based on convolution and Transformer models, the problem of low fault detection accuracy of single infrared images is effectively solved. By combining visible light images, image information is enriched, and the convolution and Transformer models can better achieve feature fusion between multimodal images, effectively improving the accuracy of fault detection and accurately identifying the fault area and fault type of the photovoltaic panel. Attached Figure Description

[0043] Figure 1 This is a diagram showing the segmentation of the photovoltaic panel mask area in an embodiment of the present invention;

[0044] Where (a1~a3) and (b1~b3) are the photovoltaic panel images after threshold segmentation, erosion dilation, and contour detection and filling, respectively;

[0045] Figure 2 These are images obtained after feature extraction and image registration in the embodiments of the present invention;

[0046] Among them, (a~c) are: feature point annotation map after infrared light feature extraction, feature point annotation map after visible light feature extraction, and registered dual-light weighted image, respectively.

[0047] Figure 3 This is a diagram of a deep learning-based model framework in an embodiment of the present invention.

[0048] Where a is the structure diagram of the infrared and visible light image fusion algorithm based on Transformer, and b is the structure diagram of the improved YOLOv4 algorithm;

[0049] Figure 4 This is a flowchart of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0051] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0052] like Figure 1 As shown, an embodiment of the present invention provides a method comprising four main steps: First, photovoltaic region segmentation; Second, infrared and visible light image registration; Third, infrared and visible light image fusion; Fourth, automatic fault image recognition. Figure 2 As can be seen, infrared and visible light images have significant differences, and both contain rich image features.

[0053] Step one first utilizes the OTSU thresholding algorithm to obtain a suitable threshold by calculating the maximum variance, dividing the image into foreground and background. Then, the obtained image is first processed by erosion to remove environmental interference and pixel-clustered areas. Next, dilation is used to eliminate holes and reconstruct the photovoltaic region. Then, a contour detection algorithm is used to filter contour sizes and remove interference; finally, boundary tracking is performed, and pixel filling is applied to the detected inner contours to obtain the photovoltaic region mask image. Figure 1 As can be seen from (a1) to (a3) ​​and (b1) to (b3), after layer-by-layer segmentation and extraction, the photovoltaic region features become more obvious. However, many interfering factors still exist in the entire image, and there is a certain deviation in the extraction of the photovoltaic region. This indicates that step one has a certain effect on the photovoltaic region image segmentation, but further enhancement of image segmentation is still needed.

[0054] Step two utilizes a SIFT-based image feature point extraction method to extract corner features from the image, thereby achieving registration between the photovoltaic and visible light images. This method first extracts feature points from the infrared and visible light images of the photovoltaic array, and then obtains the optimal image registration result based on the minimum error function. Figure 2As can be seen from (a) to (c), after step two, the infrared and visible light images can be accurately registered, which facilitates the further fusion and feature extraction of the two-light images in step three.

[0055] Step 3 takes into account the multimodal properties of infrared and visible light, fuses infrared and visible light and extracts features, and uses the multi-head attention mechanism of Transformer to enable the deep learning model to acquire more feature information;

[0056] The model used in step four is as follows: Figure 3 As shown, it includes a CFB (Cross-Modal Fusion Skeleton) module, composed of convolutions and CFT (Cross-Modal Fusion Transformer); a Neck module, composed of SPP and PAN; and a Prediction module. The specific structure of CFB is as follows: Figure 3 As shown in (a), the connection method between modules is as follows: Figure 3 As shown in (b).

[0057] Infrared and visible light images of photovoltaic power plants captured by drones contain information on the operating status of photovoltaic modules. Using deep learning tools to effectively identify defects in photovoltaic images has significant practical engineering implications. This invention designs corresponding defect feature extraction strategies based on the texture feature information of photovoltaic images of different modalities, which greatly increases the accuracy of photovoltaic panel fault identification in drone photovoltaic inspections, while also improving the robustness and versatility of the model.

Claims

1. An automatic photovoltaic fault detection method based on multispectral fusion, characterized in that, Specifically, the following steps are included: 1) Segment the photovoltaic region of the image to obtain a photovoltaic region mask image; 2) The SIFT-based image feature point extraction method obtains image feature points, extracts corner features in the image, and then achieves accurate registration of infrared and visible light images; 3) Fusion of infrared and visible light image features based on convolution and Transformer models; 4) Improve the YOLOv4 photovoltaic fault detection model based on Transformer to identify photovoltaic panel faults; Step 4) is as follows: The photovoltaic fault detection model based on the Transformer improves YOLOv4. The structure includes a cross-modal fusion skeleton CFB module, which is composed of convolution and CFT cross-modal fusion Transformer; a Neck module, which is composed of SPP and PAN; and a Prediction module. The input first passes through the CFB module, then through the SPP and PAN, and then through the Prediction module for detection.

2. The automatic photovoltaic fault detection method based on multispectral fusion as described in claim 1, characterized in that, Step 1) is as follows: Considering the significant color difference in photovoltaic images and the obvious peak attributes in their grayscale histograms, an image photovoltaic region segmentation and extraction method based on the OTSU threshold segmentation algorithm is used. By calculating the maximum variance, a suitable threshold is obtained to divide the image into foreground and background. For each grayscale value Calculate one Find the largest The grayscale at this time The formula for the separation threshold is as follows: ； in, This indicates the probability of a pixel with a grayscale value smaller than this value appearing. This indicates the probability of a pixel with a grayscale value greater than this value appearing. This represents the overall grayscale mean of the image; This represents the average gray value of the pixels in the image that are smaller than this gray value; This represents the average gray value of pixels in the image that are greater than this gray value.

3. The automatic photovoltaic fault detection method based on multispectral fusion as described in claim 1, characterized in that, Step 2) is as follows: First, feature points are extracted from the infrared and visible light images of the photovoltaic array: a Gaussian scale space is constructed, and difference-of-Gaussian (DOG) is used to detect feature points in the images. Considering the rectangular characteristics of photovoltaic images, which have obvious extremum properties, and the fact that photovoltaic photos are all overhead shots with fixed feature directions, the mathematical description of using difference-of-Gaussian to detect image feature points is as follows: ； in, The scaling factor between two adjacent Gaussian scale spaces; The scale-variable Gaussian function has the following formula: ； in, The scale space factor is the standard deviation of the Gaussian normal distribution. The size of is positively correlated with the smoothness of the image. The larger the 𝜎 value, the smoother the image, that is, the more blurred the image. 𝐿(𝑥, 𝑦, 𝜎) represents the Gaussian scale of the image; Then, the optimal image registration result is obtained based on the minimum error function; registration is performed on the photovoltaic array infrared image and the natural light image, where the photovoltaic array infrared image is used... This indicates that the photovoltaic array uses natural light images. Indicates that features were extracted using the Harris feature extraction method. and The feature point sets are represented as follows: ； ； in , Infrared images of photovoltaic arrays and natural light images The number of extracted feature points; the infrared image of the photovoltaic array and natural light images Place them sequentially on the same coordinate axis to form a New images, and The slope between any two feature points is expressed as: ; The method for registering infrared images and natural light images using slope constraints based on paired feature points from two images is as follows: right For any feature point in the array, calculate its relationship with... The Euclidean distance between all feature points is used to select the point with the smallest Euclidean distance as the registration point pair for coarse matching. Sort all coarsely matched registration point pairs by Euclidean distance from smallest to largest, and then delete them. Registration of multiple feature points Coarsely register point pairs for the same feature point, then use the remaining feature points... and To indicate; choose and The first 20 registration point pairs, considering the new image size is Calculate the slope between the two points. ； For the new set of slopes Sort by frequency of occurrence in descending order and find the slope value that appears most frequently. ; Traversal calculation and All of them satisfy [ -0.5, All feature point pairs within +0.5] form a new set 𝑝𝑜𝑠𝑘; Finally, the number of matching point pairs obtained from different matrices is counted, and the case with the most matching point pairs is selected to obtain the optimal matching parameters.

4. The automatic photovoltaic fault detection method based on multispectral fusion as described in claim 1, characterized in that, Step 3) is as follows: Considering that the model input image contains both infrared and visible light modalities, it is necessary to fuse the different modalities to effectively extract intra-modal and inter-modal features, described as follows: ； and These represent the input RGB image and the input infrared image, respectively. Represents the RGB feature map. Representing infrared feature maps using network feature extraction functions and Feature mapping is performed on input images of different modalities. This is a fused feature map; modified from the basic YOLOv5 network framework, the input is images of different modalities. The traditional single-input-stream YOLO backbone is replaced with a two-stream network, a multimodal fusion backbone is implemented, and CFT is embedded within it. The feature maps of the two modalities after convolution are flattened and concatenated, denoted as . The input to the CFT is composed of positional encoding; the Transformer mechanism enables features from different modalities to interact, concatenating multimodal features into a sequence, and then the Transformer automatically performs intramodal and intermodal information fusion simultaneously, robustly capturing potential interaction information between infrared and visible light; Make three copies and multiply them respectively by the corresponding linear transformation weight matrix W to obtain Q, K, and V, described as follows: ； ； The formula calculates the inner product of each row vector of matrices Q and K. To prevent the inner product from becoming too large, it is divided by . .

Images