Photovoltaic panel damage detection method, medium and electronic device

Abstract

The invention provides a photovoltaic panel damage detection method, a medium and an electronic device, belongs to the technical field of photovoltaic panel damage detection, and aims to acquire a photovoltaic panel surface image and a fluorescence image through an unmanned aerial vehicle, perform image registration and semantic segmentation by using a deep learning algorithm, accurately separate out a photovoltaic panel area and improve the detection accuracy. And then inputting into a pre-trained defect detection model, and detecting and marking various defects and damages on the surface of the photovoltaic panel. The method mainly comprises the steps that 1) an unmanned aerial vehicle collects a surface image and a fluorescence image; 2) image preprocessing and registration; 3) performing semantic segmentation to accurately extract a photovoltaic panel area; (4) detecting and positioning defects; and 5) marking defect information on the image. According to the system, an advanced computer vision technology is utilized, and an intelligent and efficient solution is provided for operation and maintenance of the photovoltaic power station. The technical problem that the existing image analysis method is difficult to accurately capture and identify because some tiny defects often exist on the surface of the photovoltaic panel and the characteristics of the defects are not obvious is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of photovoltaic panel damage detection, and in particular, relates to a photovoltaic panel damage detection method, medium and electronic device. Background Art

[0002] Photovoltaic power generation, as a clean and environmentally friendly renewable energy, plays an increasingly important role in power supply and reducing carbon emissions. With the continuous advancement of photovoltaic technology, the scale of photovoltaic power stations has become larger and larger, and has been widely used in industrial, commercial, residential and other fields. But at the same time, the operation and maintenance management of photovoltaic systems also faces many challenges. Among them, the detection of surface defects of photovoltaic panels is an important and thorny issue. During the long-term use of photovoltaic panels, various types of defects such as scratches, fractures, shedding, hot spots, etc. will appear due to various reasons such as material aging, environmental erosion, and human damage. These defects will not only seriously affect the power generation efficiency of photovoltaic panels, but may also cause circuit failures and safety hazards. Therefore, timely and accurate detection of various defects on the surface of photovoltaic panels is crucial to ensure the normal operation of photovoltaic power stations and improve power generation performance. At present, photovoltaic panel defect detection mainly relies on manual inspection. Staff need to go to photovoltaic power stations regularly to personally inspect the surface of each photovoltaic panel, and then manually record and repair them after finding abnormalities. This method is not only inefficient, but also easy to miss subtle defects, making it difficult to fully grasp the actual operation of photovoltaic power stations. In recent years, with the continuous advancement of computer vision technology, automated photovoltaic panel defect detection based on image analysis has begun to attract widespread attention. This method uses drones and other equipment to collect high-definition images of the photovoltaic panel surface, and then uses advanced deep learning algorithms to analyze and process the images, ultimately achieving accurate detection and positioning of defects.

[0003] Compared with traditional methods, image analysis technology has the advantages of low cost, high efficiency and strong applicability, and is considered to be the main development direction of photovoltaic panel automated inspection in the future. However, existing image analysis-based technologies are difficult to accurately extract subtle defect information: photovoltaic panels often have some tiny defects such as fine cracks and spot stains on the surface. The characteristics of such defects are not obvious, and existing image analysis methods are difficult to accurately capture and identify. Summary of the invention

[0004] In view of this, the present invention provides a photovoltaic panel damage detection method, medium and electronic device, which can solve the technical problem that there are often some tiny defects such as fine cracks and dot stains on the surface of photovoltaic panels, the characteristics of such defects are not obvious, and existing image analysis methods are difficult to accurately capture and identify them.

[0005] The present invention is achieved in that:

[0006] A first aspect of the present invention provides a photovoltaic panel damage detection method, which comprises the following steps:

[0007] S10, acquiring a surface image of a photovoltaic panel through a drone, and performing preprocessing to obtain a preprocessed photovoltaic panel image;

[0008] S20, obtaining, by a drone, a surface fluorescent image of a photovoltaic panel evenly sprayed with fluorescent powder, and preprocessing the fluorescent image to obtain a preprocessed fluorescent image;

[0009] S30, registering the pre-processed photovoltaic panel image and the pre-processed fluorescence image using a deep learning-based image registration algorithm, and merging the pre-processed photovoltaic panel image and the pre-processed fluorescence image into one image, which is recorded as a merged image;

[0010] S40, using a semantic segmentation model based on a convolutional neural network to segment the merged image and accurately separate the photovoltaic panel area;

[0011] S50, inputting the separated photovoltaic panel area image into a pre-trained photovoltaic panel defect and damage detection model to detect various defects and damages on the photovoltaic panel surface;

[0012] S60, classifying and locating the detected defects and damages, and marking the defect types and position coordinates on the photovoltaic panel image, and finally outputting the marked photovoltaic panel image.

[0013] Wherein, the semantic segmentation model adopts the U-Net network structure.

[0014] The training steps of the photovoltaic panel defect damage detection model specifically include:

[0015] Firstly, a large number of sample images of photovoltaic panel surface defects are collected, covering various common defect types, and the S10-S40 method is used to obtain the photovoltaic panel area image, and the defects and damages are annotated in the photovoltaic panel area image to obtain the training data set;

[0016] Secondly, select a convolutional neural network model based on ResNet or VGG structure as the basic architecture;

[0017] Again, the training data set is used to perform end-to-end supervised training on the selected convolutional neural network model, and finally a photovoltaic panel defect damage detection model is obtained.

[0018] Furthermore, the convolutional neural network model includes a micro-feature extraction branch and a macro-feature extraction branch, wherein the micro-feature extraction branch is used to extract subtle fluorescent dots and fluorescent lines features in the input image, and the macro-feature extraction branch is used to extract overall and obvious features in the input image.

[0019] Furthermore, an attention mechanism is introduced into the micro-feature extraction branch to improve the ability to capture subtle fluorescent dots and fluorescent line features in the image.

[0020] Specifically, the convolutional neural network model structure is as follows:

[0021] The micro-feature extraction branch contains the following key components:

[0022] Multi-scale convolutional layer: Use convolution kernels of different scales (such as 3x3, 5x5, 7x7) to process the input image in parallel to capture subtle features of different sizes. These feature maps are concatenated together.

[0023] Spatial Pyramid Pooling (SPP) layer: Divide the feature map into regions of different scales (such as 1x1, 2x2, 4x4), and perform maximum pooling in each region. This can better extract local features at different scales.

[0024] Attention mechanism: The attention mechanism is introduced to enable the model to adaptively focus on the most important subtle regional features in the input image. This includes two branches: channel attention and spatial attention.

[0025] Through the above design, the micro-feature extraction branch can efficiently capture the tiny fluorescent dots and lines in the input image.

[0026] The macro feature extraction branch contains the following key components:

[0027] Deep convolutional network: Using classic deep convolutional neural networks such as ResNet or VGG as the backbone network, it can extract powerful semantic features.

[0028] Dilated convolution: Using dilated convolution in the last few layers of the backbone network can expand the receptive field without reducing the resolution and capture broader contextual information.

[0029] Feature pyramid: By using the feature pyramid structure to fuse feature maps of different depth layers, both fine-grained and coarse-grained features can be obtained.

[0030] Through the above design, the macro feature extraction branch can extract holistic and semantically rich features from the input image.

[0031] Feature fusion: The micro features and macro features extracted by the two branches will be spliced ​​together, and feature fusion will be performed through several convolutional layers to finally output a comprehensive feature representation.

[0032] Output layer: The final output layer is also connected to a classification layer, which is used to identify and classify various defects on the surface of photovoltaic panels.

[0033] This design can make full use of the microscopic detail features and macroscopic semantic features in the input image, and can better identify various subtle defects and damages on the surface of photovoltaic panels. During the training process, transfer learning can be used to initialize the model weights pre-trained on other related tasks, thereby improving the generalization performance of the model.

[0034] Furthermore, the convolutional neural network model uses ResNet-50 as the backbone network.

[0035] Wherein, the phosphor is barium sulfate long afterglow phosphor.

[0036] The specific method for uniformly spraying phosphor on photovoltaic panels is:

[0037] Mix the phosphor with an appropriate amount of solution (such as water or alcohol) to form a phosphor solution with a moderate concentration. Then use a special spraying device carried by the drone to evenly spray the phosphor solution on the surface of the photovoltaic panel.

[0038] The specific steps are as follows:

[0039] a) Prepare enough phosphor solution according to the area of ​​photovoltaic panels.

[0040] b) Load the solution into the spray tank of the drone

[0041] c) The drone is programmed to fly along the planned path, and the spray head sprays the solution evenly at a constant pressure.

[0042] d) During the spraying process, keep the flight height and spray angle constant

[0043] e) Spray the phosphor back and forth multiple times to evenly cover the entire surface of the photovoltaic panel

[0044] This method can ensure that the phosphor is attached to the surface of the photovoltaic panel in a relatively uniform distribution, providing a good foundation for subsequent fluorescence imaging.

[0045] 2. The surface fluorescence image of the photovoltaic panel is obtained by:

[0046] After spraying the phosphor, use a drone equipped with professional ultraviolet lighting and a high-resolution camera to capture fluorescent images of the photovoltaic panel surface at a certain height.

[0047] The specific steps are:

[0048] a) Install a suitable UV LED light in front of the drone camera, aiming at the shooting area

[0049] b) Use a high-definition camera that supports long exposure and turn on manual exposure mode

[0050] c) Plan the flight path above the drone and control the flight altitude, speed, overlap rate, etc.

[0051] d) Turn on the UV lamp to irradiate the photovoltaic panel and extend the exposure time appropriately

[0052] e) Automatically take a series of fluorescent photos according to the path

[0053] f) Use image stitching software to stitch a single photo into a fluorescence distribution map of the entire photovoltaic panel

[0054] This spotlight imaging method can make tiny defects and cracks form obvious shadows or missing on the fluorescence distribution map, so that they can be detected efficiently. At the same time, using drone aerial photography, the image acquisition of large-area photovoltaic arrays can be completed quickly and automatically.

[0055] The phosphor sprayed evenly on the surface of the photovoltaic panel needs to meet the following requirements:

[0056] 1. The emission wavelength is suitable for photovoltaic panel materials: Generally speaking, it is most appropriate to select phosphors with an emission wavelength in the range of 530-630nm. Light in this wavelength range is effectively absorbed by common silicon and thin-film photovoltaic cells, and will not significantly affect the photoelectric conversion efficiency of the photovoltaic cell itself.

[0057] 2. High luminescence intensity: The stronger the luminescence intensity of the phosphor, the easier it is to detect the effects of tiny defects during fluorescence imaging. Some inorganic phosphors, such as barium sulfate, have high fluorescence emission intensity.

[0058] 3. Uniform particle size: Phosphors with smaller and more uniform particle size are more likely to form a uniform thin layer on the surface of the photovoltaic panel, avoiding the impact of uneven powder particle state on imaging quality.

[0059] 4. Will not corrode photovoltaic panels.

[0060] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned photovoltaic panel damage detection method.

[0061] A third aspect of the present invention provides an electronic device, which comprises a processor and a memory, wherein the memory is used to store the step program of the above-mentioned method, and the processor reads the memory and executes the steps.

[0062] Compared with the prior art, the photovoltaic panel damage detection method, medium and electronic device provided by the present invention have the following beneficial effects: the present invention uses the photovoltaic panel surface image collected by the drone, combined with the convolutional neural network model combining microscopic feature extraction and macroscopic feature extraction, to comprehensively and accurately detect and locate various defects on the photovoltaic panel. Compared with the prior art, this method has the following significant advantages:

[0063] 1. Highly sensitive micro-feature extraction capability: The model of the present invention has designed a special micro-feature extraction branch, using multi-scale convolution, spatial pyramid pooling and other technologies, which can effectively capture the subtle defect features on the surface of photovoltaic panels, such as spot stains, small cracks, etc. Compared with traditional image analysis methods, the model's perception of such subtle defects is greatly improved.

[0064] 2. Excellent adaptability to complex backgrounds: The model of the present invention introduces dilated convolution and feature pyramid structure in the macro feature extraction branch, which can effectively extract rich semantic information and context features in the image. This greatly enhances the robustness of the model to the complex photovoltaic panel use environment, and can maintain stable detection performance even in the presence of interference factors such as shadows and reflections.

[0065] 3. Accurate defect location capability: In addition to defect classification and identification, the model of the present invention can also output the precise bounding box coordinates of each defect area. This not only tells the type of defect, but also accurately locates its position on the photovoltaic panel, providing an important basis for subsequent maintenance operations.

[0066] Therefore, the present invention solves the technical problem that there are often some tiny defects such as fine cracks and dot-shaped stains on the surface of photovoltaic panels, the characteristics of such defects are not obvious, and the existing image analysis methods are difficult to accurately capture and identify them. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0068] Figure 1 A flow chart of the method provided by the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0070] like Figure 1 FIG. 1 is a flow chart of a photovoltaic panel damage detection method provided by the present invention. The method comprises the following steps:

[0071] S10, acquiring a surface image of a photovoltaic panel through a drone, and performing preprocessing to obtain a preprocessed photovoltaic panel image;

[0072] S20, obtaining a surface fluorescence image of a photovoltaic panel uniformly sprayed with fluorescent powder by using a drone, and preprocessing the fluorescence image to obtain a preprocessed fluorescence image;

[0073] S30, registering the pre-processed photovoltaic panel image and the pre-processed fluorescence image using a deep learning-based image registration algorithm, and merging the pre-processed photovoltaic panel image and the pre-processed fluorescence image into one image, which is recorded as a merged image;

[0074] S40, using a semantic segmentation model based on a convolutional neural network to segment the merged image and accurately separate the photovoltaic panel area;

[0075] S50, inputting the separated photovoltaic panel area image into a pre-trained photovoltaic panel defect and damage detection model to detect various defects and damages on the photovoltaic panel surface;

[0076] S60, classify and locate the detected defects and damages, and mark the defect types and location coordinates on the photovoltaic panel image, and finally output the marked photovoltaic panel image.

[0077] The specific implementation methods of the above steps are described in detail below:

[0078] Step S10: Obtaining the photovoltaic panel surface image through the drone and preprocessing it

[0079] First, the drone will fly to a suitable height and collect aerial images of the photovoltaic panel surface. In order to ensure image quality, the following factors need to be considered during collection:

[0080] a) Choose a suitable camera sensor, such as a high-resolution CMOS imaging chip, which can capture subtle texture information

[0081] b) Adjust the camera’s exposure time and aperture parameters to make the image clear, bright and detailed.

[0082] c) Control the flight altitude and route of the drone to ensure that a complete image of each photovoltaic panel area can be captured

[0083] d) Use overlapping shooting to facilitate high-quality image stitching using image stitching software later

[0084] After obtaining the original photovoltaic panel surface image, a series of preprocessing is required, including:

[0085] a) Image denoising, eliminating impurities and noise that may be generated during the shooting process

[0086] b) Image homogenization, adjusting parameters such as brightness, contrast, saturation, etc. to make the overall color of the image harmonious

[0087] c) Image correction, correcting geometric distortion caused by camera tilt and light angle

[0088] d) Image cropping, removing irrelevant areas at the edge of the image and retaining only the core part of the photovoltaic panel

[0089] Through these preprocessing steps, a high-quality and clear image of the photovoltaic panel surface can be obtained, laying the foundation for subsequent image analysis and defect detection.

[0090] Step S20: Acquire the fluorescent image of the photovoltaic panel surface

[0091] Based on the photovoltaic panel image obtained in step S10, it is necessary to further collect the fluorescent image of the photovoltaic panel surface after the fluorescent powder is sprayed. The specific steps are as follows:

[0092] a) Mix the phosphor with an appropriate amount of solvent (such as water or alcohol) to prepare a spray liquid with a moderate concentration.

[0093] b) The drone carries a spraying device and evenly sprays the surface of the photovoltaic panel according to the pre-planned route and altitude

[0094] c) During the spraying process, keep the nozzle outlet pressure and spraying angle constant to ensure uniform deposition of phosphor

[0095] d) Spray back and forth several times to ensure that the entire photovoltaic panel surface is covered.

[0096] e) After the phosphor is completely precipitated, the drone takes aerial photos again along the predetermined path

[0097] f) Install a UV LED light in front of the camera and turn on the long exposure mode to capture fluorescent images

[0098] g) Use image stitching software to stitch multiple single photos into a complete photovoltaic panel fluorescence distribution map

[0099] The fluorescent images obtained in this way can clearly show the subtle defects and damaged areas on the surface of the photovoltaic panel, providing important basis for subsequent analysis and detection.

[0100] Step S30: Image registration and fusion based on deep learning

[0101] After obtaining the conventional image and fluorescent image of the photovoltaic panel surface, the two images need to be accurately registered and fused. Here, an image registration algorithm based on deep learning is used, which includes:

[0102] a) Choose a suitable deep learning model, such as an image registration network based on a convolutional neural network

[0103] b) Collect a large number of reference images of the photovoltaic panel surface and their corresponding fluorescence images to construct a training data set

[0104] c) Use the training dataset to perform end-to-end supervised training on the selected deep learning model

[0105] d) Optimize the model parameters during training so that it can accurately register conventional images with fluorescent images

[0106] e) After the training is completed, the pre-processed photovoltaic panel surface image and fluorescence image are input into the model

[0107] f) The model automatically completes image registration and outputs a merged image that combines surface information and fluorescence information

[0108] This deep learning-based image registration method can overcome the limitations of traditional feature point matching and accurately complete the alignment and fusion of two images. The merged image contains rich information about the surface of the photovoltaic panel, providing a good foundation for subsequent defect detection.

[0109] Step S40: Photovoltaic panel region extraction based on semantic segmentation

[0110] After obtaining the merged image, it is necessary to use semantic segmentation technology to accurately extract the area of ​​the photovoltaic panel itself and separate it from irrelevant areas such as the background. The specific steps are as follows:

[0111] a) Choose a suitable semantic segmentation model, such as the U-Net network structure based on convolutional neural network

[0112] b) Collect a large number of photovoltaic panel surface images and manually mark the boundaries of the photovoltaic panels to build a training data set

[0113] c) Use the training dataset to perform end-to-end supervised training on the semantic segmentation model

[0114] d) Optimize the model parameters during training so that it can accurately segment the photovoltaic panel area from the background

[0115] e) After training is completed, the merged image is input into the semantic segmentation model

[0116] f) The model automatically completes image segmentation and outputs a segmented image containing only the photovoltaic panel area

[0117] This semantic segmentation method based on deep learning can overcome the limitations of traditional image segmentation algorithms and accurately extract the photovoltaic panel area from the cluttered background. The segmented image only contains the information of the photovoltaic panel itself, providing ideal input data for subsequent defect detection analysis.

[0118] Step S50: Photovoltaic panel defect detection based on convolutional neural network

[0119] The following is a specific implementation of the training of the photovoltaic panel defect damage detection model

[0120] 1. Data preprocessing

[0121] First, the collected photovoltaic panel image data needs to be preprocessed:

[0122] 1.1 Image annotation: All images are manually annotated to mark the location and type of various defects. These annotation information will be used as supervisory signals for model training.

[0123] 1.2 Image enhancement: Use basic transformations such as rotation, flipping, scaling, and some advanced image enhancement techniques such as mixup and cutout to expand the training set to enhance the generalization ability of the model.

[0124] 1.3 Data division: Divide the labeled image dataset into training set, validation set and test set in a ratio of 7:2:1.

[0125] 1.4 Preprocessing: Perform standard preprocessing operations such as scale normalization and color space conversion on the input image to make it meet the model input requirements.

[0126] 2. Model architecture implementation

[0127] According to the overall structure designed previously, each module is implemented specifically:

[0128] 2.1 Micro-feature extraction branch

[0129] (1) Multi-scale convolutional layer:

[0130] Three parallel convolutional layers are used, using 3x3, 5x5, and 7x7 convolution kernels to capture subtle features of different scales. Each convolutional layer is followed by a batch norm layer and a ReLU activation function.

[0131] The output feature maps of the convolutional layers are concatenated together to form a rich micro-feature representation.

[0132] (2) Spatial Pyramid Pooling (SPP) layer:

[0133] The concatenated feature map is divided into three scales: 1x1, 2x2, and 4x4, and the maximum pooling operation is performed in each region. This can obtain local features at different scales and further enhance the expressiveness of micro features.

[0134] The output of the SPP layer will be flattened to form a feature vector of fixed length.

[0135] (3) Attention Mechanism:

[0136] Two branches, channel attention and spatial attention, are introduced to adaptively focus on the most important micro-region features in the input image.

[0137] The channel attention branch uses average pooling and fully connected layers to learn the importance weight of each channel feature, while the spatial attention branch uses convolutional layers to learn the importance weight of each spatial position.

[0138] The two attention weight maps are multiplied element-wise with the input feature map to enhance the expression of key micro features.

[0139] Finally, the output of the micro-feature extraction branch is a feature vector rich in detail information.

[0140] 2.2 Macro feature extraction branch

[0141] (1) Deep Convolutional Network:

[0142] ResNet-50 is selected as the backbone network, which can effectively extract powerful semantic features. The first few convolutional blocks of the ResNet-50 pre-trained model will be directly used as the feature extractor.

[0143] (2) Dilated Convolution:

[0144] In the last few convolution blocks of the ResNet backbone network, dilated convolutions are used, which can increase the receptive field without reducing the feature map resolution and capture richer contextual information.

[0145] (3) Feature Pyramid:

[0146] The feature pyramid structure is used to fuse the feature maps output by different layers of ResNet at multiple scales, so that both fine-grained and coarse-grained semantic features can be obtained.

[0147] Specifically, the output feature maps of the last few convolution blocks of ResNet will be upsampled and concatenated step by step to form a multi-scale feature pyramid, and the last 16 convolution blocks are selected by default.

[0148] Finally, the macro feature extraction branch outputs a feature vector rich in semantic information.

[0149] 2.3 Feature Fusion Module

[0150] The feature vectors output by the micro-feature extraction branch and the macro-feature extraction branch will be spliced ​​together and sent to several convolutional layers for feature fusion.

[0151] These fused convolutional layers include:

[0152] -1x1 convolution, used for dimension reduction and channel number adjustment

[0153] -3x3 convolution to further extract combined features

[0154] -Batch Norm and ReLU activation

[0155] The output of the feature fusion module is a comprehensive feature representation that contains both the microscopic details and macroscopic semantic information of the input image.

[0156] 2.4 Classification Output Layer

[0157] Finally, a classification layer is connected to identify and classify photovoltaic panel defects. The classification layer includes:

[0158] -Global average pooling, pooling the feature map into a feature vector of fixed length

[0159] -Two fully connected layers, the first one uses ReLU activation and the second one uses Softmax activation to get the final classification probability output

[0160] The loss function of the entire model uses cross entropy loss, and the model parameters are optimized through end-to-end supervised training.

[0161] 3. Training strategy

[0162] 3.1 Initialization

[0163] In order to speed up the convergence and improve the generalization ability, the transfer learning method can be used for initialization:

[0164] (1) Micro-feature extraction branch:

[0165] It can be initialized to the model weights pre-trained on other related visual tasks (such as image classification, object detection, etc.), making use of its learned underlying visual features.

[0166] (2) Macro feature extraction branch:

[0167] You can also use ResNet-50 pre-trained on other related tasks as the initialization of the backbone network.

[0168] (3) Classification output layer:

[0169] Random initialization can greatly improve the initial performance of the model and speed up convergence.

[0170] 3.2 Optimization Algorithm

[0171] The model is trained end-to-end using the Adam optimizer. The learning rate is initially set to 1e-4, and the cosine annealing strategy is used to dynamically adjust the learning rate.

[0172] Adam is an adaptive learning rate optimization algorithm that can handle sparse gradients and noisy data well and performs well in visual tasks. The cosine annealing strategy can gradually reduce the learning rate during training to help the model converge to a better solution.

[0173] 3.3 Loss Function

[0174] In addition to the commonly used cross entropy classification loss, some auxiliary losses can be introduced to further enhance the performance of the model:

[0175] (1) Positioning loss:

[0176] For each defect area, its bounding box coordinates can be predicted and the bounding box regression can be optimized using L1 loss or Smooth L1 loss. This can help the model better learn the location information of the defects.

[0177] (2) Segmentation loss:

[0178] In addition to classification, pixel-level semantic segmentation can also be performed on the defect area, and segmentation loss can be introduced to guide the model to learn more refined defect masks. This can enhance the model's ability to perceive subtle defects.

[0179] The total loss function is:

[0180] Total loss = Classification loss + λ1*Localization loss + λ2*Segmentation loss

[0181] 4. Training process

[0182] 4.1 Multi-scale Input

[0183] In order to further enhance the robustness and performance of the model, a multi-scale input training strategy can be adopted. Specifically, in each training cycle, different image scaling ratios (such as 0.5, 1.0, 1.5) are randomly selected and the input image is scaled accordingly.

[0184] The benefits of doing this are:

[0185] - Allows the model to learn feature extraction capabilities for different scales

[0186] -Increased the diversity of training samples and improved generalization ability

[0187] -Simulate the scale changes that may be encountered in real deployment to improve robustness

[0188] 4.2 Data Augmentation

[0189] In addition to basic transformations such as rotation, flipping, and scaling, you can also try some more advanced data enhancement techniques:

[0190] (1)Mixup:

[0191] In the training batch, two samples are randomly selected and linearly interpolated in a certain ratio to obtain a new synthetic sample. This can increase the diversity of training data and improve the robustness of the model to noise and interference.

[0192] (2)Cutout:

[0193] Randomly block a rectangular area on the training image to simulate partial occlusion in the input image. This can encourage the model to learn more robust feature representations.

[0194] (3)CutMix:

[0195] Combining the ideas of Mixup and Cutout, two samples are randomly selected in the training batch, and then a region is randomly cut on one of the samples and replaced with the corresponding region of the other sample. This can create more complex and diverse training samples.

[0196] Through these data enhancement techniques, the training set can be greatly expanded, the overfitting problem can be alleviated, and the generalization ability of the model can be improved.

[0197] 4.3 Multi-task Learning

[0198] In addition to classification loss, some auxiliary tasks can also be introduced to form a multi-task learning framework:

[0199] (1) Defect location:

[0200] In addition to classification, the bounding box coordinates of each defect can be predicted simultaneously. This not only obtains the category information of the defect, but also accurately locates its position.

[0201] (2) Semantic Segmentation:

[0202] In addition to classification and positioning, the model can also be required to output the defect category of each pixel, that is, to perform pixel-level semantic segmentation on the input image. This allows the model to learn more fine-grained defect features.

[0203] Through multi-task learning, the model can not only perform classification tasks, but also complete localization and segmentation tasks at the same time. These auxiliary tasks can play a regularization role and further improve the performance of the model in classification tasks.

[0204] 4.4 Model Distillation

[0205] In addition to the above strategies, model distillation can also be used to enhance model performance. Specifically:

[0206] (1) Train a larger and more complex teacher model, which can be the complete model designed previously, or a more powerful version that has been further optimized.

[0207] (2) Then train a smaller and more efficient student model so that it can effectively "absorb" the knowledge of the teacher model.

[0208] The loss function of the student model includes:

[0209] - Standard classification loss

[0210] -Distillation loss, which is the KL divergence between the output of the student model and the output of the teacher model

[0211] Through this knowledge distillation, the reasoning efficiency can be greatly improved while maintaining the model accuracy, making it more suitable for actual deployment.

[0212] 5. Inference deployment

[0213] After training is completed, the model needs to be evaluated and deployed.

[0214] 5.1 Model Evaluation

[0215] First, evaluate the performance of the model on the test set. Commonly used indicators include:

[0216] -Classification accuracy

[0217] -IoU of defect localization

[0218] -mIoU for semantic segmentation

[0219] In addition, the model's performance on different types of defects and its ability to perceive subtle defects can be further analyzed. This helps to discover the advantages and disadvantages of the model and further optimize it.

[0220] 5.2 Model Optimization

[0221] The problems exposed by the model during evaluation can be further optimized:

[0222] -Adjust the network structure, such as adding more micro-feature extraction capabilities

[0223] - Try other advanced data augmentation techniques

[0224] -Optimize loss function design and enhance learning of subtle defects

[0225] -Use model compression and acceleration technology to improve reasoning speed

[0226] Through continuous iterative optimization, a final model with high accuracy and efficiency can be obtained. This deep learning-based defect detection method can overcome the limitations of traditional image processing algorithms and accurately identify various types of defects on the surface of photovoltaic panels. By training a large number of real defect samples, the detection model can learn the typical characteristics of various defects, thereby accurately detecting defect information in new photovoltaic panel images.

[0227] Step S60: Defect classification and location and result annotation

[0228] After completing the defect detection, the detection results need to be further classified and located, and the defect information needs to be marked on the original photovoltaic panel image to form the final output result. Specifically, it includes:

[0229] a) According to the output of the defect detection model, different types of defects are classified, such as scratches, breaks, and detachments.

[0230] b) Calculate the precise coordinates of each type of defect in the photovoltaic panel image and provide the spatial positioning information of the defect

[0231] c) The classification and positioning results are superimposed on the original photovoltaic panel surface image through image annotation

[0232] d) Output photovoltaic panel images with various defect information to provide a basis for subsequent maintenance and management

[0233] This method of defect classification, location and result annotation can intuitively present various defects on the surface of photovoltaic panels and provide a visual reference for maintenance personnel. By superimposing annotations on the original image, the specific type and location of the defect can be clearly indicated, which is conducive to quickly locating the problem and taking targeted maintenance measures.

[0234] In summary, this photovoltaic panel defect detection method based on drone aerial photography and deep learning algorithm can automatically complete the entire process from image acquisition, image preprocessing, image registration and fusion, region extraction, defect detection to result annotation. This method makes full use of advanced computer vision and deep learning technology, overcomes the limitations of traditional methods, and can accurately and efficiently identify various defects on the surface of photovoltaic panels.

[0235] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned photovoltaic panel damage detection method.

[0236] A third aspect of the present invention provides an electronic device, which comprises a processor and a memory, wherein the memory is used to store the step program of the above-mentioned method, and the processor reads the memory and executes the steps.

[0237] In order to better implement the present invention, the following provides a specific embodiment of the present invention developed and run on a computer in combination with a specific formula:

[0238] 1. Step S10: Obtaining the photovoltaic panel surface image through the drone and preprocessing it

[0239] In this step, the drone first hovers at a suitable height above the photovoltaic panel to collect a high-resolution panoramic image of the photovoltaic panel surface. The collected original image is recorded as I0. In order to eliminate the overexposure or underexposure problem caused by poor lighting conditions, it is necessary to adjust the parameters such as exposure time and white balance of image I0. Specifically, the following formula can be used for image correction:

[0240] I c =α·I0+β

[0241] Among them, α and β are brightness gain and offset coefficient respectively, which can be adaptively determined by histogram analysis and other methods. After adjustment, the preprocessed image I1 is obtained.

[0242] Next, the image I1 needs to be geometrically corrected to eliminate the distortion caused by the camera tilt. An image registration algorithm based on corner detection, such as SIFT or ORB features, can be used to solve the camera's internal and external parameter matrix, and then a perspective transformation is performed based on these parameters to obtain the corrected image I2.

[0243] In order to further remove the impurities and noise in the image, a bilateral filter [3] can be used to smooth the image I2:

[0244]

[0245] Where W(x, y) is the normalization factor, Ω(x, y) is the filter window centered at (x, y), and G s and G r are spatial Gaussian kernel and grayscale Gaussian kernel respectively. Through this step, the denoised image I3 can be obtained.

[0246] Finally, in order to accurately extract the photovoltaic panel area, a method based on Canny edge detection and Hough line transform can be used to first extract the edge information in image I3, then identify the straight line edge, and thus segment the photovoltaic panel area. The segmented photovoltaic panel area image is recorded as I4.

[0247] In summary, step S10 obtains a high-quality photovoltaic panel surface image I4 through a series of image preprocessing techniques, including exposure correction, geometric correction, denoising and region segmentation, which lays a foundation for subsequent defect detection and analysis.

[0248] 2. Step S20: Acquiring a fluorescent image of the photovoltaic panel surface

[0249] Based on the photovoltaic panel image obtained in step S10, it is necessary to further collect a fluorescent image after the surface of the photovoltaic panel is sprayed with fluorescent powder. Specifically, the fluorescent powder is first mixed with an appropriate amount of solvent (such as water or alcohol) to prepare a spray liquid with moderate concentration. Then, the drone carries a special spraying device to evenly spray the surface of the photovoltaic panel according to the pre-planned route and altitude. In order to ensure that the deposition distribution of the fluorescent powder is more uniform, a constant nozzle outlet pressure and spraying angle need to be maintained during the spraying process. By spraying back and forth multiple times, the entire surface of the photovoltaic panel can be evenly covered.

[0250] After spraying the fluorescent powder, the drone will take aerial photos again along the predetermined path to collect fluorescent images of the photovoltaic panel surface. During the shooting process, a suitable UV LED lamp needs to be installed in front of the drone camera and the long exposure mode is turned on. According to the camera parameters and optical path characteristics, the following camera imaging model can be established:

[0251]

[0252] Wherein, I5 is the final fluorescence image, and L(x, y, t) is the light intensity at the (x, y) coordinate at time t. In order to obtain a clearer fluorescence imaging effect, the exposure time T can be adjusted appropriately.

[0253] At the same time, the SLAM algorithm based on visual odometer can be used to perform three-dimensional reconstruction on the continuously shot fluorescent image sequence to obtain the three-dimensional geometric information of the photovoltaic panel surface. The reconstructed three-dimensional point cloud is denoted as P.

[0254] In summary, by spraying phosphor and using ultraviolet irradiation imaging, a fluorescent image I5 that clearly reflects the surface defect information of the photovoltaic panel and the corresponding three-dimensional point cloud P can be obtained. These data will provide an important basis for subsequent image registration and defect detection.

[0255] 3. Step S30: Image Registration and Fusion Based on Deep Learning

[0256] After obtaining the conventional image I4 and the fluorescent image I5 of the photovoltaic panel surface, the two images need to be accurately registered and fused. Here, an image registration algorithm based on deep learning is used, which specifically includes the following steps:

[0257] First, a large number of reference images of the photovoltaic panel surface and their corresponding fluorescence images are collected to construct a training dataset. In order to enhance the generalization ability of the model, data enhancement can be performed on the training samples, such as rotation, scaling, translation and other transformations.

[0258] Next, we design an image registration model based on a convolutional neural network. The input is the photovoltaic panel surface image I4 and the fluorescence image I5, and the output is the transformation parameter θ between them. The following loss function can be defined:

[0259]

[0260] in, Represents the geometric transformation operation of the image. Through end-to-end supervised learning, the network parameters Θ can be optimized so that it can accurately complete the registration between the two images.

[0261] After the training is completed, the images I4 and I5 obtained in steps S10 and S20 are input into the trained deep learning model to automatically complete the image registration and output a merged image I6 that combines the surface information and the fluorescence information.

[0262] To further enhance the performance of the model, a constraint based on feature consistency can be added to the loss function:

[0263]

[0264] Among them, F(·) represents the image feature extraction function and λ is a hyperparameter. This can not only optimize the geometric registration accuracy, but also enhance the consistency of semantic information.

[0265] Through the above-mentioned deep learning-based image registration method, the conventional image and the fluorescent image of the photovoltaic panel surface can be accurately aligned and fused to obtain a merged image I6 containing rich information, which provides ideal input data for subsequent defect detection analysis.

[0266] Step S40: Photovoltaic panel region extraction based on semantic segmentation

[0267] After obtaining the merged image I6, it is necessary to use semantic segmentation technology to accurately extract the area of ​​the photovoltaic panel itself and separate it from irrelevant areas such as the background. Specifically, a U-Net semantic segmentation model based on a convolutional neural network can be used.

[0268] First, a large number of photovoltaic panel surface images are collected and the boundaries of the photovoltaic panels are manually marked to construct a training dataset. Among them, M i The mask is annotated for the segmentation of the corresponding image. In order to enhance the robustness of the model, data augmentation operations such as geometric transformation and color adjustment can be performed on the training samples.

[0269] Next, define the loss function of the U-Net network as cross entropy loss:

[0270]

[0271] Among them, W is the network parameter, and P(·|I6;W) represents the probability output that the pixel belongs to the photovoltaic panel area. Through end-to-end supervised training, the network parameter W is optimized so that it can accurately segment the photovoltaic panel area from the background.

[0272] After the training is completed, the merged image I6 is input into the trained semantic segmentation model to obtain a binary segmentation image M containing only the photovoltaic panel area. In order to further improve the segmentation accuracy, the conditional random field (CRF) algorithm based on graph reasoning can be combined with image prior knowledge for post-processing optimization.

[0273] In summary, step S40 utilizes the semantic segmentation technology based on deep learning to accurately extract the area of ​​the photovoltaic panel itself from the complex background, providing ideal input data for subsequent defect detection analysis.

[0274] 5. Step S50: Photovoltaic panel defect detection based on convolutional neural network

[0275] After obtaining the photovoltaic panel area image M extracted in the previous step, it can be input into the pre-trained photovoltaic panel defect detection model for automatic defect recognition. Here, a defect detection model based on a convolutional neural network is used, which specifically includes the following steps:

[0276] First, we collected a large number of defect sample images of photovoltaic panels, covering all common defect types, and constructed a training dataset. Where yi is the defect category label of the i-th sample. In order to further enhance the detection performance, the training samples can be expanded through methods such as data enhancement.

[0277] Next, design a defect detection network based on ResNet or VGG structure. Let the network parameter be θ and define the following classification loss function:

[0278]

[0279] Among them, P(y i |M i; θ) represents the network's response to the input image M i Through end-to-end supervised training, the network parameters θ are optimized to enable it to accurately identify different types of defects.

[0280] In order to improve the detection performance, a positioning loss term based on bounding box regression can also be added to the loss function:

[0281]

[0282] Among them, b i is the true bounding box coordinate of the i-th sample, is the bounding box coordinate predicted by the network, and λ is a hyperparameter. This not only allows defect recognition, but also gives the exact location of the defect in the image.

[0283] After the training is completed, the extracted photovoltaic panel area image M is input into the trained detection model to automatically identify various defects. The detection output result is recorded as where y j and b j denote the j-th detected defect category and bounding box coordinates, respectively.

[0284] In summary, step S50 utilizes the deep learning-based defect detection technology to accurately identify various defect types and their location information on the surface of the photovoltaic panel, providing an important basis for subsequent maintenance management.

[0285] 6. Step S60: Defect classification and location and result annotation

[0286] After completing the defect detection, the detection results need to be further classified and located, and the defect information needs to be marked on the original photovoltaic panel image to form the final output result. The specific steps include:

[0287] First, according to the defect detection result output in step S50 Clustering algorithm is used to classify different types of defects. The classification result is where c k represents the k-th defect, The index collection corresponding to this category.

[0288] For each type of defect, calculate its exact position coordinates in the original photovoltaic panel image I0. The normalized bounding box b obtained by detection can be converted into j Mapping back to the original image space:

[0289]

[0290]

[0291]

[0292]

[0293] Among them, W0 and H0 are the width and height of the original image I0 respectively. Through this mapping, the precise position coordinates of each defect on I0 can be obtained.

[0294] Finally, the classification and positioning results are superimposed on the original photovoltaic panel surface image I0 through image annotation. Different visual annotation methods can be used for different types of defects, such as using bounding boxes and polylines of different colors. At the same time, quantitative parameters such as the area and length of the defect can also be given in the annotation information. After this step, a photovoltaic panel image I7 with various defect information annotated is output, providing an intuitive reference for subsequent maintenance and management.

[0295] In summary, the photovoltaic panel defect detection method based on drones and deep learning proposed in the present invention makes full use of advanced computer vision technology and can automatically and efficiently complete the entire process from image acquisition to defect location and annotation. This method overcomes the limitations of traditional manual inspection and can quickly and accurately detect various defects on the surface of photovoltaic panels, providing strong support for the operation and maintenance management of photovoltaic power generation systems.

[0296] In the specific implementation, step S10 obtains a high-quality surface image of the photovoltaic panel through pre-processing techniques such as exposure correction, geometric correction, denoising and regional segmentation. Step S20 obtains a fluorescent image of the photovoltaic panel that clearly reflects defect information by spraying fluorescent powder and irradiating with ultraviolet light. Step S30 uses an image registration algorithm based on deep learning to accurately merge the two images. Step S40 uses semantic segmentation technology to extract the photovoltaic panel area from the complex background. Step S50 uses a convolutional neural network for automated defect detection. Finally, step S60 classifies and locates the detection results and marks the information of various defects on the original image.

[0297] Specifically, the principle of the present invention is that the causes of surface defects of photovoltaic panels usually involve complex physical and chemical processes. During the manufacturing and use process, photovoltaic panels are prone to various tiny cracks, defects, etc., which are often difficult to observe with the naked eye. However, these minor defects will gradually expand and eventually affect the power generation performance of photovoltaic panels. It is worth noting that these minor defects usually show some special visual characteristics under specific environmental conditions. For example, when the surface of the photovoltaic panel is coated with fluorescent powder or fluorescent coating, small cracks or depressions will gather a large number of fluorescent powder particles due to surface tension and other effects. These highly concentrated fluorescent powder areas will appear as obvious bright spots or areas when imaging, highlighting the location of the minor defects.

[0298] Based on this physical mechanism, the photovoltaic panel defect detection method of the present invention adopts a special image acquisition and processing strategy. First, a uniform layer of phosphor or fluorescent coating is applied on the surface of the photovoltaic panel. Then, a professional imaging device, such as a high-definition digital camera under ultraviolet light, is used to capture the processed surface image of the photovoltaic panel. In this way, subtle defects that were originally difficult to detect will stand out in the form of bright fluorescent spots, providing important clues for subsequent image analysis. Next, these images containing subtle defect information will be input into the deep learning model proposed by the present invention for automated detection. The core innovation of this model is that it can effectively extract the fusion representation of microscopic detail features and macroscopic semantic features in the image.

[0299] Specifically, the model designs a micro-feature extraction branch, which uses multi-scale convolution, spatial pyramid pooling and other technologies to accurately capture the visual features of micro defects such as small cracks and dents. At the same time, it also contains a macro feature extraction branch, which extracts rich semantic information and contextual features in the image through deep convolutional networks, dilated convolutions, feature pyramids and other means.

[0300] After the two types of features are fused, the model can integrate microscopic details and macroscopic semantics to form a more comprehensive defect representation. Ultimately, by optimizing the entire end-to-end model through supervised learning, it is possible to accurately detect and locate various subtle defects in photovoltaic panels.

[0301] The key to adopting this detection method based on fluorescence-enhanced image acquisition and deep learning is:

[0302] 1. The aggregation effect of phosphor can effectively highlight the location information of microscopic defects and provide important basis for subsequent image analysis.

[0303] 2. The deep learning model can fully learn and extract the visual features of these subtle defects, making up for the limitations of traditional image processing technology.

[0304] 3. The fusion of micro-features and macro-features can enhance the model's adaptability to complex photovoltaic panel environments and improve the accuracy and stability of detection.

[0305] Among them, in order to realize the above-mentioned advanced photovoltaic panel defect detection function, the present invention proposes a feature fusion model based on deep learning, and its key technical solutions are as follows:

[0306] 1. Micro feature extraction branch

[0307] This branch aims to extract subtle defect features from the input image. It contains the following core modules:

[0308] (1) Multi-scale convolutional layer: Use convolution kernels of different sizes (such as 3x3, 5x5, 7x7) to process the input image in parallel and capture detailed features of different scales.

[0309] (2) Spatial Pyramid Pooling (SPP) layer: The feature map is divided into multiple regions, and maximum pooling is performed in each region to extract local features at different scales.

[0310] (3) Attention mechanism: Channel attention and spatial attention are introduced to enable the model to adaptively focus on the most critical micro-region features.

[0311] 2. Macro feature extraction branch

[0312] This branch aims to extract the overall semantic features in the image. It consists of the following key components:

[0313] (1) Deep convolutional network: Using the pre-trained ResNet-50 as the backbone network, it can effectively extract powerful semantic features.

[0314] (2) Dilated convolution: Using dilated convolution in the last few convolution blocks of ResNet can increase the receptive field without reducing the resolution and capture richer contextual information. The last 16 convolution blocks are selected by default.

[0315] (3) Feature pyramid: The feature maps output by different layers of ResNet are fused at multiple scales, taking into account both fine-grained and coarse-grained semantic features.

[0316] 3. Feature Fusion Module

[0317] The outputs of the micro-feature extraction branch and the macro-feature extraction branch will be concatenated and sent to several convolutional layers for feature fusion, which can fully combine detail information and semantic information to form a more comprehensive feature representation.

[0318] 4. Classification output layer

[0319] Finally, a classification layer is connected, including global pooling and fully connected layers, to identify and classify photovoltaic panel defects.

[0320] The entire model is optimized through end-to-end supervised training, and the loss functions include classification loss, positioning loss, and segmentation loss.

[0321] The principle of this technical solution is as follows:

[0322] 1. Importance of micro-feature extraction

[0323] There are various subtle defects on the surface of photovoltaic panels, such as spot stains, small cracks, etc. These defects are not obvious and difficult to be effectively captured by traditional image processing methods.

[0324] The present invention can fully extract these detailed features by designing a special micro-feature extraction branch and using multi-scale convolution, spatial pyramid pooling and other technologies. At the same time, the attention mechanism is introduced to adaptively enhance the most critical micro-region features. This greatly improves the model's ability to perceive subtle defects.

[0325] 2. Auxiliary role of macro feature extraction

[0326] In addition to microscopic features, the overall semantic information in the image is also crucial for defect detection. For example, the surface texture of the photovoltaic panel and the installation environment will affect the appearance characteristics of the defect.

[0327] The present invention uses deep convolutional networks, dilated convolutions, and feature pyramids in the macro feature extraction branch to extract rich context information. The fusion of these semantic features with micro features can enhance the model's adaptability to complex background environments and improve the accuracy and stability of detection.

[0328] In summary, the photovoltaic panel defect detection technology solution proposed in the present invention cleverly combines the two key technologies of fluorescence imaging and deep learning, which can fundamentally solve the problem that existing methods are difficult to accurately extract subtle defect information.

[0329] In order to better implement the present invention, an embodiment of a specific use scenario of the present invention is provided below. The total installed capacity of the photovoltaic power station in this embodiment reaches 10MW, with a total of 7,000 photovoltaic panels and an area of ​​about 200,000 square meters. The specific steps in this embodiment are as follows:

[0330] 1. Data Collection

[0331] First, a drone was used to conduct a comprehensive aerial photography of the 10MW power station. The drone's hovering height was set to 10 meters, and the flight speed was controlled at about 5m / s. According to the actual scale of the photovoltaic power station, the drone's flight path was pre-planned, and overlapping shooting was used to ensure that each photovoltaic panel area could be fully covered. The entire aerial photography process lasted about 2 hours, and a total of 12,000 high-definition images were collected.

[0332] After acquiring the conventional photovoltaic panel surface image, the phosphor was sprayed and fluorescent imaging was performed. Specifically, the phosphor was mixed with water to prepare a spray liquid with an appropriate concentration, which was loaded into the drone's special spraying device. Then, the entire photovoltaic power station was evenly sprayed according to the predetermined flight path. After the phosphor was fully deposited, the drone was used again for aerial photography, this time using a camera equipped with ultraviolet LEDs and long exposure function. In this way, a total of 3,000 high-quality fluorescent images of the photovoltaic panel surface were collected.

[0333] All collected image data are stored on the server, providing a basis for subsequent image preprocessing and defect detection.

[0334] 2. Image Preprocessing

[0335] After obtaining the original photovoltaic panel surface image and fluorescence image, a series of preprocessing operations were first performed on them.

[0336] For the photovoltaic panel surface image, exposure correction was first performed. According to the collected histogram statistical information, it was found that most images had uneven brightness, that is, some areas were overexposed and some areas were underexposed. To this end, the brightness gain coefficient α and offset coefficient β of each image were calculated, and the formula I was used to calculate the brightness gain coefficient α and offset coefficient β of each image. c =α·I0+β to correct the image. After this step, the overall brightness and contrast of all images are significantly improved.

[0337] Next, the image was geometrically corrected to eliminate the distortion caused by camera tilt. An image registration algorithm based on SIFT feature matching was used. First, the key points in the image were detected, then the camera's internal and external parameter matrices were calculated, and finally a perspective transformation was performed to obtain the corrected image. This ensures that all images are aligned in geometric space.

[0338] In order to remove the noise in the image, a bilateral filter is applied to smooth the image. Bilateral filtering can not only effectively eliminate noise, but also well preserve edge details. The default setting of the spatial Gaussian kernel standard deviation σ s =3, Gray Gaussian kernel standard deviation σ r =0.1, the image is filtered.

[0339] Finally, the boundary contour of the photovoltaic panel area is accurately extracted by using Canny edge detection and Hough line transform.First, the edge information of the image is detected by using the Canny operator, and then the straight line edge is identified by applying Hough transform, and finally the precise boundary of the photovoltaic panel area is determined.

[0340] Through the above preprocessing steps, a series of high-quality photovoltaic panel surface images were obtained, laying the foundation for subsequent image registration and defect detection.

[0341] At the same time, similar preprocessing operations were performed on the acquired fluorescence images, including geometric correction, denoising, and region extraction, etc. This ensures the quality of the fluorescence images and their alignment with the conventional images.

[0342] 3. Image registration and fusion

[0343] After obtaining the preprocessed photovoltaic panel surface image and fluorescence image, the next step is to accurately align and fuse them.

[0344] To this end, 500 pairs of conventional and fluorescent images of photovoltaic panels were first collected to construct a training dataset. During the training process, data enhancement was performed on the samples, including rotation, scaling, translation and other transformations, to enhance the generalization ability of the model.

[0345] Then, a convolutional neural network-based image registration model is designed. The input of the model is a pair of photovoltaic panel images to be registered, and the output is the transformation parameter θ between them. The following loss function is defined:

[0346]

[0347] in, and are the conventional image and fluorescence image of the i-th training sample, respectively, F(·) is the image feature extraction function, and λ is a hyperparameter. Through end-to-end supervised learning, the network parameters are optimized so that it can accurately complete the registration of the two images.

[0348] After the training is completed, the photovoltaic panel surface image and fluorescence image obtained in the previous steps are input into the trained registration model. The model automatically completes the image registration and outputs a high-quality merged image that combines surface information and fluorescence information.

[0349] In order to further verify the performance of the method of the present invention, a quantitative evaluation was performed on 400 test images. The results showed that the average overlap rate of image alignment using the deep learning registration algorithm of the present invention reached 95.7%, which was significantly better than the traditional method based on feature point matching (the average overlap rate was only 87.4%). This shows that the registration technology of the present invention can accurately complete the registration fusion between conventional images of photovoltaic panels and fluorescent images.

[0350] 4. Photovoltaic panel area extraction

[0351] After obtaining the registered merged image, the semantic segmentation technology is then used to accurately extract the area of ​​the photovoltaic panel itself.

[0352] First, 2,000 photovoltaic panel surface images were collected and the boundaries of the photovoltaic panels were manually annotated to construct a training dataset. During the training process, data augmentation operations such as geometric transformation and color adjustment were performed on the samples to enhance the robustness of the model.

[0353] Then, a semantic segmentation model based on the U-Net network was adopted. The loss function of this model is the cross entropy loss:

[0354]

[0355] Where W is the network parameter, P(·|I6;W) represents the probability output of the pixel belonging to the photovoltaic panel area, and M i The segmentation mask of the i-th training sample is annotated. Through end-to-end supervised learning, the network parameters are optimized so that it can accurately segment the photovoltaic panel area from the background.

[0356] In the test phase, the registered merged image I6 is input into the trained semantic segmentation model, and a binary segmentation result M containing only the photovoltaic panel area is obtained. In order to further improve the segmentation accuracy, a post-processing method based on conditional random field (CRF) is also applied to perform spatial smoothing optimization on the segmentation results.

[0357] In order to verify the performance of the method of the present invention, an evaluation was conducted on 500 test images. The results show that the average segmentation accuracy of the photovoltaic panel area using the semantic segmentation technology of the present invention reached 93.2%, which is much higher than the traditional edge detection-based method (only 84.6%). This shows that the region extraction technology of the present invention can accurately segment the region of the photovoltaic panel itself from the complex background.

[0358] 5. Defect Detection

[0359] After obtaining the photovoltaic panel area image M extracted in the previous steps, deep learning technology is used to perform automated defect detection on it.

[0360] First, 1,500 defective sample images of photovoltaic panels were collected, covering various common defect types such as scratches, breaks, and detachments, and a training data set was constructed. During the training process, data enhancement operations such as rotation, scaling, and brightness adjustment were performed on the samples to improve the generalization ability of the model.

[0361] Then, a defect detection network based on ResNet-50 backbone is designed. The loss function of this network contains classification loss and bounding box regression loss:

[0362]

[0363] Among them, y i is the defect category label of the i-th training sample, b i and are the real bounding box and the network predicted bounding box, respectively, and λ is a hyperparameter. Through end-to-end supervised learning, the network parameter θ is optimized so that it can accurately identify various defects and give precise location coordinates.

[0364] In the testing phase, the extracted photovoltaic panel area image M is input into the trained defect detection model. The model automatically completes the defect identification and location, and outputs a series of detection results. where y j represents the jth detected defect category, b j are its bounding box coordinates.

[0365] In order to evaluate the performance of the method of the present invention, a quantitative analysis was performed on 500 test images. The results show that the average recognition accuracy of common defect types (such as scratches, breaks, and detachments) using the deep learning defect detection technology of the present invention reached 91.3%, which is much higher than the traditional method based on artificial features (only 78.5%). At the same time, the average error between the detected defect position coordinates and the actual coordinates is only 1.8%, indicating that the positioning technology of the present invention has also achieved a high degree of accuracy. This shows that the defect detection method of the present invention can accurately identify various defects on the surface of photovoltaic panels, providing a reliable basis for subsequent maintenance.

[0366] 6. Defect marking

[0367] After completing defect detection, the last step is to visually annotate the detection results and mark the location and type of various defects on the original photovoltaic panel image.

[0368] First, the different types of defects detected are classified using the clustering algorithm, denoted as Then, the exact coordinates of each defect on the original image I0 are calculated:

[0369]

[0370]

[0371]

[0372]

[0373] Where W0 and H0 are the width and height of the original image respectively.

[0374] Finally, the classification and positioning results are superimposed on the original photovoltaic panel image I0 through image annotation. Different types of defects are annotated with bounding boxes of different colors. At the same time, quantitative parameters such as the area and length of the defects are also given in the annotation information. After this step, a final output image I7 with various types of defect information annotated is obtained.

[0375] In order to verify the labeling technology of the present invention, 5 maintenance experts were invited to conduct subjective evaluation on 100 labeled images. The results showed that the experts gave high scores to the clarity, completeness and operability of the labeling, with an average score of 4.8 points (out of 5 points). This shows that the defect labeling method of the present invention can intuitively present various defect information on the surface of photovoltaic panels, providing a reliable basis for subsequent maintenance management.

[0376] 7. Technical effect analysis

[0377] Through the above experimental verification, it can be seen that the photovoltaic panel defect detection method based on drones and deep learning proposed in the present invention has achieved significant performance improvements in various key technical links and has obvious advantages over traditional methods. Specifically:

[0378] 1) In terms of image acquisition, the use of drone aerial photography can cover the entire photovoltaic power station in a short time and obtain large-scale high-definition image data, which is more than 10 times more efficient than traditional manual photography. At the same time, by rationally planning the flight path of the drone, it can ensure that every photovoltaic panel is fully photographed to avoid omissions. This lays a solid foundation for subsequent defect detection.

[0379] 2) In terms of image preprocessing, the present invention uses a series of advanced computer vision algorithms, including exposure correction, geometric correction, denoising, and regional segmentation, which effectively solves the impact of factors such as light conditions and camera tilt on image quality. After these preprocessing steps, high-quality photovoltaic panel surface images and fluorescence images are obtained. Compared with the traditional method that relies on manual basket selection, the image quality of the method of the present invention is more stable and reliable.

[0380] 3) In terms of image registration and fusion, the deep learning algorithm is used to achieve accurate registration between conventional images and fluorescent images of photovoltaic panels. Test results show that the image overlap rate of the method of the present invention is as high as 95.7%, which is much better than the traditional method based on feature points. This ensures that subsequent defect detection can accurately locate the problem area on the surface of the photovoltaic panel.

[0381] 4) In terms of photovoltaic panel area extraction, the present invention uses semantic segmentation technology based on deep learning, which can accurately segment the photovoltaic panel area from the complex background. Test data shows that the photovoltaic panel area segmentation accuracy (IoU) of this method reaches 93.2%, which is significantly higher than the traditional edge detection-based method. This is crucial for subsequent defect detection.

[0382] 5) In terms of defect detection, the present invention uses an automated method based on convolutional neural networks to accurately identify various photovoltaic panel defects, such as scratches, breaks, and detachments. Test results show that the average recognition accuracy of this method is as high as 91.3%, which is much better than the traditional method based on artificial features. At the same time, the detected defect position coordinates are also very accurate, with an average error of only 1.8%.

[0383] 6) In terms of result annotation, the visual annotation technology proposed in this invention can intuitively mark the location and type of various defects on the original photovoltaic panel image and provide quantitative defect parameters. The subjective evaluation of maintenance experts shows that this annotation method is clear and complete, providing an operational basis for subsequent maintenance.

[0384] In general, by efficiently collecting image data with drones and combining it with advanced deep learning algorithms, the method of the present invention has achieved significant performance improvements in all key links, and has obvious advantages over traditional manual inspection methods. This not only greatly improves the efficiency and accuracy of photovoltaic power station defect detection, but also provides reliable information support for subsequent maintenance management.

[0385] 8. Statistical analysis

[0386] In order to more intuitively demonstrate the technical effect of the method of the present invention, some key performance indicators obtained during the experiment were statistically analyzed and corresponding charts were produced.

[0387] First, the efficiency improvement of the method of the present invention in image acquisition is statistically analyzed. Figure 1 As shown in the figure, compared with the traditional manual shooting method, the present invention uses drone aerial photography to complete the full coverage of a 10MW photovoltaic power station within 2 hours and obtain 12,000 high-resolution images. In the same time, manual shooting can only obtain about 1,000 images. It can be seen that the method of the present invention improves the image acquisition efficiency by nearly 12 times.

[0388] Next, the performance indicators of the method of the present invention and the traditional method in key links such as image preprocessing, registration fusion, and region extraction were compared. As shown in Table 1, the method of the present invention is significantly better than the traditional method in various indicators, including image overlap rate, segmentation accuracy (IoU), defect detection accuracy, etc. In particular, in terms of defect detection accuracy, the method of the present invention reached 91.3%, which is much higher than the 78.5% of the traditional method. This shows that the technical solution of the present invention can significantly improve the performance level of the entire detection system.

[0389] Table 1 Comparison of key performance indicators

[0390] index Method of the present invention Traditional methods Image overlap ratio 95.7％ 87.4％ Segmentation accuracy (I0u) 93.2％ 84.6％ Defect detection accuracy 91.3％ 78.5％ Defect location error 1.8％ 4.5％

[0391] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A photovoltaic panel damage detection method, characterized in that: The following steps are involved: S10, acquiring a surface image of a photovoltaic panel through a drone, and performing preprocessing to obtain a preprocessed photovoltaic panel image; S20, obtaining, by a drone, a surface fluorescent image of a photovoltaic panel evenly sprayed with fluorescent powder, and preprocessing the fluorescent image to obtain a preprocessed fluorescent image; S30, registering the pre-processed photovoltaic panel image and the pre-processed fluorescence image using a deep learning-based image registration algorithm, and merging the pre-processed photovoltaic panel image and the pre-processed fluorescence image into one image, which is recorded as a merged image; S40, using a semantic segmentation model based on a convolutional neural network to segment the merged image and accurately separate the photovoltaic panel area; S50, inputting the separated photovoltaic panel area image into a pre-trained photovoltaic panel defect and damage detection model to detect various defects and damages on the photovoltaic panel surface; S60, classifying and locating the detected defects and damages, and marking the defect types and position coordinates on the photovoltaic panel image, and finally outputting the marked photovoltaic panel image.

2. A photovoltaic panel damage detection method according to claim 1, characterized in that: The semantic segmentation model adopts the U-Net network structure.

3. A photovoltaic panel damage detection method according to claim 1, characterized in that: The training steps of the photovoltaic panel defect damage detection model specifically include: First, a large number of sample images of photovoltaic panel surface defects are collected, covering various common defect types, and the S10-S40 method is used to obtain photovoltaic panel area images, and defects and damages are annotated in the photovoltaic panel area images to obtain a training data set; Secondly, select a convolutional neural network model based on ResNet or VGG structure as the basic architecture; Again, the training data set is used to perform end-to-end supervised training on the selected convolutional neural network model, and finally a photovoltaic panel defect damage detection model is obtained.

4. A photovoltaic panel damage detection method according to claim 3, characterized in that: The convolutional neural network model includes a micro feature extraction branch and a macro feature extraction branch. The micro feature extraction branch is used to extract subtle fluorescent point and fluorescent line features in the input image, and the macro feature extraction branch is used to extract overall and obvious features in the input image.

5. A photovoltaic panel damage detection method according to claim 4, characterized in that: In the micro-feature extraction branch, an attention mechanism is also introduced to improve the ability to capture subtle fluorescent dots and fluorescent line features in the image.

6. A photovoltaic panel damage detection method according to claim 4, characterized in that: The convolutional neural network model uses ResNet-50 as the backbone network.

7. A photovoltaic panel damage detection method according to claim 1, characterized in that: The phosphor is barium sulfate long afterglow phosphor.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the photovoltaic panel damage detection method according to any one of claims 1 to 7.

9. An electronic device, characterized in that: A processor and a memory are provided, wherein the memory is used to store the step program of the method described in any one of claims 1 to 7, and the processor reads the memory to execute the steps.

Images