A single-stage photovoltaic module surface defect detection method based on improved RetinaNet

Abstract

The application discloses a single-stage photovoltaic module surface defect detection method based on an improved RetinaNet and belongs to the technical field of image recognition. The feature pyramid structure and the residual network structure of the RetinaNet are optimized, the network feature fusion method is changed, the original one-way feature fusion structure is replaced by a bidirectional fusion structure from top to bottom and from bottom to top, and the bidirectional feature fusion structure improves the ability of the network to extract small target information at different stages. In order to suppress background noise and improve the feature extraction capability, a channel attention mechanism module ECA is added to the residual structure. The improved residual block improves the feature extraction capability of the network and strengthens the attention degree to small targets. Therefore, the optimized network has excellent performance in photovoltaic module surface defect detection.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method for detecting surface defects in single-stage photovoltaic modules based on an improved RetinaNet. Background Technology

[0002] Surface defect detection of photovoltaic (PV) modules is a crucial aspect of the photovoltaic industry, directly impacting the performance and long-term stable operation of PV power generation systems. Because existing large-scale PV power plants are often located in harsh environments, solar PV modules may develop problems such as cracks and hot spots. External environmental factors can cause cracks to propagate, further exacerbating the heating of the entire PV module. The presence of hot spots and cracks significantly reduces heating efficiency and can even lead to serious damage and substantial economic losses. Therefore, in today's rapidly developing new energy landscape, surface defect detection of solar PV modules has become paramount.

[0003] Surface defect detection methods for photovoltaic (PV) modules based on image processing and computer vision technologies have been widely applied. By acquiring image data of the PV module surface and analyzing and processing this data using image processing techniques, defects such as cracks, stains, and ablation on the PV module surface can be effectively detected. This method is non-contact and highly efficient, and has become one of the mainstream methods for PV module surface defect detection. However, due to the uncertainty of defect types, different defects require different parameter adjustments, therefore this method cannot be widely applied to most PV module surface defect detection scenarios.

[0004] The application of machine learning and artificial intelligence technologies in photovoltaic module surface defect detection is becoming increasingly common. Machine learning algorithms can be used to train and learn from large amounts of photovoltaic module data, thereby achieving automatic identification and classification of surface defects. The development of deep learning technology has further improved the accuracy and efficiency of photovoltaic module surface defect detection, making the detection results more accurate and reliable. For example, Chinese patent CN202210528398.0 discloses a two-stage photovoltaic module surface defect detection method based on deep learning, including the following steps: (1) acquiring original photovoltaic module images and creating a sample dataset; (2) dividing the sample dataset and constructing a training sample set, a test sample set, and a verification sample set; (3) performing data augmentation preprocessing on the sample dataset; (4) training the target detection YOLOv5 network model using the augmented training sample set and the verification sample set; (5) the target detection YOLOv5 network model includes a foreground extraction model and a defect detection model, and the output of the foreground extraction model is the input of the defect detection model; (6) inputting the augmented test sample set into the trained target detection YOLOv5 network model, and obtaining the defect detection result after the foreground extraction stage and the defect detection stage detection output; although this method can effectively detect the surface defect area of ​​the photovoltaic module, the accuracy of small target defect identification is low.

[0005] In view of this, the present invention proposes a single-stage photovoltaic module surface defect detection method based on an improved RetinaNet.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] Includes the following steps:

[0008] A1. Use industrial cameras to collect images of photovoltaic modules and create a photovoltaic module image dataset;

[0009] A2. Divide the sample dataset into training, testing, and validation sets;

[0010] A3. Perform data augmentation preprocessing on the sample dataset;

[0011] A4. Based on the enhanced image of the defective photovoltaic module, surface defect detection is performed using the RetinaNet deep learning network model to obtain the detection results. The improved RetinaNet single-stage object detection network includes a feature extraction network and a bidirectional feature fusion structure;

[0012] A5. Input the enhanced test sample set into the trained RetinaNet target detection network model. After the feature extraction and feature fusion stages, the model is detected and finally output by the classification and regression module to obtain the defect detection results.

[0013] Preferably, in step A1, the acquisition of original photovoltaic module images and the creation of a sample dataset includes the following steps:

[0014] A11. Use mechanical equipment for detecting surface defects of photovoltaic modules and an industrial camera mounted on them to take pictures of the photovoltaic modules and collect original images of surface defects of the photovoltaic modules;

[0015] A12. Annotate the acquired raw images, use different defect labels to annotate small units and defects, and generate a document containing .xml.

[0016] A13. The generated .xml document containing annotation information and the original image are bound and saved in a txt file as a sample dataset.

[0017] Preferably, in step A3, the data augmentation preprocessing uses random brightness enhancement, image inversion, and cropping and stitching methods to augment the labeled image, thereby expanding the sample dataset.

[0018] Preferably, in step A4, the step of training the target detection RetinaNet network model using the enhanced training sample set and validation sample set specifically involves: inputting the enhanced training sample set into the improved RetinaNet network model, iteratively training the defect detection model sequentially, and saving the trained network parameters; using the validation sample set to adjust the parameters of the trained defect detection model, and finally obtaining the defect detection RetinaNet model weights.

[0019] The specific steps of the defect detection stage are as follows: the enhanced photovoltaic module image is input into the defect detection RetinaNet model. First, the input image is subjected to feature extraction. After passing through the PAN bidirectional feature fusion structure, it enters the classification module and regression module to obtain the training results and outputs candidate boxes. Then, the maximum value suppression algorithm is used to filter the candidate boxes to obtain the final defect detection results. The defect detection results are then labeled in the original photovoltaic module image.

[0020] Preferably, in step A4, the improved RetinaNet network model is obtained by optimizing the existing RetinaNet network model, including the following steps:

[0021] A41. Obtain the loss function of an existing RetinaNet network model;

[0022] A42. In the existing RetinaNet network model feature fusion FPN structure, a bottom-up path is added to form a bidirectional feature fusion result;

[0023] A43. Improve the ResNet structure in the feature extraction network by adding the ECA attention mechanism to enhance the ability to transmit deep semantic information and improve the accuracy of small target detection;

[0024] A44. Update the existing RetinaNet network parameters to reduce the total loss function, and iterate this process until the network converges to obtain the RetinaNet network model for object detection.

[0025] Preferably, the loss function L of the existing RetinaNet network model Z It consists of two parts: the classification loss function L cls and regression loss function L reg The sum is obtained, that is

[0026] L Z =L cls +L reg

[0027] The classification loss uses FocalLoss as the loss function, calculating the classification loss for all positive and negative samples. The regression loss uses smoothed L1 loss as the loss function, calculating the regression loss for all positive samples.

[0028]

[0029] The beneficial effects of this invention are as follows:

[0030] 1. The single-stage photovoltaic module surface defect detection method based on improved RetinaNet described in this invention improves the network's ability to extract information from small targets by improving the RetinaNet feature fusion mechanism and introducing a bidirectional feature fusion path.

[0031] 2. The single-stage photovoltaic module surface defect detection method based on improved RetinaNet described in this invention improves the ResNet structure of RetinaNet and enhances the deep semantic information transmission capability by leveraging the ECA attention mechanism, thereby improving the detection accuracy of small targets.

[0032] 3. The single-stage photovoltaic module surface defect detection method based on the improved RetinaNet described in this invention, compared with the traditional method of manually detecting defects, utilizes a single-stage target detection network to achieve automatic detection of photovoltaic module surface defects, and can still achieve good detection results for small targets, greatly improving the recognition accuracy and detection efficiency of the detection results. Attached Figure Description

[0033] Figure 1This is a flowchart of a single-stage photovoltaic module surface defect detection method based on an improved RetinaNet proposed in this invention;

[0034] Figure 2 This is a network structure diagram of a single-stage photovoltaic module surface defect detection method based on an improved RetinaNet proposed in this invention.

[0035] Figure 3 This is a network structure diagram of the ECA attention mechanism introduced in this invention;

[0036] Figure 4 This is a diagram of the improved ResNet network structure of this invention. Detailed Implementation

[0037] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0038] See Figures 1 to 3 This invention discloses a method for detecting surface defects in single-stage photovoltaic modules based on an improved RetinaNet.

[0039] A single-stage photovoltaic module surface defect detection method based on improved RetinaNet includes the following steps:

[0040] A1. Collect original images of photovoltaic modules and construct a sample dataset;

[0041] The specific steps are as follows:

[0042] A11. Use an industrial camera above a photovoltaic module surface defect detection machine to take pictures of the solar photovoltaic module and obtain its original image;

[0043] A12. Use the labelImg annotation software to annotate the original image. Annotate the different types of defects on the surface of the photovoltaic module, and finally save the annotation as an .xml format document;

[0044] To improve the accuracy and robustness of object detection algorithms, a large image dataset is required for annotation. The annotation process is as follows: Open the collected raw images using annotation software, use bounding boxes to annotate surface defects of photovoltaic modules, and name the bounding boxes as defect types. After annotation, save the files as .xml format documents. Each document corresponds to one image and contains the category information ki for each defect, the x and y coordinates of the center point of the bounding box xi and yi, and the width and height of the bounding box wi and hi.

[0045] This approach improves detection accuracy, data diversity, annotation information richness, and scalability. By using high-resolution industrial cameras to acquire raw images of photovoltaic modules, the clarity and detail of the image data are ensured, which helps subsequent algorithms to more accurately identify and locate surface defects. Acquiring images at different times, angles, and lighting conditions increases the diversity of the dataset, thereby improving the robustness of the algorithm in various practical application scenarios. Using annotation software to annotate defects in detail, including defect type, location, and size, provides accurate supervision information for training deep learning or machine learning models. This is key to achieving high-precision automatic defect detection. Saving the annotation information in a structured XML format facilitates data storage, retrieval, and expansion, and also makes it easier for other researchers or engineers to reuse the dataset.

[0046] A13. Bind the original image and the .xml annotation file together, and name them the same;

[0047] A2. Divide the completed dataset to construct a training set, a validation set, and a test set for photovoltaic module surface defect detection.

[0048] The specific steps are as follows: In this embodiment, the dataset, validation set, and test set are divided in a 7:3:1 ratio (this can be adjusted appropriately in actual operation). The training set is used to train the neural network, thereby obtaining the weights of each neuron in the network. The validation set is used to verify the network's detection performance after each round of network training, and the hyperparameters in the model are adjusted based on the detection performance. The test set is used to evaluate the network's final prediction performance. By reserving a portion of the data as a test set, it can be used to measure the model's performance on unseen data. This helps to evaluate the model's generalization ability, i.e., the model's predictive ability for new samples. The test set does not participate in the training process and can be used for a fair performance evaluation after the model is finalized, avoiding performance evaluation bias caused by the model overfitting the training data. The validation set can be used for hyperparameter selection and early stopping of the model, thus avoiding overfitting of the model on the training set and finding the optimal model complexity and hyperparameter settings.

[0049] A3. Perform data augmentation preprocessing on the sample dataset;

[0050] Specifically, random brightness enhancement, image inversion, and cropping / stitching are used to augment labeled images, expanding the sample dataset and improving the generalization ability of the neural network. This makes the trained network parameters more suitable for real-world scenarios. By creating diverse data variations, data augmentation forces the model to learn a wider range of features, thus making it more robust when encountering novel data in the real world. By artificially expanding the training set, data augmentation reduces the model's dependence on a small number of specific samples, lowering the model's tendency to overfit the training data. More training samples usually provide richer information, helping the model capture more complex patterns and thus improving its prediction accuracy. The additional samples generated by data augmentation increase the diversity of the original dataset, which helps simulate various situations that may be encountered in real-world applications. Enhancement methods such as brightness adjustment and image inversion allow the model to maintain a good recognition rate when facing changes in lighting and differences in viewing angle.

[0051] A4. Use the data-augmented training set samples and validation set samples to train and validate the RetinaNet object detection network model;

[0052] The RetinaNet network model for object detection includes a feature extraction structure and a feature fusion structure. The network uses the feature extraction output as the input to the feature fusion model.

[0053] The specific operation involves inputting the augmented training set samples into the RetinaNet target detection network model, iteratively training it to extract features and classify defect targets in the dataset, and saving the training results for each round. In each training round, the saved network weights are used to test the validation set samples. The test results are analyzed, and the hyperparameters of the network model are adjusted, ultimately resulting in the optimized RetinaNet neural network for detecting surface defects in photovoltaic modules. As a deep learning model, RetinaNet, by learning from a large number of augmented data samples, can more accurately identify and locate defects on the surface of photovoltaic modules. Training with augmented data samples helps the model learn more types of defect features, enhancing its generalization detection ability for new samples. RetinaNet's feature extraction and feature fusion structures enable it to efficiently process image information, maintaining both high accuracy and fast detection speed. Using the validation set for testing in each round allows for timely identification of problems during model training and adjustment of hyperparameters to optimize model performance.

[0054] The training steps are as follows: input the enhanced training sample set into the RetinaNet network model for object detection, referring to... Figure 2This is a schematic diagram of the RetinaNet network structure, which is a classic single-stage network structure of Backbone + Neck + Head. The Backbone uses ResNet, the Neck uses FPN, and the Head consists of a class subnet and a box subnet. The ResNet structure has three convolutional kernels with sizes of 1×1, 3×3, and 1×1, which are used for dimensionality compression, convolution processing, and dimensionality restoration, respectively.

[0055] Specifically, the optimized RetinaNet network model for object detection is obtained through the following steps:

[0056] A41. The ECA (Efficient Channel Attention Network) attention mechanism is introduced to enhance the network's attention to small objects. The core idea of ​​ECANet is to introduce a channel attention mechanism into the convolutional operation, effectively capturing the relationships between channels in the image and improving feature representation capabilities. The goal of the channel attention mechanism is to adaptively adjust the weights of channel features, allowing the network to better focus on important features and suppress unimportant ones. Through this mechanism, ECANet can effectively enhance the network's representation capabilities without increasing the number of parameters or computational cost. The structure of ECANet is as follows... Figure 3 As shown, the input feature map size is

[0057] [H,W,C]. Global average pooling is used to compress the two-dimensional features of each channel into a single real number, transforming the feature map into a vector of [1,1,C]. Next, the size of the adaptive one-dimensional convolutional kernel is calculated based on the number of channels in the feature map.

[0058]

[0059] Where |t|odd represents the odd number closest to k, b=1, y=2, and C is the number of input feature channels. Weights are obtained through a convolutional kernel of size k, and after normalization, these weights are combined with the input feature map to finally generate the output feature map.

[0060] A42. When an image is input, the feature extraction network outputs Conv3Block, Conv4Block, and Conv5Block layers, which are then passed to the FPN module for feature fusion. C5 is the high-level output of the feature extraction network, directly outputting feature map P5 after two convolutions. However, P5 has already undergone multiple convolutions and contains very little target information, especially for small targets. Therefore, in this study, the FPN structure of the RetinaNet algorithm is improved using the PANet structure, adding a top-down feature fusion path to the FPN structure to improve information extraction capabilities. Subsampling weighting is applied to the feature maps C3_1, C4_1, and C5_1 output by the FPN. Based on this fusion method, the output of each layer is convolved to obtain the optimized FPN outputs P3-P7. Finally, the five feature maps P3-P7 are output to the classification and regression module for target prediction.

[0061] A43. Obtain the loss function of an existing RetinaNet network model;

[0062] In this embodiment, the loss function of the existing RetinaNet network model is obtained by summing the classification loss and the regression loss, i.e., L Z =L cls +L reg

[0063] The classification loss uses FocalLoss as the loss function, and the formula for calculating the classification loss function Lcls is as follows:

[0064] L cls (p t )=-α(1-p t ) γ logp t

[0065]

[0066] In the formula, pt is the score of the predicted box classification; α represents the parameter used to control the balance of positive and negative samples, with a value of 0.25; y represents the modulation factor parameter, which is a parameter to reduce the proportion of easily classified samples in the learning process, with a value of 2.

[0067] The regression loss uses smoothed L1 loss as the loss function.

[0068]

[0069] Here, x and y are the model's output and label, respectively, and |xy| represents the difference between them. When |xy| is less than 1, the squared error is used; otherwise, the linear error is used.

[0070] A44. Update the existing RetinaNet network parameters to reduce the total loss function, and iterate this process until the network converges to obtain the RetinaNet network model for object detection.

[0071] A5. Input the enhanced test sample set into the trained RetinaNet target detection network model. After the feature extraction and feature fusion stages, the model is detected and finally output by the classification and regression module to obtain the defect detection results.

[0072] Specifically, the enhanced test sample set is input into the trained improved RetinaNet network model for defect detection. The detection is divided into three stages: feature extraction, feature fusion, classification, and regression modules.

[0073] (1) Feature extraction module: Input the enhanced test sample set into the RetinaNet feature extraction model and output the defect feature information of each layer.

[0074] (2) Feature fusion module: It fuses bottom-up semantic data with top-down semantic data and utilizes the underlying positioning details.

[0075] (3) The fused feature layers are fed into the ClassSubnet and BoxSubnet. The ClassSubnet first performs four 3×3 convolution operations with a kernel of 256, and after each convolution, a ReLU activation function is used, followed by a 3×3 convolution operation with a kernel of K×A, where K is the number of target categories and A is the number of anchor boxes. Finally, the Softmax function is used to obtain the confidence of each anchor box. During training, anchor boxes that reach the confidence threshold are selected and FocalLoss is used to calculate the classification loss and optimize the network. The ClassSubnet and BoxSubnet do not share parameters, but the feature layers share parameters.

[0076] In summary, compared with traditional manual inspection methods, this invention improves the RetinaNet network by adopting a bidirectional fusion backbone structure to enhance the extraction capability of small targets, and introduces an ECA attention mechanism to propose an improved ResNet structure, thereby improving the network's accuracy and efficiency in identifying surface defects of photovoltaic modules.

[0077] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concept, should be covered within the scope of protection of the present invention.

Claims

1. A method for detecting surface defects in a single-stage photovoltaic module based on an improved RetinaNet, characterized in that, The method includes the following steps: A1. Use industrial cameras to collect images of photovoltaic modules and create a photovoltaic module image dataset; A2. Divide the sample dataset into training, testing, and validation sets; A3. Perform data augmentation preprocessing on the sample dataset; A4. Based on the enhanced image of the photovoltaic module with defects, surface defect detection is performed using the RetinaNet deep learning network model to obtain the detection results. The improved RetinaNet single-stage target detection network includes a feature extraction network and a bidirectional feature fusion structure. A5. Input the enhanced test sample set into the trained RetinaNet target detection network model. After the feature extraction stage and feature fusion stage, the detection is performed, and finally the defect detection result is output through the classification and regression module. In step A4, the improved RetinaNet network model is obtained by optimizing the existing RetinaNet network model, including the following steps: Obtain the loss function of an existing RetinaNet network model; In the existing RetinaNet network model feature fusion FPN structure, a bottom-up path is added to form a bidirectional feature fusion structure; The ResNet structure in the feature extraction network is improved by adding an ECA attention mechanism to the residual structure to enhance the ability to transmit deep semantic information and improve the detection accuracy of small targets. Update the existing RetinaNet network parameters to reduce the total loss function, and iterate this process until the network converges to obtain the RetinaNet network model for object detection. In step A5, the specific steps of the defect detection stage are as follows: The enhanced photovoltaic module image is input into the defect detection RetinaNet model. First, the input image is subjected to feature extraction. After passing through the PAN bidirectional feature fusion structure, it enters the classification module and regression module to obtain the training results, outputs candidate boxes, and then uses the maximum value suppression algorithm to filter the candidate boxes to obtain the final defect detection results. The defect detection results are then labeled in the original photovoltaic module image. The loss function consists of two parts: the classification loss function L. cls and regression loss function L reg The sum is obtained as L Z =L cls +L reg The classification loss uses Focal Loss as the loss function, calculating the classification loss for all positive and negative samples. The regression loss uses Smoothing L1 loss as the loss function, calculating the regression loss for all positive samples. ； The enhanced test sample set is input into the trained improved RetinaNet network model for defect detection. The detection is divided into three stages: feature extraction, feature fusion, classification and regression modules. (1) Feature extraction module: Input the enhanced test sample set into the RetinaNet feature extraction model and output the defect feature information of each layer; (2) Feature fusion module: fuses bottom-up semantic data with top-down semantic data, and utilizes the underlying positioning details; (3) The fused feature layers are fed into the classification sub-network and the bounding box sub-network. The classification sub-network first performs four 3×3 convolution operations with a kernel of 256, and after each convolution, a ReLU activation function is used, followed by a 3×3 convolution operation with a kernel of K×A, where K is the number of target categories and A is the number of anchor boxes. Finally, the Softmax function is used to obtain the confidence of each anchor box. During training, anchor boxes that reach the confidence threshold are selected and FocalLoss is used to calculate the classification loss and to learn and optimize the network. The classification sub-network and the bounding box sub-network do not share parameters, but the feature layers share parameters.

2. The method for detecting surface defects of a single-stage photovoltaic module based on improved RetinaNet according to claim 1, characterized in that: In step A1, original photovoltaic module images are acquired, and a sample dataset is created, including the following steps: A11. Use mechanical equipment and industrial cameras to detect surface defects in photovoltaic modules and take pictures of the photovoltaic modules to collect original images of the photovoltaic modules; A12. Annotate the acquired raw images, use different defect labels to annotate small units and defects, and generate a document containing .xml. A13. The generated .xml document containing annotation information and the original image are bound and saved in a txt file as a sample dataset.

3. The method for detecting surface defects of a single-stage photovoltaic module based on improved RetinaNet according to claim 1, characterized in that: In step A3, the data augmentation preprocessing uses random brightness enhancement, image inversion, and cropping and stitching methods to augment the labeled images, thereby expanding the sample dataset.

4. The method for detecting surface defects of a single-stage photovoltaic module based on improved RetinaNet according to claim 2, characterized in that: In step A4, the target detection RetinaNet network model is trained using the enhanced training set and validation set. The specific steps are as follows: input the enhanced training sample set into the improved RetinaNet network model, iteratively train the defect detection model, and save the trained network parameters. The parameters of the trained defect detection model were adjusted using a validation sample set to obtain the defect detection RetinaNet model.

Images