Photovoltaic module surface defect detection and grading method based on multi-modal deep learning

Abstract

The application relates to the technical field of surface defect detection, in particular to a photovoltaic module surface defect detection and grading method based on multi-modal deep learning, which collects photovoltaic module images and metadata containing defect information, operation parameters and environmental data, realizes the detection and instance segmentation of photovoltaic module surface defects by using an improved MaskR-CNN network through a two-stage deep learning architecture, extracts photovoltaic module image and photovoltaic module operation metadata features, fuses the features through an attention mechanism, and finally outputs the'slight-medium-severe' grading result through MLP classification decision. The application solves the problems of low detection precision, lack of defect grading capability and poor adaptability to complex environments in the prior art, realizes automatic and high-precision defect recognition and grading, and is suitable for large-scale photovoltaic power station operation and maintenance management.

Description

Technical Field

[0001] This invention relates to the field of surface defect detection technology. Background Technology

[0002] In the operation and maintenance of photovoltaic power plants, defect identification is a crucial step in ensuring the efficient and stable operation of the system. With the widespread application of photovoltaic power generation systems, the demand for automated detection of surface defects in modules is increasing. Traditional manual inspection methods are not only inefficient but also susceptible to subjective judgment errors, making it difficult to meet the real-time monitoring requirements of large-scale photovoltaic power plants.

[0003] In recent years, deep learning technology has made significant progress in the field of industrial vision inspection and has been widely applied to various surface defect detection tasks. However, defect detection of photovoltaic modules still faces many challenges. First, the types of defects are diverse. Defects that may exist on the surface of photovoltaic modules include cracks, stains, aging, hot spots, etc., and their shapes, sizes, and locations vary, increasing the difficulty of detection. Second, the lighting conditions are complex. In outdoor environments, the light intensity varies greatly, and shadow interference is severe, affecting image quality and thus detection accuracy.

[0004] Currently, some research has attempted to use convolutional neural networks (CNNs) for defect detection in photovoltaic modules, mainly focusing on image classification or object detection methods. For example, some studies use YOLO or Faster R-CNN for defect localization, while others use semantic segmentation networks (such as U-Net) for pixel-level defect recognition. However, most of these methods only focus on the presence or absence of defects and fail to further classify their severity effectively.

[0005] Furthermore, existing technologies typically do not fully utilize multi-source information fusion strategies. In addition to image data, operating parameters of photovoltaic modules (such as temperature, current, and voltage) and environmental data (such as light intensity and humidity) can also be used as auxiliary features to improve the accuracy and robustness of defect identification, but most current research has not yet effectively fused these metadata with image features.

[0006] In summary, existing photovoltaic module defect identification methods have the following shortcomings: (1) lack of comprehensive analysis capabilities for defect types and severity; (2) poor adaptability to complex lighting conditions and background noise; (3) failure to fully utilize multimodal data (images + metadata) for joint modeling; and (4) lack of efficient model deployment schemes to support real-time detection in large-scale scenarios. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, this invention provides a method for detecting and classifying surface defects in photovoltaic modules based on multimodal deep learning, comprising the following steps:

[0008] S1. Collect images and operational metadata of photovoltaic modules;

[0009] S2. Input the photovoltaic module image into the improved MaskR-CNN network to obtain defect instance information and output the type, location, and area of ​​surface defects of the photovoltaic module.

[0010] S3. Based on defect instance information, extract image features and simultaneously extract feature information from runtime metadata. Use an attention mechanism to achieve multimodal fusion of the two types of features and obtain a fused feature vector.

[0011] S4. Input the fused feature vector into the MLP classification decision module for classification decision, and output the defect severity level after processing by the cross-entropy loss function.

[0012] Preferably, in step S1, a high-resolution industrial camera is used to acquire images of the photovoltaic module with a resolution of 1024×1024 pixels. The metadata includes defect coordinates, defect size, defect area, module model, voltage, temperature, current, light intensity, humidity, vertical position, and horizontal sector number.

[0013] Preferably, step S1 further includes preprocessing the photovoltaic module image by using ColorJitter to perturb the image's color, randomly changing the image's brightness, contrast, saturation, and hue, standardizing the pixel values ​​to the range of [0,1], and subtracting the mean and dividing by the standard deviation for each channel.

[0014] Preferably, in step S2, improving the MaskR-CNN network includes:

[0015] Two new branches have been added: MaskIoU prediction branch and BboxIoU prediction branch. The MaskIoU prediction branch is used to predict the intersection-union ratio between the generated mask and the ground truth label, and the BboxIoU prediction branch is used to predict the intersection-union ratio between the bounding box and the ground truth box.

[0016] The backbone network uses ResNet-50 or ResNet-101 and combines it with Feature Pyramid Network (FPN) to extract multi-scale features; candidate regions are generated through region proposal network, and after RoIAlign layer maps RoIs of different sizes to feature maps of the same scale, the classification head predicts the category of each RoI, the bounding box regression head optimizes the location information of RoI, and the masking head generates the binary mask corresponding to each RoI.

[0017] During network training, the weights trained using photovoltaic module-related images from the COCO dataset are used to initialize the network parameters. A regional fine-tuning strategy is adopted to train the model separately for the frame, cell, and solder ribbon regions of the photovoltaic module.

[0018] In the inference and post-processing stages, GPU-accelerated online inference is used, and redundant detection results are removed through threshold filtering, non-maximum suppression, and IoU-guided screening.

[0019] Preferably, in step S2, the overall loss function of the improved MaskR-CNN network is composed of a weighted sum of multiple sub-losses:

[0020] ;

[0021] in, For RPN classification loss, For RPN regression loss, Classification loss for the target For bounding box regression loss, For mask loss, For BboxIoU losses, Due to Mask IoU loss, For weights.

[0022] Preferably, in step S3, image feature extraction includes cropping image blocks of defective regions, encoding them using a ResNet18 model, and capturing the spatial relationships of image blocks using a Bi-GRU; feature extraction of metadata includes embedding categorical features, normalizing and linearly transforming numerical features, and extracting structured features through an MLP containing Dropout and BatchNorm layers.

[0023] Preferably, in step S3, the attention mechanism formula is:

[0024] ;

[0025] in, For query vectors, For key vectors, These are value vectors, all generated by linear transformations of the input sequence;

[0026] Cross-modal interaction is achieved by applying a cross-attention mechanism: image features are used as query vectors, metadata features are used as key vectors and value vectors, allowing image information to guide metadata filtering; and metadata features are used as query vectors, image features are used as key vectors and value vectors, allowing metadata to help filter the image regions most relevant to the current defect type.

[0027] Preferably, in step S4, cross-entropy loss is used for calculation:

[0028] ;

[0029] in, For the sample size, For severity levels, For the first The sample belongs to the first The true label of the class, It is a predicted probability.

[0030] The beneficial effects of this invention are as follows:

[0031] This invention integrates image features and equipment operation metadata to construct an end-to-end process from defect detection to severity assessment, achieving automated and high-precision defect identification and grading, and improving adaptability to complex lighting conditions and background noise. It employs an improved MaskR-CNN architecture, adding MaskIoU and BboxIoU prediction branches, combined with a region-specific fine-tuning training strategy to adapt to texture differences in different regions of photovoltaic modules, improving model robustness. This enables high-precision identification of various defect types, such as cracks, stains, and hot spots, even in real-world scenarios with uneven lighting and complex backgrounds. Furthermore, by combining image features with photovoltaic module operation metadata, this invention achieves dynamic cross-modal information interaction through an attention mechanism, guiding image and tabular data to enhance key feature extraction capabilities and improve the scientific rigor and interpretability of defect severity assessment, supporting user-defined grading standards. Finally, this invention constructs a two-stage end-to-end architecture, from raw image and metadata input to final defect grading output, requiring no manual intervention throughout. It can be deployed on edge computing devices for real-time detection, adapting to the operation and maintenance needs of large-scale photovoltaic power plants and improving maintenance efficiency. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the overall process of an embodiment of the present invention. Detailed Implementation

[0033] Embodiments of the present invention provide a method for surface defect detection and classification of photovoltaic modules based on multimodal deep learning, such as... Figure 1 As shown, it includes the following steps:

[0034] S1. Collect photovoltaic module images and operational metadata, specifically:

[0035] Images of photovoltaic modules taken using high-resolution industrial cameras, with a resolution of 1024×1024 pixels, can clearly capture various defect details on the surface of photovoltaic modules.

[0036] The photovoltaic module images are preprocessed by using ColorJitter to perturb the colors of the images. By randomly changing the brightness, contrast, saturation and hue of the images, different lighting conditions are simulated to improve the robustness of the model in complex lighting environments. Robustness is the ability to cope with complex lighting problems. At the same time, the pixel values ​​are standardized to the range of [0,1] and the mean is subtracted and the standard deviation is divided by each channel to unify the data scale.

[0037] The runtime metadata includes information such as defect coordinates, defect size, defect area, component model, voltage, temperature, current, light intensity, humidity, vertical position, and horizontal sector number.

[0038] S2. Input the preprocessed photovoltaic module image into the improved MaskR-CNN network. The improved MaskR-CNN network includes MaskIoU and BboxIoU prediction branches, a ResNet+FPN backbone network, and a region-based fine-tuning training strategy to obtain defect instance information and output the type, location, and area of ​​defects on the photovoltaic module surface, specifically:

[0039] Improvements to the Mask R-CNN network include:

[0040] The improved MaskR-CNN network is optimized based on the standard MaskR-CNN by adding MaskIoU prediction branch and BboxIoU prediction branch. The MaskIoU prediction branch is used to predict the intersection-union ratio (IoU) between the generated mask and the ground truth label to measure the accuracy of the segmentation result. The BboxIoU prediction branch is used to predict the IoU between the bounding box and the ground truth box to help filter high-quality detection results. Together, they provide a more reliable confidence index and reduce false detections.

[0041] The structure is improved by using ResNet-50 or ResNet-101 in the backbone network and combining it with FeaturePyramid Network (FPN) to extract multi-scale features. Candidate regions are generated through Region Proposal Network (RPN). After RoIAlign layer maps RoIs of different sizes to feature maps of the same scale, the classification head predicts the category (defect / non-defect) of each RoI, the bounding box regression head optimizes the position information of RoI, and the masking head generates a binary mask for each RoI to achieve pixel-level defect segmentation.

[0042] The training strategy was improved by using weights from photovoltaic module-related images in the COCO dataset to initialize network parameters during network training, thereby accelerating convergence and improving generalization ability. Considering the significant differences in defect features across different regions of the photovoltaic module (such as borders, cells, and solder strips), a zone-by-zone fine-tuning strategy was adopted to train the model separately for each region, adapting it to local texture features and defect distribution characteristics.

[0043] Design a loss function, the overall loss function is composed of a weighted sum of multiple sub-losses:

[0044] ;

[0045] in, The classification loss for RPN is cross-entropy loss. This is the RPN regression loss (Smooth L1 Loss). The target classification loss is (cross-entropy loss). The bounding box regression loss is the same as the RPN regression loss. This is the mask loss (binary cross-entropy loss). This is the BboxIoU loss (mean squared error). This is the Mask IoU loss (same as Bbox IoU loss). The weights are dynamically adjusted during training based on the convergence of each sub-loss. By jointly optimizing the word loss, the coordinated learning of all components of the model is ensured, improving the accuracy of defect localization, classification, and segmentation, while also enhancing the self-evaluation capability of prediction quality.

[0046] Mask R-CNN is a multi-task learning framework that simultaneously performs multiple sub-tasks, including Region Proposal Generation (RPN), object classification, precise bounding box localization (regression), and pixel-level segmentation mask generation. Each sub-loss function corresponds to and optimizes a specific functional module or output of the model. In addition, the newly added BboxIoU loss and MaskIoU loss introduce direct supervision signals for prediction quality (accuracy of bounding boxes and masks). Therefore, combining these sub-losses ensures that all components of the model learn in the right direction, achieving coordinated and efficient overall performance, ultimately outputting accurate defect locations (bounding boxes), shapes (masks), types (classification), and an evaluation of the quality of these predictions (IoU prediction).

[0047] Improvements to the inference and post-processing stages: GPU-accelerated online inference is adopted to support real-time detection; redundant detection results are removed through threshold filtering, non-maximum suppression (NMS), and IoU-guided filtering, and finally, information such as the bounding box of the defect instance, pixel-level defect mask, classification probability, MaskIoU predicted value, BboxIoU predicted value, defect area, defect center coordinates, and sector number are output.

[0048] S3. Based on defect instance information, extract image features; extract feature information from runtime metadata, and achieve multimodal fusion of the two types of features through an attention mechanism to obtain a fused feature vector, specifically:

[0049] During image feature extraction, the corresponding defect region image patch is first cropped from the defect bounding box output in step S2. Then, a pre-trained ResNet18 model is used to encode the image patch and extract high-level semantic features (512 dimensions). Finally, a bidirectional gated recurrent unit (Bi-GRU) is introduced to capture the spatial relationship between multiple image patches, outputting an image feature vector with a dimension of 256. ,in, The number of image patches, This is a feature dimension used to enhance the model's understanding of defect distribution patterns.

[0050] When extracting feature information from runtime metadata, categorical features are embedded (dimensions 16-128), and numerical features are normalized and linearly transformed. All processed features are concatenated and fed into an MLP containing Dropout and BatchNorm layers to extract structured features and output a tabular feature vector. ,in, The total dimension of the features; the extracted feature names and corresponding descriptions are shown in the table below:

[0051]

[0052] To effectively integrate image and tabular feature data, multimodal fusion employs an attention mechanism to achieve cross-modal information interaction. The core formula of the attention mechanism is:

[0053] ;

[0054] in, For query vectors, For key vectors, The values ​​are vectors, all generated by linear transformations of the input sequence. Self-attention mechanisms are applied to image features and table features respectively to focus on key information within each modality. Specifically, self-attention is applied to image patch sequences to focus on the most discriminative local regions, and self-attention is applied to table features to filter the variables most important for severity judgment.

[0055] Cross-modal interaction is achieved by applying a cross-attention mechanism, using image features as... Table features as and This allows image information to guide metadata filtering, while using metadata as... Image features as and This allows metadata to help filter image regions most relevant to the current defect type, and this two-way interaction mechanism enhances the model's ability to understand complex defect patterns. Finally, the fused features are concatenated and input into a fully connected classification head to obtain a fused feature vector. ,in, To fuse feature dimensions.

[0056] S4. Input the fused feature vector into the MLP classification decision module for classification decision, and output the defect severity level after processing by the cross-entropy loss function.

[0057] The classification decision module uses cross-entropy loss for calculation:

[0058] ;

[0059] in, For the sample size, For severity levels, For the first The sample belongs to the first The true label of the class, It is a predicted probability.

[0060] In this embodiment, the output severity level includes three levels: "Slight," "Moderate," and "Severe," and more levels can be supported according to actual needs. The severity level is determined based on the comprehensive analysis results of these multimodal input features by professionals. The specific conditions for outputting "Slight," "Moderate," or "Severe" are jointly determined by the severity labels marked in the training data and the feature weights learned by the model. For example, a large crack defect located in a critical position that causes a significant drop in voltage is classified as "Severe"; a small stain located at the edge and having a negligible impact on operating parameters is classified as "Slight."

[0061] Furthermore, during model optimization, SHAP values ​​are used to analyze important features and assist in model interpretation; Optuna is used for automated hyperparameter search, with F1-score as the primary evaluation metric, adapting to class imbalance scenarios. This method can be deployed on edge computing devices to achieve real-time defect detection and classification, and supports integration with operations and maintenance platforms to provide data support for maintenance priority ranking and resource scheduling.

[0062] This invention has been described through embodiments. Those skilled in the art will understand that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of this invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, this invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of this invention.

Claims

1. A method for detecting and classifying surface defects in photovoltaic modules based on multimodal deep learning, characterized in that, Includes the following steps: S1. Collect images and operational metadata of photovoltaic modules; S2. Input the photovoltaic module image into the improved MaskR-CNN network to obtain defect instance information and output the type, location, and area of ​​surface defects of the photovoltaic module. S3. Based on defect instance information, extract image features and simultaneously extract feature information from runtime metadata. Use an attention mechanism to achieve multimodal fusion of the two types of features and obtain a fused feature vector. S4. Input the fused feature vector into the MLP classification decision module for classification decision, and output the defect severity level after processing by the cross-entropy loss function.

2. The method for surface defect detection and classification of photovoltaic modules based on multimodal deep learning according to claim 1, characterized in that, In step S1, a high-resolution industrial camera is used to acquire images of the photovoltaic module with a resolution of 1024×1024 pixels. The metadata includes defect coordinates, defect size, defect area, module model, voltage, temperature, current, light intensity, humidity, vertical position, and horizontal sector number.

3. The method for surface defect detection and classification of photovoltaic modules based on multimodal deep learning according to claim 1, characterized in that, Step S1 also includes preprocessing the photovoltaic module image by using ColorJitter to perturb the image color, randomly changing the image brightness, contrast, saturation and hue, standardizing the pixel values ​​to the range of [0,1], and subtracting the mean and dividing by the standard deviation for each channel.

4. The method for surface defect detection and classification of photovoltaic modules based on multimodal deep learning according to claim 1, characterized in that, In step S2, the improvements to the Mask R-CNN network include: Two new branches have been added: MaskIoU prediction branch and BboxIoU prediction branch. The MaskIoU prediction branch is used to predict the intersection-union ratio between the generated mask and the ground truth label, and the BboxIoU prediction branch is used to predict the intersection-union ratio between the bounding box and the ground truth box.

5. The method for surface defect detection and classification of photovoltaic modules based on multimodal deep learning according to claim 1, characterized in that, In step S2, the improvements to the Mask R-CNN network include: The backbone network uses ResNet-50 or ResNet-101 and combines it with Feature Pyramid Network (FPN) to extract multi-scale features; candidate regions are generated through region proposal network, and after RoIAlign layer maps RoIs of different sizes to feature maps of the same scale, the classification head predicts the category of each RoI, the bounding box regression head optimizes the location information of RoI, and the masking head generates the binary mask corresponding to each RoI. During network training, the weights trained using photovoltaic module-related images from the COCO dataset are used to initialize the network parameters. A regional fine-tuning strategy is adopted to train the model separately for the frame, cell, and solder ribbon regions of the photovoltaic module. In the inference and post-processing stages, GPU-accelerated online inference is used, and redundant detection results are removed through threshold filtering, non-maximum suppression, and IoU-guided screening.

6. The method for surface defect detection and classification of photovoltaic modules based on multimodal deep learning according to claim 1, characterized in that, In step S2, the overall loss function of the improved Mask R-CNN network is a weighted sum of multiple sub-losses. composition: ； in, For RPN classification loss, For RPN regression loss, Classification loss for the target, For bounding box regression loss, For mask loss, For BboxIoU losses, Due to Mask IoU loss, For weights.

7. The method for surface defect detection and classification of photovoltaic modules based on multimodal deep learning according to claim 1, characterized in that, In step S3, image feature extraction includes cropping image blocks of defective regions, encoding them using a ResNet18 model, and capturing the spatial relationships of image blocks using a Bi-GRU. Feature extraction of metadata includes embedding categorical features, normalizing and linearly transforming numerical features, and extracting structured features using an MLP with Dropout and BatchNorm layers.

8. The method for detecting and classifying surface defects of photovoltaic modules based on multimodal deep learning according to claim 1, characterized in that, In step S3, the formula for the attention mechanism is: ； in, For query vectors, For key vectors, These are value vectors, all generated by linear transformations of the input sequence.

9. The method for detecting and classifying surface defects of photovoltaic modules based on multimodal deep learning according to claim 8, characterized in that, In step S3, the multimodal fusion of the two types of features through the attention mechanism is specifically achieved by: applying a cross-attention mechanism, using image features as the query vector and metadata features as the key vector and value vector; and using metadata features as the query vector and image features as the key vector and value vector; to achieve multimodal fusion of the two types of features and obtain a fused feature vector.

10. The method for detecting and classifying surface defects of photovoltaic modules based on multimodal deep learning according to claim 1, characterized in that, In step S4, cross-entropy loss is used for calculation: ； in, For the sample size, For severity levels, For the first The sample belongs to the first The true label of the class, It is a predicted probability.

Images