Photovoltaic panel defect detection method, system, equipment and medium

Abstract

The invention discloses a photovoltaic panel defect detection method, system and device and a medium, and relates to the technical field of photovoltaic defect detection.The method comprises the steps that an infrared image and a visible light image of a photovoltaic module are obtained; visual features are extracted and fused; marking defect areas in the fused image data set, storing marking information of the defect areas, and constructing a training data set; inputting the training data set into a CLIP model for processing, and generating embedded representation; adopting the embedded expression to initialize the layer weight of the YOLOv11 model, and performing fine adjustment on the YOLOv11 model; inputting image data of a to-be-detected photovoltaic module into the trained photovoltaic defect detection model, and detecting to obtain a defective area in the image data; according to the method, the problem that a large amount of data for photovoltaic detection cannot be obtained in a remote area, so that model training is difficult is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of photovoltaic defect detection, and in particular to a photovoltaic panel defect detection method, system, equipment and medium. Background Art

[0002] Photovoltaic (PV) systems play a pivotal role in the global transition to renewable energy as a sustainable and environmentally friendly source of electricity. Recent advances in deep learning and computer vision show great promise in automating the detection of defects in photovoltaic (PV) panels. Object recognition algorithms are specifically designed to accurately identify and classify defects in PV.

[0003] The detection methods currently used include YOLO target detection: collecting and labeling photovoltaic panel image data, building a training environment, configuring the YOLO model and training it, evaluating the model performance through the validation set and optimizing it, testing the model to ensure its accuracy and then deploying it to the actual detection system, and finally continuously monitoring the model performance and providing necessary technical support and maintenance.

[0004] However, PV defect detection must effectively handle a wide range of detection, identify various types of defects, and provide high-precision real-time processing; but the currently used models require a large amount of labeled data sets for training, are usually tailored for specific tasks, and cannot perform satisfactorily in different real-world scenarios; especially in remote areas, it becomes more difficult to detect environmental changes, and it is impossible to obtain a large amount of data for photovoltaic detection, resulting in the inability of the trained models to achieve large-scale and detailed detection. Summary of the invention

[0005] In view of the problem that the existing technology cannot obtain a large amount of data for photovoltaic detection, resulting in the inability of the trained model to achieve large-scale and detailed detection, the present invention proposes a photovoltaic panel defect detection method, system, device and medium, thereby initializing the YOLOv11 layer by utilizing the multimodal understanding capability of CLIP, and performing fine-tuning training on a small amount of target domain data, thereby solving the problem of difficulty in model training due to the lack of a large amount of labeled data.

[0006] A photovoltaic panel defect detection method comprises the following steps: Acquire image data containing different lighting conditions, different defect types, and different photovoltaic module types; the image data includes a plurality of infrared images and a plurality of visible light images; Extract visual features from each infrared image and visible light image, and fuse all visual features to obtain a fused image dataset; annotate defective areas in the fused image dataset, and save text information of the annotated defective areas; construct a training dataset using the fused image dataset and text information of the annotated defective areas; Input the training data set into the CLIP model, use a text editor to process the text information of the marked defect area, and generate a coding vector ; Use the image encoder to process the fused image data set and generate the encoding vector ; According to the encoding vector and the encoding vector Generate an embedded representation; use the embedded representation to initialize the layer weights of the YOLOv11 model, adjust the first layer of the YOLOv11 model through the initialized layer weights, and obtain a photovoltaic defect detection model; The visual features of the infrared image and the visible light image of the photovoltaic module to be tested are fused, and the fused image data is input into the photovoltaic defect detection model to detect the defective area in the image data.

[0007] Furthermore, before extracting the visual features of each infrared image and visible light image, the infrared image and the visible light image are aligned using a SIFT algorithm, which specifically includes the following steps: The scale of each infrared image and visible light image is continuously transformed using a Gaussian function; the Gaussian function G ( x , y , s ) is expressed as: ; in( x , y ) represents pixel coordinates; s is the scale parameter, which determines the blurriness of the image; For the original image I ( x , y ) is different from s The convolution operation is performed on the Gaussian function of the value. The formula for the convolution operation is: ; in, L ( x , y , s ) indicates that the scale s The image below shows that * represents the convolution operation; By changing sThe value of is used to obtain a series of images with different blur levels, thus forming a multi-scale spatial sequence; The local extreme points of the multi-scale space sequence are detected by the Gaussian difference DOG method. Specifically, for each pair of adjacent images in the multi-scale space sequence, the difference between them is calculated to obtain the DOG response value image : ; in, k is a constant that is a multiple of the adjacent scale space; In the DOG response value image, local extreme points are detected by comparing each image pixel with the pixel values ​​in its neighborhood; The detected local extreme points are located, and the key points with contrast lower than the set value and unstable edge response points are removed by fitting a three-dimensional quadratic function; and the gradient of the pixels in the neighborhood of the feature point is used to determine its direction parameter; By statistically analyzing the gradient histogram, the stable direction of the local structure of the key point is found; based on the position and direction of the image gradient in the neighborhood of each key point, a 128-dimensional feature descriptor is constructed; The infrared image and the visible light image are aligned according to the feature descriptor.

[0008] Furthermore, a dual encoder is used to process each infrared image and visible light image in parallel to obtain the visual features in the two images; a deep interpretable iterative algorithm is used to fuse all the visual features to obtain a fused image dataset.

[0009] Furthermore, the method further includes optimizing the fused image dataset by using the image fusion module loss and the semantic segmentation module loss before constructing the training dataset; wherein the image fusion module loss L F It is expressed as: ; in, H and W Represents the height and width of the image respectively, Target( ir , vi ) represents the fusion target image, The cross entropy loss is used as the semantic segmentation module loss, and its formula is: ; in, ( h , w )and ( h , w ) respectively represent the positions ( h ,w ) Category c The predicted labels and true labels of It is the set of all categories in the semantic segmentation task; is an index variable whose value ranges from 0 to " classes -1”; represents the cross entropy loss function; The obtained image fusion module loss and semantic segmentation module loss are minimized to optimize the fused image dataset; the minimization process is specifically expressed as: ; Among them, arg min means finding the optimal model parameters by minimizing the loss function; represents the L1 norm; HW represents the product of the image height H and width W, and represents the total number of pixels in the image; L F Represents the total image fusion loss, which is composed of intensity loss and loss of detail Weighted composition; Φ( ir , vi ) represents the fusion model, input infrared image ir and visible light images vi , output fused image f - max ( ir , vi ); , is the normalized weight parameter of the fusion loss, which is used to adjust the loss magnitude; G(⋅) is the gradient extraction operator; α represents the coefficient for balancing the intensity loss and detail loss; L S represents semantic segmentation loss, using the cross entropy loss function; Ψ( f ) represents the semantic segmentation model, and the input fusion image f , output semantic segmentation results t f ; l The coefficient to balance the fusion loss and semantic segmentation loss; L cross-entropy is the cross entropy loss function; c ∈[0, classes ) means traversing all semantic categories, classes kind; f =Φ( ir , vi ) represents the fused image f must be generated by the fusion model Φ, t f =Ψ( f ) represents the semantic segmentation resultt f must be generated by the segmentation model Ψ.

[0010] Further, the layer weights of the YOLOv11 model are initialized by using the embedded representation, and the first layer of the YOLOv11 model is adjusted by the initialized layer weights, specifically comprising the following steps: The output of the text encoder and the image encoder output Add together and get e = + ; e After re-adjusting the weights, N weights are generated; the dimension after re-adjusting the weights is D=h*w*c, where h is the height, w is the width, and c is the number of channels; specifically: e = + =CLIP_Embed(X i ,Y i );X i Represents the image information input to the CLIP model, Y i Represents text information input into the CLIP model; according to e Generate the final weight W1, and use W1 to adjust the first layer of YOLOv11.

[0011] The present invention also provides a photovoltaic panel defect detection system, comprising: An acquisition module, used to acquire image data containing different lighting conditions, different defect types and different photovoltaic module types; the image data includes a plurality of infrared images and a plurality of visible light images; The training set construction module is used to extract the visual features of each infrared image and visible light image, and fuse all the visual features to obtain a fused image data set; annotate the defective areas in the fused image data set, and save the text information of the annotated defective areas; and construct a training data set through the fused image data set and the text information of the annotated defective areas; The training module is used to input the training data set into the CLIP model, use a text editor to process the text information of the marked defect area, and generate a coding vector ; Use the image encoder to process the fused image data set and generate the encoding vector ; According to the encoding vector and the encoding vector Generate an embedded representation; use the embedded representation to initialize the layer weights of the YOLOv11 model, adjust the first layer of the YOLOv11 model through the initialized layer weights, and obtain a photovoltaic defect detection model; The detection module is used to fuse the visual features of the infrared image and the visible light image of the photovoltaic module to be tested, input the fused image data into the photovoltaic defect detection model, and detect the defective area in the image data.

[0012] The present invention also provides a photovoltaic panel defect detection computer device, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor implements the steps of the photovoltaic panel defect detection method when executing the computer program.

[0013] The present invention also provides a readable storage medium, wherein the readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, they are used to execute the steps of the photovoltaic panel defect detection method.

[0014] The present invention provides a photovoltaic panel defect detection method, which has the following beneficial effects: The present invention collects infrared and visible light images of various lighting conditions, defect types, and different types of photovoltaic modules, and extracts visual features for efficient fusion, thereby generating high-quality images with rich details and high information content, thereby improving the accuracy of the trained photovoltaic defect detection model; at the same time, considering the problem of limited labeled data in photovoltaic defect detection, an architecture combining the CLIP model with the YOLOv11 model is proposed, and the embedded representation for initializing the YOLOv11 layer weights is generated by processing the fused image data set and the text information of the labeled defect area using CLIP, and fine-tuning training is performed on a small amount of target domain data, solving the problem of difficulty in model training due to the inability to obtain a large amount of data for photovoltaic detection in remote areas. This method not only improves the detection performance of the model, but also greatly shortens the training time, providing a more efficient and reliable solution for the maintenance and monitoring of photovoltaic systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of a photovoltaic panel defect detection method according to an embodiment of the present invention; Figure 2 Schematic diagram of the architecture of a multimodal data fusion module in an embodiment of the present invention; Figure 3 Flow chart of initialization of YOLOv11 in an embodiment of the present invention; Figure 4 Schematic diagram of the YOLOv11 fine-tuning process in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0017] The present invention proposes a photovoltaic panel defect detection method, which uses CLIP+YOLO+multimodal data fusion method to perform photovoltaic defect detection; the method ensures the consistency of resolution and size of infrared and visible light images through image alignment processing, and uses SIFT algorithm to eliminate shooting angle and viewing angle differences; then, through multimodal data fusion technology, images of different sensors are received, dual private encoders are used to extract features, and feature fusion is performed using a deep interpretable iterative algorithm, and finally the image is optimized and reconstructed through a specific loss function. In the data set annotation and division link, the labelme tool is used to annotate the fused data set, and the annotation information is converted into a format acceptable to YOLO, and the training set, validation set and test set are divided according to the ratio of 8:1:1. In terms of model building, combining the fast target detection capability of YOLO and the multimodal understanding capability of CLIP, the layer weights of YOLOv11 are initialized through CLIP, and fine-tuned to improve the performance in small sample learning scenarios; the method specifically includes the following steps: S1. Obtain infrared and visible light images of the same scene with consistent image resolution, size, and other parameters. These images cover various lighting conditions, defect types, and different photovoltaic module types.

[0018] S2. Image alignment processing: Use the SIFT (Scale Invariant Feature Transform) feature matching algorithm to align infrared and visible light images to eliminate image misalignment caused by differences in shooting angles, viewing angles, etc.

[0019] SIFT is a descriptor used in the field of image processing. It is scale invariant and has good stability to object scale changes, rigid body transformations, light intensity and occlusion. It can detect key points in an image and is a local feature descriptor. The process of processing an image using the SIFT algorithm includes the following steps: The Gaussian blur function is used to continuously transform the scale parameters of the infrared image and the visible light image to obtain a multi-scale spatial sequence. G ( x , y , s ) is used to construct the scale space, where ( x , y ) represents pixel coordinates, s is the scale parameter, which determines the blur degree of the image. The specific formula of the Gaussian function is: (1); Formula (1) describes how to s The weighted value of each pixel in the image is calculated to achieve image blurring.

[0020] In order to obtain a multi-scale spatial sequence, the original image needs to be I ( x , y ) is different from s The convolution operation is performed on the Gaussian function of the value. The formula for the convolution operation is: (2); in, L ( x , y , s ) indicates that the scale s The image below shows that * indicates the convolution operation. By changing s By adjusting the value of , a series of images with different blur levels can be obtained, thus forming a multi-scale spatial sequence.

[0021] In these scale spaces, the local extreme points are detected by the Difference of Gaussian (DOG) method, which are potential feature points. For each pair of adjacent images in the multi-scale space sequence, the difference between them is calculated to obtain the DOG response value image, in which each pixel value reflects the degree of difference between the images at adjacent scales.

[0022] DOG response value image: (3); in, k is a constant that is a multiple of the adjacent scale space, usually taken as .

[0023] In the DOG response value image, local extreme points are detected by comparing each pixel with the pixel values ​​in its neighborhood. Local extreme points include local maximum and local minimum, which are usually regarded as potential feature points.

[0024] The detected extreme points are precisely located, and low-contrast key points and unstable edge response points are removed by fitting a three-dimensional quadratic function. The gradient of the pixels in the neighborhood of the feature point is used to determine its direction parameter. By statistically analyzing the gradient histogram, the stable direction of the local structure of the key point is found, which helps to achieve the rotation invariance of the image. For each key point, a 128-dimensional feature descriptor is constructed. The descriptor is based on the position and direction of the image gradient in the neighborhood of the key point, and has rotation invariance and high uniqueness.

[0025] S3, multimodal data fusion, such as Figure 2 As shown in Figure 1, its core goal is to efficiently fuse infrared images (IR) and visible light images (VI) to generate a high-quality image that contains rich details and high information content. The module includes a dual encoder for extracting features, a fusion strategy for promoting feature fusion, and a decoder for reconstructing the results.

[0026] S3.1. In the data input stage, infrared images and visible light images from different sensors are received. These two images have their own characteristics. Infrared images are good at capturing the thermal radiation information of objects, which is particularly effective for object recognition at night or in low light conditions. Visible light images are good at their rich colors and detailed information, and can clearly show the texture and structural characteristics of objects. To ensure the smooth progress of the fusion process, the two images will be aligned (as shown in S2), thereby eliminating the adverse effects caused by differences in shooting conditions or equipment, and laying a solid foundation for subsequent feature extraction and fusion.

[0027] S3.2, feature extraction module, this module adopts a dual private encoder architecture, this design is to make full use of the unique information of infrared and visible light images. Each encoder is specially trained to accurately capture the visual features in the image; visual features include high-frequency and low-frequency features. High-frequency features mainly include tiny information such as edges and details, while low-frequency features cover more macro information such as contours and backgrounds. Through the parallel processing of dual encoders, the system can simultaneously obtain important visual features in both images, providing comprehensive and detailed information support for subsequent feature fusion.

[0028] S3.3, in the feature fusion stage, a deep interpretable iterative algorithm is introduced. This algorithm achieves deep integration of the two modal features by cascading multiple small convolution filters (such as 3x3 convolution kernels). The activation function uses the Sigmoid function. This design not only reduces the spatial dimension of the parameters and the complexity of the model, but also enables the network to capture key information in the image more efficiently during the training process. Through iterative optimization, the algorithm can gradually optimize the fusion results to ensure that the output image retains details while taking into account the integrity of the overall information.

[0029] S3.4. Generate the final fused image through the reconstruction step. A specific loss function is used to guide the reconstruction process to ensure that the output image is close to the optimal combination of infrared and visible light images in terms of visual effect. This loss function combines the fusion module loss and the segmentation module loss. The loss function of the segmentation module (cross entropy loss) is introduced to improve the performance of the fused image in high-level semantic tasks. By introducing the semantic segmentation task, the model can pay more attention to the semantic structure in the image, thereby improving the performance of the fused image in subsequent high-level semantic tasks. This design aims to ensure that the final generated image not only meets the visual quality requirements, but also supports the execution of high-level semantic tasks. Through the combination of adaptive learning and regularization constraints, the deep interpretable iterative algorithm has demonstrated excellent performance in the image fusion process, bringing new breakthroughs to the field of image fusion.

[0030] Image fusion module loss L F Focus on improving the visual effect of the fused image, ensuring that the fused image retains the thermal radiation information of the infrared image and integrates the rich colors and details of the visible light image.

[0031] The loss function can be designed as: (4); in, H and W Represents the height and width of the image respectively. Target ( ir , vi ) represents the fusion target image. Here, the maximum value in the input image is used as the fusion target, that is, max( ir , vi ). This design aims to ensure that the fused image is close to the optimal value in the input image in terms of brightness while retaining important detail information, thereby optimizing the overall visual effect.

[0032] Semantic segmentation module loss L S The focus is on preserving and improving high-level semantic information. By introducing the semantic segmentation task, the model can pay more attention to the semantic structure in the image during the fusion process, thereby improving the performance of the fused image in subsequent high-level semantic tasks.

[0033] The present invention adopts cross entropy loss as the semantic segmentation module loss, and its formula is: (5); in, ( h , w )and ( h , w ) respectively represent the positions (h , w ) The predicted label and true label of category c, It is the set of all categories in the semantic segmentation task; is an index variable whose value ranges from 0 to " classes -1”; stands for cross entropy loss function. In classification tasks such as semantic segmentation, it is used to measure the difference between the probability distribution of the category labels predicted by the model and the probability distribution of the actual category labels. By minimizing this loss function, the model performance can be optimized and its predictions can be more in line with the actual situation. The cross entropy loss evaluates the performance of the model in the semantic segmentation task by comparing the difference between the predicted labels and the actual labels. By minimizing this loss, the model can learn more accurate semantic information, thereby better retaining this high-level information during the image fusion process.

[0034] (6); Among them, arg min means finding the optimal model parameters by minimizing the loss function; represents the L1 norm (sum of absolute values), which is used to calculate the difference between the predicted value and the reference value; HW represents the product of the image height (H) and width (W), which represents the total number of pixels in the image; L F Represents the total image fusion loss, which is composed of intensity loss and loss of detail Weighted composition; Φ( ir , vi ) represents the fusion model, input infrared image ir and visible light images vi , output fused image f - max ( ir , vi ); , is the normalized weight parameter of the fusion loss, which is used to adjust the loss magnitude; G(⋅) is the gradient extraction operator, which is used to calculate the gradient details (edges, textures) of the image; α represents the coefficient for balancing the intensity loss and detail loss; L S represents semantic segmentation loss, using the cross entropy loss function; Ψ( f ) represents the semantic segmentation model, and the input fusion image f , output semantic segmentation results t f ; l The coefficient to balance the fusion loss and semantic segmentation loss; L cross-entropy is the cross entropy loss function. c∈[0,classes): traverse all semantic categories (such as roads, vehicles, pedestrians, etc.), a total of classes; f =Φ( ir , vi ) represents the fused image f must be generated by the fusion model Φ, t f =Ψ( f ) represents the semantic segmentation result t f must be generated by the segmentation model Ψ.

[0035] Balance coefficient l It plays a vital role in the loss function. It is used to regulate the loss of the image fusion module. L F and semantic segmentation module loss L S By adjusting l The value of can balance the trade-off between visual effect and semantic information retention, so as to obtain a fused image that better meets the needs of practical applications. By optimizing the details and intensity distribution, the overall quality of the fused image can be further improved, making it more in line with the needs of practical applications.

[0036] In summary, Figure 2 The image fusion module demonstrated achieves high-quality fusion of infrared and visible light images through carefully designed preprocessing, feature extraction, feature fusion, and reconstruction steps. This module not only improves the quality of the fused image and the integrity of the semantic information, but also provides more reliable and rich information support for subsequent visual tasks. This innovative design not only improves the efficiency and accuracy of image fusion technology, but also opens up a new path for the development of the image fusion field and shows broad application prospects.

[0037] S4. Dataset annotation.

[0038] After a large amount of data is fused according to S3, the diversity and comprehensiveness of the data set are improved. The generalization ability of the model is enhanced; the fused data set is labeled. Data set labeling is the process of assigning labels to each object or area in the data set so that the machine learning model can learn these labels and make predictions. Labelme is an open source image labeling tool that supports multiple labeling shapes such as polygons, rectangles, circles, etc. The present invention is used for target detection, so it is labeled with a rectangular box. Photovoltaic panel defect types include diodes, open circuits, and hot spot defects. Use the tools provided by labelme to draw the labeled area on the image as required, and assign a label to each defect. After the labeling is completed, the labeling information is saved as a JSON format file, which provides the necessary supervision information for model training.

[0039] Labelme is used for labeling. The labeled files (location information and types of labeled photovoltaic defects) are saved as json format files. The open source code is used to convert them into txt files acceptable to YOLO, and the training set, validation set, and test set are divided according to the ratio of 8:1:1. YOLO (You Only Look Once) is a popular target detection algorithm that accepts txt files in a specific format as input. These files contain information about the category, location, and size of the target in the image. Use programming languages ​​such as Python to write scripts to read the annotation information in the JSON file and convert it to the txt format accepted by YOLO. Such conversion tools or libraries already exist in the open source community. Dividing the converted txt and the fused images into training sets, validation sets, and test sets in a ratio of 8:1:1 means that 80% of the data is used to train the model, 10% of the data is used to verify the model (adjust parameters during training), and another 10% of the data is used to test the model (evaluate performance after training). The data used for training is the aligned data set.

[0040] S4. Build CLIP and YOLOv11 models.

[0041] YOLO is known for its exceptional fast and effective object detection. The model employs a framework consisting of many convolutional layers followed by fully connected layers. Each grid cell in the model is tasked with detecting objects within its designated region. CLIP is a multimodal model that effectively combines textual and visual inputs to produce embeddings capable of performing zero-shot learning. The architecture consists of two main components: a text encoder and an image encoder. The text encoder processes written descriptions, while the image encoder processes visual information. Both encoders produce embeddings in a shared latent space. CLIP has the ability to simultaneously analyze and understand written and visual data in this collaborative setting. The model has the ability to generate embeddings for any possible pairing of images and text, enabling it to accomplish a wide range of tasks without the need for specific training data for each individual task. The zero-shot capability is particularly beneficial for applications that lack labeled data, providing a general solution for a wide range of AI tasks.

[0042] The approach of combining CLIP with YOLOv11 requires leveraging the multimodal capabilities of CLIP to improve the startup and training process of YOLO. The embeddings generated by CLIP on the provided representative dataset are used to initialize the layers of YOLOv11, thus providing the model with comprehensive contextual understanding from the beginning, such as Figure 3 This strategy allows the use of more informative weights, thus enhancing the effectiveness of the training process.

[0043] like Figure 3 As shown, Xi Y represents the input image data, which is the feature vector representing a specific image extracted from the photovoltaic defect detection dataset (aligned dataset). i Represents bounding box information associated with an image. These bounding boxes are used to identify the location and size of defects in the image, usually expressed in coordinate form, indicating the specific area of ​​the defect in the image. CLIP (Contrastive Language-Image PreTraining) is a multimodal model that can process image and text data. It generates rich embedding representations by learning the relationship between images and corresponding text. By calling, the CLIP model takes the image feature vector and its corresponding bounding box information as input and generates an embedding representation e This embedding represents e It contains the visual features of the image and the position information related to the bounding box, which can provide context information and feature initialization for the subsequent YOLOv11n model. e The layer weights will be used to initialize the YOLOv11n model, thereby enhancing the model's understanding and detection of image features. This process can improve the model's performance in small-shot learning scenarios, especially when data is limited. By initializing the initial layers with CLIP embeddings, YOLOv11 is modified to include the most important features. In addition, the use of CLIP embeddings improves the efficiency of YOLOv11 in few-shot learning, allowing the model to quickly adapt to new tasks with less data ( Figure 4 ). This minimizes the need for large dataset generation and reduces the time required for training.

[0044] Figure 3 The process of generating weights based on CLIP and initializing the first layer of YOLOv11 includes the following steps: (1) CLIP embedding extraction: Input part: The bounding box is input into the text encoder for processing, and the fused image is sent into the image encoder for processing.

[0045] Encoding process: The text encoder receives the bounding box as input and outputs the encoded vector after processing. ; The image encoder encodes the fused image and generates a coding vector .

[0046] (2) Comprehensive embedding calculation: The output of the text encoder and the image encoder output Add together and get + , further get the comprehensive embedding e .

[0047] (3) Weight initialization: Initialize the weights of the first convolutional layer of the YOLOv11 model to the comprehensive embedding e ;Will e The value of is directly assigned to the convolution kernel parameters of the first layer of the model. This initialization method breaks the traditional random initialization mode and enables the model to start learning from the pre-trained multimodal feature space. e After re-adjusting the weights, the dimension after re-adjusting the weights is D=h*w*c (h is the height, w is the width, and c is the number of channels), generating N weights ( Figure 3 These weights are generated using CLIP, specifically: e = + =CLIP_Embed(X i ,Y i );X i Represents the image information input to the CLIP model, Y i Represents the text information input into the CLIP model. The final generated weight W1 (W1= e ) is used to initialize the first layer of YOLOv11. The whole process aims to initialize the first layer of the target detection model YOLOv11 by fusing the information of the image and the bounding box, and using CLIP to generate appropriate weights to improve the performance of the model in subsequent detection tasks.

[0048] (4) Fine-tuning optimization: After initialization, the standard back-propagation algorithm is used to fine-tune the entire model network: The model is trained using the selected representative dataset. All network weights are iteratively updated by minimizing the detection loss function. The Adam optimizer is used to select an appropriate learning rate and batch size. The weights are updated by backpropagation, and the detection loss function used to minimize the detection loss function is: , by optimizing the weight matrix W , so that the model is suitable for all data points ( x i , y i ) prediction error (loss function L ) The sum is minimized. Where min represents the minimization operation; Σ is the summation symbol, which means that all data points are accumulated; L ( x i , y i ; W ) is the loss function, which calculates a single data point ( xi , y i )exist W The loss value under x i is the input feature; y i is the true label, and in target detection it is the bounding box and category label.

[0049] The steps to use the Adam optimizer include: Initialization: Weights W Initial value is CLIP synthesis embedded e .

[0050] Forward propagation: input data x i Through the weighted W The model outputs the prediction results (bounding box and class probability).

[0051] Calculate loss: Calculate the loss for each sample L ( x i , y i ; W ), and sum to get the total loss Σ L .

[0052] Back propagation: The total loss is about W Find the gradient ▽ W ·∑ L , determine the weight update direction.

[0053] Update weights through optimizer Adam: W1:= W- η*▽W1·∑ L , where η is the learning rate, and it is repeated for many times until convergence.

[0054] Repeat the above steps until the loss function no longer decreases significantly or reaches the preset number of iterations to generate updated weights.

[0055] The fine-tuning process mainly includes two scenarios: (1) Training from scratch: Train the YOLOv11n model from scratch using representative images.

[0056] (2) Pre-trained YOLOv11: The YOLOv11n model was fine-tuned using the pre-trained weights as a starting point.

[0057] Training from scratch requires an adaptive strategy to learn discriminative information from images, especially when there is less data than the pre-trained model. The combination of CLIP embeddings with YOLOv11n produces significant enhancements. First, the rich contextual information of CLIP embeddings should help the model perform well with fewer training samples. Leveraging CLIP’s multimodal understanding, YOLOv11n will be more adaptable to new tasks. The combination of CLIP embeddings with YOLOv11n greatly improves the performance of the model, making it a reliable and efficient solution for PV defect detection. This approach minimizes the requirement for large amounts of data and training time, facilitating wider applications in the field of maintenance and monitoring of PV energy systems. Extending the model to include real-time defect detection in operating PV systems will be an important milestone towards achieving practical implementation. Ultimately, expanding the model’s capabilities to identify a wider range of defects and irregularities in other categories of PV energy systems will increase its usefulness and efficiency in the field of sustainable energy.

[0058] The present invention provides a photovoltaic panel defect detection method, which has the following beneficial effects: 1) The SIFT algorithm is used for image alignment processing to ensure the consistency of infrared and visible light images in resolution and size, eliminate the misalignment caused by shooting angle and viewing angle differences, and improve the accuracy and stability of image processing. 2) Through the dual private encoder architecture and the deeply interpretable iterative algorithm, the efficient fusion of infrared and visible light images is achieved, the high-frequency and low-frequency features are extracted and integrated, and high-quality images with rich details and high information content are generated, providing comprehensive feature support for subsequent defect detection. The efficiency and comprehensiveness of multimodal data fusion are improved. 3) Considering the problem of limited labeled data in photovoltaic defect detection, an architecture combining CLIP with YOLOv11n is proposed. By using the multimodal understanding ability of CLIP to initialize the layers of YOLOv11n and fine-tuning training on a small amount of target domain data, the problem of difficulty in model training due to the lack of a large amount of labeled data is solved. This method not only improves the detection performance of the model, but also greatly shortens the training time, providing a more efficient and reliable solution for the maintenance and monitoring of photovoltaic systems.

[0059] Based on the same inventive concept, the present invention also proposes a photovoltaic panel defect detection system, comprising: The acquisition module is used to acquire image data containing different lighting conditions, different defect types and different photovoltaic component types; the image data includes multiple infrared images and multiple visible light images.

[0060] The training set construction module is used to extract the visual features of each infrared image and visible light image, and fuse all the visual features to obtain a fused image data set; mark the defective areas in the fused image data set, and save the text information of the marked defective areas; and construct a training data set through the fused image data set and the text information of the marked defective areas.

[0061] The training module is used to input the training data set into the CLIP model, use a text editor to process the text information of the marked defect area, and generate a coding vector ; Use the image encoder to process the fused image data set and generate the encoding vector ; According to the encoding vector and the encoding vector Generate an embedded representation; use the embedded representation to initialize the layer weights of the YOLOv11 model, adjust the first layer of the YOLOv11 model through the initialized layer weights, and obtain a photovoltaic defect detection model.

[0062] The detection module is used to fuse the visual features of the infrared image and the visible light image of the photovoltaic module to be tested, input the fused image data into the photovoltaic defect detection model, and detect the defective area in the image data.

[0063] The present invention also proposes a photovoltaic panel defect detection computer device, comprising: a memory, a processor, and a computer program stored in the memory, and the processor implements the steps of the photovoltaic panel defect detection method when executing the computer program.

[0064] The present invention also proposes a readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, they are used to execute the steps of the photovoltaic panel defect detection method.

[0065] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A photovoltaic panel defect detection method, characterized in that: The following steps are involved: Acquire image data containing different lighting conditions, different defect types, and different photovoltaic module types; the image data includes a plurality of infrared images and a plurality of visible light images; Extract visual features from each infrared image and visible light image, and fuse all visual features to obtain a fused image dataset; annotate defective areas in the fused image dataset, and save text information of the annotated defective areas; construct a training dataset using the fused image dataset and text information of the annotated defective areas; Input the training data set into the CLIP model, use a text editor to process the text information of the marked defect area, and generate a coding vector ; Use the image encoder to process the fused image data set and generate the encoding vector ; According to the encoding vector and the encoding vector Generating Embedded Representations e = + ; e After re-adjusting the weights, N weights are generated; The dimension after re-adjusting the weight is D=h*w*c, where h is the height, w is the width, and c is the number of channels; specifically: e = + =CLIP_Embed(X i ,Y i );X i Represents the image information input to the CLIP model, Y i Represents the text information input to the CLIP model; according to the embedding representation e Generate the final weight W1, adjust the first layer of YOLOv11 through W1, and obtain the photovoltaic defect detection model; The visual features of the infrared image and the visible light image of the photovoltaic module to be tested are fused, and the fused image data is input into the photovoltaic defect detection model to detect the defective area in the image data.

2. A photovoltaic panel defect detection method according to claim 1, characterized in that: Before extracting the visual features of each infrared image and visible light image, the infrared image and the visible light image are aligned using the SIFT algorithm, which specifically includes the following steps: The scale of each infrared image and visible light image is continuously transformed using a Gaussian function; the Gaussian function G ( x , y , σ ) is expressed as: ； in( x , y ) represents pixel coordinates; σ is the scale parameter, which determines the blurriness of the image; For the original image I ( x , y ) is different from σ The convolution operation is performed on the Gaussian function of the value. The formula for the convolution operation is: ； in, L ( x , y , σ ) indicates that the scale σ The image below shows that * represents the convolution operation; By changing σ The value of is used to obtain a series of images with different blur levels, thus forming a multi-scale spatial sequence; The local extreme points of the multi-scale space sequence are detected by the Gaussian difference DOG method. Specifically, for each pair of adjacent images in the multi-scale space sequence, the difference between them is calculated to obtain the DOG response value image : ； in, k is a constant that is a multiple of the adjacent scale space; In the DOG response value image, local extreme points are detected by comparing each image pixel with the pixel values ​​in its neighborhood; The detected local extreme points are located, and the key points with contrast lower than the set value and unstable edge response points are removed by fitting a three-dimensional quadratic function; and the gradient of the pixels in the neighborhood of the feature point is used to determine its direction parameter; By statistically analyzing the gradient histogram, the stable direction of the local structure of the key point is found; based on the position and direction of the image gradient in the neighborhood of each key point, a 128-dimensional feature descriptor is constructed; The infrared image and the visible light image are aligned according to the feature descriptor.

3. A photovoltaic panel defect detection method according to claim 1, characterized in that: A dual encoder is used to process each infrared image and visible light image in parallel to obtain the visual features of the two images; All visual features are fused using a deep interpretable iterative algorithm to obtain a fused image dataset.

4. A photovoltaic panel defect detection method according to claim 1, characterized in that: The method also includes optimizing the fused image dataset by using image fusion module loss and semantic segmentation module loss before constructing the training dataset; wherein the image fusion module loss L F It is expressed as: ； in, H and W Represents the height and width of the image respectively, Target( ir , vi ) represents the fusion target image, The cross entropy loss is used as the semantic segmentation module loss, and its formula is: ； in, ( h , w )and ( h , w ) respectively represent the positions ( h , w ) Category c The predicted labels and true labels of It is the set of all categories in the semantic segmentation task; is an index variable whose value ranges from 0 to " classes -1”; represents the cross entropy loss function; The obtained image fusion module loss and semantic segmentation module loss are minimized to optimize the fused image dataset; the minimization process is specifically expressed as: ； Among them, arg min means finding the optimal model parameters by minimizing the loss function; represents the L1 norm; HW represents the product of the image height H and width W, and represents the total number of pixels in the image; L F Represents the total image fusion loss, which is composed of intensity loss and loss of detail Weighted composition; Φ( ir , vi ) represents the fusion model, input infrared image ir and visible light images vi , output fused image f - max ( ir , vi ); , is the normalized weight parameter of the fusion loss, which is used to adjust the loss magnitude; G(⋅) is the gradient extraction operator; α represents the coefficient for balancing the intensity loss and detail loss; L S represents semantic segmentation loss, using the cross entropy loss function; Ψ( f ) represents the semantic segmentation model, and the input fusion image f , output semantic segmentation results t f ; λ The coefficient to balance the fusion loss and semantic segmentation loss; L cross-entropy is the cross entropy loss function; c ∈[0, classes ) means traversing all semantic categories, classes kind; f =Φ( ir , vi ) represents the fused image f must be generated by the fusion model Φ, t f =Ψ( f ) represents the semantic segmentation result t f must be generated by the segmentation model Ψ.

5. A photovoltaic panel defect detection system, characterized in that: include: An acquisition module, used to acquire image data containing different lighting conditions, different defect types and different photovoltaic module types; the image data includes a plurality of infrared images and a plurality of visible light images; The training set construction module is used to extract the visual features of each infrared image and visible light image, and fuse all the visual features to obtain a fused image data set; annotate the defective areas in the fused image data set, and save the text information of the annotated defective areas; and construct a training data set through the fused image data set and the text information of the annotated defective areas; The training module is used to input the training data set into the CLIP model, use a text editor to process the text information of the marked defect area, and generate a coding vector ; Use the image encoder to process the fused image data set and generate the encoding vector ; According to the encoding vector and the encoding vector Generating Embedded Representations e = + ; e After re-adjusting the weights, N weights are generated; The dimension after re-adjusting the weight is D=h*w*c, where h is the height, w is the width, and c is the number of channels; specifically: e = + =CLIP_Embed(X i ,Y i );X i Represents the image information input to the CLIP model, Y i Represents the text information input to the CLIP model; according to the embedding representation e Generate the final weight W1, adjust the first layer of YOLOv11 through W1, and obtain the photovoltaic defect detection model; The detection module is used to fuse the visual features of the infrared image and the visible light image of the photovoltaic module to be tested, input the fused image data into the photovoltaic defect detection model, and detect the defective area in the image data.

6. A photovoltaic panel defect detection computer device, characterized in that: include: A memory, a processor and a computer program stored in the memory, wherein the processor implements the steps of the photovoltaic panel defect detection method according to any one of claims 1 to 4 when executing the computer program.

7. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, they are used to execute the steps of the photovoltaic panel defect detection method according to any one of claims 1 to 4.

Images