An open-vocabulary-based photovoltaic panel damage detection method, system and medium

Abstract

The application discloses a photovoltaic panel damage detection method and system based on an open vocabulary, and a medium, solves the problems of high model training cost and long training time in the prior art, has the beneficial effects of reducing training cost and improving recognition accuracy, and the specific scheme is as follows: a photovoltaic panel damage detection method based on an open vocabulary, comprising constructing a training data set; performing knowledge distillation pre-training, using the pre-trained open vocabulary target detection model as the teacher model, using the detection model to be trained as the student model, distilling the intermediate layer features and class prediction distribution of the teacher model respectively, and the student model learns basic semantic understanding ability through the first pre-training data set; freezing the knowledge distillation module of the student model, collecting images of photovoltaic panels to be detected and preprocessing, and inputting the fused multi-scale feature map and the updated text embedding vector into the detection output module.

Description

Technical Field

[0001] This invention relates to the field of defect detection technology, and in particular to a method, system and medium for detecting damage to photovoltaic panels based on open vocabulary. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Photovoltaic power generation, as an important form of clean energy, has been widely used globally. As the core component of a photovoltaic power generation system, the operating status of photovoltaic panels directly affects power generation efficiency and system safety. However, during long-term operation, photovoltaic panels can suffer various types of damage due to environmental factors and manufacturing defects, such as hot spots, cracks, and stains. Timely and accurate detection of these damages is crucial for ensuring the safe operation and economic benefits of photovoltaic power plants.

[0004] Traditional photovoltaic (PV) panel defect detection methods primarily rely on manual inspection, which suffers from low efficiency, high cost, and susceptibility to subjective factors. With the development of computer vision and deep learning technologies, automated defect detection methods based on deep learning have gradually become a research hotspot. Currently, object detection algorithms such as Fast R-CNN (Fast Region Convolutional Neural Network), Mask R-CNN (Mask Region Convolutional Neural Network), DETR (Object Detection Model), and YOLO (Object Detection Model) have been widely applied in the field of PV panel defect detection. These methods are typically trained on datasets with closed-set categories and perform well in identifying defect types within predefined categories.

[0005] However, existing defect detection methods based on closed set categories have the following shortcomings: First, in order to enable machines to acquire comprehensive and multi-category image-text understanding capabilities, most existing models are trained from scratch using large-scale image-text datasets. This makes the model training cost extremely high, resulting in extremely high requirements for computing resources and a lengthy training cycle, which is not conducive to industrial application and rapid iteration.

[0006] Secondly, in order to improve the generalization ability to specific targets, existing open vocabulary detection algorithms require an increase in the number of model parameters, resulting in a huge number of model parameters that are difficult to deploy on resource-constrained terminal devices.

[0007] Furthermore, since the datasets used in the pre-training stage of existing open vocabulary detection models mainly target specific objects in daily life and do not include defect datasets, defect classes are easily misclassified as base classes or ignored due to insufficient background representation. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the purpose of this invention is to provide a photovoltaic panel damage detection method based on open vocabulary, which enables open vocabulary defect detection, shortens the training cycle, reduces the number of model parameters and training costs, and improves the targeting and accuracy of photovoltaic panel defect detection.

[0009] To achieve the above objectives, the present invention is implemented through the following technical solution: A photovoltaic panel damage detection method based on an open vocabulary includes the following: To construct a training dataset, semantic annotations are added to the publicly available photovoltaic defect dataset and self-taken photovoltaic panel defect images to form a defect dataset. The basic image dataset and a portion of the defect dataset are mixed in a preset ratio to form the first pre-training dataset, and the remaining defect dataset is used as the second pre-training dataset. Knowledge distillation pre-training is performed, using the pre-trained open vocabulary target detection model as the teacher model and the detection model to be trained as the student model. The intermediate layer features and category prediction distribution of the teacher model are distilled separately, and the student model learns basic semantic understanding ability through the first pre-training dataset. Freeze the knowledge distillation module of the student model, and use the second pre-training dataset to perform defect-specific training on the student model to obtain pre-training weights; Images of the photovoltaic panels to be detected are acquired and preprocessed to obtain fused multi-scale feature maps and updated text embedding vectors. The fused multi-scale feature maps and updated text embedding vectors are input to the detection output module. The detection output module regresses bounding boxes and target embedding vectors by decoupling the detection head. By calculating the similarity between the target and the text and the cross-entropy between the target and the text, a region-text contrast loss is constructed, and the total training loss is obtained. Rectangular localization boxes with category labels or user prompts and guaranteed probabilities are generated.

[0010] As described above, the photovoltaic panel damage detection method based on open vocabulary involves acquiring an image of the photovoltaic panel to be detected and preprocessing it to obtain a fused multi-scale feature map and an updated text embedding vector, including the following: The preprocessed image is input into the defect feature extraction module, which includes a minimum convolutional unit module and a multi-convolutional layer aggregation network. The minimum convolutional unit module includes deformable convolutional layers, batch normalization layers and defect enhancement activation functions. The multi-convolutional layer aggregation network outputs multi-scale feature maps. The multi-scale feature maps and text embedding vectors are input into the image-text fusion module. The image-text fusion module reorganizes the pixel values ​​of the feature maps at different scales into the channel dimension through spatial depth transformation to achieve resolution alignment. Then, the image features and text features are fused through text-guided Ghost convolution and cross-attention mechanism, and the fused multi-scale feature maps and updated text embedding vectors are output.

[0011] As described above, the photovoltaic panel damage detection method based on open vocabulary includes a defect feature extraction module that further includes a DS submodule. The DS submodule is divided into two branches: the left branch is used for max pooling, and the right branch performs a 3×3 convolution. The two feature maps are then concatenated and output. The multi-convolutional layer aggregation network includes two branches. The left branch contains six DBD modules and one Concat module. The four 3×3 convolutional modules in the DBD module are used to obtain feature maps at different levels and aggregate features at different levels through dense residual connections. The right branch is the total residual connection of the multi-convolutional layer aggregation network to help the multi-convolutional layer aggregation network perform gradient backpropagation better.

[0012] As described above, in the photovoltaic panel damage detection method based on open vocabulary, the expression for the defect enhancement activation function is:

[0013] Where α is a hyperparameter greater than 0, and τ is the gradient threshold; The defect enhancement activation function employs two activation modes: when a high gradient region (defect edge) is detected in the feature map, exponential activation is used to enhance the defect feature response; linear activation is used for uniform background regions to suppress noise amplification. The gradient threshold adjustment strategy is as follows: The gradient map G is obtained by automatic differentiation of the input feature map x; Calculate the global average gradient:

[0014] make

[0015] Where i and j represent the height and width of the gradient map G, respectively, H is the height of the input feature map, and W is the width of the input feature map. λ is the global average gradient, and λ is a hyperparameter that determines the model's sensitivity to image gradients.

[0016] As described above, the photovoltaic panel damage detection method based on open vocabulary includes an image-text fusion module comprising two sub-modules: SD-Ghost and I-Pooling Attention. Through SDT / DST operations, the pixel values ​​of the multi-scale feature map output by the defect feature extraction module are completely reconstructed to the channel dimension, preserving all pixel values ​​and spatial positional relationships to achieve lossless resolution conversion. The fully aligned feature map obtained through spatial depth transformation is then used as input into the text-guided Ghost convolution module to achieve text-image feature fusion.

[0017] The photovoltaic panel damage detection method based on open vocabulary, as described above, fuses image features and text features through text-guided Ghost convolution and cross-attention mechanism, outputting a fused multi-scale feature map and an updated text embedding vector, including the following: Given a text embedding vector T and image features After the Ghost convolution operation, a maximum sigmoid attention mechanism is added to aggregate text features into image features in the following way:

[0018] Where δ is the Sigmoid function, C is the vocabulary size, and T is the lexicon. j Let represent the vector representation of the j-th word, ( ) T For transpose operator; Unified scale feature map that incorporates semantics After undergoing the inverse SDT / DST transformation, the image is transformed again into a multi-scale feature map, which is then input into the I-Pooling Attention submodule. The I-Pooling Attention submodule applies max pooling to the multi-scale features to obtain 2×2 regions, thereby generating multiple image patch labels. The text embedding vector is updated in the following way:

[0019] in, W is the updated value of the text embedding vector. Multihead-Attention refers to the multihead attention calculation mechanism, which uses the query vector as a reference to calculate the associated feature weights of the key vector and performs weighted fusion on the value vector, thereby adaptively focusing on the key feature information of the defect. The image marker block obtained after the pooling operation.

[0020] The calculation process for the total training loss in the photovoltaic panel damage detection method based on open vocabulary described above is as follows: The similarity between the target and the text is obtained using the following formula. :

[0021] in, It is an L2 normalization operation, t j It is the j-th text embedding vector, where α is a learnable scaling factor and β is a learnable offset factor; The region-text contrast loss L is obtained using the following formula. con for:

[0022] Where, N pos The number of positive sample prediction boxes (obtained from label assignment); s i,yi For the i-th positive sample prediction box and the ground truth text label y i Similarity score; s i,j The similarity score between the i-th predicted bounding box and the j-th text. Using cross-union ratio loss L iou and distribution focus loss L dfl To perform bounding box regression, the total training loss is... Defined as:

[0023] in, It is an indicator factor.

[0024] The photovoltaic panel damage detection method based on open vocabulary, as described above, involves distilling the intermediate layer features and category prediction distributions of the teacher model, including the following: The intermediate layer features of the teacher model are distilled using the feature distillation loss function:

[0025] Where l represents the feature of the l-th layer. , H represents the multi-scale features of the intermediate layers of the teacher model and the student model, respectively. l W l D l These represent the height, width, and number of channels of the feature in layer l, respectively. Distillation of the class prediction distribution of the teacher model is performed using a soft-label distillation loss function:

[0026] in, Output an object-text similarity matrix for the teacher model. , Output the values ​​of coordinates (k, j) and (k, m) in the object-text similarity matrix for the teacher model. , Output the values ​​of coordinates (k, j) and (k, m) in the object-text similarity matrix for the student model. Output a similarity matrix for the student model, where K is the number of prediction boxes, C is the vocabulary size, and t is the temperature parameter.

[0027] Secondly, the present invention provides a lightweight intelligent damage detection system for photovoltaic panels based on open vocabulary, comprising: The training data construction module adds semantic annotations to the publicly available photovoltaic defect dataset and self-taken photovoltaic panel defect images to form a defect dataset. The basic image dataset and a portion of the defect dataset are mixed according to a preset ratio to form the first pre-training dataset, and the remaining defect dataset is used as the second pre-training dataset. The knowledge distillation module uses a pre-trained open vocabulary target detection model as the teacher model and the detection model to be trained as the student model. It distills the intermediate layer features and category prediction distribution of the teacher model respectively. The student model learns basic semantic understanding ability through the first pre-training dataset and uses the second pre-training dataset to perform defect-specific training on the student model to obtain pre-training weights. The image processing module acquires images of the photovoltaic panel to be detected and performs preprocessing to obtain a fused multi-scale feature map and an updated text embedding vector. The fused multi-scale feature map and the updated text embedding vector are then input to the detection output module. The detection output module decouples the detection head from the bounding box and the target embedding vector. It constructs the region-text contrast loss by calculating the cross-entropy between the similarity between the target and the text and the distribution relationship between the target and the text, and obtains the total training loss. It then generates rectangular localization boxes with category labels or user prompts and guaranteed probabilities.

[0028] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned photovoltaic panel damage detection method based on open vocabulary.

[0029] The beneficial effects of the present invention are as follows: The detection method provided by this invention integrates publicly available photovoltaic defect datasets and real-world private defect datasets, strengthens semantic annotation of defect location and shape, and uses basic image datasets for training, enabling the model to fully learn defect features. It directly distills basic semantic understanding capabilities from a pre-trained open-vocabulary target detection model, eliminating the need to build image-text associations from scratch and significantly saving computational resources. A two-stage training strategy is employed: knowledge distillation pre-training followed by targeted training on the defect dataset. The distillation stage rapidly acquires basic semantic capabilities such as direction, size, and shape, shortening the overall training cycle by more than 70% compared to training from scratch, thus lowering the technical and resource barriers for industrial application. The fused multi-scale feature map and the updated text embedding vector are input to the detection output module. The detection output module decouples the detection head to regress the bounding box and the target embedding vector, calculates the cross-entropy between the similarity between the target and text and the distribution relationship between the target and text, constructs the region-text contrast loss, and obtains the total training loss. Compared to existing methods, this method improves the accuracy of defect class recognition by more than 25%. Attached Figure Description

[0030] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0031] Figure 1 This is a flowchart illustrating a photovoltaic panel damage detection method based on open vocabulary.

[0032] Figure 2 This is a schematic diagram of a photovoltaic panel damage detection system based on open vocabulary.

[0033] Figure 3 This is a schematic diagram of the defect feature extraction module based on deformable convolution.

[0034] Figure 4 This is a schematic diagram of the image-text fusion module based on feature map spatial depth transformation.

[0035] Figure 5 This is a structural diagram of the knowledge distillation module.

[0036] Figure 6 A schematic diagram of the detection output module that incorporates regional text contrast loss.

[0037] Figure 7 A schematic diagram illustrating the principle of tensor index update for spatial depth transformation operations. Detailed Implementation

[0038] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0039] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless otherwise expressly indicated by the invention, the singular form is also intended to include the plural form. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. As introduced in the background section, existing technologies suffer from high model training costs and long training times. To address these technical issues, this invention proposes a photovoltaic panel damage detection method based on an open vocabulary.

[0040] Example 1 In a typical embodiment of the present invention, reference is made to Figure 1 As shown, a photovoltaic panel damage detection method based on open vocabulary includes the following: 1) Construct a training dataset. After adding semantic annotations to the publicly available photovoltaic defect dataset and self-taken photovoltaic panel defect images, a defect dataset is formed. The basic image dataset and a portion of the defect dataset are mixed according to a preset ratio to form the first pre-training dataset, and the remaining defect dataset is used as the second pre-training dataset. 2) Specifically, high-resolution industrial cameras are used to capture video of the photovoltaic panel surface in real time. The camera's lens is parallel to the photovoltaic panel surface to acquire video. The final acquired video is preprocessed in terms of frame extraction, color, brightness, rotation, and image sharpness before being transmitted to a defect feature extraction module based on deformable convolution. Color adjustment converts the image from RGB color space to grayscale color space, as photovoltaic panel defect detection mainly focuses on texture and shape features, and grayscale images can reduce the amount of computation. Brightness adjustment enhances the contrast of the image through histogram equalization, making the defect features more prominent. The rotation operation rotates the image to a standard orientation, ensuring that the edges of the photovoltaic panel are parallel to the edges of the image. Image sharpness enhancement enhances the edge information of the image through a sharpening filter, improving the clarity of the defect edges. In addition, some preprocessed images can be returned to the dataset preparation stage to expand the defect dataset. The dataset construction integrates public photovoltaic defect datasets and real-world private defect datasets, and strengthens the semantic annotation of defect location and shape. It is also used with basic image datasets to assist training, allowing the model to fully learn the defect features.

[0041] Real photovoltaic panel defect images were captured and processed, and combined with manual fine-grained annotation and global semantic annotation to generate a private dataset; all data annotations were saved in JSON file format. The expanded dataset of all basic graphic shapes and a portion of the defect data were mixed 1:1 as the training dataset for the first pre-training stage, with this stage using approximately 40% of the total training data. Specifically, assuming the basic image dataset contains 1000 images, 1000 images are randomly selected from the public photovoltaic defect dataset and the private photovoltaic defect dataset, and the two are mixed to obtain 2000 images as the first pre-training dataset, accounting for 40% of the total training data. The remaining 3000 photovoltaic defect images are used as the second pre-training dataset, accounting for 60% of the total training data.

[0042] 2) Perform knowledge distillation pre-training, using the pre-trained open vocabulary target detection model as the teacher model, referencing... Figure 5 As shown, YOLO-world-L (YOLO-World model proposed by Tencent AILab Computer Vision Center, an open vocabulary object detection model for everyday object detection) is used as the teacher model, and the detection model to be trained is used as the student model. The intermediate layer features and category prediction distribution of the teacher model are distilled separately. The student model learns basic semantic understanding ability through the first pre-training dataset. At this stage, the student model will obtain basic image semantic understanding ability from the teacher model, such as concepts of direction, size, and shape. Distillation is performed on the intermediate layer features and class prediction distributions of the teacher model, including the following: Knowledge transfer at the feature level is crucial for semantic alignment. The intermediate features of the teacher model are distilled using a feature distillation loss function. (1) Among them, L feat The loss function is the feature distillation function. "feat" refers to the feature function, "stu" refers to the student function, and "tea" refers to the teacher function. , H represents the multi-scale features of the intermediate layers of the teacher model and the student model, respectively. l W l D l These represent the height, width, and number of channels of the feature in layer l, respectively. Loss function through soft label distillation Distillation of the class prediction distribution of the teacher model: (2) In this model, the student model learns the category prediction distribution (soft labels) from the teacher model, rather than just hard labels, in order to improve generalization ability. Output an object-text similarity matrix for the teacher model. , Output the values ​​of coordinates (k, j) and (k, m) in the object-text similarity matrix for the teacher model. , Output the values ​​of coordinates (k, j) and (k, m) in the object-text similarity matrix for the student model. Output a similarity matrix for the student model, where K is the number of predicted boxes, C is the vocabulary size, and t is the temperature parameter, which controls the smoothness of the soft labels. Although both defect detection and object detection tasks require open-vocabulary detection, defect detection demands a much lower depth of semantic understanding than object detection. This makes many weights in existing open-vocabulary detection models unnecessary or even detrimental to defect detection. Therefore, distilling the basic semantic understanding capabilities from existing large-scale visual language models can fulfill the task requirements, significantly reducing the pre-training cost of the model. In other words, by introducing a knowledge distillation module, using YOLO-world-L as the teacher model, its basic semantic understanding capabilities can be directly distilled, eliminating the need for the model to build image-text associations from scratch and greatly saving computational resources.

[0043] 3) Freeze the knowledge distillation module of the student model, use the second pre-training dataset to perform defect-specific training on the student model, and obtain pre-training weights. The obtained pre-training weights will guide the model in subsequent open vocabulary detection tasks. It's important to explain that weights are trainable parameters in deep neural networks, used to quantify the contribution and influence of preceding feature inputs on the output of subsequent neurons. They are core parameters in network operations. When a deep learning model first starts training, the network weights are random numbers and cannot identify defects at all. The training process involves continuously refining the weights using labeled datasets until the model can accurately detect targets. After training, the model saves the best-performing set of weights. The next time a defect detection task is performed, this weight file is directly loaded, allowing the model to access all the knowledge learned during the previous training phase.

[0044] It should be noted that this step and the previous step adopt a two-stage training strategy of knowledge distillation pre-training + defect-specific training. The distillation stage quickly acquires basic semantic capabilities such as direction, size, and shape. Subsequently, only the distillation module needs to be frozen for photovoltaic defect-specific training, which greatly shortens the overall training cycle, reduces the technical and resource threshold for industrial implementation, and facilitates rapid iteration and optimization of the model. 4) Acquire images of the photovoltaic panels to be detected and preprocess them to obtain fused multi-scale feature maps and updated text embedding vectors. Input the fused multi-scale feature maps and updated text embedding vectors into the detection output module. The detection output module decouples the detection head to regress the bounding box and the target embedding vector. By calculating the similarity between the target and the text and the cross-entropy between the target and the text, it constructs the region-text contrast loss and obtains the total training loss. It then generates rectangular localization boxes with category labels or user prompts and guaranteed probabilities, which are output as the detection results.

[0045] The process involves acquiring and preprocessing images of the photovoltaic panels to be inspected to obtain a fused multi-scale feature map and an updated text embedding vector, including the following: 4-1) The preprocessed image is input into the defect feature extraction module. This module is based on deformable convolution and includes a DBD (Deformable convolution-Batch normalization-DEA) minimum convolutional unit module, an MLAN (Multi-Layer Aggregation Network) multi-convolutional layer aggregation network, and a DS (Down Sampling) module. (See reference...) Figure 3 As shown, the smallest convolutional unit module includes deformable convolutional layers, batch normalization layers, and defect enhancement activation functions, and outputs multi-scale feature maps through the aggregation network of multiple convolutional layers. Defect-Enhanced Activation (DEA) is a dual-mode activation function proposed to better target the abstract goal of defects. (3) Specifically, the activation function employs two activation modes: when a high gradient region (defect edge) is detected in the feature map, exponential activation is used to enhance the defect feature response; linear activation is used for uniform background regions to suppress noise amplification and improve stability against complex backgrounds.

[0046] Here, α is a hyperparameter greater than 0, used to adjust the magnitude of the model's enhancement of defect features, and τ is the gradient threshold, dynamically adjusted to adapt to different defect types. The specific adjustment strategy is as follows: The gradient map G is obtained by automatic differentiation of the input feature map x; Calculate the global average gradient: (4) make (5) Where H is the height of the input feature map and W is the width of the input feature map. The global average gradient is given by λ, a hyperparameter that determines the model's sensitivity to image gradients and is adjusted through training or experience (set to 1 in the experiment). Compared to the widely used SiLU activation function in image processing, DEA is specifically designed for defect detection scenarios. It distinguishes defect edges from background regions by varying gradients, using exponential activation to enhance the response at defect edges and linear activation to suppress noise in the background. This approach more effectively highlights defect features, improves defect detection accuracy, and is more targeted for defect detection tasks.

[0047] Because multi-convolutional layer feature fusion combined with residual connections is an efficient network structure, the MLAN submodule (Multi-Convolutional Layer Aggregation Network) was constructed based on this. MLAN mainly contains two branches. The left branch, the main branch, includes six DBD modules and one Concat module (a concatenation module that combines two feature maps of the same size into one for better feature fusion). Four 3×3 convolutional modules are used to obtain feature maps at different levels and aggregate these levels of features through dense residual connections. The right branch provides a general residual connection for this module to help the network perform better gradient backpropagation and accelerate model convergence.

[0048] It should be noted that the main function of the DS submodule is to achieve cross-layer transition and scale adjustment of feature maps, laying the foundation for subsequent multi-layer feature aggregation. The DS submodule is also divided into two branches: the left branch performs max pooling, and the right branch performs a 3×3 convolution. The two feature maps are then concatenated for output. The use of deformable convolution and defect enhancement activation functions gives the feature extraction module of this method a stronger feature extraction capability for irregular and varied abstract defect targets.

[0049] 4-2) Input the multi-scale feature map and text embedding vector into the image-text fusion module, refer to... Figure 4 and Figure 7As shown, the image-text fusion module reassembles the pixel values ​​of feature maps at different scales to the channel dimension through spatial depth transformation operations to achieve resolution alignment. The image-text fusion module is an image-semantic fusion module based on feature map spatial depth transformation. It includes two sub-modules: SD-Ghost (space to depth transformation-Ghostconvolution, i.e., a feature fusion module combining spatial depth transformation and ghost convolution, which avoids repeated downsampling and upsampling operations to align the input feature maps and reduces the computational cost of the feature fusion stage) and I-Pooling Attention (image pooling attention mechanism). Through SDT / DST (spatial-depth transformation / depth-spatial transformation) operations, it completely reassembles the pixel values ​​of the multi-scale feature maps output by the defect feature extraction module to the channel dimension, preserving all pixel values ​​and spatial relationships, achieving lossless resolution conversion. The fully aligned feature maps obtained through spatial depth transformation achieve lossless resolution conversion. This process only uses tensor index rearrangement to achieve "one-time cross-layer stitching," quickly aligning the resolution of feature maps at different levels and avoiding the overhead of FPN (Feature Pyramid). Multi-level repetitive operations of network (pyramid network) type network, the fully aligned feature map obtained by spatial depth transformation is used as input into the text-guided Ghost convolution (a lightweight convolution module) to achieve text-image feature fusion.

[0050] Specifically, the core process of SDT / DST operation includes two steps: 1) Define the size of the partitioned spatial block as s×s. For the input feature map... The SDT transform divides it into non-overlapping s×s spatial blocks; 2) Determine the mapping relationship from space to channels. Reassemble the pixel values ​​within each s×s spatial block sequentially to the channel dimension, so that the number of channels becomes the original s. 2 The width and height are multiplied by 1 / s of their original values, and the transformed feature map is: The tensor indexes will be rearranged according to the following rules: (6) (7) Where L is the tensor index of the input feature map, and i, j, c refer to the height, width, and number of channels of the gradient map G (a tensor, which is a multidimensional array that uniformly represents data and parameters in deep learning. It is the basic data container for neural network operations. All images, text, audio, model weights, and gradients are tensors). SDT is the spatial depth transformation calculation. / / is the floor operator, and % is the modulo operator; 4-3) Then, the fusion of image features and text features is completed through text-guided Ghost convolution and cross-attention mechanism. Specifically, given the text embedding T and image features... In this embodiment, a maximum Sigmoid attention mechanism is added after the Ghost convolution operation to aggregate text features into image features in the following way: (8) Where δ is calculated using the Sigmoid function, X l Let C be the input image features, C be the vocabulary size, and T be the given text embedding vector. j Let represent the vector representation of the j-th word, ( ) T For transpose operator; Unified scale feature map that incorporates semantics After undergoing an inverse SDT / DST transformation, the image is transformed again into a multi-scale feature map, which is then input into the I-Pooling Attention submodule. This module updates the text embedding vector by aggregating image features. Instead of directly applying cross-attention to the image features, this module applies max pooling to the multi-scale features to obtain 2×2 regions, resulting in a total of 12 image patch labels. Then, the text embedding vector is updated in the following way: (9) in, W is the updated value of the text embedding vector (the text vector after learning image information through a maximum sigmoid attention mechanism). Multihead-Attention refers to the multihead attention computing mechanism, which uses the query vector as a reference to calculate the associated feature weights of the key vector and performs weighted fusion on the value vector, thereby adaptively focusing on the key feature information of the defect. The image marker block obtained after the pooling operation.

[0051] In the image semantic fusion module based on feature map spatial depth transformation, the above feature fusion process will be executed twice, and the final output will be a multi-scale feature map containing rich semantic information and a text embedding vector containing image features.

[0052] It should be noted that the reference Figure 6 As shown, the working process of the detection output module is as follows: The detection output module has a decoupling head with two 3×3 convolutional layers for regressing bounding boxes. and target embedding vector Where K represents the number of targets, and given the multi-dimensional feature map and text embedding output by the image semantic fusion module, this detection head will output K target prediction results. and annotations Using a task-aligned label assignment method, the predicted results are matched with the ground truth labels, and a text index is assigned as a classification label for each positive sample prediction. A region-text contrast loss Li is constructed by calculating the cross-entropy between the similarity between the target and the text and the assignment relationship between the target and the text. con Similarity between target and text (s) k,j It is calculated using the following formula: (10) in, It is an L2 normalization operation, t j It is the j-th text embedding vector; α and β are the learnable scaling factor and offset factor, respectively, which play an important role in the text contrast training of stable regions.

[0053] This yields the region-text contrast loss L. con for: (11) Where, N pos The number of positive sample prediction boxes (obtained from label assignment); s i,yi For the i-th positive sample prediction box and the ground truth text label y i Similarity score; s i,j The similarity score between the i-th predicted bounding box and the j-th text is given.

[0054] In addition, the head will also employ cross-union loss L. iou and distribution focus loss L dfl To perform bounding box regression, the total training loss is defined as: (12) in, It is an indicator factor that is set to 1 when the input image comes from detection or target localization data, and to 0 when the input image comes from image-text data.

[0055] To further verify the performance of the method of this invention on the open vocabulary detection task, a novel defect detection experiment was conducted. A certain type of defect (e.g., "irregular cracks at the joint of battery cells") was removed from the training dataset. During the testing phase, this type of defect was detected using the user prompt "detect irregular cracks at the joint of battery cells". Experimental results show that the detection method provided in this embodiment achieves an accuracy of 85.3% for novel defects not covered in the training set, significantly outperforming traditional defect detection methods based on closed-set categories (with an accuracy of 0% because they cannot identify defects outside the training set) and the YOLO-world-L method (with an accuracy of 62.1% due to a lack of targeted training on a defect dataset). This verifies the advantages of the method of this invention on the open vocabulary detection task, enabling the identification of novel defects outside the training set based on natural language descriptions, and demonstrating stronger generalization ability and flexibility.

[0056] Furthermore, the detection method provided in this embodiment has a training time of 12 hours, significantly less than the 40 hours required for training from scratch. This demonstrates that the knowledge distillation pre-training strategy proposed in this invention can significantly shorten the model's training time and reduce training costs. Compared to the training from scratch method, the detection method reduces training time by 70%, verifying that knowledge distillation pre-training can quickly provide the model with basic semantic understanding capabilities, avoiding the need to build image-text associations from scratch and significantly saving training time and computational resources.

[0057] Example 2 This embodiment provides a lightweight intelligent damage detection system for photovoltaic panels based on open vocabulary, referencing... Figure 2 As shown, it includes: The training data construction module adds semantic annotations to the publicly available photovoltaic defect dataset and self-taken photovoltaic panel defect images to form a defect dataset. The basic image dataset and a portion of the defect dataset are mixed according to a preset ratio to form the first pre-training dataset, and the remaining defect dataset is used as the second pre-training dataset. The knowledge distillation module uses a pre-trained open vocabulary target detection model as the teacher model and the detection model to be trained as the student model. It distills the intermediate layer features and category prediction distribution of the teacher model respectively. The student model learns basic semantic understanding ability through the first pre-training dataset and uses the second pre-training dataset to perform defect-specific training on the student model to obtain pre-training weights. The image processing module acquires images of the photovoltaic panel to be detected and performs preprocessing to obtain a fused multi-scale feature map and an updated text embedding vector. The fused multi-scale feature map and the updated text embedding vector are then input to the detection output module. The detection output module decouples the detection head from the bounding box and the target embedding vector. It constructs the region-text contrast loss by calculating the cross-entropy between the similarity between the target and the text and the distribution relationship between the target and the text, and obtains the total training loss. It then generates rectangular localization boxes with category labels or user prompts and guaranteed probabilities.

[0058] The system provided in this embodiment achieves feature transformation through spatial depth transformation operations. Figure 1 One-time cross-layer splicing replaces the multi-layer convolution and sampling operations of traditional FPN networks, significantly reducing the number of model parameters. The total number of model parameters is reduced by more than 60% compared to existing open vocabulary detection algorithms, making it easier to deploy on resource-constrained terminal devices. By integrating public photovoltaic defect datasets and real-world private defect datasets, and strengthening the semantic annotation of defect location and shape, and using basic image datasets to assist training, the model can fully learn defect features. The defect enhancement activation function distinguishes defect edges and background regions through gradients, enhancing the response to defect features and suppressing background noise. Deformable convolution adapts to the feature extraction requirements of irregular defects in photovoltaic panels, improving the defect class recognition accuracy by more than 25% compared to existing methods, effectively solving the problem of defects being easily misclassified or ignored.

[0059] Example 3 This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the photovoltaic panel damage detection method based on open vocabulary described in Embodiment 1. For details, please refer to Embodiment 1.

[0060] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A photovoltaic panel damage detection method based on open vocabulary, characterized in that, Includes the following: To construct a training dataset, semantic annotations are added to the publicly available photovoltaic defect dataset and self-taken photovoltaic panel defect images to form a defect dataset. The basic image dataset and a portion of the defect dataset are mixed in a preset ratio to form the first pre-training dataset, and the remaining defect dataset is used as the second pre-training dataset. Knowledge distillation pre-training is performed, using the pre-trained open vocabulary target detection model as the teacher model and the detection model to be trained as the student model. The intermediate layer features and category prediction distribution of the teacher model are distilled separately, and the student model learns basic semantic understanding ability through the first pre-training dataset. Freeze the knowledge distillation module of the student model, and use the second pre-training dataset to perform defect-specific training on the student model to obtain pre-training weights; Images of the photovoltaic panels to be detected are acquired and preprocessed to obtain fused multi-scale feature maps and updated text embedding vectors. The fused multi-scale feature maps and updated text embedding vectors are input to the detection output module. The detection output module regresses bounding boxes and target embedding vectors by decoupling the detection head. By calculating the similarity between the target and the text and the cross-entropy between the target and the text, a region-text contrast loss is constructed, and the total training loss is obtained. Rectangular localization boxes with category labels or user prompts and guaranteed probabilities are generated.

2. The photovoltaic panel damage detection method based on open vocabulary according to claim 1, characterized in that, The process involves acquiring and preprocessing images of the photovoltaic panel to be detected to obtain a fused multi-scale feature map and an updated text embedding vector, including the following: The preprocessed image is input into the defect feature extraction module, which includes a minimum convolutional unit module and a multi-convolutional layer aggregation network. The minimum convolutional unit module includes deformable convolutional layers, batch normalization layers and defect enhancement activation functions. The multi-convolutional layer aggregation network outputs multi-scale feature maps. The multi-scale feature maps and text embedding vectors are input into the image-text fusion module. The image-text fusion module reorganizes the pixel values ​​of the feature maps at different scales into the channel dimension through spatial depth transformation to achieve resolution alignment. Then, the image features and text features are fused through text-guided Ghost convolution and cross-attention mechanism, and the fused multi-scale feature maps and updated text embedding vectors are output.

3. The photovoltaic panel damage detection method based on open vocabulary according to claim 2, characterized in that, The defect feature extraction module also includes a DS submodule, which is divided into two branches. The left branch is used to perform max pooling, and the right branch performs a 3×3 convolution. The two feature maps are then concatenated and output. The multi-convolutional layer aggregation network includes two branches. The left branch contains six DBD modules and one Concat module. The four 3×3 convolutional modules in the DBD module are used to obtain feature maps at different levels and aggregate features at different levels through dense residual connections. The right branch is the total residual connection of the multi-convolutional layer aggregation network to help the multi-convolutional layer aggregation network perform gradient backpropagation better.

4. The photovoltaic panel damage detection method based on open vocabulary according to claim 2, characterized in that, The expression for the defect-enhancing activation function is: Where α is a hyperparameter greater than 0, and τ is the gradient threshold; The defect enhancement activation function employs two activation modes: when a high gradient region (defect edge) is detected in the feature map, exponential activation is used to enhance the defect feature response; linear activation is used for uniform background regions to suppress noise amplification. The gradient threshold adjustment strategy is as follows: The gradient map G is obtained by automatic differentiation of the input feature map x; Calculate the global average gradient: make Where i and j refer to the height and width of the gradient map G, respectively, H is the height of the input feature map, and W is the width of the input feature map. λ is the global average gradient, and λ is a hyperparameter that determines the model's sensitivity to image gradients.

5. The photovoltaic panel damage detection method based on open vocabulary according to claim 2, characterized in that, The image-text fusion module includes two sub-modules: SD-Ghost and I-Pooling Attention. Through SDT / DST operations, the pixel values ​​of the multi-scale feature map output by the defect feature extraction module are completely reconstructed to the channel dimension, preserving all pixel values ​​and spatial positional relationships, thus achieving lossless resolution conversion. The fully aligned feature map obtained through spatial depth transformation is used as input into the text-guided Ghost convolution module to achieve text-image feature fusion.

6. The photovoltaic panel damage detection method based on open vocabulary according to claim 5, characterized in that, The text-guided Ghost convolution and cross-attention mechanism fuses image and text features, outputting a fused multi-scale feature map and an updated text embedding vector, including the following: Given a text embedding vector T and image features After the Ghost convolution operation, a maximum sigmoid attention mechanism is added to aggregate text features into image features in the following way: Where δ is the Sigmoid function, C is the vocabulary size, and T is the lexicon. j Let represent the vector representation of the j-th word, ( ) T For transpose operator; Unified scale feature map that incorporates semantics After undergoing the inverse SDT / DST transformation, the image is transformed again into a multi-scale feature map, which is then input into the I-Pooling Attention submodule. The I-Pooling Attention submodule applies max pooling to the multi-scale features to obtain 2×2 regions, thereby generating multiple image patch labels. The text embedding vector is updated in the following way: in, W is the updated value of the text embedding vector. Multihead-Attention refers to the multihead attention calculation mechanism, which uses the query vector as a reference to calculate the associated feature weights of the key vector and performs weighted fusion on the value vector, thereby adaptively focusing on the key feature information of the defect. The image marker block obtained after the pooling operation.

7. The photovoltaic panel damage detection method based on open vocabulary according to claim 1, characterized in that, The calculation process for the total training loss is as follows: The similarity between the target and the text is obtained using the following formula. : in, It is an L2 normalization operation, t j It is the j-th text embedding vector, where α is a learnable scaling factor and β is a learnable offset factor; The region-text contrast loss L is obtained using the following formula. con for: Where, N pos The number of positive sample prediction boxes (obtained from label assignment); s i,yi For the i-th positive sample prediction box and the ground truth text label y i Similarity score; s i,j The similarity score between the i-th predicted bounding box and the j-th text. Using cross-union ratio loss L iou and distribution focus loss L dfl To perform bounding box regression, the total training loss is... Defined as: in It is an indicator factor.

8. The photovoltaic panel damage detection method based on open vocabulary according to claim 1, characterized in that, The distillation of the intermediate layer features and category prediction distributions of the teacher model includes the following: Loss function through characteristic distillation Distillation of intermediate layer features in the teacher model: Where l represents the feature of the l-th layer. , H represents the multi-scale features of the intermediate layers of the teacher model and the student model, respectively. l W l D l These represent the height, width, and number of channels of the feature in layer l, respectively. Loss function through soft label distillation Distillation of the class prediction distribution of the teacher model: in, Output an object-text similarity matrix for the teacher model. , Output the values ​​of coordinates (k, j) and (k, m) in the object-text similarity matrix for the teacher model. , Output the values ​​of coordinates (k, j) and (k, m) in the object-text similarity matrix for the student model. Output a similarity matrix for the student model, where K is the number of prediction boxes, C is the vocabulary size, and t is the temperature parameter.

9. A lightweight intelligent damage detection system for photovoltaic panels based on open vocabulary, characterized in that, include: The training data construction module adds semantic annotations to the publicly available photovoltaic defect dataset and self-taken photovoltaic panel defect images to form a defect dataset. The basic image dataset and a portion of the defect dataset are mixed according to a preset ratio to form the first pre-training dataset, and the remaining defect dataset is used as the second pre-training dataset. The knowledge distillation module uses a pre-trained open vocabulary target detection model as the teacher model and the detection model to be trained as the student model. It distills the intermediate layer features and category prediction distribution of the teacher model respectively. The student model learns basic semantic understanding ability through the first pre-training dataset and uses the second pre-training dataset to perform defect-specific training on the student model to obtain pre-training weights. The image processing module acquires images of the photovoltaic panel to be detected and performs preprocessing to obtain a fused multi-scale feature map and an updated text embedding vector. The fused multi-scale feature map and the updated text embedding vector are then input to the detection output module. The detection output module decouples the detection head from the bounding box and the target embedding vector. It constructs the region-text contrast loss by calculating the cross-entropy between the similarity between the target and the text and the distribution relationship between the target and the text, and obtains the total training loss. It then generates rectangular localization boxes with category labels or user prompts and guaranteed probabilities.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the photovoltaic panel damage detection method based on any one of claims 1-9.

Images