Lightweight infrared thermal imaging solar panel fault detection method, device and equipment based on deep learning and storage medium

Abstract

The invention relates to the technical field of solar panel fault detection, in particular to a lightweight infrared thermal imaging solar panel fault detection method, device and equipment based on deep learning and a storage medium, and the detection method comprises the following steps: S1, obtaining a solar panel image through a low-altitude infrared thermal imaging method; s2, establishing a fault detection model and performing training; s3, importing the solar panel image into the trained fault detection model to obtain a solar panel fault detection result; according to the invention, the advantages of DETR and YOLO are combined to design a low-altitude unmanned aerial vehicle infrared thermal imaging defect detection model; a RepHGcone backbone network adaptive to a YOLO framework is designed, and the global feature extraction capability under the complex background of a DETR class model is introduced while network parameters are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of solar panel fault detection, and specifically to a lightweight infrared thermal imaging solar panel fault detection method, device, equipment and storage medium based on deep learning. Background Art

[0002] Nowadays, the rapid development of photovoltaic power stations has put forward higher and higher requirements for on-site maintenance and fault diagnosis of photovoltaic components. In the existing technology, some deep learning methods are used to detect solar panel faults. However, these deep learning methods are faced with the contradiction between lightweight and accuracy. The transformer network with high model accuracy and high reliability has very high requirements for hardware performance and data sets, while the Yo Lo network that can detect in real time is faced with the problems of low accuracy and poor detection ability of small targets.

[0003] In summary, how to design a solar panel fault detection method so that the detection model it adopts has good overall performance, strong small target detection capability, can accurately detect faults, and has a high degree of lightweight has become a problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of the present invention is to provide a lightweight infrared thermal imaging solar panel fault detection method, device, equipment and storage medium based on deep learning to solve the above problems.

[0005] In order to achieve the above objectives, the following technical solutions are provided:

[0006] A lightweight infrared thermal imaging solar panel fault detection method based on deep learning includes the following steps:

[0007] S1: Acquire solar panel images using low-altitude infrared thermal imaging method;

[0008] S2: Establish a fault detection model and train it;

[0009] S3: importing the solar panel image into the trained fault detection model to obtain the solar panel fault detection result;

[0010] Among them, the fault detection model training steps are as follows:

[0011] Take all defect type images in infrared thermal imaging, divide the defect type images into training set, verification set and test set in the ratio of 8:1:1, train the model, and obtain the trained fault detection model;

[0012] Among them, the learning rate is 0.01, the number of data loaders is 8, and the number of training epochs is 200; the defect types include: bird droppings and dirt on the module surface, vegetation on the module surface, single-cell thermal anomaly during module operation, battery rupture, and multi-cell thermal anomaly.

[0013] Preferably,

[0014] The structure of the trained fault detection model includes:

[0015] Backbone network, Neck network and Head network;

[0016] The backbone network includes:

[0017] The input module I nput, the HGStem module, the first RepHG module, the first DWConv module, the second RepHG module, the second DWConv module, the third RepHG module, the fourth RepHG module, the fifth RepHG module, the third DWConv module, the sixth RepHG module, and the SPPF module are connected in sequence;

[0018] The neck network includes:

[0019] The first upsampling module, the first splicing module, the first CMSPC module, the second upsampling module, the second splicing module, the second CMSPC module, the first Conv module, the third splicing module, the third CMSPC module, the second Conv module, the fourth splicing module, and the fourth CMSPC module are connected in sequence;

[0020] Among them, the first upsampling module of the neck network upsamples the output result of the SPPF module in the backbone network;

[0021] The output result of the third RepHG module also enters the first splicing module to participate in splicing;

[0022] The output result of the fifth RepHG module also enters the second splicing module to participate in splicing;

[0023] The output result of the SPPF module also enters the fourth splicing module for splicing;

[0024] The head network includes:

[0025] A first DCD head module, a second DCD head module, and a third DCD head module connected in parallel;

[0026] The first DCD head module is connected to the second CMSPC module;

[0027] The second DCD head module is connected to the third CMSPC module;

[0028] The third DCD head module is connected to the fourth CMSPC module.

[0029] Preferably,

[0030] The RepHG module comprises:

[0031] Multiple parallel RLCVR modules, Squeeze Conv modules and Excitation Conv modules;

[0032] Each RLCVR module includes a Conv layer and a RepConvBNReLU layer connected in sequence, and the output results of the multiple parallel RLCVR modules are spliced ​​and connected to the Squeeze Conv module and the Excitation Conv module in sequence.

[0033] Preferably,

[0034] The CMSPC module comprises:

[0035] The 3x3 Conv BN Si LU layer, the Sp l it layer, the CMSPC black layer, the Concat layer and the 3x3 Conv BN Si LU layer are connected in sequence, and the CMSPC black layer includes multiple CMSPC blacks; the output result of the Sp l it layer and the output results of the multiple CMSPC blacks are also spliced ​​in the Concat layer;

[0036] The CMSPC block includes a 3x3 Conv BN Si LU layer, an MSPConv module and an Add layer connected in sequence, and the input content in the CMSPC block also enters the Add layer for addition.

[0037] Preferably,

[0038] In the MSPConv module, the specific operations on the input content are as follows:

[0039] First, the input channel is equally divided into the first channel, the second channel, the third channel, and the fourth channel. The first channel performs a 3×3 standard convolution operation, the second channel performs a 5×5 standard convolution operation, and the third and fourth channels remain unchanged and perform identity mapping. The first, second, third, and fourth channels are re-merged and output through a 1x1 standard convolution layer.

[0040] Preferably,

[0041] The DCD head module comprises:

[0042] The first CBS module, the second CBS module and the Concat module are connected in sequence;

[0043] The output result of the Concat module is divided into four branches, which are respectively connected to the mask&offset module, the first TaskDecomposition module, the second TaskDecomposition module, and the Sign module;

[0044] Among them, the first TaskDecomposition module is connected to the DCNV2 module and the Bbox.Loss module in sequence; the second TaskDecomposition module is connected to the Multi module and the Cls.Loss module in sequence;

[0045] The mask&offset module is connected to the DCNV2 module, and the Sigmoid module is sequentially connected to the Conv module, the Conv+ReLU module, and the Multiply module.

[0046] The present invention also provides a light-weight infrared thermal imaging solar panel fault detection device based on deep learning, which is applied to the light-weight infrared thermal imaging solar panel fault detection method based on deep learning.

[0047] include:

[0048] Infrared thermal imaging unit for acquiring images of solar panels;

[0049] A model building unit, used for building a fault detection model;

[0050] A training unit, used for training a fault detection model;

[0051] The output unit is used to output the detection result.

[0052] The present invention also provides a lightweight infrared thermal imaging solar panel fault detection device based on deep learning.

[0053] The device comprises: a processor and a memory storing computer program instructions;

[0054] When the processor executes the computer program instructions, the lightweight infrared thermal imaging solar panel fault detection method based on deep learning is implemented.

[0055] The present invention also provides a computer-readable storage medium,

[0056] The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the lightweight infrared thermal imaging solar panel fault detection method based on deep learning is implemented.

[0057] The beneficial effects of the present invention are:

[0058] 1. The present invention combines the advantages of DETR and YOLO to design a low-altitude UAV infrared thermal imaging defect detection model; designs a RepHGbone backbone network adapted to the YOLO framework, while reducing network parameters, it introduces the global feature extraction capability of the DETR-like model under complex backgrounds; at the same time, since the position of the UAV in the air is uncertain, the size of the infrared thermal imaging defect varies with the height of the UAV, and the present invention improves the network's multi-scale extraction capability for defects of different sizes, and improves the overall performance of the lightweight model and the small target detection capability;

[0059] 2. The present invention designs a lightweight dynamic coordination detection head, which is not a decoupling head that simply separates classification and detection, but rather reasonably decomposes the classification and inspection heads, which greatly improves the detection capability of small targets in complex backgrounds and the accuracy of the model; experiments show that the model has achieved state-of-the-art performance, with an accuracy of 89.14% for online data enhancement when the parameter is 3.14m; the proposed lightweight, high-performance model can be easily deployed on UAV equipment, and can accurately detect faults in the face of detections at different heights and interference from different complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A network structure diagram of a fault detection model of the present invention;

[0061] Figure 2 This is the RepHG module structure diagram;

[0062] Figure 3 This is the CMSPC module structure diagram;

[0063] Figure 4 This is the CMSPC black block structure diagram;

[0064] Figure 5 This is the structure diagram of the MSPConv module;

[0065] Figure 6 This is the DCD head module structure diagram; DETAILED DESCRIPTION

[0066] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions of the schematic diagrams in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Example:

[0068] In this embodiment, a drone is used to obtain solar panel images using a low-altitude infrared thermal imaging method. The network structure of the fault detection model is as follows: Figure 1 As shown in the figure, RepConv is introduced. The process of RepConv is to use a multi-branch structure first, which can bring higher accuracy. Then, convolution (Conv) and normalization (BN) are integrated to reduce the amount of calculation. At the same time, the structure is equivalently converted into a single-branch structure through structural reparameterization technology during reasoning, which can keep the accuracy unchanged while bringing faster reasoning speed. Considering the unreliability of RT-DETR for targets of different sizes, RepHG block is constructed, which uses multiple convolutional layers to allow each convolutional layer to extract features at different levels and the expression ability of rich features to improve the multi-scale feature extraction ability of the model, and all convolutional layers share the same hyperparameters, which can reduce the number of parameters. Then, SE attention is imitated, and compression and excitation convolution are used to reduce the model again while paying attention to more important features. It imitates HGNetV2, while extracting rich and deep multi-scale features, and maximizes the lightweight design, so that the model can better adapt to drones and give full play to the performance of the model on hardware with low computing power.

[0069] The network structure details of the fault detection model of this embodiment are shown in Table 1 below;

[0070] Table 1. Details of the fault detection model network structure

[0071] Layer number Layer Name Output shape C0 HGStem (32,48) C1 RepHGBlock (48,128,3) C2 DWConv (128,3,2,1,False) C3 RepHGBlock (96,512,3) C4 DWConv (512,3,2,1,False) C5 RepHGBlock (192,1024,3,True,False) C6 RepHGBlock (192,1024,3,True,True) C7 RepHGBlock (192,1024,3,True,True) C8 DWConv (1024,3,2,1,False) C9 RepHGBlock (384,2048,3,True,False) C10 SPPF (1024,5)

[0072] like Figure 2 As shown in the figure, the simplified idea of ​​RepConv is to unify convolutions of different sizes into 3x3, and then combine 3x3 convolution and BN. The main idea of ​​reparameterization is to combine convolution (Conv) and normalization (BN) to reduce the amount of calculation;

[0073] like Figure 3-5As shown in the figure, the complexity of the backbone network structure directly affects the ability to extract image features. Since the infrared images taken by drones are very complex, and faults often have different shapes and sizes, it is not easy to detect. Previous work has introduced an attention mechanism to extract features efficiently. However, since drones are small hardware devices with limited computing power and memory resources, if the model is too complex, it will be very bad for deployment.

[0074] To this end, a multi-scale partial convolution is proposed. It introduces multiple convolutions with different kernel sizes and retains half of the channels to enable it to capture spatial features of multiple scales while reducing the number of parameters. This convolution fully considers the balance between computational efficiency and multi-scale feature acquisition. Figure 5 As shown in the figure, the input channel is first divided into four equal parts, of which the first part is subjected to a 3×3 convolution operation, and the second part is subjected to a 5×5 convolution operation. At the same time, in order to enhance the information interaction between various convolutions, the features of different granularities after various convolutions are rearranged according to the channel dimension and batch dimension. Since the drone collects infrared solar panel images in the air, most of the fault heat sources are small targets, and the convolution kernels of the first and second layers are variable, convolution layers with convolution kernels of 3 and 5 are used. Therefore, the receptive field range and fine-grained features can be adjusted by simply adjusting the size and number of convolution kernels. For the remaining part, the channel is kept unchanged and an identity mapping is performed. This step is mainly to reduce the amount of calculation of the model while retaining the feature information flow. Finally, point-by-point convolution is used to aggregate cross-group information of all features to achieve multi-channel information interaction and fuse multi-channel information for output. Finally, the two parts are re-merged and output through a 1x1 convolution layer. The advantage of using a 1×1 convolution layer is that multi-channel information interaction and fusion of multi-channel information are possible for output. This can improve the perception of information of different scales while retaining low-cost convolution.

[0075] like Figure 6As shown in the figure, the existing detection head is in the form of a decoupled head, and its structure is to achieve classification and positioning of the target in parallel. Such a dual-branch design may lead to a lack of interaction between the two tasks, and the lack of correlation when executing them may lead to inconsistent predictions; the goal of classification is to learn the discriminative features of the target, and the goal of positioning is to accurately locate the boundary of the target; so the two tasks are very different, which will also lead to a certain degree of spatial misalignment between the two tasks; at the same time, due to the existence of parallel design, the number of detection head parameters is too large, which will also affect the efficiency of defect detection, which contradicts the idea of ​​realizing lightweight drone deployment; when high-altitude infrared thermal imaging detects solar panels, the best anchor points for classification and positioning are usually inconsistent and may vary greatly depending on the shape and characteristics of the defects; and the previous detection head lacks linkage between classification and detection, and it does not make optimal allocation of classification and positioning, so it may be difficult to make accurate and consistent predictions for the two tasks, resulting in frequent occurrence of misclassified defects and low precision;

[0076] To this end, a dynamic collaborative detection head is designed. First, two CBSs are used to construct the input feature module to stack the original features after processing through a series of shared convolutional layers to obtain a feature map. This can retain the multi-scale information of different detection layers and enhance the collaborative interaction between classification and detection for subsequent task decomposition and prediction. However, when merging branches, conflicts between classification and detection are inevitably induced because the classification and detection tasks focus on different things. Therefore, in order to enhance the collaborative interaction between classification and detection and reduce the conflict between the two, a task decomposition module is introduced to reasonably decompose the classification and detection tasks by dynamically calculating the features of the classification and detection tasks.

[0077] The task decomposition module can coordinate the focus of classification and localization tasks, so that the two tasks will not conflict and increase the interactivity between them. Although the two are aligned after using the task decomposition module, they still need to be operated separately according to the different focus of classification and detection. Generally speaking, the feature representations required for classification and regression tasks are different, so task decomposition can better meet the needs of these two tasks.

[0078] The original feature map can be decomposed into the feature representations required for different tasks; it helps the model learn better task-related features and improve the performance of classification and regression; however, the decoupling decomposition method of the YOLOv8 detection head does not take into account the differences between classification and positioning, and simply decomposes them; the introduced task decomposition module performs spatial alignment on the feature map of the regression task;

[0079] Then, the feature map of the classification task is subjected to classification alignment processing, and the feature map is processed through two convolutional layers to obtain the classified feature map;

[0080] Finally, CI oU and DFL LOSS are used for detection tasks, and BCE LOSS is used for classification tasks to specify the degree of inconsistency between the model's predicted value and the true value; in this way, classification and detection achieve the prediction of the classification and regression tasks of the input feature map through task decomposition, spatial alignment and fusion operations;

[0081] The solar panel fault detection experiment is as follows:

[0082] Aerial photography was performed using a drone with a resolution of 640×640, taken on a sunny day; the wind speed was about 8 m / s;

[0083] Shooting during the day makes image processing clearer; when the solar panels are running, fault hot spots will be better detected; and the probability of accidents during rainy day inspections is higher than that during sunny days, and the infrared sensing ability is too poor; inspections should also be carried out during the day, when there is sufficient light and the infrared detection effect is very good. At night, because the solar panels are not running, effective fault data cannot be collected; solar power plants are often built in sparsely populated areas with plenty of sunshine, and are often accompanied by strong winds. However, the wind speed has no effect on the drone infrared detection algorithm, but the requirements for drones are different; using drones for remote monitoring will not affect the normal operation of solar panels;

[0084] In order to collect a more comprehensive and reliable data set, infrared detection drones are used to perform infrared detection of solar panels in different airspaces; mobile work platforms are used to fly above solar cells; in this case, being far away from the target may be an advantage because a large area can be inspected at one time; in order to ensure the quality of thermal images, thermal imagers used for long-distance inspections have higher image resolution; thermal imagers are also equipped with interchangeable lenses to enable drone operators to perform low-altitude inspections at different altitudes; when the drone exceeds 100m, its infrared thermal sensing capability will drop significantly, which will greatly reduce the accuracy of defect detection, so the data collection range and drone working range are 1-100m, and its detection has higher reliability; at 1m, the fault hotspot is large and easy to detect, and at 50m, there is a smaller hotspot; when the drone is at 100m, the fault hotspot is small, which is a great test for the model's detection ability for small targets; 897 infrared thermal sensing images taken by drones at different altitudes were collected, and 312 infrared thermal sensing images of solar panels collected from the Internet were used to improve the versatility of the model; data enhancement was then performed to obtain a total of 4838 images, and then the labe l img are annotated one by one, and the data format is YOLO format; the dataset is randomly divided into training set, validation set and test set in the ratio of 8:1:1; the learning rate is 0.01, the number of data loaders is 8, and the number of training epochs is 200; the size of training and test images is set to (640, 640) using YOLO's padding function;

[0085] There are fault hot spots and hot spots in the solar panels in the collected images; 80% of the images, i.e. 3,870 images, were randomly selected as the training set, 10% of the images, a total of 484 images, were selected as the test set, and the remaining 10% of the images, a total of 484 images, were selected as the verification set; normal defect types include dirt, plants, thermal cells, multi-thermal cells, open circuits, battery cracks, DC box components, etc., but they appear as consistent hot spots and heating areas in the low-altitude thermal imaging of drones, as shown in Table 2 below:

[0086] Table 2. Defect types

[0087]

[0088] Results comparison:

[0089] In order to verify the influence of module mixing on model performance, model comparison was carried out; as shown in Table 3, the above data sets were trained respectively; 200 training epochs were used as the final measurement standard; it can be seen from Table 3 that the mAP@0.5 of YOLOv8n is 80.51%; when RepHGbone is used to replace the original backbone network, mAP@0.5 is improved, and the model size and GFLOPs are greatly reduced; the RepHGbone used is designed in combination with Repconv, and its multi-branch shared parameter operation and the operation of merging Conv and BN reduce the calculation amount and size of the model, and due to the use of compressed and expanded convolutional layers, its feature extraction ability is also enhanced; compared with B and C, mAP is improved a lot; compared with B and D, mAP is improved, and the designed dynamic collaborative detection head has a very good effect. Due to the addition of dynamic convolution in the detection head, the model has a high accuracy when detecting under complex backgrounds, and combined with the task decomposition module, classification and detection do not interfere with each other, and there is feature interaction; and the comparison of other modules added also proves the reliability of the model designed in this embodiment.

[0090] Table 3. Model comparison

[0091]

Claims

1. A lightweight infrared thermal imaging solar panel fault detection method based on deep learning, characterized in that: The following steps are involved: S1: Acquire solar panel images using low-altitude infrared thermal imaging method; S2: Establish a fault detection model and train it; S3: importing the solar panel image into the trained fault detection model to obtain the solar panel fault detection result; Among them, the fault detection model training steps are as follows: Take all defect type images in infrared thermal imaging, divide the defect type images into training set, verification set and test set in the ratio of 8:1:1, train the model, and obtain the trained fault detection model; Among them, the learning rate is 0.01, the number of data loaders is 8, and the number of training epochs is 200; the defect types include: bird droppings and dirt on the module surface, vegetation on the module surface, single-cell thermal anomaly during module operation, battery rupture, and multi-cell thermal anomaly.

2. According to the deep learning-based lightweight infrared thermal imaging solar panel fault detection method of claim 1, it is characterized in that: The structure of the trained fault detection model includes: Backbone network, Neck network and Head network; The backbone network includes: The input module Input, the HGStem module, the first RepHG module, the first DWConv module, the second RepHG module, the second DWConv module, the third RepHG module, the fourth RepHG module, the fifth RepHG module, the third DWConv module, the sixth RepHG module, and the SPPF module are connected in sequence; The neck network includes: The first upsampling module, the first splicing module, the first CMSPC module, the second upsampling module, the second splicing module, the second CMSPC module, the first Conv module, the third splicing module, the third CMSPC module, the second Conv module, the fourth splicing module, and the fourth CMSPC module are connected in sequence; Among them, the first upsampling module of the neck network upsamples the output result of the SPPF module in the backbone network; The output result of the third RepHG module also enters the first splicing module to participate in splicing; The output result of the fifth RepHG module also enters the second splicing module to participate in splicing; The output result of the SPPF module also enters the fourth splicing module for splicing; The head network includes: A first DCD head module, a second DCD head module, and a third DCD head module connected in parallel; The first DCD head module is connected to the second CMSPC module; The second DCD head module is connected to the third CMSPC module; The third DCD head module is connected to the fourth CMSPC module.

3. According to the deep learning-based lightweight infrared thermal imaging solar panel fault detection method of claim 2, it is characterized in that: The RepHG module comprises: Multiple parallel RLCVR modules, Squeeze Conv modules, and Excitation Conv modules; Each RLCVR module includes a Conv layer and a RepConvBNReLU layer connected in sequence, and the output results of the multiple parallel RLCVR modules are spliced ​​and connected to the Squeeze Conv module and the Excitation Conv module in sequence.

4. According to the method of lightweight infrared thermal imaging solar panel fault detection based on deep learning in claim 3, it is characterized in that: The CMSPC module comprises: The 3x3 Conv BN SiLU layer, Split layer, CMSPC block layer, Concat layer and 3x3 ConvBN SiLU layer are connected in sequence. The CMSPC block layer includes multiple CMSPC blocks. The output result of the Split layer and the output results of multiple CMSPC blocks are also spliced ​​in the Concat layer. The CMSPC block includes a 3x3 Conv BN SiLU layer, an MSPConv module and an Add layer connected in sequence, and the input content in the CMSPC block also enters the Add layer for addition.

5. The method for detecting solar panel faults using lightweight infrared thermal imaging based on deep learning according to claim 4, characterized in that: In the MSPConv module, the specific operations on the input content are as follows: First, the input channel is equally divided into the first channel, the second channel, the third channel, and the fourth channel. The first channel performs a 3×3 standard convolution operation, the second channel performs a 5×5 standard convolution operation, and the third and fourth channels remain unchanged and perform identity mapping. The first, second, third, and fourth channels are re-merged and output through a 1x1 standard convolution layer.

6. A light-weight infrared thermal imaging solar panel fault detection method based on deep learning according to claim 5, characterized in that: The DCD head module comprises: The first CBS module, the second CBS module and the Concat module are connected in sequence; The output result of the Concat module is divided into four branches, which are connected to the mask&offset module, the first TaskDecomposition module, the second TaskDecomposition module, and the Sigmoid module respectively; Among them, the first TaskDecomposition module is connected to the DCNV2 module and the Bbox.Loss module in sequence; the second TaskDecomposition module is connected to the Multiply module and the Cls.Loss module in sequence; The mask&offset module is connected to the DCNV2 module, and the Sigmoid module is connected to the Conv module, the Conv+ReLU module, and the Multiply module in sequence.

7. A light-weight infrared thermal imaging solar panel fault detection device based on deep learning, applied to a light-weight infrared thermal imaging solar panel fault detection method based on deep learning as claimed in claim 6, characterized in that: include: Infrared thermal imaging unit for acquiring images of solar panels; A model building unit, used for building a fault detection model; A training unit, used for training a fault detection model; The output unit is used to output the detection result.

8. A lightweight infrared thermal imaging solar panel fault detection device based on deep learning, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the deep learning-based lightweight infrared thermal imaging solar panel fault detection method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the deep learning-based lightweight infrared thermal imaging solar panel fault detection method as described in any one of claims 1 to 6 is implemented.

Images