Unmanned aerial vehicle visual angle large-scale photovoltaic module infrared image defect instance segmentation method

Abstract

The invention relates to the technical field of photovoltaic power generation, and discloses an unmanned aerial vehicle visual angle large-scale photovoltaic module infrared image defect instance segmentation method, which comprises the following steps: S1, data acquisition: carrying out infrared image acquisition on a photovoltaic module in a photovoltaic power station by using an unmanned aerial vehicle; s2, image preprocessing: carrying out preprocessing of noise removal and contrast enhancement on the acquired infrared image; s3, feature extraction: adopting StarNet as a feature extraction network to extract high-dimensional features; s4, multi-head attention mechanism introduction: multi-dimensional feature optimization is carried out by introducing MHSA; s5, instance segmentation: segmenting the feature information by using an improved YOLOv8 segmentation head PIS-SegHead, and generating an instance mask; and S6, defect identification and analysis: further analyzing each instance mask. According to the method, feature optimization is performed through the StarNet feature extraction network and star operation in combination with a multi-head self-attention mechanism (MHSA), different types of defects can be accurately recognized and classified, and the precision and applicability are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of photovoltaic power generation, and in particular to a method for segmenting defect instances of infrared images of large-scale photovoltaic components from the perspective of an unmanned aerial vehicle. Background Art

[0002] As the demand for sustainable energy continues to grow, the photovoltaic industry is experiencing remarkable and vigorous development in China and around the world. Photovoltaic technology, with its unique environmental protection and sustainability, provides a reliable energy solution for modern society and has become one of the important means to reduce dependence on fossil fuels, alleviate global climate change, and provide clean and green electricity. As an important pillar of renewable energy, photovoltaic technology is not only changing the traditional pattern of the energy field, but its future development will also have a profound impact on human society and the progress of science and technology.

[0003] However, with the rapid development of the photovoltaic industry, the widespread use of photovoltaic panels has improved the efficiency of renewable energy utilization, but has also gradually shown the challenges faced by long-term exposure to the natural environment. Harsh conditions in nature, such as wind and sand etching, rain erosion, and ice and snow loads, may cause physical damage to photovoltaic panels, such as surface cracks, layer peeling, or circuit interruptions. These defects will seriously affect the energy conversion efficiency and power output performance of photovoltaic panels, and increase potential circuit safety risks.

[0004] To address these issues, although traditional manual inspection and sensor inspection methods can diagnose defects to a certain extent, they are usually time-consuming, costly, and easily affected by human factors, resulting in certain errors. Summary of the invention

[0005] In order to make up for the above shortcomings, the present invention provides a defect instance segmentation method for large-scale photovoltaic module infrared images from a drone perspective, aiming to improve the traditional manual detection and sensor detection methods, which can perform defect diagnosis to a certain extent, but are time-consuming, costly, easily affected by human factors, and have certain errors.

[0006] In a first aspect, the present invention provides the following technical solution: a method for segmenting defect instances of infrared images of large-scale photovoltaic modules from a drone perspective, comprising the following steps:

[0007] S1. Data collection: Use the data collection platform carried by the drone to collect infrared images of photovoltaic components in the photovoltaic power station and obtain large-scale infrared image data;

[0008] S2. Image preprocessing: preprocessing the collected infrared images for noise removal and contrast enhancement;

[0009] S3. Feature extraction: StarNet is used as the feature extraction network to extract high-dimensional features;

[0010] S4. Introduction of multi-head self-attention mechanism: After feature extraction, the MHSA multi-head self-attention mechanism is introduced to perform multi-dimensional feature optimization;

[0011] S5. Instance segmentation: Use the improved YOLOv8 segmentation head PIS-SegHead to segment feature information, generate instance masks, and detect the defect location, category, size and shape of photovoltaic modules;

[0012] S6. Defect identification and analysis: Each instance mask is further analyzed to identify and annotate the location and characteristics of the defects.

[0013] Preferably, in step S1, the data acquisition platform carried by the drone is a pan-tilt camera with infrared thermal imaging function, and the pan-tilt camera with infrared thermal imaging function can acquire high-resolution infrared images.

[0014] Preferably, in the step S3, the StarNet used in the feature extraction step introduces a star operation, and the star operation maps the image features to a high-dimensional nonlinear feature space through an element-by-element product method.

[0015] Preferably, in the step S5, the PIS-SegHead segmentation head adopted in the instance segmentation step includes shared convolution and scale scaling operations.

[0016] Preferably, in step S6, defect identification includes detecting point spots, stripe spots and open circuit defects in the photovoltaic module, and generating corresponding segmentation masks.

[0017] In a second aspect, the present invention provides the following technical solution: a defect instance segmentation system for large-scale photovoltaic modules infrared images from a drone perspective, comprising:

[0018] A data acquisition module is used to collect infrared image data of photovoltaic components in a photovoltaic power station through an infrared thermal imaging camera carried by a drone;

[0019] An image preprocessing module is used to remove noise and enhance contrast of the acquired infrared images;

[0020] Feature extraction module, which uses StarNet network as the feature extraction module and combines star operation to extract high-dimensional nonlinear image features;

[0021] The attention mechanism module is used to optimize image features through the multi-head self-attention MHSA mechanism after feature extraction;

[0022] The instance segmentation module uses the improved YOLOv8 segmentation head PIS-SegHead to perform instance segmentation on the optimized image features and generate instance masks of PV module defects;

[0023] The defect recognition and analysis module is used to identify and analyze defects in photovoltaic modules, including the type, location and shape characteristics of the defects.

[0024] In the third aspect, the invention provides the following technical solution: a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the computer program, the above-mentioned method for segmenting defect instances of large-scale photovoltaic components in infrared images from a drone's perspective is implemented.

[0025] In a fourth aspect, the present invention provides the following technical solution: a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned method for segmenting defect instances of large-scale photovoltaic components in infrared images from a drone's perspective is implemented.

[0026] The present invention has the following beneficial effects:

[0027] 1. In the present invention, the StarNet feature extraction network and "star operation" are combined with the multi-head self-attention mechanism (MHSA) for feature optimization, so that the model can capture the detailed information in the infrared images of photovoltaic modules. The improved YOLOv8 segmentation head PIS-SegHead further enhances the instance-level segmentation capability, which can accurately identify and classify different types of defects, and has a significant improvement in accuracy and applicability.

[0028] 2. In the present invention, by combining the shared convolution and scale scaling operations in the StarNet network and PIS-SegHead, the amount of computation is significantly reduced, so that the algorithm can be deployed on edge devices with limited computing power. This lightweight design is particularly suitable for mobile platforms such as drones, breaking through the limitations of traditional computer vision methods in field applications and having high engineering application value.

[0029] 3. In the present invention, the embedded GPU carried by the drone can process the collected infrared images in real time to achieve automatic detection and rapid response. This automatic detection and real-time processing capability solves the problems of slow traditional manual detection, large manual errors, and difficulty in coping with large-scale application scenarios, so that the defects of photovoltaic modules can be analyzed while data is collected, greatly improving the efficiency and accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1This is the PIS-YOLO structure diagram of the defect instance segmentation method of large-scale photovoltaic module infrared images from the perspective of drones proposed in the present invention;

[0031] Figure 2 The StarNet structure diagram of the defect instance segmentation method of large-scale photovoltaic module infrared images from the perspective of drones proposed by the present invention;

[0032] Figure 3 The C2f and C2f-Star structures of the defect instance segmentation method of large-scale photovoltaic module infrared images from the perspective of a drone proposed in the present invention;

[0033] Figure 4 This is the PIS-SegHead structure diagram of the defect instance segmentation method of large-scale photovoltaic module infrared images from the perspective of drones proposed in the present invention;

[0034] Figure 5 This is the structural diagram of the MHSA attention mechanism of the defect instance segmentation method of large-scale photovoltaic module infrared images from the perspective of drones proposed in the present invention;

[0035] Figure 6 This is a comparison chart of the Box (a) and Mask (b) training curves of the various algorithms of the defect instance segmentation method of large-scale photovoltaic module infrared images from the perspective of a drone proposed in the present invention on the initial data set;

[0036] Figure 7 This is a comparison chart of Box and Mask training curves of various algorithms of the defect instance segmentation method of large-scale photovoltaic module infrared image from the perspective of drone proposed in the present invention on the enhanced data set;

[0037] Figure 8 This is a visualization comparison diagram of various algorithms for the defect instance segmentation method of large-scale photovoltaic module infrared images from the perspective of a drone proposed in the present invention;

[0038] Fig. 9 A schematic diagram of ablation experiment heat map visualization of the defect instance segmentation method of infrared images of large-scale photovoltaic modules from the perspective of a drone proposed in the present invention;

[0039] Fig.10 The different features of the data set of the defect instance segmentation method of large-scale photovoltaic module infrared images from the drone perspective proposed in the present invention, and the schematic diagram of the point spot (DB), streak (TB), and open circuit (DL) defects in the infrared images of photovoltaic modules in large-scale photovoltaic power stations. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Embodiment 1

[0042] Reference Figure 1-Figure 5 In a first embodiment of the present invention, the present invention provides a method for segmenting defect instances of infrared images of large-scale photovoltaic modules from a drone perspective, comprising the following steps:

[0043] S1. Data collection: Use the data collection platform carried by the drone to collect infrared images of photovoltaic components in the photovoltaic power station and obtain large-scale infrared image data;

[0044] S2. Image preprocessing: preprocessing the collected infrared images for noise removal and contrast enhancement;

[0045] S3. Feature extraction: StarNet is used as the feature extraction network to extract high-dimensional features;

[0046] S4. Introduction of multi-head self-attention mechanism: After feature extraction, the MHSA multi-head self-attention mechanism is introduced to perform multi-dimensional feature optimization;

[0047] S5. Instance segmentation: Use the improved YOLOv8 segmentation head PIS-SegHead to segment feature information, generate instance masks, and detect the defect location, category, size and shape of photovoltaic modules;

[0048] S6. Defect identification and analysis: Each instance mask is further analyzed to identify and annotate the location and characteristics of the defects.

[0049] Specifically, the PIS-YOLO algorithm is as follows Figure 1 As shown. First, the image is input into Backbone to extract feature information, and then the Neck part fuses the extracted feature information. Finally, the feature information is output to the segmentation head. The segmentation head generates a set of prototype masks through multiple layers of convolution, and at the same time generates mask coefficients associated with each detection box. The mask coefficients are then multiplied by the prototype mask to generate the final instance mask. The instance mask is then cropped according to the boundary of the detection box, and finally the category of each pixel is output through the activation function. The loss function uses binary cross entropy loss and Dice loss to process unbalanced data, and finally outputs a segmentation mask of the same size as the input image, indicating the probability that each pixel belongs to the target category.

[0050] In PIS-YOLO, the image is first input into the efficient feature extraction network StarNet, which innovatively introduces the "star operation" to enhance the feature extraction capability. Subsequently, the features are further optimized through the MHSA multi-head attention mechanism. In order to more effectively utilize the feature information extracted by Backbone, the C2f-Star module is proposed in combination with the "star operation" and applied to the Neck part of the network, replacing the original C2f module, further improving the effect of feature fusion. Finally, the rich feature information is input into the PIS-SegHead lightweight segmentation head, which outputs the category information of each pixel, generates a segmentation mask, and outputs the prediction result.

[0051] As an efficient convolutional neural network, StarNet innovatively introduces the "star operation". The "star operation" maps image features to a high-dimensional nonlinear feature space through element-wise multiplication. This method can significantly improve the feature expression capability without increasing the network width. The network structure of StarNet is as follows: Figure 2 As shown in the figure, its structure is very simple and mainly consists of convolutional layers and StarBlock modules. In the StarBlock module, the "star operation" is integrated to achieve efficient feature extraction and fusion. This network not only retains the advantages of traditional convolutional neural networks, but also enhances the high dimensionality and nonlinearity of feature representation through star operations.

[0052] The design goal of StarNet is to maximize the ability of feature representation while maintaining the compactness of the network. This ensures that the entire model can more accurately extract and express key information in the input data while running efficiently. Experiments have shown that StarNet not only has significant advantages in performance, but also maintains low latency when computing resources are limited, making it suitable for real-time application scenarios. The introduction of StarNet as a feature extraction network brings many significant advantages. First, the overall number of parameters of the model is significantly reduced, and the computational complexity is reduced, thereby accelerating the inference speed of the model. Second, StarNet's efficient feature expression capability ensures the accuracy of the model.

[0053] In order to make more effective use of the feature information extracted by Backbone, the "star operation" is integrated into the C2f module, and the C2f-Star module is proposed. The structure of the C2f-Star module is as follows: Figure 3As shown. By introducing the innovative "star operation" in StarNet, the C2f-Star module can map image features to a high-dimensional nonlinear feature space through element-wise multiplication, thereby significantly improving feature expression capabilities without increasing network width. This fusion not only retains the high efficiency of the C2f module, but also further enhances its feature extraction and expression effects, making the network perform better when processing complex image tasks. Through this improvement, the C2f-Star module achieves a balance between efficiency and accuracy while maintaining lightweight. Figure 3 (a) and (b) show the module structure diagrams of C2f and C2f-Star respectively.

[0054] The segmentation head of YOLOv8 draws on the idea of ​​YOLACT and decomposes the instance segmentation task into two steps. First, a set of prototype masks (PrototypeMasks) is generated, and then each detection head outputs a set of coefficients for each instance target, and these prototype masks are weighted and combined to obtain the final instance segmentation result. However, the segmentation head of YOLOv8 has obvious shortcomings. First, the segmentation head uses shared prototype masks, which are common to all instances. Although this method is computationally efficient, it may not capture the detailed features of the target when dealing with targets that require fine features, resulting in insufficient segmentation results. Secondly, since the prototype masks are globally shared, for targets of different scales, small targets may be ignored in the shared masks, and large targets may not be accurately segmented. In scenes with dense targets, the masks of multiple instances may overlap with each other, affecting the segmentation accuracy. Finally, the segmentation head of YOLOv8 contains a large number of convolutional layers and feature extraction layers, which makes its parameter volume very large. After actual testing, the segmentation head of YOLOv8n accounts for 30.7% of the parameters of the entire network. To address the above problems, a series of innovative designs are introduced and a new and efficient segmentation head is proposed: PIS-SegHead. Its network structure is as follows: Figure 4 shown.

[0055] The PIS-SegHead segmentation head introduces shared convolution, GroupNorm, and Scale scaling operations. First, BatchNorm is replaced by GroupNorm. Compared with BatchNorm, GroupNorm does not depend on the batch size, and is particularly effective for training small batches or even single images. The introduction of GroupNorm in the segmentation head can stably improve the accuracy of detection and segmentation under various batch sizes. Secondly, to address the problem of excessive computational complexity of the segmentation head, a shared convolution strategy is adopted. This strategy greatly reduces the number of parameters in the model, making the segmentation head more lightweight, thereby improving the computational efficiency of the model. Finally, to address the problem of inconsistent scales, while using shared convolution, a Scale layer is introduced to scale features. The Scale layer can appropriately scale features according to the requirements of different detection heads, thereby ensuring that each detection head can perform target detection and segmentation at the optimal scale. This method effectively alleviates the problem of decreased accuracy caused by inconsistent target scales. Finally, the original segmentation head and the optimized segmentation head of YOLOv8n were actually tested. The results showed that the number of parameters of the optimized segmentation head was 0.25M, the total model parameter number was 1.7M, and it accounted for 14.4% of the total calculation. In contrast, the number of parameters of the original segmentation head of YOLOv8 was 1,00M, and the total model parameter number was 3.26M, accounting for 30.7% of the total calculation. Through optimization, the number of segmentation head parameters was reduced by about 75.6%, and the total model parameter number was reduced by nearly half. This optimization significantly reduced the amount of calculation while maintaining a high degree of accuracy.

[0056] In step S1, the data acquisition platform carried by the drone is a gimbal camera with infrared thermal imaging function, which can obtain high-resolution infrared images.

[0057] Specifically, in the data collection stage, the drone selected the DJI M300RTK model, which has a long single flight time of up to 55 minutes and a load capacity of 2.7 kg. This makes it very suitable for inspections of large-scale photovoltaic power stations. This drone is equipped with a gimbal camera with infrared thermal imaging function, such as the DJI Zenmuse H20T. The gimbal camera integrates a variety of sensors, including a high-resolution (20 million pixel) visible light camera, and a 640×512 pixel infrared thermal imaging module for temperature imaging, with high-precision laser ranging function. These features enable it to capture high-resolution infrared images of photovoltaic modules, and can effectively identify hot spots or defects on photovoltaic panels.

[0058] In order to collect data efficiently, drones need to plan reasonable flight routes according to the layout of photovoltaic power stations. The flight path planning should ensure full coverage of photovoltaic modules and avoid blind spots that are not collected. The flight altitude, flight speed and viewing angle of the gimbal camera of the drone should be adjusted according to the arrangement and area of ​​the photovoltaic modules to ensure the clarity and effective coverage of the collected infrared images. During the flight, the acquisition platform needs to adjust the camera's shooting angle in real time to ensure that each component is in the best shooting position, thereby improving the quality of the collected images and the accuracy of subsequent detection.

[0059] In step S3, the StarNet used in the feature extraction step introduces the star operation, which maps image features to a high-dimensional nonlinear feature space through the element-by-element product method.

[0060] Specifically, in the feature extraction stage, the StarNet network is used. An important feature of the StarNet network is the introduction of the "Star Operation". The Star Operation maps image features to a high-dimensional nonlinear feature space by element-by-element multiplication. Specifically, after each convolution operation, the Star Operation performs element-by-element multiplication on the output features to generate more complex high-dimensional features. This process can better capture the detailed features in the image, especially when processing complex infrared images, which can help extract more effective information.

[0061] The StarNet network itself consists of multiple modules, mainly including convolutional layers, star operation layers, and pooling layers. The convolutional layer is responsible for extracting basic features of the image, such as edges and textures, while the star operation layer uses element-by-element multiplication to enhance the convolutional features, thereby forming a more complex high-dimensional feature expression, further improving the effect of feature extraction. The pooling layer is used to reduce the dimensionality of the extracted features, thereby reducing the computational complexity. Through the combination of these modules, the StarNet network ensures the ability to express features while also controlling the amount of computation, making it suitable for efficient real-time processing on mobile platforms such as drones.

[0062] In step S5, the PIS-SegHead segmentation head used in the instance segmentation step includes shared convolution and scale scaling operations.

[0063] Specifically, in the instance segmentation stage, an improved YOLOv8 segmentation head, called PIS-SegHead, is used. Shared convolution and scale scaling operations are introduced in PIS-SegHead to optimize the segmentation process. The design idea of ​​shared convolution is to reduce computational redundancy. Specifically, convolution operations are usually computationally intensive. Through shared convolution technology, multi-layer features can be shared, reducing the amount of computation and storage resources required for the model. This is very helpful for realizing a lightweight segmentation head, allowing PIS-SegHead to maintain efficient operation under limited computing power.

[0064] In addition, the scaling operation is designed to address the problem of different sizes of PV module defects in images. PV module defects vary in size, and scaling can be used to adaptively adjust features to facilitate the simultaneous detection of large and small defects, thereby improving segmentation accuracy. PIS-SegHead also adds a lightweight feature fusion layer to effectively fuse features extracted from different convolutional layers, which enables it to obtain more refined segmentation results for defects while maintaining computational efficiency.

[0065] In step S6, defect identification includes detecting point spots, stripe spots and open circuit defects in photovoltaic modules and generating corresponding segmentation masks.

[0066] Specifically, during the defect identification and analysis stage, the PIS-YOLO network generates a segmentation mask for each defect to accurately identify the location, type, and shape of the defect. The main defects of photovoltaic modules include spots, stripes, and open circuits, all of which affect the efficiency of the module. Specifically, spots are usually caused by overheating of silicon wafers or surface contaminants in local areas. These spots form local hot spots on the module, which may cause localized decreases in efficiency. Stripe defects are usually caused by broken or detached solder strips, which affect the normal flow of current and may further cause local failures of the module. The open circuit defect is caused by the disconnection of the circuit, which causes the entire module to fail to work properly.

[0067] During the defect identification process, the system generates a segmentation mask for each defect. The mask is a binary image used to describe the shape and area of ​​the defect. Through the segmentation mask, the system can further analyze the size and shape of the defect, such as the size of the spot, the length of the stripe, and the specific location of the open circuit. This information is very important for the subsequent formulation of repair plans. For example, smaller spots may only require surface cleaning, while larger or multiple spot components may require more complex processing methods. Through the analysis of the segmentation mask, those defects that have the greatest impact on power generation efficiency can be prioritized during maintenance, thereby improving the overall operating efficiency of the photovoltaic power station.

[0068] Experimental setup and results analysis:

[0069] Experimental environment and parameter settings

[0070] All experiments covered in this paper were completed on a deep learning workstation with a GPU configuration of NVIDIA RTX2080Ti11GB. Based on the above experimental conditions, we set the training epochs to 300, the batch size to 16, the initial learning rate to 0.01, the momentum to 0.937, the weight decay coefficient to 0.0005, and the SGD optimizer was used for optimization, and the momentum of SGD was set to 0.8. The input image size was 640×640, the number of workers was 8, and mosaic was used for data enhancement.

[0071] Comparative Analysis of PIS-YOLO and Advanced Instance Segmentation Algorithms

[0072] In order to fully demonstrate the superiority of the PIS-YOLO instance segmentation algorithm, this paper quantitatively analyzes PIS-YOLO and other four most advanced instance segmentation algorithms on the photovoltaic module infrared image instance segmentation dataset and its enhanced dataset. The comparison methods include: YOLOv5-seg, YOLOv7-seg, YOLOv8-seg, Gelan-seg. The experimental results of the initial dataset are shown in Table 1 and Figure 6 As shown in Table 2 and Figure 7 shown.

[0073] In the experimental results of YOLO instance segmentation, there are two parts in the indicators: Box and Mask. The indicators such as P, R, and mAP of Box are calculated by the intersection of detection boxes, and the calculation formulas are (2.7) to (2.12). The indicators such as P, R, and mAP of Mask are calculated by the intersection of detection boxes, and the calculation formulas are also (2.7) to (2.12). Since the target detection task is relatively simple and mainly focuses on the location and category judgment of the target, it usually has a higher FPS. The instance segmentation task not only requires the positioning and classification of the target object, but also requires the prediction of the pixel-level segmentation mask of each target object. Therefore, the FPS of instance segmentation is usually much lower than that of target detection. It can be seen from Table 1 that on the initial data set, PIS-YOLO has a precision rate P of 95%, a recall rate R of 87.3%, and a mAP@0.5 of 90.1% in Box. With a precision of 82.7% for Mask, a recall of 80.5% for Mask, and mAP@0.5 of 81.2%, PIS-YOLO performed well and achieved the best performance indicators of the two tasks. Compared with the baseline model YOLOv8n-seg, PIS-YOLO improved mAP@0.5 of Box by 4.7% and mAP@0.5 of Mask by 2.4%. On the enhanced dataset, the performance of each algorithm has improved to a certain extent, which shows that our dataset enhancement strategy is effective. At the same time, PIS-YOLO once again achieved the best performance indicators of the two tasks. In addition, PIS-YOLO achieved 48.4% model lightweight, Gflops decreased by 31.7%, and FPS only dropped from 71.5 to 70.5, but it can still meet the requirements of real-time instance segmentation. This shows that PIS-YOLO not only improves the accuracy of detection and segmentation, but also performs well in computational efficiency and resource consumption.

[0074] Compared with PIS-YOLO and GVT-YOLO (the network of the present invention), the mAP@0.5 of the detection box (Box) increased by 3.1% on the initial data set and by 0.6% on the enhanced data set. Another advantage of PIS-YOLO over GVT-YOLO is the added instance segmentation function. However, the speed of PIS-YOLO (FPS=70.5) is much lower than that of GVT-YOLO (FPS=249), and the two can be applied to different application scenarios. PIS-YOLO can not only detect the location and category of defects, but also accurately obtain the size and shape of defects. Therefore, it can be used in situations where the size and shape of defects need to be accurately obtained but the speed requirements are slightly lower. GVT-YOLO can only detect the location and category of defects, but the speed is very fast, and can be used in situations where the speed requirements are very high but only the location and category of defects need to be detected.

[0075] Table 1 Performance comparison of PIS-YOLO and other advanced instance segmentation algorithms on the initial dataset

[0076]

[0077] Table 2 Performance comparison of PIS-YOLO and other advanced instance segmentation algorithms on enhanced datasets

[0078]

[0079] In order to more intuitively demonstrate the superiority of PIS-YOLO, this paper visualizes and analyzes the detection results of different detection algorithms on the photovoltaic module infrared image defect dataset. The visualization results are as follows: Figure 8 shown.

[0080] From the visual comparison chart, we can see that tiny defect DBs pose a huge challenge to each algorithm. Overall, the Gelan-c-seg and PIS-YOLO models perform particularly well in the recognition and edge detection of small defect areas, especially in fine detection. Other algorithms all have varying degrees of over-detection and missed detection. This further proves the advantage of PIS-YOLO in segmentation tasks. However, PIS-YOLO still has some shortcomings. The DB confidence in some areas is low, indicating that its detection results may be unstable. In summary, PIS-YOLO has a strong ability to capture details and segmentation accuracy in photovoltaic panel defect detection.

[0081] Ablation experiment analysis

[0082] In order to verify the effectiveness of each improvement strategy in PIS-YOLO, this section conducts an ablation experiment analysis. The specific experimental results are shown in Table 3.

[0083] Table 3 PIS-YOLO ablation experiment results

[0084]

[0085] From the results in Table 3, we can see that the overall performance of the model has been significantly improved by gradually introducing different modules. First, after the introduction of the StarNet module, although the performance of the Mask task has slightly decreased, the Precision and mAP@0.5 indicators of the Box task have been improved, and the number of model parameters has been greatly reduced. After the introduction of the C2f-Star module, the feature fusion capability of the model has been further improved. This improvement is not only due to the C2f-Star module itself, but also due to the synergy with StarNet in Backbone, which also has "star operation". Through this multi-level feature fusion mechanism, the model can more effectively integrate feature information at different levels, thereby significantly improving the performance in both Box and Mask tasks. It shows that the combination of C2f-Star and StarNet not only improves the performance of the model, but also optimizes the overall structure of the model. When the PIS-SegHead segmentation head is introduced, the model has been significantly improved in the mAP@0.5 indicator of the Mask task, which shows that the PIS-SegHead segmentation head has a significant effect on improving the segmentation accuracy of the model. In addition, the number of model parameters is also greatly reduced, which further verifies the effectiveness of the PIS-SegHead segmentation head in optimizing the model structure. Finally, the introduction of the MHSA attention mechanism further improves the performance of the model in the Box and Mask tasks. Although the number of model parameters has increased slightly, this increase is worthwhile compared to the performance improvement. The MHSA attention mechanism significantly enhances the model's ability to capture contextual information and improves the overall accuracy of the task. Overall, the final model improves the mAP@0.5 indicators of the Box task and the Mask task by 4.7% and 2.4% respectively compared with the baseline method. The ablation experiment fully demonstrates the effectiveness of each module in enhancing model performance and improving computational efficiency.

[0086] The present invention also introduces CAM heat map visualization to verify the effectiveness of each strategy. Fig. 9 As shown in the figure, it can be seen that the performance of different methods in photovoltaic panel defect detection is significantly different. As a baseline method, the defect area is blurred and noisy. Compared with the baseline defect detection, the accuracy and concentration of methods A and B have been significantly improved. Method C further improves the clarity and detection accuracy of the defect area and can better handle complex defects. Method D performs best among all methods, has the strongest defect localization ability, can clearly separate the defect area, and effectively reduce background noise. Overall, method D shows the best performance in the photovoltaic module defect detection task.

[0087] The data collection process involves four main parts: photovoltaic arrays in natural scenes, data collection platforms carried by drones, carefully designed flight routes, and collected images of photovoltaic arrays. The collection process mainly relies on the DJI M300RTK drone flight platform and the equipped DJI Zenmuse H20T gimbal camera. The M300RTK drone has a single cruise time of 55 minutes and a flight load of 2.7 kg, which ensures the efficiency and reliability of photovoltaic array defect image acquisition. The H20T gimbal camera integrates a variety of sensors, including 20 million pixel zoom, 12 million pixel wide angle, 1200 meter laser ranging and 640×512 pixel thermal imaging functions, which can fully meet the needs of photovoltaic array infrared image acquisition. The infrared images of photovoltaic components collected by the drone are input into the GVT-YOLO network invented in this article, and the algorithm outputs the location and category of the defects, such as Fig.10 shown.

[0088] Embodiment 2:

[0089] In a second embodiment of the present invention, the present invention provides a defect instance segmentation system for large-scale photovoltaic module infrared images from a drone perspective, comprising:

[0090] A data acquisition module is used to collect infrared image data of photovoltaic components in a photovoltaic power station through an infrared thermal imaging camera carried by a drone;

[0091] An image preprocessing module is used to remove noise and enhance contrast of the acquired infrared images;

[0092] Feature extraction module, which uses StarNet network as the feature extraction module and combines star operation to extract high-dimensional nonlinear image features;

[0093] The attention mechanism module is used to optimize image features through the multi-head self-attention MHSA mechanism after feature extraction;

[0094] The instance segmentation module uses the improved YOLOv8 segmentation head PIS-SegHead to perform instance segmentation on the optimized image features and generate instance masks of PV module defects;

[0095] The defect recognition and analysis module is used to identify and analyze defects in photovoltaic modules, including the type, location and shape characteristics of the defects.

[0096] Specifically, the system is used to achieve defect instance segmentation of large-scale photovoltaic module infrared images from the perspective of drones, including the entire process from data collection to defect identification and analysis. Specifically, this system consists of multiple modules, each of which performs its own duties and cooperates with each other to form a complete solution for real-time detection and analysis of photovoltaic modules.

[0097] Data acquisition module

[0098] The data acquisition module is the core starting point of the system. Its function is to collect infrared images of photovoltaic modules through the infrared thermal imaging camera carried by the drone. The drone uses the DJI M300RTK, an industrial-grade drone with long flight time and high load capacity. In order to achieve efficient inspection of photovoltaic modules, the drone is equipped with a DJI Zenmuse H20T gimbal camera, which integrates a high-resolution visible light camera, infrared thermal imaging camera, laser rangefinder and other sensors to obtain detailed infrared image data. The data collection of the drone is not only about collecting the image itself, but also includes the planning of the flight path to ensure that every photovoltaic module can be effectively covered and there will be no blind spots for image collection.

[0099] Image preprocessing module

[0100] The image preprocessing module is used to perform basic processing on the infrared images obtained from the data acquisition module. The module first removes noise to reduce the interference of random noise in the image on subsequent analysis. Common methods include Gaussian filtering and median filtering. Then contrast enhancement is performed to make the infrared features of photovoltaic modules more obvious. The goal of image preprocessing is to convert the collected original image into a format and quality that is more suitable for subsequent feature extraction, so as to better detect defects in photovoltaic modules. These preprocessing steps are an indispensable part of the entire analysis chain and lay the foundation for subsequent efficient and accurate detection.

[0101] Feature extraction module

[0102] The feature extraction module is designed based on the StarNet network of deep learning. This module extracts effective features from infrared images of photovoltaic modules through a series of convolution operations. The StarNet network uses "Star Operation", which maps image features to a high-dimensional nonlinear feature space through an element-by-element product method, which can significantly enhance the expressiveness of features with a limited number of parameters. Through the feature extraction module, key details, edge information, texture information, etc. of the image are effectively extracted, providing high-quality feature representation for subsequent instance segmentation. StarNet not only improves the quality of feature expression, but also has lightweight feature extraction capabilities, which makes it very suitable for deployment on drone platforms with limited computing resources.

[0103] Attention mechanism module

[0104] The attention mechanism module is used to optimize the image features extracted by the feature extraction module. Specifically, by introducing the MHSA (multi-head self-attention) mechanism, the module can establish more effective relationships between different areas of the image, thereby paying more precise attention to defective areas. The role of the multi-head self-attention mechanism is to enhance the model's understanding of the contextual information of different areas in the image, so that the model can distinguish tiny defects in photovoltaic modules, such as fine cracks or local hotspots. In addition, the multi-head self-attention mechanism can also improve the globality of feature representation, so that the model can not only focus on local details, but also comprehensively consider the image information of the entire component, helping to perform subsequent defect segmentation and identification more accurately.

[0105] Instance Segmentation Module

[0106] The instance segmentation module uses the PIS-SegHead segmentation head, which is an improvement on the traditional YOLOv8 segmentation head. PIS-SegHead reduces the amount of computation by sharing convolutions, allowing multiple feature layers to be processed by the same convolution kernel, reducing redundant computations. In addition, in order to solve the problem of inconsistent defect sizes in photovoltaic modules, a scale scaling operation is introduced in the module. This operation can dynamically adjust the scale of the feature map so that both large-area defects and small-scale defects can be effectively detected. The purpose of the instance segmentation module is to accurately locate and depict the outline, size, and shape of each defect by generating a segmentation mask, so as to provide more detailed information for subsequent analysis.

[0107] Defect Identification and Analysis Module

[0108] The defect recognition and analysis module is used to further analyze the mask generated by the instance segmentation module. This module can identify different types of defects in PV modules, including spots, stripes, and open circuits. Spot defects may be caused by local overheating, which is usually reflected as local high temperature areas in infrared images; stripe defects are related to broken solder strips or poor connections in the circuit; open circuit defects are caused by the lack of current inside the module, resulting in the inability to effectively generate electricity in local areas. This module uses segmentation masks to evaluate the area, shape, and distribution of defects. Through quantitative analysis of these defect features, detailed fault reports can be generated to provide maintenance personnel with a clear reference basis for timely fault repair. The defect recognition and analysis module can also compare the detection results with the historical data of the PV power station to determine the development trend of the defects, so as to make predictive maintenance decisions and ensure the long-term and efficient operation of the PV power station.

[0109] Overall collaboration and application of the system

[0110] The modules of the entire system are highly collaborative. From data collection to defect identification, the modules gradually process and transmit data to form a closed-loop intelligent analysis process. The drone collects images during flight and transmits them to the preprocessing module in real time for processing. The processed images enter the feature extraction and attention optimization stages, and then the instance segmentation module generates a segmentation mask, which ultimately leads to detailed defect identification and analysis.

[0111] The advantage of this system is that it integrates a variety of advanced computer vision and deep learning technologies, and can achieve real-time defect detection of large-scale photovoltaic modules on mobile platforms such as drones. Through automated inspections and intelligent defect identification, the system significantly improves the detection efficiency of photovoltaic modules, reduces the cost and error of manual inspections, and provides an efficient and reliable solution for the daily operation and maintenance of photovoltaic power stations.

[0112] Embodiment 3

[0113] The third embodiment of the present invention is based on the same inventive concept. The present invention proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the method for segmenting defect instances of large-scale photovoltaic components infrared images from a drone perspective in the above-mentioned embodiment.

[0114] Embodiment 4

[0115] The fourth embodiment of the present invention is based on the same inventive concept. The present invention proposes a computer device, which includes: a processor and a memory; the processor and the memory communicate with each other; the memory is used to store instructions; the processor is used to execute the instructions in the memory to execute the method for segmenting defect instances of large-scale photovoltaic components infrared images from the perspective of a drone of the above embodiment.

[0116] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0117] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A defect instance segmentation method for large-scale photovoltaic module infrared images from a drone perspective, characterized by: The following steps are involved: S1. Data collection: Use the data collection platform carried by the drone to collect infrared images of photovoltaic components in the photovoltaic power station and obtain large-scale infrared image data; S2. Image preprocessing: preprocessing the collected infrared images for noise removal and contrast enhancement; S3. Feature extraction: StarNet is used as the feature extraction network to extract high-dimensional features; S4. Introduction of multi-head self-attention mechanism: After feature extraction, the MHSA multi-head self-attention mechanism is introduced to perform multi-dimensional feature optimization; S5. Instance segmentation: Use the improved YOLOv8 segmentation head PIS-SegHead to segment feature information, generate instance masks, and detect the defect location, category, size and shape of photovoltaic modules; S6. Defect identification and analysis: Each instance mask is further analyzed to identify and annotate the location and characteristics of the defects.

2. The defect instance segmentation method of large-scale photovoltaic module infrared image from the perspective of a drone according to claim 1 is characterized in that: In the step S1, the data acquisition platform carried by the drone is a pan-tilt camera with an infrared thermal imaging function, and the pan-tilt camera with an infrared thermal imaging function can obtain high-resolution infrared images.

3. The defect instance segmentation method of large-scale photovoltaic module infrared image from the perspective of a drone according to claim 1 is characterized in that: In the S3 step, the StarNet used in the feature extraction step introduces a star operation, and the star operation maps the image features to a high-dimensional nonlinear feature space through an element-by-element product method.

4. The defect instance segmentation method of large-scale photovoltaic module infrared image from the perspective of a drone according to claim 1 is characterized in that: In the step S5, the PIS-SegHead segmentation head used in the instance segmentation step includes shared convolution and scale scaling operations.

5. The defect instance segmentation method of large-scale photovoltaic module infrared image from the perspective of a drone according to claim 1 is characterized in that: In the step S6, defect identification includes detecting point spots, stripe spots and open circuit defects in the photovoltaic module, and generating corresponding segmentation masks.

6. A large-scale photovoltaic module infrared image defect instance segmentation system from a drone perspective, characterized by: The method for segmenting defect instances of infrared images of large-scale photovoltaic modules from the perspective of a drone as described in any one of claims 1 to 5 comprises: A data acquisition module is used to collect infrared image data of photovoltaic components in a photovoltaic power station through an infrared thermal imaging camera carried by a drone; An image preprocessing module is used to remove noise and enhance contrast of the acquired infrared images; Feature extraction module, which uses StarNet network as the feature extraction module and combines star operation to extract high-dimensional nonlinear image features; The attention mechanism module is used to optimize image features through the multi-head self-attention MHSA mechanism after feature extraction; The instance segmentation module uses the improved YOLOv8 segmentation head PIS-SegHead to perform instance segmentation on the optimized image features and generate instance masks of PV module defects; The defect recognition and analysis module is used to identify and analyze defects in photovoltaic modules, including the type, location and shape characteristics of the defects.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for segmenting defect instances of infrared images of large-scale photovoltaic modules from the perspective of a drone is implemented.

8. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for segmenting defect instances of infrared images of large-scale photovoltaic modules from a drone perspective as described in any one of claims 1 to 5 is implemented.

Images