A photovoltaic panel defect inspection method based on a UAV and an improved YOLO algorithm

Abstract

The present application relates to photovoltaic panel defect detection technical field, especially to a kind of photovoltaic panel defect inspection method based on unmanned aerial vehicle and improved YOLO algorithm, comprising: planning inspection route and image acquisition trigger point;Unmanned aerial vehicle obtains photovoltaic panel's visible light image and infrared image and positioning and attitude data;Image pre-processing and space alignment are obtained after registration infrared image;Based on photovoltaic panel geometry priori, registration residual is explicitly estimated, local registration credibility map is generated, and multi-source image data are obtained;Multi-source image data are input into the defect detection model constructed based on improved YOLO algorithm, the position information and category information of photovoltaic panel defect in image are obtained, are mapped as geographic coordinate, and the geographic position information and defect category information of photovoltaic panel defect are obtained.The present application realizes the automatic, high-precision, real-time detection and accurate geographic positioning of photovoltaic panel defect, and provides a solution for the intelligent operation and maintenance of photovoltaic power station.

Description

Technical Field

[0001] This invention relates to the field of photovoltaic panel defect detection technology, and in particular to a photovoltaic panel defect inspection method based on UAVs and an improved YOLO algorithm. Background Technology

[0002] As a crucial component of clean energy, photovoltaic (PV) power generation relies on regular inspections of the health of PV panels for stable operation and efficient power generation. With the widespread adoption of PV technology, the operation and maintenance of large-scale PV power plants faces significant challenges. PV panels, exposed to the outdoors for extended periods, are prone to various defects such as hot spots, microcracks, fragmentation, snail trails, and dust accumulation. These defects can significantly reduce power generation efficiency and even cause fires. Traditional manual inspection methods are inefficient, costly, and susceptible to missed detections and subjective judgment errors. Existing automated inspection solutions, such as installing fixed monitoring cameras or inspection robots, suffer from limited coverage, high deployment costs, and poor terrain adaptability. In recent years, drone inspection technology has been widely adopted, but most systems only automate data acquisition; subsequent image analysis and defect assessment still require substantial manual processing. While general target detection algorithms exist, the vast differences in PV panel defect scales and the indistinct defect features against complex backgrounds lead to high rates of missed and false detections. Visible light and infrared image information are complementary, but existing methods often employ single sensors or simple post-processing fusion, failing to achieve deep feature-level fusion to improve detection robustness and resulting in insufficient utilization of multi-source information. Most solutions adopt a "collection-transmission-cloud processing" model, which suffers from high latency and relies on image stitching and manual calibration for defect location accuracy. This fails to meet the needs of rapid response and precise maintenance in large-scale power plants, resulting in poor real-time performance and positioning accuracy. Furthermore, there is a lack of end-to-end automation capabilities from task planning and real-time detection to data management and model iteration, leading to a missing intelligent closed-loop system.

[0003] Traditional image processing methods improve the ability to identify and extract defects in photovoltaic modules by performing image preprocessing operations. In their journal article "A Defect Detection Method for Photovoltaic Panels Based on Improved K-means Clustering," Zhao Qiang et al. proposed an improved K-means clustering image processing method based on the HSV spatial model. This method can quickly detect and locate hot spots and statistically determine the degree of damage to photovoltaic panels, exhibiting high accuracy, good sensitivity, and stability; however, its detection efficiency needs further improvement.

[0004] In recent years, drone inspection has become a research hotspot due to its flexibility, efficiency, and wide coverage. Meanwhile, deep learning-based target detection algorithms, especially the YOLO series, are widely used in industrial defect detection due to their speed and high accuracy. Patent publication number CN121169862A discloses a deep learning-based method for detecting and identifying defects in solar photovoltaic panels, proposing an improved YOLO11-DRP model for detecting defects in photovoltaic panel images. However, this method has low detection accuracy and is difficult to meet practical needs. Directly applying the general YOLO model for photovoltaic defect detection faces the following technical challenges: 1) Large differences in the scale of photovoltaic defects; for example, when photovoltaic panels have large areas of dust accumulation and small-sized hidden cracks, the general model's detection effect is poor. 2) Complex outdoor environmental interference, such as changes in lighting, shadows, and reflections, affects the judgment of results. 3) Images captured by drones have angular distortion and inconsistent scales, increasing the difficulty of detection. 4) Limited defect sample data, imbalanced categories, and insufficient model generalization ability. Furthermore, existing technologies typically treat drones merely as image acquisition tools, with detection algorithms loosely coupled with inspection tasks, failing to form an efficient, accurate, end-to-end automated solution. Therefore, there is an urgent need for an intelligent method and system that integrates autonomous inspection, real-time analysis, and accurate identification. Summary of the Invention

[0005] In view of this, the present invention provides a photovoltaic panel defect inspection method based on UAV and improved YOLO algorithm to solve the problems faced by the existing general YOLO model in photovoltaic defect detection, such as large scale difference, strong outdoor interference, image distortion and sample imbalance, and lack of end-to-end automated solution deeply coupled with inspection task.

[0006] The technical solution of this invention is implemented as follows: This invention provides a method for inspecting defects in photovoltaic panels based on unmanned aerial vehicles (UAVs) and an improved YOLO algorithm, comprising the following steps: S1. Plan the drone inspection route and image acquisition trigger point based on the digital map of the photovoltaic power station; S2. The UAV flies along the inspection route and acquires visible light and infrared images of the photovoltaic panels using a synchronous acquisition method, and acquires the positioning and attitude data corresponding to each frame of the image. S3. The acquired visible light and infrared images are preprocessed, and the preprocessed visible light and infrared images are spatially aligned using an image registration method to obtain a registered infrared image. Based on the geometric prior of the photovoltaic panel, the registration residual is explicitly estimated to generate a local registration confidence map. The visible light image, the registered infrared image, and the local registration confidence map constitute multi-source image data. S4. Input the multi-source image data into the defect detection model built based on the improved YOLO algorithm to obtain the location and category information of photovoltaic panel defects in the image; S5. Post-process the location information and combine it with the positioning and attitude data. Use the spatial analysis method to map the location information into geographic coordinates to obtain the geographic location information and defect category information of the photovoltaic panel defects.

[0007] Based on the above technical solutions, preferably, step S1 specifically includes: A three-dimensional topographic map of the photovoltaic power station is constructed based on the digital orthophoto image and digital elevation model. The position, boundary and orientation information of each photovoltaic panel string is extracted from the digital orthophoto image using image recognition methods. Based on the centerline of each string and the spacing between adjacent strings, the UAV inspection route is planned using the centerline route method. The flight altitude, flight speed and shooting parameters of the dual-light camera are set according to the type of target defect to determine the image acquisition trigger point.

[0008] Based on the above technical solutions, preferably, step S3 specifically includes: S31. The visible light image and the infrared image are preprocessed respectively. The visible light image is subjected to grayscale transformation, noise filtering and threshold segmentation to obtain a binary image, and the visible light structure skeleton map is extracted by the direction-constrained Sobel operator. The infrared image is subjected to non-uniformity correction and radiometric calibration. S32. A feature-point-based registration algorithm is used to calculate the global homography transformation matrix of the infrared image relative to the visible light image. The infrared image is resampled according to the transformation matrix to obtain the registered infrared image. Contour search is performed on the binary image to extract the coordinates of the four corner points of each photovoltaic panel. Perspective transformation is then performed to correct the image to an approximate front view to obtain the perspective-corrected image. S33. Using the four corner points of the photovoltaic panel detected in the image after perspective transformation correction as anchor points, an ideal grid is constructed based on the geometric prior of the photovoltaic panel. The template matching method is used to estimate the registration residual and matching confidence of each grid node. Spatial interpolation is performed with the matching confidence as weight to obtain the dense registration residual field of the whole image. After normalization processing, a local registration confidence map is generated. The visible light image, the registered infrared image and the local registration confidence map constitute multi-source image data.

[0009] Based on the above technical solutions, preferably, the generation of the full-image dense registration residual field in step S33 specifically includes: S331. Using the four corner points of the photovoltaic panel detected in the image after perspective transformation correction as anchor points, construct an ideal grid in the visible light image coordinate system according to the known photovoltaic panel size and panel spacing parameters, and obtain the ideal position coordinates of each grid node. S332. For each grid node, a local template centered on the node is cut out from the visible light structure skeleton image. Normalized cross-correlation template matching is performed in the corresponding search window of the registered infrared image. The registration residual vector of the node is recorded by the difference between the main peak position and the ideal node position. The matching confidence of the node is calculated based on the difference between the main and secondary peak response values. S333. Using the registration residual modulus of each grid node as the data fitting target, and the square of the matching confidence as the node weight, a dense registration residual field with the same resolution as the image is generated in the entire map range by using confidence-weighted regularization interpolation.

[0010] Based on the above technical solution, preferably, in step S332, the matching confidence level is calculated using the following formula: ; in, To match confidence levels, The dominant peak value of the normalized cross-correlation response within the search window corresponding to the k-th grid node is... To exclude the secondary peak after excluding the neighborhood of the main peak; In step S333, the dense registration residual field is calculated using the following formula: ; in, For dense registration residual field, To find the solution that minimizes the objective functional for the function f; Let be the dense residual field function of the whole graph to be determined. Let k be the image coordinates of the kth grid node. To register the magnitude of the residual vector for the k-th grid node, Let k be the contribution weight of the k-th node to the data item. For function f The Hessian matrix; The bending energy of the thin-plate spline is used to constrain the spatial smoothness of the interpolated surface. To smooth out the regularization coefficients, N is the total number of grid nodes.

[0011] Based on the above technical solutions, preferably, in step S4, the defect detection model includes a backbone network, a neck network, and a detection head connected in sequence; the backbone network is a Swing Transformer network, which extracts multi-scale feature maps of the input multi-source image data through a shift window self-attention mechanism; the neck network is an efficient multi-scale attention module, which performs weighted calibration and fusion on the multi-scale feature maps output by the backbone network; the detection head uses a shape-aware bounding box regression loss function to calculate regression loss during the training phase, and outputs the category and location information of each defect during the inference phase.

[0012] Based on the above technical solutions, preferably, the Swin Transformer backbone network comprises four sequentially connected stages: The first stage divides the multi-source image data into non-overlapping 4×4 pixel blocks, which are then flattened and linearly projected to obtain the initial feature map. In the second to fourth stages, each block merging layer stitches together adjacent 2×2 feature blocks from the previous stage and then performs downsampling through linear projection, thereby halving the feature map resolution and doubling the number of channels in each stage. Each stage contains several alternating W-MSA and SW-MSA modules. The input of each sub-module is first processed by layer normalization before entering the attention calculation. The attention calculation result is added to the module input residual. The residual addition result is then processed by layer normalization and input to a multilayer perceptron. The multilayer perceptron output is then added to its input residual again. The four stages output feature maps of different resolutions, which together constitute a multi-scale feature map. The efficient multi-scale attention module divides the multi-scale feature map into several groups along the channel dimension. After each group of features is reshaped to the batch dimension, it is fed into two parallel branches for processing. The first branch uses 1×1 convolution followed by one-dimensional global average pooling along the height and width directions to extract directional contextual features; the second branch uses 3×3 convolution to extract multi-scale spatial structure features; the outputs of the two branches are then normalized by global average pooling and Softmax, respectively, and then generated by matrix dot product operation to produce spatial attention weights; simultaneously, after each training step, the current attention weights and the attention weights of the previous time step are updated by exponential moving average with a decay factor to generate channel attention weights; the spatial attention weights and channel attention weights are activated by Sigmoid and then multiplied with the input feature map to complete the dual weighted calibration of channel and spatial dimensions, resulting in a calibrated fused feature map.

[0013] Based on the above technical solution, preferably, in step S4, the shape-aware bounding box regression loss function is calculated using the following formula: ; Where L is the total Shape-IoU loss value; This is the intersection-union ratio (IU) between the predicted bounding box and the ground truth bounding box. The shape-aware center distance penalty term is calculated using the following formula: ; in, , The coordinates of the center of the prediction box. , Let be the coordinates of the center of the ground truth bounding box, and c be the diagonal length of the minimum bounding box between the predicted and ground truth bounding boxes. This is the shape weighting coefficient in the horizontal direction. This is the shape weight coefficient in the vertical direction, and its value is determined by the aspect ratio of the actual bounding box; The shape-aware contour penalty term is calculated using the following formula: ; Where w and h are the width and height of the prediction box, respectively. This is a weighted value representing the difference in size between the predicted bounding box and the true bounding box along the t-direction. As a penalty index, The value is 4.

[0014] Based on the above technical solutions, preferably, the execution process of the defect detection model includes the following steps: S41. After scaling the multi-source image data to a fixed size, input it into the backbone network of the defect detection model. After four stages of feature extraction, multi-scale feature maps with different resolutions are obtained. S42. Input the multi-scale feature map into the efficient multi-scale attention module for feature fusion to generate a weighted and calibrated fused feature map. S43. Input the calibrated fused feature map into the detection head, and output the predicted position value and category confidence score of each defect candidate box; S44. Perform non-maximum suppression on the predicted location value and category confidence score to remove redundant detection boxes and obtain the location and category information of photovoltaic panel defects in the image.

[0015] Based on the above technical solutions, preferably, step S5 specifically includes: using location information, combined with the positioning and attitude data corresponding to each frame image and camera intrinsic parameters, to calculate the pixel coordinates of the detection box into latitude and longitude coordinates in the geodetic coordinate system; further, stitching together the visible light images of all frames into a power station orthophoto map, and using the absolute geographic accuracy of the panoramic orthophoto map as a benchmark, correcting the latitude and longitude coordinates of each defect to obtain the geographic location information of the photovoltaic panel defect.

[0016] The present invention has the following advantages over the prior art: (1) This invention proposes a photovoltaic panel defect inspection method based on UAVs and an improved YOLO algorithm. It organically combines UAV automated flight path planning, dual-light synchronous acquisition, multi-source image fusion, and deep learning defect detection to construct a complete closed-loop process from data acquisition to defect geolocation. Compared with traditional manual inspection and fixed monitoring schemes, this invention has significant improvements in inspection coverage, detection accuracy, positioning accuracy, and system maintainability, and is suitable for the daily operation and maintenance of large-scale ground-mounted photovoltaic power plants.

[0017] (2) In the image preprocessing stage, this invention introduces an explicit estimation mechanism for registration residuals based on the geometric prior of the photovoltaic panel. By performing normalized cross-correlation template matching at the grid nodes, the local registration error at each spatial location is quantified. A dense local registration reliability map of the whole image is generated by thin plate spline interpolation, which is used as the third channel to form a multi-source input together with the visible light image and the registered infrared image. This mechanism enables the subsequent detection model to perceive the registration reliability of the two-light image at each spatial location and adaptively suppress cross-modal feature fusion in areas with poor registration quality, thereby reducing the interference of registration residuals on the detection results and improving the reliability of two-light fusion detection.

[0018] (3) This invention optimizes the YOLO model, using the Swing Transformer as the backbone network. Through the alternating stacking of multi-head self-attention within a window and multi-head self-attention in a shifted window, global context modeling across windows is achieved with linear computational complexity, and multi-scale semantic representation is obtained through hierarchical feature map extraction. Compared with convolutional backbones with limited receptive fields, this structure has a stronger feature representation capability for defects that are distributed in a dispersed manner on photovoltaic panels with large scale differences, which helps to improve the detection performance of small target defects and complex backgrounds.

[0019] (4) This invention introduces an efficient multi-scale attention module into the neck network. Through channel grouping, dual parallel branches (directional one-dimensional pooling and depthwise separable convolution), and an exponential moving average channel weight update mechanism, it simultaneously models long-range dependencies in the horizontal and vertical directions and local spatial structure features without compressing the channel dimension, and implements dual weighted calibration for the channel and spatial dimensions. This module enhances the response of defect-related regions in the multi-scale feature map while maintaining low computational overhead, which helps to improve the quality of feature fusion.

[0020] (5) This invention employs a shape-aware bounding box regression loss function, incorporating the shape and scale attributes of the predicted box itself into the loss calculation. Through orientation-adaptive center distance penalty terms and shape-aware contour penalty terms, the regression gradient can be adjusted differentially in different directions according to the aspect ratio of the target's true bounding box. For targets with obvious aspect ratio characteristics, such as slender hidden cracks and strip-shaped stains commonly found in photovoltaic panel defects, this loss function can provide a more discriminative regression signal compared to traditional IoU series losses, which helps improve the positioning accuracy of the bounding box.

[0021] (6) This invention combines RTK centimeter-level positioning data, IMU attitude data and camera intrinsic parameters, and uses collinear equations to map the defect detection box in the image coordinate system to the geodetic coordinate system. Furthermore, it performs global accuracy correction on the single-frame positioning results through panoramic image stitching and bundle adjustment, accurately associating the defect with the corresponding photovoltaic panel number and its position on the panel, providing maintenance personnel with defect geographic information that can be directly used for on-site positioning, and reducing the workload of manual inspection. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the photovoltaic panel defect inspection method of the present invention; Figure 2 This is a drone inspection route planning diagram according to an embodiment of the present invention; Figure 3 This is a detailed diagram of the unmanned aerial vehicle (UAV) inspection route planning according to an embodiment of the present invention; Figure 4 This invention provides an embodiment of a drone for capturing high-resolution RGB images. Figure 5 This is an infrared thermal image captured by a drone according to an embodiment of the present invention; Figure 6 This is the original image before matrix transformation according to an embodiment of the present invention; Figure 7 This is an approximate front view of an embodiment of the present invention after matrix transformation; Figure 8 This is a network structure diagram of the defect detection model in one embodiment of the present invention; Figure 9 This is a diagram of the Swing Transformer network framework for the defect detection model of this invention. Figure 10 This is a schematic diagram of the Swing Transformer block module of the defect detection model of this invention. Figure 11 This is a diagram of the EMAttention network framework of the defect detection model of this invention; Figure 12 This is a schematic diagram of the Shape-IoU loss function of the defect detection model of this invention; Figure 13 This is a diagram showing the precise location of defects according to an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0025] like Figure 1 As shown, this invention provides a photovoltaic panel defect inspection method based on UAVs and an improved YOLO algorithm, including the following steps: S1. Plan the drone inspection route and image acquisition trigger point based on the digital map of the photovoltaic power station; S2. The UAV flies along the inspection route and acquires visible light and infrared images of the photovoltaic panels using a synchronous acquisition method, and acquires the positioning and attitude data corresponding to each frame of the image. S3. The acquired visible light and infrared images are preprocessed, and the preprocessed visible light and infrared images are spatially aligned using an image registration method to obtain a registered infrared image. Based on the geometric prior of the photovoltaic panel, the registration residual is explicitly estimated to generate a local registration confidence map. The visible light image, the registered infrared image, and the local registration confidence map constitute multi-source image data. S4. Input the multi-source image data into the defect detection model built based on the improved YOLO algorithm to obtain the location and category information of photovoltaic panel defects in the image; S5. Post-process the location information and combine it with the positioning and attitude data. Use the spatial analysis method to map the location information into geographic coordinates to obtain the geographic location information and defect category information of the photovoltaic panel defects.

[0026] In one embodiment of the present invention, step S1 specifically includes: Aerial photography using drones is used to pre-acquire digital orthophoto maps (DOMs) and digital elevation models (DEMs) of the photovoltaic power station. The DOM is orthorectified aerial photograph of the ground, with a uniform scale and geographic coordinates, providing precise planar locations of features. The DEM is a raster-based digital model representing ground elevation information. By fusing the DOM and DEM, a 3D topographic map of the power station is generated, providing crucial data support for subsequent drone-based terrain-following flight and precise positioning.

[0027] Image recognition methods are used to automatically identify the location, boundary, and orientation information of all photovoltaic panel strings from digital orthophoto maps, assign a unique number to each photovoltaic panel, and associate its location information with an electronic map.

[0028] Based on the layout of the photovoltaic panels, a centerline route method is used to plan the UAV inspection route. Specifically, for each string of photovoltaic panels, the average coordinates of the center points of all panels are calculated to determine the centerline of the string, while the average interval between adjacent strings is also calculated. A single route is generated through the center points of each string, parallel to the length of the photovoltaic panels. Starting from the outermost string of the power plant, its centerline is used as the initial route; then, the search proceeds towards adjacent strings with a step size smaller than the string interval. Once a new photovoltaic panel is identified, its centerline is calculated as the next route; this process is repeated until all strings are traversed, and finally, all routes are connected end-to-end to form a complete inspection path. The centerline route method allows the UAV to fly between adjacent strings, and the field of view of the dual-light camera can simultaneously cover strings on both sides, effectively improving the coverage efficiency of a single inspection. Figure 2 The inspection route is obtained by using the centerline route method. The white dots in the figure represent the obtained strings, which are connected end to end to form a complete inspection path. Figure 3 The drone remote controller page after the inspection route has been planned.

[0029] Based on the target defect type (such as hot spots, microcracks, occlusion, etc.), the flight altitude, flight speed, and shooting parameters such as exposure and gain of the dual-light camera are set. Data acquisition trigger points are planned to ensure that shooting is performed near the center of each component. The spacing between trigger points is set according to the flight speed and camera frame rate to ensure sufficient overlap between adjacent frames. Based on the electronic map of the photovoltaic power station, the full coverage planning of the UAV automated inspection route is completed.

[0030] In one embodiment of the present invention, step S2 specifically includes: The drone carries a dual-light payload: a visible light camera and an infrared thermal imager, both mounted on the same gimbal, achieving near-coaxial positioning in space. The visible light camera captures visible defects on photovoltaic panels, such as cracks, snail trails, and fragments, while the infrared thermal imager detects thermal anomalies like hot spots and bypass diode malfunctions. The drone flies automatically along a preset route. When it reaches a preset shooting trigger point, the flight control system sends a precise PWM trigger signal through its auxiliary output channel. This signal is simultaneously connected to the shutter cables of both the visible light camera and the infrared thermal imager via a simple MOSFET switching circuit. When the pulse arrives, the shutter cables of both cameras are momentarily short-circuited, simulating the action of pressing the shutter button. This achieves millisecond-level or even microsecond-level synchronous triggering, acquiring high-definition RGB images and infrared thermal images of the same area. This ensures that the RGB and infrared images are highly consistent in terms of acquisition time and shooting angle, reducing the difficulty of subsequent registration. Figure 4 This is a high-resolution RGB image captured during the inspection process. Figure 5 This is an infrared thermal image taken during the inspection process.

[0031] To reduce perspective distortion, the camera's optical axis should ideally be perpendicular to the photovoltaic panel surface. The system is set with a maximum angular deviation tolerance. When the drone flies to the trigger area but its attitude exceeds the tolerance, the system delays slightly until the drone adjusts to the optimal alignment before triggering the shot, sacrificing minimal time accuracy for higher image quality.

[0032] Each time a photo is taken, the flight control system records a precise timestamp, spatial location, and drone attitude information. Specifically, this includes centimeter-level latitude, longitude, and altitude data provided by the high-precision RTK positioning module, as well as pitch, roll, and heading angles measured by the IMU. This data is stored in association with the image file, providing information for subsequent image georegistration and geographic coordinate calculation.

[0033] In one embodiment of the present invention, step S3 specifically includes: S31. The visible light image and the infrared image are preprocessed respectively. The visible light image is subjected to grayscale transformation, noise filtering and threshold segmentation to obtain a binary image, and the visible light structure skeleton map is extracted by the direction-constrained Sobel operator. The infrared image is subjected to non-uniformity correction and radiometric calibration.

[0034] Specifically, the visible light image is sequentially subjected to linear grayscale transformation, Gaussian filtering for noise reduction, and threshold segmentation: linear grayscale transformation stretches the image's grayscale range, making the contrast between the photovoltaic panel and the background more distinct; Gaussian filtering reduces image noise, preventing excessive stray edges from being generated during subsequent edge detection; global threshold segmentation converts the grayscale image into a black-and-white binary image, facilitating subsequent contour extraction. Simultaneously, automatic white balance and contrast enhancement processing are applied to the visible light image to further improve image quality. Based on this, the direction-constrained Sobel operator is used to enhance the edges of the binary image, extracting the visible light structural skeleton map. The direction-constrained Sobel operator retains only the gradient response consistent with the edge direction of the photovoltaic panel, suppressing stray edges and obtaining a skeleton image with the photovoltaic panel's border lines as the main content. Non-uniformity correction is performed on the infrared image to eliminate fixed pattern noise caused by inconsistent detector responses, followed by radiometric and temperature calibration to convert grayscale values ​​into equivalent temperature values, improving the contrast of thermal anomaly areas.

[0035] S32. A feature-point-based registration algorithm is used to calculate the global homography transformation matrix of the infrared image relative to the visible light image. The infrared image is resampled according to the transformation matrix to obtain the registered infrared image. Contour search is performed on the binary image to extract the coordinates of the four corner points of each photovoltaic panel. Perspective transformation is then performed to correct the image to an approximate front view to obtain the perspective-corrected image.

[0036] Due to the different physical positions of the two cameras, the acquired views exhibit translational and rotational differences. ORB feature point detection and RANSAC robust estimation methods are employed to extract matching feature point pairs between the visible light and infrared images, and the global homography transformation matrix of the infrared image relative to the visible light image is calculated. The infrared image is then resampled using bilinear interpolation based on this matrix to obtain a registered infrared image aligned with the spatial coordinates of the visible light image. Global homography registration can eliminate the difference in viewpoint between the two cameras, but due to lens distortion and the non-planarity of the plate surface, local residual registration errors still exist.

[0037] After registration, contour searching is performed on the binary image obtained in step S31. The largest contour with an area satisfying a preset threshold is selected, and its minimum bounding rectangle is calculated to obtain the coordinates of the four corner points of the photovoltaic array. This process is repeated after obtaining the coordinates of the four corner points of the original image. to Face the goal squarely Figure 4 After establishing the corner coordinate correspondence, the perspective transformation matrix is ​​calculated, and a perspective transformation is performed on the image to correct the trapezoidal distortion caused by the skewed shooting angle, resulting in an approximate front view. The corrected image is then obtained. The original image is shown below. Figure 6 As shown, the approximate front view is as follows: Figure 7 As shown, by Figure 6 and Figure 7 It can be seen that perspective transformation correction can significantly improve trapezoidal distortion caused by skewed shooting angle. The transformation process uses bilinear interpolation as the default interpolation method, achieving a good balance between computational efficiency and image smoothness, resulting in a more natural transformed image.

[0038] S33. Using the four corner points of the photovoltaic panel detected in the image after perspective transformation correction as anchor points, an ideal grid is constructed based on the geometric prior of the photovoltaic panel. The template matching method is used to estimate the registration residual and matching confidence of each grid node. Spatial interpolation is performed with the matching confidence as weight to obtain the dense registration residual field of the whole image. After normalization processing, a local registration confidence map is generated. The visible light image, the registered infrared image and the local registration confidence map constitute multi-source image data.

[0039] Specifically, the generation of the dense registration residual field of the whole image includes: S331. Using the four corner points of the photovoltaic panel detected in the perspective-corrected image as anchor points, and based on the known actual width, height, and spacing parameters of the photovoltaic panel, calculate the row and column grid node coordinates of each photovoltaic panel in the visible light image coordinate system according to the equal spacing rule, construct an ideal grid, and obtain the ideal position coordinates of each grid node. The ideal grid reflects the precise position that the corner points of each panel should appear under the condition of no registration error.

[0040] Step S332: For each grid node, use the ideal position coordinates of grid node k. Centered on the visible light structure skeleton diagram, a cut-out size of [size missing] is taken. The local template (P is set according to the feature size of the board corner points, typically 32 pixels) is used in the registered infrared image. Centered on, with dimensions of Within the search window (S>P, typically 64 pixels), normalized cross-correlation (NCC) template matching is performed. The main peak position and... The difference is the registration residual vector of that node. Its mold length This characterizes the local registration error at that node. The matching confidence of the k-th node is calculated using the following formula: ; in, Let be the matching confidence score of the k-th grid node. The dominant peak value of the NCC response surface within the search window corresponding to the k-th grid node. To exclude the main peak and its neighboring secondary peaks, the denominator As a normalization factor, ensure The greater the difference between the primary and secondary peaks, the higher the uniqueness of the match and the higher the confidence level. When the primary and secondary peaks are close, it means that there are multiple similar features, the reliability of the matching result is low, and the confidence level is close to zero.

[0041] Step S333: Using the registration residual modulus of each grid node as the data fitting target, and the square of the matching confidence as the node weight, confidence-weighted regularized interpolation is used to generate a dense registration residual field with the same resolution as the image across the entire map. Dense Registration Residual Field The following variational optimization problem is obtained by performing confidence-weighted regularized interpolation on the sparse residuals of all N grid nodes: ; in, For dense registration residual field, To find the solution that minimizes the objective functional for the function f; Let be the dense residual field function of the whole graph to be determined. Let k be the image coordinates of the kth grid node. To register the magnitude of the residual vector for the k-th grid node, Let k be the contribution weight of the k-th node to the data item. For function f The Hessian matrix (second-order partial derivative matrix). The bending energy of the thin-plate spline is used to constrain the spatial smoothness of the interpolated surface. >0 represents the smoothing regularization coefficient, controlling the trade-off between data fitting accuracy and surface smoothness; N is the total number of grid nodes. The analytical solution to the above variational problem is Thin-Plate Spline (TPS) interpolation, which can efficiently generate dense registration residual fields with the same resolution as the image across the entire map. This yields... Then, min-max normalization is performed to map the value range to [0,1]. The smaller the residual, the higher the confidence level, thus generating a local registration confidence map. The local registration confidence map, together with the visible light image and the registered infrared image, constitutes three-channel multi-source image data, which serves as the input for the subsequent detection model.

[0042] Traditional dual-light image fusion methods directly superimpose the registered infrared image with the visible light image, ignoring the spatial non-uniformity of registration accuracy. This method explicitly calculates the registration residual at grid nodes and generates a full-image density reliability map using thin-plate spline interpolation. This enables the detection model to perceive the registration reliability at each spatial location when fusing multi-source images, and adaptively suppresses feature fusion in poorly registered areas, thereby reducing the interference of registration errors on defect detection results.

[0043] In one embodiment of the present invention, in step S4, the defect detection model includes a backbone network, a neck network, and a detection head connected in sequence. The backbone network is a Swing Transformer network, which extracts multi-scale feature maps of the input multi-source image data through a shift window self-attention mechanism; the neck network is an efficient multi-scale attention module, which performs weighted calibration and fusion on the multi-scale feature maps output by the backbone network; the detection head calculates the regression loss using a shape-aware bounding box regression loss function during the training phase, and outputs the category and location information of each defect during the inference phase.

[0044] Specifically, the defect detection model is as follows: Figure 8As shown, the entire network consists of three interconnected parts from left to right: a backbone network, a neck network, and a detection head. The leftmost input receives a three-channel image (Images) with a size of H×W×3. First, it undergoes patch partitioning and linear embedding operations to complete image segmentation and feature dimension projection, transforming the original pixel image into a sequential feature representation that can be processed by the Transformer. Subsequently, the features flow sequentially through four stages (Stage 1 to Stage 4), each containing several SwinTransformer Block modules (yellow squares). Adjacent stages are downsampled using patch merging layers (gray squares), halving the resolution and doubling the number of channels at each stage, achieving hierarchical extraction from shallow detail features to deep semantic features. An efficient multi-scale attention module (EMA, orange square) is inserted after the output of Stage 2 and before the input of Stage 3, performing dual channel and spatial weighting calibration on the multi-scale feature map to enhance the feature fusion capability of the neck network. The stacking numbers of the Swin Transformer Blocks at each stage are 2, 2, 6, and 2, respectively, labeled "×2" and "×6" in the figure. The network ultimately passes through layer normalization, average pooling, a fully connected layer (FC layer), and a softmax classification layer, outputting the predicted probabilities for each category to complete the defect classification task. In this invention, the YOLOv8n architecture is optimized by introducing the Swin Transformer as the backbone network. By constructing hierarchical feature maps and employing a shift-window multi-head self-attention mechanism, the high computational complexity and lack of multi-scale feature modeling capabilities of the standard Vision Transformer are addressed.

[0045] After scaling the three-channel multi-source image data to a fixed size (e.g., 640×640 pixels), it is fed into the backbone network. The backbone network uses the Swin Transformer, whose network structure consists of four sequentially connected stages. The first stage divides the multi-source image data into non-overlapping 4×4 pixel patches. Each patch is flattened and linearly projected to obtain a C-dimensional vector. The initial feature map is obtained; in the second to fourth stages, each stage uses a patch merging layer to stitch together adjacent 2×2 feature blocks from the previous stage and then downsamples them through linear projection, thereby halving the feature map resolution and doubling the number of channels in each stage, forming a hierarchical feature map. Specifically, let the input image size be... The size of the first-stage output feature map is (C is the base number of channels), the sizes of the output feature maps for the second, third, and fourth stages are respectively: , , .

[0046] Specifically, the framework structure of the Swin Transformer backbone network is as follows: Figure 9 As shown, the leftmost image represents the original input image (Images), sized H×W×3, indicating a three-channel color image with height H and width W. The input image first undergoes two operations: patch partitioning and linear embedding. This divides the image into non-overlapping blocks and projects them into a high-dimensional feature space, resulting in an initial feature map (H / 4×W / 4×48) with a resolution of H / 4×W / 4 and 48 channels. Subsequently, the network extracts features sequentially from Stage 1 to Stage 4. Each stage entry point has a patch merging layer that downsamples the feature map output from the previous stage, halving the resolution and doubling the number of channels. Each stage contains several stacked Swin Transformer Blocks (yellow squares). Specifically, Stage 1 takes the initial feature map as input, processes it through two Swin Transformer Blocks, and outputs a feature map with a size of H / 4×W / 4×C (where C is the baseline number of channels, such as 96). Stage 2 first downsamples the feature map output from Stage 1 to H / 8×W / 8×2C through Patch Merging, and then processes it through two blocks, maintaining the output at H / 8×W / 8×2C. Similarly, Stage 3 first downsamples the output of Stage 2 to H / 16×W / 16×4C through Patch Merging, and then performs feature extraction through six blocks, outputting a size of H / 16×W / 16×4C. Stage 4 downsamples the output of Stage 3 to H / 32×W / 32×8C through Patch Merging, and finally outputs a feature map with a size of H / 32×W / 32×8C after processing through two blocks. The arrows and size labels at the top of the figure clearly illustrate the process where the feature map resolution halves with each level of network depth, while the number of channels doubles with each level. This hierarchical design enables the Swin Transformer to construct a multi-scale feature pyramid similar to that of a convolutional neural network. The multi-scale feature maps output from its four stages together form the foundation required for feature fusion and enhancement in the subsequent neck network.

[0047] Each stage contains several sequentially alternating window-based multi-head self-attention modules (W-MSA) and shift-window-based multi-head self-attention modules (SW-MSA). The input to each sub-module is first processed by layer normalization (LN) before entering the attention calculation. The attention calculation result is added to the module input residual; the residual sum is then processed by layer normalization again before being input to a multilayer perceptron (MLP), and the MLP output is again added to its input residual. l The working principle of each block is as follows: ; ; ; ; in, Output features for the W-MSA module (the first) l (blocks) Output features for MLP modules (the first) l (blocks) For the first l The input features of each block, Output features for the SW-MSA module (the first) l+ 1 block), Output features for MLP modules (the first) l+ 1 block), LN(·) represents layer normalization; MLP(·) represents multilayer perceptron; SW-MSA(·) represents shifted window multi-head self-attention. W-MSA is in a fixed Self-attention is computed within a window (typically M=7), reducing the computational complexity of global attention from the square of the image resolution to linear; SW-MSA shifts the entire window based on W-MSA. Each pixel enables cross-window information interaction between adjacent windows, breaking down the isolation between windows and establishing connection channels for image blocks within different windows. This simulates the effect of global attention and compensates for the interruption of the receptive field at the window boundaries of W-MSA. The four stages output feature maps of different resolutions, collectively forming a multi-scale feature map.

[0048] Furthermore, Figure 10 The diagram, grouped into pairs of adjacent Transformer blocks, details the internal computational flow of alternating stacks of window-based multi-head self-attention modules (W-MSA) and shift-window multi-head self-attention modules (SW-MSA), as well as the residual connection structure of each block. The left side of the diagram shows the... l The first block, the one on the right is the [number]th block. l +1 block, both are structurally symmetrical, the only difference being the way attention is calculated. (The left side...) lTaking a block as an example, its input is the output feature of the previous block. First, the features undergo layer normalization (LN, blue square) to eliminate internal covariate bias. Then, they are fed into a multi-head self-attention module (W-MSA, orange square) within a window. Self-attention is computed within a local window, replacing the quadratic complexity of global self-attention with linear complexity; W-MSA output and module input. Intermediate features are obtained by adding the residuals (⊕ symbol). After layer normalization (LN), the features are fed into a multilayer perceptron (MLP, green square) for nonlinear feature transformation. Similarly, the output of the MLP is added to the features before entering the MLP through residual connections, and finally the features of the current block are output. The right side of the first l +1 block structure and the first l Each block corresponds completely, the difference being that the attention module is replaced by the shifted window multi-head self-attention module (SW-MSA, yellow square). By shifting the entire window by half the window size, the isolation between adjacent windows is broken, enabling global information interaction across windows and compensating for the lack of receptive field interruption at the window boundary of W-MSA.

[0049] The neck network employs an efficient multi-scale attention module (EMA) to weight, calibrate, and fuse the multi-scale feature maps output by the backbone network. This efficiently captures multi-scale spatial features and inter-channel relationships without reducing channel dimensionality, avoiding information loss that might result from channel dimensionality reduction. Let the input feature map size be... The EMA divides it evenly into g groups along the channel dimension, with each group having c / g channels, and the feature size of each group is [missing information]. The features of each group are reshaped to the batch dimension and stacked as follows: The tensors allow each group to be processed independently in subsequent calculations, with each group focusing on learning different aspects of features, which helps the model capture diverse defect patterns.

[0050] Each reshaped feature set is fed into two parallel branches for processing: the first branch first... Convolution performs channel transformation, borrowing the idea of ​​coordinate attention, and then performs one-dimensional global average pooling along the height and width directions respectively, yielding... and The directional context vector, after being activated by a sigmoid function and multiplied with the feature map, helps to capture long-range dependencies and local detail features in the horizontal and vertical directions; the second branch adopts... Depthwise separable convolutions have a larger receptive field, focusing on capturing long-range dependencies and multi-scale spatial structure features. The outputs of the two branches are added and fused, then compressed using global average pooling. The vector, after being normalized by Softmax, is then subjected to a matrix dot product operation, and combined with the spatial features (size) output by the second branch. Generate spatial attention weights Its size is This process can capture pixel-level pairwise relationships and aggregate global contextual information to generate more accurate attention weights.

[0051] Furthermore, Figure 11 This demonstrates the complete data flow path and dual-branch parallel computation mechanism within the EMA module. The size of the input feature map (Input) is... (Height h, width w, number of channels c), firstly, it is evenly divided into g groups along the channel dimension (shown by the dashed box on the left side of the figure), with each group having c / g channels. The features of each group are then reshaped to the batch dimension and stacked. The tensor (denoted as X) is used to process each group independently in subsequent calculations. X then enters two parallel branches simultaneously. The upper branch embodies the coordinate attention concept: the feature X is subjected to global average pooling (X AvgPool, Y AvgPool) along the X direction (width direction) and Y direction (height direction), respectively, yielding... and The directional context vectors are concatenated and then subjected to a 1×1 convolution for channel transformation. After being activated by two separate sigmoid functions, they are multiplied with the feature map to effectively capture long-range dependencies in the horizontal and vertical directions. The weighted features output from the two sigmoid functions are summed using the ⊕ operator and then fed into a branch containing group normalization. Finally, they are compressed using global average pooling (AvgPool) to... The global description vector, after Softmax normalization, is compared with the spatial features of the lower branch ( Spatial attention weights are generated through matrix dot product operations (Matmul, the orange square above). (size The lower branch uses a 3×3 depthwise separable convolution (3×3, cyan squares) to extract local multi-scale spatial structure features. Its output is then normalized by Softmax and global average pooling (AvgPool) to obtain the channel description vector. Then, the matrix dot product (Matmul, orange square below) is multiplied by the spatial attention weights of the upper branch to complete the spatial dimension weighting. The outputs of the two branches are then fused by adding them together (⊕), activated by a sigmoid function, and added to the original input X from the skip connection. The final output is a feature map (Output) that has undergone dual channel and spatial calibration, and its size is restored to [value missing]. .

[0052] At the same time, after each training step, with a decay factor (0< <1) Update the attention weights generated at the current time step with the attention weights generated at the previous time step using an exponential moving average to generate the channel attention weights, i.e.: ; in, Let be the channel attention weights updated in step t. For the channel attention weights from the previous step, These are the attention weights calculated from the feature map in the current step. This is the decay factor. The exponential moving average serves to suppress drastic fluctuations in attention weights during single-step training, improving the temporal stability of the weights. Ultimately, the spatial attention weights... With channel attention weights After being activated by the Sigmoid algorithm, the samples are multiplied with the input feature map to achieve a dual weighted calibration of both channel and spatial dimensions. ; in, For the input feature map, It is the Sigmoid activation function. The calibrated fused feature map is then fed into the detection head for further processing.

[0053] The detection head adopts a YOLOv8 decoupled detection head structure, which calculates the classification branch and the regression branch independently, and outputs the category confidence score and bounding box position prediction value of each anchor point respectively.

[0054] The training process for the defect detection model includes: labeling each image in the pre-collected dataset using Labelimg annotation software, and then dividing the labeled images into training and test sets in an 8:2 ratio. The experimental environment configuration during training was as follows: batch size of 16, initial learning rate of 0.01, final learning rate of 0.001, learning rate momentum of 0.937, weight decay of 0.0005, training epoch of 200, and iterative training using the SGD optimizer. After training, the model automatically saves the final weight file and the weight file with the best training results, selecting the weights with the best performance on the validation set for subsequent detection tasks. After training, the defect detection model is deployed on an onboard edge computing unit. The hardware platform uses an embedded GPU module with powerful AI computing capabilities to meet the power consumption and size constraints of real-time image processing and deep learning inference, enabling real-time defect detection and classification of multi-source input image data.

[0055] During the training phase, the bounding box regression loss uses the Shape-Aware IoU (Shape-IoU) loss function. The principle of the Shape-IoU loss function is as follows: Figure 12 As shown in the figure, the orange rectangular area (GT, or Ground Truth) represents the actual labeled defect boundary box, denoted as... The coordinates of its center point (blue solid dot) are marked as follows: The width and height of the real frame are respectively... and The yellow rectangle (Prediction) represents the predicted bounding box output by the model, denoted as B, with its center point (blue solid dot) labeled as (x, y). The width and height of the predicted box are labeled as W and h, respectively, at the bottom and right of the figure. The ground truth and predicted boxes partially overlap in the figure, while the gray shaded areas where the two boxes do not overlap visually represent the positional and shape differences between the current prediction and the ground truth labels. In this invention, the Shape-IoU loss function is based on the above geometric relationship, further introducing a shape-aware center distance penalty term on top of the traditional Intersection over Union (IoU) ratio, which measures the degree of overlap between the two boxes. and shape-aware contour penalty : The horizontal and vertical center point offset penalty weights are dynamically adjusted based on the aspect ratio of the real bounding box, so that the loss function has directional adaptability to the center alignment error of targets with different shapes. By penalizing the fine-grained constraints on the outline shape of the prediction box through exponential size difference penalties, it can provide more discriminative regression gradients for photovoltaic panel defects with significant aspect ratio characteristics, such as slender hidden cracks and irregular hot spots, thereby improving the overall positioning accuracy and shape fitting ability of the defect bounding box.

[0056] This loss function breaks away from the traditional IoU series loss functions that only focus on the relative geometric relationships between bounding boxes, and innovatively incorporates the shape and scale attributes of the bounding boxes themselves into the loss calculation, as shown in the following formula: ; Where L is the total Shape-IoU loss value; The intersection-union ratio (IU) of the predicted bounding box and the ground truth bounding box is used to measure the degree of overlap between the two boxes. The shape-aware center distance penalty term is calculated using the following formula: ; in, , The coordinates of the center of the prediction box. , Let be the coordinates of the center of the ground truth bounding box, and c be the diagonal length of the minimum bounding box between the predicted and ground truth bounding boxes. and These are shape weight coefficients for the horizontal (x-axis) and vertical (y-axis) directions, respectively. Their values ​​are determined by the aspect ratio of the true bounding box; the larger the aspect ratio, the greater the shape weight. The smaller The larger the value, the smaller the value, which allows the center distance penalty to be adaptively adjusted according to the target shape in different directions, dynamically assigning different importance to the center point deviation in different directions based on the true box shape; The shape-aware contour penalty term is calculated using the following formula: ; Where w and h are the width and height of the prediction box, respectively. For the t direction ( The weighted value representing the difference in size between the predicted bounding box and the ground truth bounding box (where w corresponds to the width direction and h corresponds to the height direction). The penalty index is set to 4. The exponential form results in a weaker penalty for small errors and a rapid increase in penalty for large errors, which helps to accurately fit the shape of the target contour. The Shape-IoU loss function can perfectly adapt to the shape characteristics of defects with obvious directionality on the photovoltaic panel (such as microcracks, edge damage, and irregular stains), ensuring high-precision and tight positioning. Its scale-aware characteristics enable it to naturally enhance the regression ability for small targets such as microcracks, reduce the false negative rate, and help the model balance the detection performance of targets of different sizes in the same scene, adapting to complex backgrounds and multi-scale detection scenarios.

[0057] The execution process of the defect detection model includes the following steps: S41. After scaling the three-channel multi-source image data to a fixed size (640×640 pixels), input it into the backbone network of the defect detection model. It then undergoes four stages of feature extraction to obtain images with resolutions of [resolution values ​​missing]. , , , Multi-scale feature maps. The first embedding layer of the Swin Transformer backbone network has 3 input channels to receive a three-channel input tensor. Due to the three-channel input tensor... Includes local registration confidence map In the self-attention calculation of the shift window at each stage of the backbone network, The spatial distribution information can interact with the dual-light features, guiding the backbone network to automatically learn to reduce the inductive bias of cross-modal feature association weights in regions with poor registration quality.

[0058] S42. Input the multi-scale feature map into the efficient multi-scale attention module for feature fusion, generating a weighted and calibrated fused feature map. When performing spatial attention-weighted fusion, the EMAttention module... The provided spatial credibility distribution serves as an explicit geometric constraint, causing the fusion weights to tilt towards single-modal features in regions with large registration residuals. This fundamentally suppresses the propagation of registration residual errors during the feature fusion stage, preventing the forced fusion of misaligned dual-light features.

[0059] S43. Input the calibrated fused feature map into the decoupled detection head. The classification branch and regression branch are calculated independently, and the predicted position values ​​(center coordinates, width and height of the bounding box) and category confidence scores corresponding to each defect candidate box are output. The accurate regression of the slender hidden crack bounding box by the Shape-IoU loss function also gains more robust feature support due to the spatial explicitness of the upstream feature reliability.

[0060] S44. Perform non-maximum suppression (NMS) on the location prediction value and category confidence score. For multiple candidate boxes with an overlap exceeding a set threshold, only the one with the highest confidence is retained, and the rest are suppressed to ensure that each defect corresponds to only one most reliable detection box. After removing redundant detection boxes, the location information and category information of photovoltaic panel defects in the image are obtained.

[0061] In one embodiment of the present invention, step S5 specifically includes the following:

[0062] After post-processing the location information output in step S4, the pixel coordinates of the center point of each defect detection box are converted into latitude and longitude coordinates in the geodetic coordinate system by combining the positioning and attitude data corresponding to each frame image and the camera intrinsic parameters. Specifically, the collinearity equation describes the geometric relationship of collinearity among the image point, the camera center, and the object point. A system of equations is established simultaneously using the pixel coordinates of the defect, the corresponding UAV RTK positioning coordinates (centimeter-level latitude, longitude, and altitude), IMU attitude angles (pitch, roll, and heading angles), and camera intrinsic parameters (focal length, principal point coordinates, and distortion coefficients) to solve for the latitude and longitude coordinates of the defect in the ground geodetic coordinate system. Combined with the photovoltaic panel grid index, the defect box in the image coordinate system is mapped to the actual geographic coordinate system of the power station, accurately locating the photovoltaic panel number to which the defect belongs and its approximate location on the panel. Figure 13 To successfully locate a defective photovoltaic panel, red spots represent defective panels, and green boxes are used as positioning frames. If the red spots are all within the green boxes, the positioning is successful.

[0063] Furthermore, to overcome the limitations of single-image field of view and cumulative positioning errors, a panoramic image stitching positioning method is adopted: all visible light images acquired from all frames are processed by feature matching and bundle adjustment to stitch together a complete orthophoto map of the power plant. When a defect is found in an original image, the system first accurately locates the image on the panoramic base map, and then reads the absolute geographic coordinates of the defect from the base map. This effectively utilizes the overall absolute geographic accuracy of the panoramic image, reduces the cumulative positioning error of a single frame, and obtains the final geographic location information and defect category information of the photovoltaic panel defect, enabling maintenance personnel to accurately locate and implement targeted repairs.

[0064] In one embodiment of the present invention, the intelligent inspection system for photovoltaic panel defects that implements the above method includes the following units: (a) Unmanned Aerial Vehicle (UAV) Inspection Unit The DJI Matrice 350 RTK industrial-grade multi-rotor drone is selected, featuring a maximum flight time of 55 minutes, IP55 protection rating, six-way binocular vision and infrared sensing system, support for dual-control mode, and a built-in high-precision RTK module. The visible light camera uses the DJI Zenmuse P1 full-frame aerial surveying gimbal camera with 45 megapixels and a 35mm fixed-focus lens, designed for high-precision surveying and inspection, providing excellent image detail for capturing visible defects such as cracks, snail trails, and fragments on photovoltaic panels. The infrared thermal imager uses the DJI Zenmuse H20T hybrid sensor payload, whose infrared portion is... 30Hz resolution thermal imager, temperature measurement accuracy or Integrated coaxially with a visible light camera, it detects thermal anomalies such as hot spots and bypass diode failures, achieving pixel-level synchronization. The high-precision positioning module utilizes the drone's built-in DJI D-RTK 2 rover, combined with third-party RTK network services such as Qianxun Location, to provide real-time centimeter-level (horizontal ±1cm+1ppm, vertical ±1.5cm+1ppm) positioning data and pitch, roll, and yaw attitude angle information for each frame of image. The flight control and data transmission module uses the drone's original O3 image transmission industry version, paired with a DJI enhanced image transmission module, ensuring stable communication in complex terrain. The flight control module receives ground commands, executes automatic flight tasks, and transmits flight status information and low-bandwidth metadata in real time via a high-speed data transmission link, while also receiving emergency commands from the edge computing unit.

[0065] (ii) Ground control and communication unit The hardware utilizes a mobile workstation compatible with the DJI Pilot 2 cloud platform for mission planning, flight monitoring, data reception, and command issuance. The mission planning software uses the DJI Pilot 2 App, supporting the import of KML range files or 2D orthophotos of the power station, automatically planning terrain-following flight paths to ensure full coverage of the target photovoltaic array and optimal imaging angles. The real-time monitoring interface displays the drone's flight status, battery level, visible light / infrared image transmission, and a preliminary overlay of detection results. A stable control link and high-speed downlink are established and maintained between the drone and the 4G / 5G cellular network module for receiving raw image data streams and detection result data.

[0066] (III) Edge Computing and Defect Detection Unit The hardware platform utilizes an NVIDIA Jetson AGX Orin 32GB module, providing up to 275 TOPS of AI computing power with configurable power consumption between 15W and 60W, suitable for real-time AI inference on airborne or ground-based mobile devices. The software environment runs Ubuntu 20.04 LTS, uses PyTorch 1.12.0 as the AI ​​framework, and TensorRT 8.5.1 as the inference engine. An improved YOLO model, accelerated and quantized by TensorRT, is deployed in this unit to achieve real-time defect detection and classification of the input image stream. This unit also handles the online computation tasks of registration residual estimation and local registration confidence map generation in step S3: TPS interpolation and exponential decay normalization are lightweight numerical computation operations, with computation time on the Jetson AGX Orin controlled to the millisecond level, without affecting the overall real-time performance of the system.

[0067] (iv) Data Management Unit A local server is deployed, configured with an Intel Xeon Silver 4310 processor, 64GB of RAM, a 2TB NVMe SSD system disk, and a 16TB HDD data disk array. PostgreSQL 14 is used to store structured data, and an NVIDIA RTX A6000 graphics card is configured for incremental model training. The database stores raw visible light / infrared images, corresponding RTK position and pose data, model detection results, and manually reviewed and annotated data. The annotation tool provides a user-friendly interface, allowing experts to review, correct, and supplement the automatic detection results, forming a high-quality incremental dataset. The incremental model training platform periodically or triggered by the newly added annotated data to fine-tune the defect detection model at the edge, achieving continuous optimization and adaptation of the model. This ensures the detection model continuously improves with use, adapting to defect patterns in different power plants and seasons, possessing adaptability and evolvability.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for inspecting defects in photovoltaic panels based on unmanned aerial vehicles (UAVs) and an improved YOLO algorithm, characterized in that, Includes the following steps: S1. Plan the drone inspection route and image acquisition trigger point based on the digital map of the photovoltaic power station; S2. The UAV flies along the inspection route and acquires visible light and infrared images of the photovoltaic panels using a synchronous acquisition method, and acquires the positioning and attitude data corresponding to each frame of the image. S3. The acquired visible light and infrared images are preprocessed, and the preprocessed visible light and infrared images are spatially aligned using an image registration method to obtain a registered infrared image. Based on the geometric prior of the photovoltaic panel, the registration residual is explicitly estimated to generate a local registration confidence map. The visible light image, the registered infrared image, and the local registration confidence map constitute multi-source image data. S4. Input the multi-source image data into the defect detection model built based on the improved YOLO algorithm to obtain the location and category information of photovoltaic panel defects in the image; S5. Based on location information and positioning and attitude data, a spatial analysis method is used to map the location information into geographic coordinates to obtain the geographic location information and defect category information of photovoltaic panel defects; Step S3 specifically includes: S31. The visible light image and the infrared image are preprocessed respectively. The visible light image is subjected to grayscale transformation, noise filtering and threshold segmentation to obtain a binary image, and the visible light structure skeleton map is extracted by the direction-constrained Sobel operator. The infrared image is subjected to non-uniformity correction and radiometric calibration. S32. A feature-point-based registration algorithm is used to calculate the global homography transformation matrix of the infrared image relative to the visible light image. The infrared image is resampled according to the transformation matrix to obtain the registered infrared image. Contour search is performed on the binary image to extract the coordinates of the four corner points of each photovoltaic panel. Perspective transformation is then performed to correct the image to an approximate front view to obtain the perspective-corrected image. S33. Using the four corner points of the photovoltaic panel detected in the perspective-corrected image as anchor points, an ideal grid is constructed based on the geometric prior of the photovoltaic panel. The template matching method is used to estimate the registration residuals and matching confidence of each grid node. Spatial interpolation is performed with the matching confidence as weight to obtain the dense registration residual field of the whole image. After normalization processing, a local registration confidence map is generated. The visible light image, the registered infrared image, and the local registration confidence map constitute multi-source image data. The generation of the full-image dense registration residual field in step S33 specifically includes: S331. Using the four corner points of the photovoltaic panel detected in the image after perspective transformation correction as anchor points, construct an ideal grid in the visible light image coordinate system according to the known photovoltaic panel size and panel spacing parameters, and obtain the ideal position coordinates of each grid node. S332. For each grid node, a local template centered on the node is cut out from the visible light structure skeleton image. Normalized cross-correlation template matching is performed in the corresponding search window of the registered infrared image. The registration residual vector of the node is recorded by the difference between the main peak position and the ideal node position. The matching confidence of the node is calculated based on the difference between the main and secondary peak response values. S333. Using the registration residual modulus of each grid node as the data fitting target, and the square of the matching confidence as the node weight, a dense registration residual field with the same resolution as the image is generated in the entire map range by using confidence-weighted regularization interpolation.

2. The photovoltaic panel defect inspection method based on UAV and improved YOLO algorithm as described in claim 1, characterized in that, Step S1 specifically includes: A three-dimensional topographic map of the photovoltaic power station is constructed based on the digital orthophoto image and digital elevation model. The position, boundary and orientation information of each photovoltaic panel string is extracted from the digital orthophoto image using image recognition methods. Based on the centerline of each string and the spacing between adjacent strings, the UAV inspection route is planned using the centerline route method. The flight altitude, flight speed and shooting parameters of the dual-light camera are set according to the type of target defect to determine the image acquisition trigger point.

3. The photovoltaic panel defect inspection method based on UAV and improved YOLO algorithm as described in claim 1, characterized in that, In step S332, the matching confidence is calculated using the following formula: ； in, To match confidence levels, The dominant peak value of the normalized cross-correlation response within the search window corresponding to the k-th grid node. To exclude the secondary peak after excluding the neighborhood of the main peak; In step S333, the dense registration residual field is calculated using the following formula: ； in, For dense registration residual field, To find the solution that minimizes the objective functional for the function f; Let be the dense residual field function of the whole graph to be determined. Let k be the image coordinates of the kth grid node. To register the magnitude of the residual vector for the k-th grid node, Let k be the contribution weight of the k-th node to the data item. For function f The Hessian matrix; The bending energy of the thin-plate spline is used to constrain the spatial smoothness of the interpolated surface. To smooth out the regularization coefficients, N is the total number of grid nodes.

4. The photovoltaic panel defect inspection method based on UAV and improved YOLO algorithm as described in claim 1, characterized in that, In step S4, the defect detection model includes a backbone network, a neck network, and a detection head connected in sequence. The backbone network is a Swing Transformer network, which extracts multi-scale feature maps of the input multi-source image data through a shift window self-attention mechanism. The neck network is an efficient multi-scale attention module that performs weighted calibration and fusion on the multi-scale feature maps output by the backbone network. During the training phase, the detection head uses a shape-aware bounding box regression loss function to calculate the regression loss, and during the inference phase, it outputs the category and location information of each defect.

5. The photovoltaic panel defect inspection method based on UAV and improved YOLO algorithm as described in claim 4, characterized in that, The Swin Transformer backbone network comprises four sequentially connected stages: The first stage divides the multi-source image data into non-overlapping 4×4 pixel blocks, which are then flattened and linearly projected to obtain the initial feature map. In the second to fourth stages, each block merging layer stitches together adjacent 2×2 feature blocks from the previous stage and then performs downsampling through linear projection, thereby halving the feature map resolution and doubling the number of channels in each stage. Each stage contains several alternating W-MSA and SW-MSA modules. The input of each sub-module is first processed by layer normalization before entering the attention calculation. The attention calculation result is added to the module input residual. The residual addition result is then processed by layer normalization and input to a multilayer perceptron. The multilayer perceptron output is then added to its input residual again. The four stages output feature maps of different resolutions, which together constitute a multi-scale feature map. The efficient multi-scale attention module divides the multi-scale feature map into several groups along the channel dimension. After each group of features is reshaped to the batch dimension, it is fed into two parallel branches for processing. The first branch uses 1×1 convolution followed by one-dimensional global average pooling along the height and width directions to extract directional contextual features; the second branch uses 3×3 convolution to extract multi-scale spatial structure features. The outputs of the two branches are respectively processed by global average pooling and Softmax normalization, and then generated by matrix dot product operation to generate spatial attention weights. At the same time, after each training step, the current attention weights and the attention weights of the previous time step are updated by exponential moving average with a decay factor to generate channel attention weights. The spatial attention weights and channel attention weights are activated by Sigmoid and then multiplied with the input feature map to complete the dual weighted calibration of channel and spatial dimensions, and obtain the calibrated fused feature map.

6. The photovoltaic panel defect inspection method based on UAV and improved YOLO algorithm as described in claim 4, characterized in that, In step S4, the shape-aware bounding box regression loss function is calculated using the following formula: ； Where L is the total Shape-IoU loss value; This is the intersection-union ratio (IoU) between the predicted bounding box and the ground truth bounding box. The shape-aware center distance penalty term is calculated using the following formula: ； in, , The coordinates of the center of the prediction box. , Let be the coordinates of the center of the ground truth bounding box, and c be the diagonal length of the minimum bounding box between the predicted and ground truth bounding boxes. This is the shape weighting coefficient in the horizontal direction. This is the shape weight coefficient in the vertical direction, and its value is determined by the aspect ratio of the actual bounding box; The shape-aware contour penalty term is calculated using the following formula: ； Where w and h are the width and height of the prediction box, respectively. This is a weighted value representing the difference in size between the predicted bounding box and the true bounding box along the t-direction. This is the penalty index.

7. The photovoltaic panel defect inspection method based on UAV and improved YOLO algorithm as described in claim 4, characterized in that, The execution process of the defect detection model includes the following steps: S41. After scaling the multi-source image data to a fixed size, input it into the backbone network of the defect detection model. After four stages of feature extraction, multi-scale feature maps with different resolutions are obtained. S42. Input the multi-scale feature map into the efficient multi-scale attention module for feature fusion to generate a weighted and calibrated fused feature map. S43. Input the calibrated fused feature map into the detection head, and output the predicted position value and category confidence score of each defect candidate box; S44. Perform non-maximum suppression on the predicted location value and category confidence score to remove redundant detection boxes and obtain the location and category information of photovoltaic panel defects in the image.

8. The photovoltaic panel defect inspection method based on UAV and improved YOLO algorithm as described in claim 1, characterized in that, Step S5 specifically includes: using location information, combining the positioning and attitude data corresponding to each frame image and camera intrinsic parameters, calculating the pixel coordinates of the detection box into latitude and longitude coordinates in the geodetic coordinate system; further, stitching together the visible light images of all frames into a power station orthophoto map, using the absolute geographic accuracy of the panoramic orthophoto map as a benchmark, correcting the latitude and longitude coordinates of each defect to obtain the geographic location information of the photovoltaic panel defects.

Images