UAV photovoltaic inspection algorithm based on dual-light fusion

Abstract

The present invention relates to the field of unmanned aerial vehicle inspection technology, and discloses a unmanned aerial vehicle photovoltaic inspection detection algorithm based on dual-light fusion, comprising: acquiring image data; calibrating the internal parameters of a dual-light camera to obtain a camera internal parameter matrix and a distortion coefficient; calibrating image registration-related parameters to obtain a homography matrix; establishing a dynamic offset compensation model to calculate a pixel offset; obtaining a defect detection result based on a dual-light image defect detection module; obtaining the coordinates of a detection frame after registration based on the camera calibration parameters and the homography matrix; fusing the dual-light detection results based on an IoU matching algorithm to obtain a fused detection result; and achieving GPS positioning and component number matching based on pixel coordinate conversion to obtain a final detection result. The present invention significantly improves the accuracy and reliability of photovoltaic defect detection, can comprehensively detect a variety of defect types, and effectively solves the technical problems of large limitations of traditional single-spectrum detection, high cross-modal registration failure rate, and insufficient dynamic error compensation.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) inspection technology, and more specifically, to an unmanned aerial vehicle (UAV) photovoltaic inspection detection algorithm based on dual-light fusion. Background Art

[0002] With the rapid development of the photovoltaic industry and the continued expansion of PV power plants, the need for efficient operation and maintenance is becoming increasingly urgent. Traditional manual inspection methods, when applied to large-scale PV power plants, have exposed numerous issues, such as high costs, low efficiency, and difficulty ensuring data accuracy. These methods are no longer able to meet the requirements of refined operation and maintenance for modern PV power plants. Therefore, the use of automated methods, such as drone inspections, has become an inevitable trend.

[0003] Traditional drone-based photovoltaic inspections rely solely on visible light image analysis, failing to identify internal faults in photovoltaic panels. Relying solely on thermal imaging makes it difficult to detect actual obstructions, cracks, or attachments on the panel surface. Both of these single inspection methods have limitations. For example, they cannot simultaneously capture both surface stains and hot spot defects, making it impossible to determine the cause of hot spots (whether they are caused by stains). This makes it difficult to comprehensively and accurately detect photovoltaic panel defects, failing to meet the complex demands of refined photovoltaic power plant operation and maintenance. Furthermore, existing methods face technical bottlenecks in dual-light image registration. Traditional automatic feature matching algorithms suffer from high matching failure rates in cross-modal scenarios due to significant differences in texture features. Manual calibration methods, while reliable, are inefficient. Furthermore, dynamic error compensation during drone flight has not been effectively addressed, resulting in reduced registration accuracy.

[0004] Therefore, a method that integrates multiple detection methods is needed to improve the accuracy and comprehensiveness of photovoltaic inspections. Summary of the Invention

[0005] The present invention provides a photovoltaic inspection and detection algorithm for unmanned aerial vehicles based on dual-light fusion to solve the technical problems in the above-mentioned related technologies.

[0006] The present invention provides a UAV photovoltaic inspection and detection algorithm based on dual-light fusion, including:

[0007] Get image data;

[0008] Based on Zhang Zhengyou's calibration method, the intrinsic parameters of the bifocal camera are calibrated to obtain the camera's intrinsic parameter matrix and distortion coefficients.

[0009] Based on the structured key point annotation method, the image registration related parameters are calibrated to obtain the homography matrix;

[0010] A dynamic offset compensation model is established based on the posture extrapolation method to calculate the pixel offset;

[0011] Based on GroundingDINO, dual-light image defect detection modules are designed to obtain defect detection results;

[0012] Bi-optical image registration is achieved based on camera calibration parameters and homography matrix to obtain the coordinates of the registered detection frame;

[0013] Based on the IoU matching algorithm, the dual-light detection results are fused to obtain the fused detection results;

[0014] GPS positioning and component number matching are achieved based on pixel coordinate conversion to obtain the final detection result.

[0015] Furthermore, the image data includes:

[0016] Calibration plate data: The calibration plate is held in hand and photographed at different positions, angles, and distances. Each camera collects 20 sets of image pairs.

[0017] Hovering data: image pairs taken when the drone is in a hovering state, with 10 consecutive image pairs collected at each shooting point;

[0018] Flight simulation inspection data, captured in flight, includes image pairs obtained when the drone's flight speed is parallel and perpendicular to the nose direction; GPS coordinates, heading angle, pitch angle, and roll angle data corresponding to each set of images are also recorded;

[0019] GPS positioning training data: Aerial photography requirements: Infrared cameras and visible light cameras capture 50 image pairs at three altitudes: 6m, 10m, and 16m, covering different viewing angles and positions. Ground photography requirements: The drone captures a set of image pairs at each object's placement.

[0020] The acquired image pair refers to a pair of images obtained by taking infrared and visible light respectively.

[0021] Furthermore, the intrinsic parameter calibration of the bifocal camera based on the Zhang Zhengyou calibration method includes:

[0022] Preprocess the collected infrared and visible light images, including grayscale conversion and Gaussian filtering;

[0023] Apply sub-pixel Harris corner detection algorithm to extract checkerboard corners;

[0024] Sub-pixel corner detection is performed using a quadratic curve fitting method based on iterative search.

[0025] Construct the coordinates of the checkerboard corners in the world coordinate system;

[0026] Solve the camera intrinsic parameter matrix and distortion coefficient by minimizing the reprojection error function;

[0027] The camera parameters are optimized with the help of Levenberg-Marquardt nonlinear optimization algorithm.

[0028] Furthermore, the method for calibrating image registration related parameters based on the structured key point annotation method includes:

[0029] The dual-spectral images of photovoltaic panels were acquired by hovering drones, and N = 16 spatially evenly distributed geometric feature points were manually annotated.

[0030] Set the key point coordinates in the infrared image to , the coordinates of the corresponding points in the visible light image are , establish the homography matrix relationship ;

[0031] Solve the homography matrix using the DLT algorithm , parameter solution is achieved by minimizing algebraic error;

[0032] The RANSAC algorithm is used to remove mismatched points, and the point pairs that pass the consistency test are retained for accurate calculation of the homography matrix.

[0033] Furthermore, the dynamic offset compensation model is established based on the posture extrapolation method, including:

[0034] Calibration of trigger delay of visible light and infrared cameras by LED flash synchronization ;

[0035] set up The drone's position at the moment, Assuming that the drone moves at a constant speed during the time interval, calculate The position at the moment;

[0036] Calculate the pixel offset on the image plane based on the pose change:

[0037] ;

[0038] ;

[0039] in 、 They are the pixel offsets in the horizontal direction (X-axis direction) and vertical direction (Y-axis direction) on the image plane respectively; 、 are the focal length parameters of the camera in the X-axis and Y-axis directions respectively; is the coordinate of the camera principal point; To set the altitude for the drone; 、 are the position changes of the UAV in the x and y directions respectively; is the heading angle change; is the pitch angle change; For the current moment; For time delay;

[0040] The calculated offset is applied to the infrared image coordinates to achieve precise alignment with the visible light image.

[0041] Furthermore, the method of fusing the bi-optical detection results based on the IoU matching algorithm includes:

[0042] For infrared detection frame and visible light detection frame , calculate the intersection-over-union ratio:

[0043] ;

[0044] when When , the same target is considered to be detected and fused into a composite defect;

[0045] when When the error is detected, manual judgment is performed or the confidence level is lowered;

[0046] when When , it is considered to be a different defect target;

[0047] For the fused detection results, the confidence calculation formula is:

[0048] ;

[0049] in For intersection and comparison; is the infrared detection frame; is the visible light detection frame; is the area function; is the intersection operation; is the union operation; is the confidence after fusion; is the infrared detection confidence; is the visible light detection confidence; 、 、 are the first, second and third confidence weight coefficients respectively.

[0050] A UAV photovoltaic inspection and detection device based on dual-light fusion, comprising:

[0051] A data acquisition module, used for acquiring image data;

[0052] Calibration module, used to calibrate the intrinsic parameters of the bifocal camera based on Zhang Zhengyou calibration method;

[0053] Registration parameter calibration module, used to calibrate image registration related parameters based on structured key point annotation method;

[0054] The offset compensation module is used to establish a dynamic offset compensation model based on the posture extrapolation method and calculate the pixel offset;

[0055] Defect detection module, used to design dual-light image defect detection modules based on GroundingDINO and perform defect detection;

[0056] Registration module, used to realize bi-optical image registration based on camera calibration parameters and homography matrix;

[0057] Fusion module, used to fuse bi-photon detection results based on IoU matching algorithm;

[0058] The positioning and matching module is used to match GPS positioning with component numbers based on pixel coordinate conversion.

[0059] A photovoltaic inspection and detection device for drones based on dual-light fusion includes: a memory and at least one processor, wherein the memory stores instructions;

[0060] The at least one processor calls the instructions in the memory so that the UAV photovoltaic inspection and detection device based on dual-light fusion executes any of the above-mentioned UAV photovoltaic inspection and detection algorithms based on dual-light fusion.

[0061] A computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, implement any of the above-mentioned UAV photovoltaic inspection and detection algorithms based on dual-light fusion.

[0062] The beneficial effects of the present invention are:

[0063] This invention uses dual-light fusion technology to combine the advantages of infrared and visible light images. It obtains a high-precision camera intrinsic parameter matrix and distortion coefficients through the Zhang Zhengyou calibration method. It solves the problem of cross-modal image registration based on the structured key point annotation method, and uses the posture extrapolation dynamic offset compensation model to eliminate spatiotemporal misalignment errors during flight. It uses GroundingDINO to construct dual-light defect detection modules and realizes intelligent fusion through the IoU matching algorithm, ultimately achieving accurate conversion of pixel coordinates to GPS coordinates and component number matching. Compared with the existing technology, this invention significantly improves the accuracy and reliability of photovoltaic defect detection, can comprehensively detect multiple defect types, and effectively solves technical problems such as the large limitations of traditional single-spectrum detection, high cross-modal registration failure rate, and insufficient dynamic error compensation. It provides an efficient, accurate, and reliable technical solution for the intelligent operation and maintenance of large-scale photovoltaic power stations. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a flow chart of the UAV photovoltaic inspection and detection algorithm based on dual-light fusion of the present invention;

[0065] Figure 2 This is an image registration process of a registration module in a UAV photovoltaic inspection and detection device based on dual-light fusion of the present invention. DETAILED DESCRIPTION

[0066] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that these embodiments are discussed solely to enable those skilled in the art to better understand and implement the subject matter described herein, and that the functions and arrangements of the elements discussed may be varied without departing from the scope of this specification. Various examples may omit, substitute, or add various processes or components as needed. Furthermore, features described in some examples may be combined in other examples.

[0067] The terms "first," "second," "third," "fourth," and so forth (if any) in the description and claims of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that shown or described herein.

[0068] At least one embodiment of the present invention discloses a UAV photovoltaic inspection and detection algorithm based on dual-light fusion, such as Figure 1 As shown, the following steps are included:

[0069] Step 100, acquiring image data;

[0070] Image data specifically includes: calibration plate data, hovering data, flight simulation inspection data, and GPS positioning training data. The specific acquisition methods are as follows:

[0071] Calibration plate data: Hold the calibration plate (avoid obstructions) and photograph it at different positions, angles, and distances. Each camera should collect at least 20 pairs of images to cover a sufficient range of pose variations and improve calibration accuracy.

[0072] Hovering data: Image pairs captured while the drone is in a hovering state. Based on the camera's smaller field of view, the drone's orientation is adjusted to ensure that all key points cover the field of view. Ten consecutive image pairs are collected at each shooting point.

[0073] The collection method for flight simulation inspection data is basically the same as that for hovering data. The difference is that the drone's speed is not zero during the shooting. The data is taken in flight and RTK mode is enabled. Complete flight data includes image pairs obtained when the drone's flight speed is parallel and perpendicular to the nose direction. The GPS coordinates, heading angle, pitch angle, roll angle, and other data corresponding to each image set are also recorded.

[0074] GPS positioning training data: Eight cone-shaped reference objects are placed on the ground, spaced at least 2 meters apart, forming a regular grid. Aerial photography requirements: Infrared and visible light cameras are used at altitudes of 6m, 10m, and 16m, capturing 50 image pairs at each altitude, covering different viewing angles and positions. Ground photography requirements: A drone captures a pair of images at each object's location. Real-time GPS positioning is obtained using RTK-GPS to obtain centimeter-level accuracy, using the WGS84 coordinate system.

[0075] The acquired image pair refers to a pair of images obtained by taking infrared and visible light respectively.

[0076] Step 200, calibrating the collected chessboard infrared image and visible light image based on Zhang Zhengyou calibration method;

[0077] The specific steps include:

[0078] First, the captured infrared and visible light images are preprocessed using the OpenCV image processing library, including grayscale conversion and filtering, to improve image quality and facilitate corner extraction. Then, a corner detection algorithm is applied to extract checkerboard corners and refine them to sub-pixel resolution. Because infrared images have different grayscale characteristics than visible light images, parameters of the corner detection algorithm, such as the threshold and Gaussian filter kernel size, need to be adjusted to ensure accurate corner extraction. Finally, the coordinates of the checkerboard corners are constructed in the world coordinate system. Assuming that the side length of each small square on the checkerboard is d, the world coordinates of the checkerboard corners are (i×d,j×d,0), where i and j are the row and column indices of the corner point on the checkerboard. For each image, the extracted corner coordinates and the corresponding world coordinates are combined into a data set. These data sets are then input into the Zhang Zhengyou calibration algorithm to solve the camera intrinsic parameter matrix and distortion coefficients by minimizing the reprojection error.

[0079] It should be noted that the internal parameter matrix Expressed as:

[0080] ;

[0081] in 、 are the focal length parameters of the camera in the X-axis and Y-axis directions respectively; is the camera principal point coordinate.

[0082] Distortion coefficient vector Expressed as:

[0083] ;

[0084] in 、 、 are the first, second and third radial distortion coefficients respectively, 、 are the first and second tangential distortion coefficients respectively.

[0085] At the same time, the Levenberg-Marquardt algorithm is used to further optimize the solved camera parameters.

[0086] Step 300, calibrating image registration related parameters based on a homography alignment matrix calculation method based on structured key point annotation to obtain a homography matrix;

[0087] When using traditional image automatic feature matching algorithms, the matching failure rate is high in cross-modal (visible light / infrared) scenarios due to large differences in texture features and misalignment between thermal radiation and optical features. Manual calibration of key points is reliable but inefficient. Existing methods cannot balance accuracy and engineering practicality.

[0088] This paper proposes a homography alignment matrix calculation method based on structured key point annotation. The method uses a hovering drone to acquire bispectral images of photovoltaic panels, manually annotates N spatially evenly distributed geometric feature points, and constructs a cross-modal mapping relationship to address the failure of automatic registration algorithms in bispectral images. The method specifically includes the following sub-steps:

[0089] Step 301, key point annotation: Obtain dual-spectral images of photovoltaic panels through hovering drone photography, and manually annotate N = 16 spatially evenly distributed geometric feature points. Select significant features such as photovoltaic panel corners and component boundary intersections as annotation points to ensure good recognizability in both infrared and visible light images.

[0090] Step 302, homography matrix calculation: Let the key point coordinates in the infrared image be , the coordinates of the corresponding points in the visible light image are , homography matrix satisfy:

[0091] ;

[0092] in is the homogeneous coordinate of the key point in the infrared image; are the homogeneous coordinates of the corresponding points in the visible light image; is the pixel coordinate of the key point in the infrared image; are the pixel coordinates of the corresponding point in the visible light image; is the homography matrix.

[0093] Homography matrix for matrix:

[0094] ;

[0095] in to are the 9 elements of the homography matrix.

[0096] Step 303, least squares solution: use the DLT (Direct Linear Transformation) algorithm to solve the homography matrix , by minimizing the algebraic error:

[0097] ;

[0098] in For the The coefficient matrix of matching points, is the homography matrix The vector form of The index of the matching point pair.

[0099] Step 304, RANSAC optimization: Use the RANSAC algorithm to remove mismatched points, with 1000 iterations and an inlier threshold of 3 pixels. Finally, point pairs that pass the consistency check are retained for accurate calculation of the homography matrix.

[0100] Step 400: establishing a dynamic offset compensation model based on a posture extrapolation method and calculating pixel offset;

[0101] When a drone is in flight, the triggering delay and posture change of the visible light and infrared cameras cause spatiotemporal misalignment in the bi-optical image, and the static cross-modal mapping relationship becomes invalid. Traditional methods do not consider the spatiotemporal coupling error in motion, resulting in a drop in dynamic scene registration accuracy of >50%. The present invention proposes a dynamic offset compensation model based on posture extrapolation. By integrating GPS positioning, gimbal Euler angles, and camera delay calibration data, a spatiotemporal mapping relationship is constructed to predict and correct the relative offset of the bi-optical image. The model specifically includes the following sub-steps:

[0102] Step 401, time synchronization calibration: calibrate the trigger time delay of the visible light and infrared camera through LED flash synchronization , typical value is 50-100ms.

[0103] Step 402, pose extrapolation model: Assume The drone's position at this moment is ,in 、 、 are the roll angle, pitch angle and heading angle respectively. During the time interval, assuming that the UAV moves at a constant speed, then The pose at this moment is:

[0104] ;

[0105] ;

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] in 、 、 are the first, second and third linear velocity components, respectively. 、 、 are the first, second and triangular velocity components respectively, 、 、 for The position coordinates of the drone at the moment, 、 、 They are The roll angle, pitch angle and heading angle of the drone at all times, For time delay.

[0111] Step 403, offset compensation calculation: Calculate the pixel offset on the image plane based on the pose change:

[0112] ;

[0113] ;

[0114] in 、 They are the pixel offsets in the horizontal direction (X-axis direction) and vertical direction (Y-axis direction) on the image plane respectively; 、 are the focal length parameters of the camera in the X-axis and Y-axis directions respectively; is the coordinate of the camera principal point; To set the altitude for the drone; 、 are the position changes of the UAV in the x and y directions respectively; is the heading angle change; is the pitch angle change; For the current moment; For time delay.

[0115] Step 404, real-time correction: applying the calculated pixel offset to the infrared image coordinates to achieve precise alignment with the visible light image.

[0116] Step 500: Designing dual-light image defect detection modules based on GroundingDINO and performing defect detection;

[0117] The dual-light image defect detection module includes: a first photovoltaic defect detection module based on infrared images and a second photovoltaic defect detection module based on visible light images. Based on groundingDINO, the present invention designs the first photovoltaic defect detection module based on infrared images and the second photovoltaic defect detection module based on visible light images. Paired infrared and visible light images are fed into the first and second photovoltaic defect detection modules, respectively, for defect detection. The module detects infrared defects (hot spots, diode failures, etc.) and visible light defects (grass and tree obstructions, bird droppings, etc.). Specifically, the module includes:

[0118] Network Structure: GroundingDINO is used as the basic detection network, which combines DETR (Detection Transformer) and language model to perform open vocabulary object detection. The network mainly includes: visual feature extractor, text encoder, feature fusion module and detection head;

[0119] Visual feature extractor: uses Swin Transformer as the backbone;

[0120] Text encoder: uses BERT to encode text descriptions;

[0121] Feature fusion module: fuses visual and text features through attention mechanism;

[0122] Detection head: outputs bounding boxes and confidence scores;

[0123] The input, detection category, confidence threshold, and NMS (non-maximum suppression) threshold of the first photovoltaic defect detection module and the second photovoltaic defect detection module are as follows:

[0124] The first photovoltaic defect detection module (infrared defect detection):

[0125] Input: infrared image;

[0126] Detection categories: hot spots, diode failure, component damage, component short circuit, and string drop;

[0127] Confidence threshold: 0.5;

[0128] NMS threshold: 0.4;

[0129] Second photovoltaic defect detection module (visible light defect detection):

[0130] Input: visible light image;

[0131] Detection categories: obstruction by vegetation, bird droppings stains, dust accumulation, component damage, component detachment;

[0132] Confidence threshold: 0.6;

[0133] NMS threshold: 0.4;

[0134] Training parameters:

[0135] Learning rate: 1e-4, using cosine annealing scheduling;

[0136] Batch size: 8

[0137] Number of training rounds: 100;

[0138] Optimizer: AdamW, weight decay 1e-4;

[0139] Data enhancement: random flipping, rotation, brightness adjustment, contrast adjustment.

[0140] Output format: Each test result contains ;

[0141] in is the coordinate of the upper left corner of the detection box; is the coordinate of the lower right corner of the detection box; is the defect category identifier; is the confidence score.

[0142] Step 600, performing bi-optical image registration based on camera calibration parameters and homography matrix;

[0143] Based on the camera intrinsic parameters and distortion coefficients obtained in step 200, the detected defect boxes are dedistorted respectively. Then, based on the homography alignment mapping matrix output in step 300, the defect boxes on the infrared image are mapped so that they are preliminarily aligned with the visible light image. Finally, the dynamic offset factor in step 400 is used to correct the box coordinates of the defect target detected on the infrared image or visible light image so that they are completely aligned with the other image. Based on the above steps, the original pair of images is converted into a scale-aligned image pair, such as Figure 2 As shown, it specifically includes the following sub-steps:

[0144] Step 601, dedistortion processing: Apply the distortion correction formula to the coordinates of the four corner points of the detection frame:

[0145] ;

[0146] ;

[0147] in is the normalized image coordinate, are the coordinates after dedistortion, 、 、 are the first, second and third radial distortion coefficients respectively, 、 are the first and second tangential distortion coefficients respectively.

[0148] Step 602: Homography transformation: transform the infrared image detection frame corner coordinates Through the homography matrix Transform to the visible light image coordinate system:

[0149] ;

[0150] in is the homogeneous coordinates of the corner points of the detection box in the infrared image; is the homogeneous coordinate of the detection box corner points in the transformed visible light image coordinate system; is the homography matrix.

[0151] Recalculate the minimum circumscribed matrix of the bounding box after the transformation.

[0152] Step 603, dynamic offset correction: according to the pixel offset calculated in step 400 Make a final correction to the coordinates:

[0153] ;

[0154] ;

[0155] in is the coordinate after homography transformation, are the final corrected coordinates.

[0156] Step 604, coordinate verification: check whether the corrected coordinates are within the valid range of the visible light image, and perform boundary clipping on coordinates that exceed the boundary.

[0157] Step 700: fusing bi-photon detection results based on an IoU matching algorithm;

[0158] Based on the aligned infrared and visible light defect categories and detection box coordinates obtained in step 600, IOU matching (threshold is 0.6) is performed to finally obtain the matched and fused defective photovoltaic panel components (infrared defects (5 categories), visible light defects (5 categories), and the coexistence of infrared and visible light defects), specifically including:

[0159] IoU calculation: for infrared detection frame and visible light detection frame , calculate the intersection-over-union ratio:

[0160] ;

[0161] in For intersection and comparison; is the infrared detection frame; is the visible light detection frame; is the area function; is the intersection operation; It is a union operation.

[0162] Matching strategy:

[0163] when When , the same target is considered to be detected and fused into a composite defect;

[0164] when When the error is detected, manual judgment is performed or the confidence level is lowered;

[0165] when , they are considered to be different defect targets.

[0166] Fusion rules:

[0167] Infrared defects only: output infrared detection results, and the category remains unchanged;

[0168] Visible light defects only: Output visible light detection results, the category remains unchanged

[0169] Simultaneous dual-light detection: bounding boxes are fused (union), confidence scores are averaged, and the category is marked as "composite defect"

[0170] Confidence update: For the fused detection results, the confidence calculation formula is:

[0171] ;

[0172] in is the confidence after fusion; is the infrared detection confidence; is the visible light detection confidence; 、 、 They are the first, second and third confidence weight coefficients, and their default values ​​are 0.4, 0.4 and 0.2 respectively; It is the intersection and comparison.

[0173] The final output contains 10 defect categories: 5 categories of infrared defects, 5 categories of visible light defects, and possible composite defects.

[0174] Step 800: Matching GPS positioning with component numbers based on pixel coordinate conversion to obtain the final detection result;

[0175] First, among the several major defective PV modules in step 700, the GPS of those with only infrared defects is calculated using the infrared camera parameters and the drone's attitude (roll, pitch, and heading). The GPS of those with only visible light defects is calculated using the visible light camera parameters and the drone's attitude. The GPS longitude and latitude of PV modules with both infrared and visible light defects are calculated using the infrared inspection box results. Then, the GPS of each defective PV module is indexed and matched in the XML file containing the module number to obtain the corresponding module number. Finally, the GPS and module number of each defective PV module are output as the basis for downstream task component defect elimination. This specifically includes the following sub-steps:

[0176] Step 801, GPS coordinate conversion: convert pixel coordinates to GPS coordinates, the conversion formula is:

[0177] ;

[0178] ;

[0179] in is the target GPS latitude coordinate; is the target GPS longitude coordinate; is the GPS latitude coordinate of the drone; is the GPS longitude coordinate of the drone; 、 are the coordinate components of the X-axis and Y-axis in the world coordinate system; 111320 is the constant in meters corresponding to one degree of latitude on the earth.

[0180] GPS conversion principle: First calculate the pixel direction vector, and then convert the pixel coordinates Convert to the direction vector in the camera coordinate system:

[0181] ;

[0182] in 、 is the pixel coordinate component; is the inverse matrix of the camera intrinsic parameter matrix; is the direction vector in the camera coordinate system.

[0183] Then through the rotation matrix and translation vectors Convert the camera coordinate system to the world coordinate system:

[0184] ;

[0185] in is the rotation matrix; is the direction vector in the camera coordinate system; is the direction vector in the world coordinate system.

[0186] Finally, assuming the ground height is , calculate the world coordinates by the intersection of the ray and the ground:

[0187] ;

[0188] ;

[0189] in 、 are the coordinate components of the X-axis and Y-axis in the world coordinate system respectively; 、 、 are the position components of the camera in the world coordinate system; 、 、 is the component of the direction vector in the world coordinate system.

[0190] Step 802, component number matching: establish a spatial index of the PV power station component numbers using an R-tree data structure; perform a spatial query in the component number XML file based on the calculated GPS coordinates; set the matching tolerance to ±2 meters to account for GPS positioning errors.

[0191] Step 803, quality control: Mark the results with GPS positioning accuracy less than 5 meters; establish a historical detection record database to avoid repeated detection; perform statistical analysis of the detection results to identify systematic deviations.

[0192] The final inspection results include: PV module number, GPS coordinates, defect type, confidence level and inspection timestamp.

[0193] This invention uses dual-light fusion technology to combine the advantages of infrared and visible light images, significantly improving the accuracy and reliability of photovoltaic defect detection, solving the limitations of single spectrum detection, and providing effective technical support for the intelligent operation and maintenance of large-scale photovoltaic power stations.

[0194] The above describes an embodiment of the present invention, but this embodiment is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Ordinary technicians in this field can also make more forms of equivalent embodiments based on the inspiration of this embodiment, all of which are protected by this embodiment.

Claims

1. A photovoltaic inspection method for drones based on dual-light fusion, characterized in that: The UAV photovoltaic inspection and detection method based on dual-light fusion includes: Get image data; Based on Zhang Zhengyou's calibration method, the intrinsic parameters of the bifocal camera are calibrated to obtain the camera's intrinsic parameter matrix and distortion coefficients. Based on the structured key point annotation method, the image registration related parameters are calibrated to obtain the homography matrix; Based on the pose extrapolation method, a dynamic offset compensation model is established and the pixel offset is calculated; specifically, the trigger delay of the visible light and infrared cameras is calibrated synchronously by LED flash. ; Calculate the pixel offset on the image plane based on the pose change; Apply the calculated offset to the infrared image coordinates to achieve precise alignment with the visible light image; Based on GroundingDINO, a dual-light image defect detection module is designed and defect detection is performed; Bi-optical image registration is achieved based on camera calibration parameters and homography matrix to obtain the coordinates of the registered detection frame; Based on the IoU matching algorithm, the dual-light detection results are fused to obtain the fused detection results; Based on the pixel coordinate conversion method, GPS positioning and component number matching are achieved to obtain the final detection results.

2. The photovoltaic inspection method based on dual-light fusion by unmanned aerial vehicle according to claim 1 is characterized in that: The image data includes: Calibration plate data: The calibration plate is held in hand and photographed at different positions, angles, and distances. Each camera collects 20 sets of image pairs. Hovering data: The drone is in a hovering state during shooting, and 10 consecutive image pairs are collected at each shooting point; Flight simulation inspection data: capture image pairs with the drone's flight speed parallel to and perpendicular to the nose direction; and record the GPS coordinates, heading angle, pitch angle, and roll angle data corresponding to each set of images. GPS positioning training data: Aerial photography requirements: Infrared cameras and visible light cameras capture 50 image pairs at three altitudes: 6m, 10m, and 16m, covering different viewing angles and positions. Ground photography requirements: The drone captures a set of image pairs at each object's placement. The acquired image pair refers to a pair of images obtained by taking infrared and visible light respectively.

3. The photovoltaic inspection method based on dual-light fusion by unmanned aerial vehicle according to claim 1 is characterized in that: The method of calibrating the intrinsic parameters of the bifocal camera based on the Zhang Zhengyou calibration method includes: Preprocess the collected infrared and visible light images, including grayscale conversion and Gaussian filtering; Apply sub-pixel Harris corner detection algorithm to extract checkerboard corners; Sub-pixel corner detection is performed using a quadratic curve fitting method based on iterative search. Construct the coordinates of the checkerboard corners in the world coordinate system; Solve the camera intrinsic parameter matrix and distortion coefficient by minimizing the reprojection error function; The camera parameters are optimized with the help of Levenberg-Marquardt nonlinear optimization algorithm.

4. The photovoltaic inspection method based on dual-light fusion by unmanned aerial vehicle according to claim 1 is characterized in that: The method for calibrating image registration-related parameters based on the structured key point annotation method includes: The dual-spectral images of photovoltaic panels were acquired by hovering drones, and N = 16 spatially evenly distributed geometric feature points were manually annotated. Set the key point coordinates in the infrared image to , the coordinates of the corresponding points in the visible light image are , establish the homography matrix relationship ; Solve the homography matrix using the DLT algorithm , parameter solution is achieved by minimizing algebraic error; The RANSAC algorithm is used to remove mismatched points, and the point pairs that pass the consistency test are retained for accurate calculation of the homography matrix.

5. The photovoltaic inspection method based on dual-light fusion by unmanned aerial vehicle according to claim 1 is characterized in that: The method for calculating the pixel offset on the image plane based on the posture change is: set up The drone's position at the moment, Assuming that the drone moves at a constant speed during the time interval, calculate The position at the moment; ； ； in 、 are the pixel offsets in the horizontal and vertical directions on the image plane, respectively; 、 are the focal length parameters of the camera in the X-axis and Y-axis directions respectively; is the coordinate of the camera principal point; To set the altitude for the drone; 、 are the position changes of the UAV in the x and y directions respectively; is the heading angle change; is the pitch angle change; For the current moment; For time delay.

6. The photovoltaic inspection method based on dual-light fusion by unmanned aerial vehicle according to claim 1 is characterized in that: The IoU matching algorithm is used to fuse the bi-optical detection results, including: For infrared detection frame and visible light detection frame , calculate the intersection-over-union ratio: ； when When , the same target is considered to be detected and fused into a composite defect; when When the error is detected, manual judgment is performed or the confidence level is lowered; when When , it is considered to be a different defect target; For the fused detection results, the confidence calculation formula is: ； in For intersection and comparison; is the infrared detection frame; is the visible light detection frame; is the area function; is the intersection operation; is the union operation; is the confidence after fusion; is the infrared detection confidence; is the visible light detection confidence; 、 、 are the first, second and third confidence weight coefficients respectively.

7. A UAV photovoltaic inspection and detection device based on dual-light fusion, characterized in that: include: A data acquisition module, used for acquiring image data; Calibration module, used to calibrate the intrinsic parameters of the bifocal camera based on Zhang Zhengyou calibration method; Registration parameter calibration module, used to calibrate image registration related parameters based on structured key point annotation method; The offset compensation module is used to establish a dynamic offset compensation model based on the posture extrapolation method and calculate the pixel offset; Specifically include: calibrating the trigger delay of visible light and infrared cameras through LED flash synchronization ; Calculate the pixel offset on the image plane based on the pose change; Apply the calculated offset to the infrared image coordinates to achieve precise alignment with the visible light image; Defect detection module, used to design dual-light image defect detection modules based on GroundingDINO and perform defect detection; Registration module, used to realize bi-optical image registration based on camera calibration parameters and homography matrix; Fusion module, used to fuse bi-photon detection results based on IoU matching algorithm; The positioning and matching module is used to match GPS positioning with component numbers based on the pixel coordinate conversion method.

8. A UAV photovoltaic inspection and detection equipment based on dual-light fusion, characterized in that: include: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the UAV photovoltaic inspection and detection device based on dual-light fusion to execute the UAV photovoltaic inspection and detection method based on dual-light fusion as described in any one of claims 1-6.

9. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the UAV photovoltaic inspection and detection method based on dual-light fusion according to any one of claims 1 to 6 is implemented.

Images