Health diagnosis method and system based on multi-modal unmanned aerial vehicle inspection

Abstract

The present application relates to a kind of health diagnosis method and system based on multi-modal unmanned aerial vehicle inspection.The multi-modal inspection data of photovoltaic field area is collected, image time registration is completed, visible light image is combined to construct the three-dimensional geometric model of photovoltaic array with geometric constraint, the accurate temperature measurement of component subdivision area of infrared image pixel is realized, infrared temperature reflection suppression and working condition normalization correction are completed relying on environmental observation data, standardization temperature rise data, visible light image feature and electrical monitoring data are fused to realize fine heat defect diagnosis output, which helps to eliminate the infrared temperature measurement interference caused by photovoltaic glass mirror reflection, complex weather conditions and electrical operation conditions, effectively distinguish environmental shielding false anomaly and component real electrical and thermal fault, greatly improve the photovoltaic module heat defect identification precision, positioning granularity and detection versatility, realize the upgrading of photovoltaic module from extensive temperature anomaly screening to fine, hierarchical and quantifiable intelligent health diagnosis.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection and multimodal data fusion diagnostic technology for photovoltaic power plants, and more specifically, to a health diagnosis method and system based on multimodal drone inspection. Background Technology

[0002] With the large-scale construction of centralized photovoltaic power plants, mountain photovoltaic systems, fishery-photovoltaic hybrid systems, agricultural-photovoltaic hybrid systems, and distributed rooftop photovoltaic systems, problems such as hot spots on photovoltaic modules, abnormal bypass diodes, string mismatch, overheating of junction boxes, dust accumulation due to shading, and overheating of connectors are frequently occurring, seriously affecting photovoltaic power generation efficiency and posing fire safety hazards. Drones equipped with visible light cameras and infrared thermal imaging cameras can achieve large-scale, high-efficiency inspections of photovoltaic fields, gradually becoming the mainstream method for checking the condition of photovoltaic modules. Conventional drone infrared inspections can identify areas with abnormal temperatures based on infrared images, outputting basic inspection information such as hot spot images and location latitude and longitude, meeting the basic needs of large-scale surveys of photovoltaic power plants.

[0003] In the process of realizing this invention, the inventors discovered that the existing photovoltaic drone infrared inspection solution still has a core defect that is difficult to overcome: relying solely on two-dimensional images to identify temperature anomalies cannot accurately map infrared thermal anomalies to the internal sub-strings, cells, and other sub-structures of the module. At the same time, environmental reflections on the photovoltaic glass surface, external weather conditions, and electrical operating conditions can significantly interfere with the infrared temperature measurement results, easily leading to false detections and temperature measurement distortion. It is difficult to accurately locate and effectively identify thermal anomalies, and cannot provide a refined fault basis for photovoltaic operation and maintenance. Summary of the Invention

[0004] Based on this, and addressing the aforementioned issues, this invention provides a health diagnosis method and system based on multimodal drone inspection. By collecting multimodal inspection data from photovoltaic (PV) fields and completing image time-series registration, a geometrically constrained three-dimensional geometric model of the PV array is constructed using visible light images. This enables precise temperature measurement of component subdivisions in infrared image pixels. Infrared temperature reflection suppression and operating condition normalization correction are achieved based on environmental observation data. The system integrates standardized temperature rise data, visible light image features, and electrical monitoring data to achieve refined thermal defect diagnosis output. This helps eliminate infrared temperature measurement interference caused by PV glass mirror reflection, complex meteorological conditions, and electrical operating conditions. It solves industry pain points such as inaccurate two-dimensional positioning, drift in repetitive texture reconstruction, and inability to accurately match fault levels in traditional inspections. It effectively distinguishes between environmental shading pseudo-anomalies and genuine electrical thermal faults in the components, significantly improving the accuracy, granularity, and versatility of PV module thermal defect identification. This upgrades PV modules from coarse-grained temperature anomaly screening to refined, hierarchical, and quantifiable intelligent health diagnosis, providing stable and reliable technical support for automated drone inspections, precise thermal defect management, safe operation and maintenance, and power generation efficiency optimization for various PV sites.

[0005] In a first aspect, the present invention provides a health diagnosis method based on multimodal unmanned aerial vehicle (UAV) inspection. The method includes: collecting multimodal inspection data and performing time-series registration on the multimodal images in the multimodal inspection data; extracting visible light images from the multimodal images to construct a photovoltaic array topology; performing three-dimensional reconstruction based on the geometric constraints of the photovoltaic array topology to obtain a three-dimensional geometric model of the module; extracting infrared images from the multimodal images and mapping the infrared image pixels within the infrared images to the three-dimensional geometric model of the module to obtain the original temperature information of the module's subdivided regions; performing reflection suppression and operating condition normalization processing on the original temperature information based on environmental observation data from the multimodal inspection data to obtain standardized temperature rise data; and combining the standardized temperature rise data, visible light images, and electrical monitoring data from the multimodal inspection data to determine thermal defects and output diagnostic results.

[0006] Optionally, multimodal inspection data is collected, and the multimodal images in the multimodal inspection data are time-series registered. This includes: planning two types of inspection routes, including orthogonal routes and low-angle oblique routes, based on the layout boundary of the photovoltaic field, and outputting a UAV cruise route plan; controlling the UAV to simultaneously collect visible light image sequences, infrared image sequences, environmental observation data, attitude data, and electrical monitoring data according to the cruise route plan, and summarizing them to obtain multimodal inspection data; using the visible light image sequence and infrared image sequence as multimodal images, frame matching is completed through hardware synchronization triggering or timestamp alignment to perform time-series registration of the multimodal images; wherein, the visible light image sequence consists of multiple visible light images, and the infrared image sequence contains multiple infrared images, each infrared image containing several infrared image pixels. By planning a combined inspection route of orthogonal and low-angle oblique views, simultaneously collecting multi-source inspection data, and completing multimodal image time-series registration based on hardware triggering and timestamp alignment, it can take into account both the details of the photovoltaic module viewed from the front and the coverage of temperature measurement on the oblique surface. It can achieve precise spatiotemporal alignment of visible light images, infrared images, environmental parameters, and electrical data, effectively solving the problems of incomplete coverage, asynchronous acquisition of multiple types of data, and temperature measurement matching deviation caused by misalignment of image frames in traditional UAV single-view inspections. It avoids the defects of component boundary confusion and hot spot mismatch across regions under oblique view shooting, ensuring the data homogeneity and temporal consistency of subsequent 3D reconstruction, temperature mapping, and defect judgment, and providing a high-precision and high-matching original data base for refined health diagnosis of photovoltaic modules.

[0007] Optionally, visible light images are extracted from multimodal images to construct a photovoltaic array topology. Based on the geometric constraints of the photovoltaic array topology, a 3D geometric model of the module is obtained, including: sequentially reading time-registered visible light images and extracting arrangement features; where arrangement features include photovoltaic module frame lines, module corners, and features of inter-row channels and support endpoints; establishing a photovoltaic array topology based on the extracted arrangement features, containing unique identifiers at multiple levels: site, array, row, column, and module; extracting geometric constraints from the photovoltaic array topology regarding module frame parallelism, module spacing consistency, and coplanarity of modules in the same row; introducing these geometric constraints into the motion-restored structure 3D reconstruction operation, solving for the photovoltaic module plane using bundle adjustment, and generating a 3D geometric model of the module. By extracting photovoltaic (PV) array features from visible light images, a PV array topology with multi-level unique identifiers is constructed. The inherent geometric constraints of the array are introduced to assist in motion recovery and 3D reconstruction. This effectively overcomes the problems of drift, feature mismatch, and misaligned component numbers in traditional 3D reconstruction caused by texture repetition and high similarity of PV module arrays. It accurately solves the plane of each PV module and generates a high-precision 3D geometric model of the module. This enables hierarchical spatial modeling of PV power plants from the overall field area to individual modules, providing a high-precision, high-matching 3D spatial benchmark for subsequent precise spatial mapping of infrared pixels, temperature matching of subdivided areas of the module, and precise location of thermal defects.

[0008] Optionally, infrared images are extracted from multimodal images, and the infrared image pixels within the infrared images are mapped to the three-dimensional geometric model of the module to obtain the original temperature information of the module's subdivided regions. This includes: reading the time-registered infrared image sequence and extracting all infrared image pixels carrying temperature values; traversing all infrared image pixels, constructing a spatial observation ray for each individual infrared image pixel, performing an intersection operation between the spatial observation ray and the module plane within the three-dimensional geometric model of the module to obtain three-dimensional intersection points; converting the three-dimensional intersection points to the local normalized coordinate system of a single photovoltaic module, dividing the subdivided regions to which the three-dimensional intersection points belong based on a preset coordinate interval; configuring a unique identifier for each subdivided region, binding the unique identifier of the subdivided region of the infrared image pixel to the original temperature measurement value of that infrared image pixel, and summarizing to obtain the original temperature information of the module's subdivided regions. By constructing spatial observation rays for each infrared image pixel and mapping them to the 3D geometric model of the component, the 2D pixel coordinates of the image are converted into spatial positions in the local coordinate system of the component. This accurately divides the component sub-regions corresponding to the pixels and establishes a unique binding relationship between temperature data and the sub-regions. This effectively solves the problem that traditional 2D infrared detection cannot accurately correspond thermal anomalies to specific components, sub-strings, and local areas. It also eliminates the cross-regional positioning deviation of hot spots caused by oblique angle imaging and dense component arrangement, and realizes pixel-level fine temperature information collection for photovoltaic modules.

[0009] Optionally, based on environmental observation data from the multimodal inspection data, the original temperature information is subjected to reflection suppression and operating condition normalization processing to obtain standardized temperature rise data. This includes: reading environmental observation data containing imaging observation geometric parameters, visible light pixel brightness, real-time irradiance, ambient temperature, and wind speed from the multimodal inspection data, and reading the inverter load from the electrical monitoring data; determining the infrared thermometry reflection pollution weight by combining the imaging observation geometric parameters and visible light pixel brightness; performing reflection suppression processing on the original temperature information based on the reflection pollution weight to remove artificially inflated temperatures and obtain the temperature after reflection suppression; selecting healthy components in the same row, with the same orientation, no obstruction, and no historical defects, and using the robust median after removing extreme values ​​to determine the reference temperature of the components in the same row; and performing operating condition normalization on the temperature after reflection suppression by combining the reference temperature, real-time irradiance, ambient temperature, wind speed, and the inverter load to output standardized temperature rise data. By fusing imaging observation geometric parameters, visible light pixel brightness, and various site environment and electrical operation parameters to construct environmental observation data, and relying on reflection pollution weighting to complete the reflection suppression processing of the original temperature, the interference of temperature inflation caused by environmental reflection of photovoltaic glass mirrors is effectively eliminated, and the false hot spot false detection problem caused by glare, cloud shadows, and sky reflection is avoided. At the same time, a baseline temperature is established by robustly statistically analyzing healthy modules in the same row, and the operating condition is normalized by combining irradiance, wind speed, ambient temperature, and inverter load to eliminate temperature deviation caused by differences in different inspection periods, meteorological conditions, and operating conditions. This enables the comparison, quantification, and reuse of module temperature rise data under different voyages and operating conditions, greatly improving the authenticity and consistency of temperature data, and providing a more robust and standardized temperature rise data foundation for subsequent multimodal joint defect judgment.

[0010] Optionally, thermal defect determination can be performed by combining standardized temperature rise data, visible light images, and electrical monitoring data from multimodal inspection data. This includes: extracting multi-scale temperature features at the pixel, sub-string, and component levels based on standardized temperature rise data; identifying occlusion-type visual features based on texture features from visible light images; and performing multi-scale joint thermal defect classification by fusing multi-scale temperature features, occlusion visual features, and electrical monitoring data to identify different types of component thermal anomalies and obtain defect types and defect determination confidence levels. By extracting multi-scale temperature features at the pixel, sub-string, and module levels, and combining visible light image occlusion visual features with electrical monitoring data, multi-scale joint thermal defect classification is achieved. This overcomes the limitations of traditional single temperature threshold judgment. It can simultaneously consider the microscopic hot spot morphology, sub-string abnormal propagation characteristics, and overall module temperature rise status. It effectively distinguishes between external occlusion interference such as dust accumulation and temporary shadows and real thermal anomalies caused by electrical faults in the module itself. Combined with electrical operation characteristics, it achieves refined identification and classification of multiple types of thermal defects and outputs quantitative defect judgment confidence. This significantly improves the accuracy and robustness of photovoltaic thermal defect detection, avoids false detection and missed detection problems caused by single infrared detection, and provides a reliable defect judgment basis for generating accurate and practical photovoltaic operation and maintenance diagnostic results.

[0011] In the above implementation process, multi-scale joint thermal defect classification is performed by integrating multi-scale temperature features, occlusion visual features, and electrical monitoring data. This includes: using a rule-based judgment model and a machine learning model for collaborative classification. The rule-based judgment model is used for rapid identification of high-confidence faults, while the machine learning model is used to distinguish defect samples with ambiguous boundaries and mixed features. By using a rule-based judgment model and a machine learning model to perform multi-scale joint thermal defect classification, the efficiency of rule-based judgment and the generalization and discrimination capabilities of machine learning are balanced. This approach can quickly and accurately screen typical faults with clear features and high confidence based on fixed judgment rules, improving the efficiency of routine defect detection. At the same time, it can effectively distinguish difficult defect samples with ambiguous boundaries and mixed features by leveraging machine learning models. This solves the problem that a single judgment mode is difficult to distinguish similar thermal anomalies and has poor classification tolerance, significantly improving the precision and adaptability of photovoltaic thermal defect classification under complex operating conditions.

[0012] Optionally, the diagnostic results can be output, including: assessing the power generation loss caused by defects by combining defect type and standardized temperature rise data; determining maintenance priorities by comprehensively considering power generation loss, safety impact, and operation and maintenance cost information; matching treatment recommendations based on maintenance priorities; and generating standardized photovoltaic array diagnostic results by integrating component location, visible light and infrared images, standardized temperature rise data, defect type, power generation loss, and treatment recommendations. By quantitatively assessing the power generation loss corresponding to defects by combining defect type and standardized temperature rise data, determining maintenance priorities and matching corresponding treatment recommendations by comprehensively considering multi-dimensional information such as power generation loss, safety impact, and operation and maintenance costs, and integrating component spatial location, multi-modal image data, temperature rise characteristics, defect information, and operation and maintenance plans to generate standardized photovoltaic diagnostic results, this approach changes the traditional inspection model that only outputs images of abnormal points and lacks quantitative assessment and operation and maintenance guidance. It enables a closed-loop output of the entire process of photovoltaic thermal defects, from anomaly identification to loss quantification, risk rating, and operation and maintenance decision-making, effectively solving the problems of existing inspection reports failing to support accurate work assignment by operation and maintenance personnel, disordered priorities, and lack of data basis for governance decisions.

[0013] In the above implementation process, after outputting the diagnostic results, the following steps are also included: unifying and storing the diagnostic results, multimodal inspection data, component 3D geometric models, and visible light and infrared images corresponding to defects, as well as power generation losses, to construct a 3D health map of the photovoltaic array; and outputting an operation and maintenance reference list and periodic retesting scheme according to the type of photovoltaic power station. By unifying and archiving the diagnostic results, multimodal inspection data, component 3D geometric models, defect image data, and power generation loss calculation parameters to construct a 3D health map of the photovoltaic array, integrated and associated storage of photovoltaic power station defect data, spatial models, and inspection data is achieved. This solves the problems of scattered traditional inspection data, poor ledger correlation, and difficulty in tracing and comparing historical defects, forming an iterative and traceable digital health archive for the power station. At the same time, differentiated operation and maintenance reference lists and periodic retesting schemes are output for different types of photovoltaic power stations, which can adapt to the operation and maintenance needs of centralized, distributed, and other photovoltaic scenarios, and realize a refined operation and maintenance model that combines fixed-point defect management with long-term status tracking.

[0014] Secondly, the present invention also provides a health diagnosis system based on multimodal UAV inspection. This system includes: a data acquisition and registration module for acquiring multimodal inspection data and performing time-series registration of the multimodal images in the multimodal inspection data; a topology reconstruction module for extracting visible light images from the multimodal images, constructing a photovoltaic array topology, and performing three-dimensional reconstruction based on the geometric constraints of the photovoltaic array topology to obtain a three-dimensional geometric model of the component; an infrared mapping temperature measurement module for extracting infrared image pixels from the multimodal images, mapping the infrared image pixels to the three-dimensional geometric model of the component, and obtaining the original temperature information of the component's subdivided regions; a temperature correction module for performing reflection suppression and operating condition normalization processing on the original temperature information based on environmental observation data from the multimodal inspection data to obtain standardized temperature rise data; and a defect diagnosis output module for combining the standardized temperature rise data, visible light images from the multimodal images, and electrical monitoring data from the multimodal inspection data to determine thermal defects and output diagnostic results.

[0015] According to the technical solution of this invention, by collecting multimodal inspection data of photovoltaic fields and completing image time-series registration, and combining visible light images to construct a three-dimensional geometric model of the photovoltaic array with geometric constraints, precise temperature measurement of component subdivision areas of infrared image pixels is achieved. Infrared temperature reflection suppression and operating condition normalization correction are completed based on environmental observation data. Standardized temperature rise data, visible light image features, and electrical monitoring data are integrated to achieve refined thermal defect diagnosis output. This helps eliminate infrared temperature measurement interference caused by photovoltaic glass mirror reflection, complex meteorological conditions, and electrical operating conditions. It solves industry pain points such as inaccurate two-dimensional positioning, drift in repetitive texture reconstruction, and inability to accurately match fault levels in traditional inspections. It effectively distinguishes between environmental shading pseudo-anomalies and real electrical thermal faults of components, significantly improving the accuracy, granularity, and versatility of photovoltaic module thermal defect identification. This upgrades photovoltaic modules from coarse temperature anomaly screening to refined, hierarchical, and quantifiable intelligent health diagnosis, providing stable and reliable technical support for automated drone inspections, precise thermal defect management, safe operation and maintenance, and power generation efficiency optimization of various photovoltaic power plants. Attached Figure Description

[0016] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a health diagnosis method based on multimodal unmanned aerial vehicle (UAV) inspection provided in an embodiment of the present invention; Figure 2This is a flowchart of the data acquisition and time-series registration method provided in the embodiments of the present invention; Figure 3 This is a flowchart of the array topology construction and constraint reconstruction method provided in the embodiments of the present invention; Figure 4 This is a flowchart of the component partition temperature measurement method provided in the embodiments of the present invention; Figure 5 This is a flowchart of the temperature reflection suppression and normalization method provided in the embodiments of the present invention; Figure 6 This is a flowchart of the multimodal joint thermal defect determination method provided in the embodiments of the present invention; Figure 7 This is a flowchart of the diagnostic result generation method provided in the embodiments of the present invention; Figure 8 This is a flowchart of the three-dimensional health map construction method provided in the embodiments of the present invention; Figure 9 This is a schematic diagram of the structure of a health diagnosis system based on multimodal drone inspection provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will now be described with reference to the accompanying drawings. For example, the flowcharts and block diagrams in the drawings illustrate the architecture, functions, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0019] Please refer to Figure 1 , Figure 1 This is a flowchart of a health diagnosis method based on multimodal drone inspection provided in an embodiment of the present invention. The health diagnosis method based on multimodal drone inspection includes: Step S100: Collect multimodal inspection data and perform temporal registration on the multimodal images in the multimodal inspection data.

[0020] In step S100 above, two types of inspection routes are first planned based on the boundaries of the photovoltaic field and the arrangement direction of the modules: orthogonal and low-angle oblique views. The orthogonal route is used to collect the complete array view to support the construction of the photovoltaic array topology. The low-angle oblique view route supplements the three-dimensional geometric information of the module frame, junction box, and bracket side. At the same time, the optimal shooting angle is planned by combining the solar azimuth angle and the module tilt angle to avoid specular reflection. Areas that cannot be avoided are marked with high reflection risk indicators for subsequent temperature correction and weighting processing. Then, according to the generated cruise route, the drone is controlled to synchronously collect visible light image sequences and infrared image sequences through hardware synchronous triggering or timestamp alignment. GPS (Global Positioning System), RTK (Real-Time Kinematics) elevation, and IMU (Inertial Measurement Unit) are recorded simultaneously. Environmental observation data and electrical monitoring data, including attitude quaternions, camera intrinsic parameters, flight altitude, gimbal pitch angle, ambient temperature, wind speed, irradiance, and inverter operating parameters synchronously uploaded from the electrical monitoring terminal, are all collected and packaged into an inspection data package with a unique index. The timing error of synchronous acquisition by visible light and infrared cameras is controlled within 50ms. Visible light image sequences and infrared image sequences are used as multimodal images, each containing multiple single-frame images. Each infrared image carries several infrared image pixels with original temperature measurement values. Timing registration is achieved by frame-by-frame matching based on hardware synchronization signals and timestamps. The entire acquisition synchronization process ensures the spatiotemporal reference consistency of multi-source data from visible light, infrared, meteorology, and electrical sources.

[0021] It is worth noting that during the data acquisition phase, the electrical monitoring terminal, which is simultaneously connected to the site's backend, collects a complete set of electrical monitoring data, including inverter output power, real-time current and voltage of each string, MPPT operating status, and connector load. This electrical monitoring data is used as a core component of the multimodal inspection data and is strictly bound to the same timestamp as visible light and infrared images, achieving complete spatiotemporal synchronization between optical imaging data and electrical monitoring data. While image morphology information can only identify surface obstructions, the synchronously acquired electrical monitoring data can provide the actual electrical operating conditions at the component level. This provides key electrical dimension criteria for distinguishing different defects such as dust accumulation and pseudo-temperature rise, string mismatch, and junction box electrical overheating, and is a necessary foundation for achieving accurate multimodal joint classification.

[0022] Step S200: Extract the visible light image from the multimodal image and construct the photovoltaic array topology; perform three-dimensional reconstruction based on the geometric constraints of the photovoltaic array topology to obtain the three-dimensional geometric model of the component.

[0023] In step S200 above, the time-registered visible light images are read one by one, and layout features such as component border lines, component corners, field row-to-row channels, bracket endpoints, and inverter and combiner box positions are extracted. Based on the extracted features, a photovoltaic array topology map containing unique identifiers at multiple levels, including field, array, row, column, and component, is constructed. Wherein, siteId is the unique identifier for the site, blockId is the array identifier, rowId is the array row identifier, colId is the array column identifier, and moduleId is the unique identifier for each photovoltaic module. Three types of geometric constraints are extracted from the topology: parallel module borders, consistent module spacing, and coplanarity of modules in the same row. The total array constraint error is introduced into the 3D reconstruction calculation of the structure for motion restoration (SfM). Constraint reconstruction process, where E point E represents the feature point reprojection error. line Indicates the component border line reprojection error; E parallel E represents the parallel constraint error of the component borders in the same row; spacing E represents the consistency constraint error in the spacing between adjacent components; plane λ1, λ2, λ3, and λ4 represent the approximate coplanar constraint error of the same row of modules; λ1, λ2, λ3, and λ4 are the weight coefficients corresponding to each constraint. Addressing the engineering challenges of highly homogenized photovoltaic module appearances and the tendency for large-area array textures to easily lead to feature mismatches, causing 3D reconstruction drift and module numbering misalignment, the feature matching process no longer relies solely on local feature descriptor similarity as the selection criterion. After completing the initial coarse feature matching, four constraint dimensions are simultaneously introduced to conduct a comprehensive evaluation of matching quality: local feature similarity, epipolar line consistency score under epipolar geometry constraints, grid consistency score corresponding to the row and column distribution pattern of the photovoltaic array, and spatial arrangement continuity score of the same row of modules. The feature matching score is calculated using the expression: MatchScore = α·s desc + β·s epi +γ·s grid + δ·s row Where α, β, γ, and δ are the fusion weights corresponding to the four constraints, and satisfy the normalization constraint α+β+γ+δ=1, s desc Characterizes the similarity between local gray-level / gradient description vectors of feature points; s epi Evaluate whether the matched feature pairs satisfy the epipolar geometry constraint and filter out erroneous matches that violate the imaging geometry; grid Utilizing the prior knowledge of the regular row and column grid of the photovoltaic array, the spatial coordinates of the matching features are constrained to conform to the grid distribution pattern of the field area; rowBased on the topological prior that the components in the same row are in an approximately plane, long-distance erroneous feature associations across rows and across arrays are suppressed. A matching score threshold is pre-configured, and all coarse matching feature pairs are traversed and the MatchScore is calculated one by one. Only feature matching pairs with a matching score greater than the preset threshold are allowed to be sent to the bundle adjustment process to participate in the component plane iterative solution. Matching pairs with scores lower than the threshold are directly eliminated to avoid low-confidence matching data interfering with the reconstruction optimization process. Subsequently, high-precision 3D geometric models of all components are generated in batches to solve the engineering problems of reconstruction drift and component numbering misalignment of regular photovoltaic arrays from the root.

[0024] Step S300: Extract the infrared image from the multimodal image, map the infrared image pixels in the infrared image to the three-dimensional geometric model of the component, and obtain the original temperature information of the component subdivision area.

[0025] In step S300 above, the time-registered infrared image sequence is read and all infrared image pixels carrying the original temperature measurement values ​​are extracted. Using the device's factory-calibrated extrinsic parameters [R0,t0] and the UAV IMU attitude as initial values, visible light-infrared dynamic registration is completed by relying on the component frame, junction box (core electrical heating component), infrared edge thermal gradient line, and local hot spot center as cross-modal anchor points, and a registration optimization objective function is constructed: Where ρ is the robust loss function; R is the rotation matrix of the visible / infrared camera extrinsic parameters to be optimized; t is the translation vector of the visible / infrared camera extrinsic parameters to be optimized; R pred The predicted rotational extrinsic parameter corresponding to the IMU attitude estimate; t pred The predicted translation extrinsic parameter corresponding to the IMU attitude prediction; Represents the visible light camera projection model; P is the corresponding 3D spatial point in the infrared image; u is the pixel coordinate of the visible light image; line rgb Line segment features extracted from visible light images; ir ν represents the line segment features extracted from the infrared image; μ and ν are the sub-item weight coefficients.

[0026] The spatial transformation relationship between visible light and infrared images is corrected by iteratively optimizing the objective function; for each infrared pixel u ir Reverse construction of space observation ray Ray(u ir ), intersecting the ray with the component plane within the component's 3D geometric model. The intersection point calculation is performed to obtain the corresponding three-dimensional intersection point P, which is the real three-dimensional spatial coordinate of the infrared pixel on the surface of the module. The three-dimensional intersection point is transformed into the local normalized coordinate system of a single photovoltaic module. The subdivided area of ​​the module is determined according to the module layout configuration file. For example, the left / middle / right sub-string, upper / middle / lower cell, junction box, etc., to which the intersection point belongs are divided according to the coordinate interval, and a unique identifier is configured for each subdivided area. The subdivided area identifier corresponding to the infrared pixel is bound to the original temperature measurement value of the pixel. The original temperature information of the subdivided area of ​​the entire module is summarized according to the module level, so as to realize the accurate mapping of the two-dimensional infrared temperature measurement pixel to the internal subdivided structure of the module.

[0027] Step S400: Based on the environmental observation data in the multimodal inspection data, the original temperature information is subjected to reflection suppression and operating condition normalization processing to obtain standardized temperature rise data.

[0028] In step S400 above, a complete set of observation data, including imaging observation geometric parameters, visible light pixel brightness, real-time irradiance, ambient temperature, wind speed, and inverter load, is read from the multimodal inspection data package. A reflection contamination model is constructed based on the observation angle, solar incidence angle, glass reflectivity, visible light pixel brightness, sky visibility range, and reflection direction of adjacent components. The reflection contamination weights are calculated using the Sigmoid function. Where σ is the Sigmoid activation function; The angle of incidence of the sun; is the camera observation angle; I is the visible light pixel brightness; SkyView is the sky visibility factor; DOP is the reflection polarization stability index obtained by approximating the brightness changes from multiple viewpoints; k1-k5 are preset fitting coefficients.

[0029] Furthermore, differential reflection suppression processing is performed on the original temperature information in conjunction with the reflection contamination weight. For areas where the reflection contamination weight is lower than a preset threshold (for example, the threshold can be set to 0.6), the original infrared observation value is converted into radiance, and the equivalent radiance component of environmental reflection is subtracted. Then, it is recalculated into temperature through Planck inverse operation to achieve quantitative correction based on radiation calibration. For high-reflection areas where the reflection contamination weight is greater than or equal to the preset threshold, the temperature value is not forcibly corrected. The corresponding pixels are marked as high-risk areas of reflection interference and bound with a confidence attenuation factor (for example, the attenuation factor is 0~1, and the higher the reflection contamination weight, the lower the attenuation factor value). In the subsequent normalization of working conditions and thermal defect judgment, the fusion weight or judgment confidence of the temperature rise data in this area is reduced to avoid the uncontrollable error introduced by fixed value correction in strong reflection scenarios. Finally, the temperature after reflection suppression and the corresponding reflection interference confidence distribution are output.

[0030] Healthy modules in the same row, facing the same direction, without visible light obstruction, and without historical defects were selected. After removing 10% of modules with extreme temperatures, the robust median was used to calculate the baseline temperature T of the modules in the same row. base Subsequently, a working condition offset fitting function was established based on the healthy samples. The physical meaning of f(·) is: under the current operating conditions (real-time irradiance G, ambient temperature T) a Under conditions of real-time wind speed v and inverter load (Load), the healthy component relative to the reference temperature T base The theoretical temperature rise offset; this fitting function is obtained by collecting historical data of healthy components in the same row under various operating conditions, and calibrating it using linear regression or local weighted regression methods, reflecting the quantitative contribution of operating condition changes to the normal temperature rise of the components; at the same time, it combines the reflection contamination weight with the infrared raw observation temperature T obs Perform reflection suppression processing to obtain the temperature T after reflection suppression. corr The normalized temperature rise calculation formula can be expressed as: in, Standardized temperature rise; T obs The original infrared observation temperature; T base The reference temperature for healthy components in the same row; T a Ambient temperature; G is real-time irradiance; υ is real-time wind speed; Load is inverter load; multi-dimensional operating condition correction is completed to eliminate temperature deviations caused by different inspection periods, weather, and operating loads, and output standardized temperature rise data that can be compared across voyages and eliminate reflection pseudo-anomalies.

[0031] Step S500: Combine standardized temperature rise data, visible light images, and electrical monitoring data from multimodal inspection data to determine thermal defects and output diagnostic results.

[0032] In step S500 above, multi-scale temperature features at the pixel, sub-string, component, and group levels are first extracted based on standardized temperature rise data. Simultaneously, visible light images are read to extract texture features and identify visual features such as dust accumulation, shadows, and bird droppings. Multi-scale joint thermal defect classification is then performed by fusing multi-scale temperature features, occlusion visual features, and electrical monitoring data within multi-modal inspection data. A rule-based judgment model and a machine learning model are used collaboratively to complete the classification. The rule-based model quickly identifies high-confidence typical faults such as junction box overheating, while the machine learning model distinguishes difficult samples with blurred boundaries and mixed features, such as microcracks, hot spots, and dust accumulation / occlusion. The classification outputs defect type (defectType), judgment confidence (confidence), evidence set (evidenceSet), and manual review flag (reviewFlag). Manual review is marked when the confidence level is below a threshold or when multiple defect scores are close. The data is then imported... A hierarchical electrical topology table is used, incorporating monitoring data from inverters, combiner boxes, string current and voltage, MPPT status, etc. An electrical fault propagation coefficient ranging from 1.0 to 8.0 is set based on the fault level. Defect judgment results are corrected by adjusting for string current consistency. A power generation loss assessment model is constructed based on the physical mechanism differences of each defect type, with specific classifications as follows: (1) For hot spots and junction box overheating defects, the evaluation model can be expressed as follows: Among them, P l To estimate power generation losses, P n N is the rated power of the component. b η is the number of bypassed substrings, N is the total number of substrings within the component, and η is the number of bypassed substrings. b This is the duty cycle loss factor after the bypass diode is turned on (0.95 for example). (2) Substring anomalies and bypass diode defects: The evaluation model can be expressed as follows: Among them, P l To estimate power generation losses, I avg I is the average current of healthy components in the same string. x V represents the actual current of the abnormal substring. m η is the voltage at the maximum power point. m This is the mismatch correction factor (dynamically calculated based on the current dispersion of each component in the string). (3) For dust accumulation and shading defects, the evaluation model can be expressed as follows: Among them, P l To estimate power generation losses, P n For the rated power of the component, A s The occlusion area is obtained from visible light image segmentation, where A is the total area of ​​the component, τ is the transmission attenuation coefficient of the occlusion (approximately 0.3~0.6 for tree shadows, approximately 0.1~0.3 for bird droppings / dust, calibrated according to visible light texture classification), and G is the real-time irradiance. r The standard test condition irradiance is 1000 W / m². (4) For string mismatch defects, the expression for the evaluation model can be: Among them, P l To estimate power generation losses, P s For the string rated power, I act For the actual MPPT tracking current, I i η is the theoretical MPPT current calculated based on real-time irradiance and temperature. t The MPPT tracking efficiency loss factor is dynamically calculated based on the degree to which the inverter MPPT voltage deviates from the optimal operating point.

[0033] Furthermore, consider the overall power generation losses and fire risk. fire(Electrical heating defect fire weight significantly increased), Defect spread risk spread The maintenance priority can be calculated using the following expression: Among them, P riority P represents the maintenance priority score. loss To estimate power generation losses; Risk fire Fire safety risk coefficient; Risk spread Cost is the defect propagation risk coefficient. repair The system prioritizes basic maintenance labor costs and maps priorities to four levels: immediate handling, planned maintenance, retesting and observation, and no handling for the time being. High-temperature defects in junction boxes and connectors are prioritized by increasing the fire risk factor. In actual operation and maintenance scenarios, the opportunity cost of downtime construction can be added as a constraint to further optimize the dispatch strategy. The system integrates component 3D coordinates, visible light and infrared image evidence, standardized temperature rise curves, defect types, estimated power generation loss, and tiered handling suggestions to generate standardized photovoltaic array operation and maintenance diagnostic results and output them. Finally, the diagnostic results, multimodal inspection raw data, component 3D geometric models, defect images, and power generation loss calculation parameters are uniformly bound and stored to construct a 3D health map of the photovoltaic array. For distributed rooftop photovoltaic output component cleaning routes and roof shadow obstacle lists, and for centralized ground-mounted photovoltaic output areas, annual power generation loss trend reports and periodic retesting plans are provided, completing a closed loop of full-process defect identification, quantitative assessment, operation and maintenance guidance, and data archiving.

[0034] Please refer to Figure 2 , Figure 2 This is a flowchart of a data acquisition and time-series registration method provided in an embodiment of the present invention; the data acquisition and time-series registration method includes: Step S20: Based on the layout boundary of the photovoltaic field, plan two types of inspection routes, including the orthogonal route and the low-angle oblique route, and output the UAV patrol route plan.

[0035] In step S20 above, the vector boundary of the photovoltaic field, the layout of the module rows and columns, the tilt angle of the module, and the annual solar azimuth change data are read. Two types of complementary inspection routes are planned. The orthogonal route is laid out vertically along the array direction to collect the complete array plane view for topology construction. The low-angle oblique route flies close to the side of the module to supplement the three-dimensional details of the junction box and bracket frame. At the same time, the optimal observation angle of the camera is calculated by combining the real-time solar incidence angle to avoid the strong specular reflection area of ​​the photovoltaic glass. Highly reflective areas that cannot be avoided are marked in the route file. Finally, a standardized UAV cruise route plan containing waypoint coordinates, gimbal pitch angle, shooting interval, and reflection risk mark is output.

[0036] Step S21: Control the UAV to synchronously collect visible light image sequences, infrared image sequences, environmental observation data, attitude data and electrical monitoring data according to the cruise route plan, and summarize them to obtain multimodal inspection data.

[0037] In step S21 above, the UAV loads the cruise route and flies autonomously. The onboard visible light camera and infrared camera use a unified trigger pulse to synchronously shoot. GPS, RTK elevation, IMU attitude quaternion, camera intrinsic parameters, real-time flight altitude, and gimbal pitch angle are collected in real time as attitude data. The real-time irradiance, ambient temperature, and wind speed are obtained by accessing the meteorological sensors at the site. The real-time current and voltage of the inverter and string are read as electrical monitoring data. All images, attitude, meteorological, and electrical data at the same shooting moment are packaged and encapsulated into an inspection data package with a unique task number and flight number, and summarized to form a complete multimodal inspection dataset.

[0038] Step S22: The visible light image sequence and the infrared image sequence are used as multimodal images. Frame matching is completed by hardware synchronization triggering or timestamp alignment to perform temporal registration of the multimodal images.

[0039] In step S22 above, the visible light image sequence consists of multiple visible light images, and the infrared image sequence contains multiple infrared images. Each infrared image contains several infrared image pixels. Millisecond-level frame alignment is achieved by relying on camera hardware synchronization pulses. When there is no hardware synchronization, the storage timestamp of each image is extracted. The visible light and infrared image frames corresponding to the same time are selected with an error threshold of 50ms to complete one-to-one matching. The timestamps of the electrical monitoring data corresponding to the time period are synchronized and aligned to achieve temporal registration of the two types of images. This ensures that the subsequent three-dimensional reconstruction and infrared pixel mapping use spatiotemporally aligned paired image data.

[0040] Please refer to Figure 3 , Figure 3 This is a flowchart of an array topology construction and constraint reconstruction method provided in an embodiment of the present invention; the array topology construction and constraint reconstruction method includes: Step S30: Read the time-registered visible light images one by one and extract the layout features; wherein, the layout features include the straight lines of the photovoltaic module frame, the corner points of the module, the inter-row channels in the field and the end features of the support.

[0041] In step S30 above, all visible light images that have completed time registration are read sequentially. Line detection and corner detection algorithms are used to identify the four border lines of the photovoltaic module, the feature corners of the four corners of the module, the outline of the passageway inside the site, and the two ends of the photovoltaic support frame by frame. Irrelevant interference lines such as weeds, fences, and debris are filtered out, and only the unique geometric features of the photovoltaic array are retained. The set of feature coordinates of a single frame image is output.

[0042] Step S31: Based on the extracted layout features, establish a photovoltaic array topology structure containing unique identifiers at multiple levels, including site, array, row, column, and component.

[0043] In step S31 above, the features of the border, corner points, channels, and supports extracted from the whole image are subjected to grid clustering to automatically divide the field station and sub-array partitions. Each row and column of components is divided into grids according to the array extension direction, and each level is assigned an independent and unique code to construct a photovoltaic array topology map. Complete the construction of a hierarchical index topology structure from the entire site to individual components.

[0044] Step S32: Extract geometric constraints from the photovoltaic array topology, including component frame parallelism, component spacing consistency, and coplanarity of components in the same row.

[0045] In step S32 above, all component meshes within the topology are traversed, and three inherent geometric constraints of the array are extracted: the upper and lower borders of components in the same row are parallel to each other, the horizontal / vertical spacing between the centers of adjacent components is equal, and all components in the same row are in the same plane. These three types of constraints are transformed into error constraints that can be calculated for the 3D reconstruction of the structure of motion restoration (SfM) to suppress the matching drift problem caused by repeated textures.

[0046] Step S33: Introduce geometric constraints into the motion recovery structure 3D reconstruction calculation, solve the photovoltaic module plane by bundle adjustment, and generate the module 3D geometric model.

[0047] In step S33 above, the three types of geometric constraints are integrated into the total array error function, the feature matching calculation logic is optimized, reliable matching points are screened by comprehensively considering four scores, only high-scoring matching pairs participate in the bundle adjustment iteration solution, the spatial plane coordinates of each photovoltaic module are calculated in batches, and a unified three-dimensional geometric model of the entire module is generated.

[0048] Please refer to Figure 4 , Figure 4 This is a flowchart of a component zone temperature measurement method provided in an embodiment of the present invention; the component zone temperature measurement method includes: Step S40: Read the time-registered infrared image sequence and extract all infrared image pixels that carry temperature values ​​within the infrared image.

[0049] In step S40 above, all infrared image sequences that are matched with the visible light time sequence are read, and all pixels in the image are traversed frame by frame. The original temperature measurement value and two-dimensional image coordinates of each pixel are extracted, and all infrared temperature measurement pixel information is completely preserved without any pixel omissions.

[0050] Step S41: Traverse all infrared image pixels, construct a spatial observation ray in reverse for a single infrared image pixel, and perform an intersection operation between the spatial observation ray and the component plane in the component's three-dimensional geometric model to obtain the three-dimensional intersection point.

[0051] In step S41 above, a visible light-infrared registration objective function is constructed based on the component frame, junction box, and thermal gradient features. The function is iteratively optimized to solve for the optimal camera transformation matrix, thus completing the spatial alignment of the two images. Subsequently, for each infrared pixel, an observation ray is generated by extending from the optical center of the infrared camera into space. The observation ray is then intersected with all the component planes output above in sequence. The intersection of the observation ray and the component plane yields a unique three-dimensional intersection point, which represents the true three-dimensional position of the infrared pixel on the photovoltaic panel surface.

[0052] Step S42: Transform the three-dimensional intersection points to the local normalized coordinate system of a single photovoltaic module, and divide the subdivided regions to which the three-dimensional intersection points belong according to the preset coordinate interval.

[0053] In step S42 above, the global coordinates of the three-dimensional intersection point are transformed to the local normalized coordinate system of the corresponding single component. The coordinate interval is divided with the lower left corner of the component as the origin. Based on the interval boundary, the intersection point can be divided into seven sub-regions: left sub-string, middle sub-string, right sub-string, upper battery cell area, middle battery cell area, lower battery cell area, and junction box area.

[0054] Step S43: Configure a unique identifier for each subdivided region, bind the unique identifier of the subdivided region of the infrared image pixel to the original temperature measurement value of the infrared image pixel, and summarize the original temperature information of the component subdivided region.

[0055] In step S43 above, a unique code identifier is set for each subdivided area, and the infrared pixel, the corresponding subdivided area code, and the original temperature measurement value are bound and stored one by one. The data are summarized in layers according to the component, row, and array levels to form a full-field original temperature dataset classified according to the subdivided areas inside the component.

[0056] Please refer to Figure 5 , Figure 5 This is a flowchart of a temperature reflection suppression and normalization method provided in an embodiment of the present invention; the temperature reflection suppression and normalization method includes: Step S50: Read environmental observation data, including imaging observation geometric parameters, visible light pixel brightness, real-time irradiance, ambient temperature, and wind speed, from the multimodal inspection data, and read the inverter load from the electrical monitoring data.

[0057] In step S50 above, all parameters required for correction are uniformly retrieved from the multimodal inspection data package, including imaging geometric parameters such as camera observation angle and solar incidence angle, visible light pixel brightness at the corresponding position, real-time irradiance, ambient air temperature, and on-site wind speed. The inverter load in the electrical monitoring data is also read, and the complete set of operating condition data synchronously collected during a single inspection is fully preserved.

[0058] Step S51: Determine the infrared thermometry reflection contamination weight by combining the imaging observation geometric parameters and the visible light pixel brightness; perform reflection suppression processing on the original temperature information based on the reflection contamination weight to remove the artificially inflated temperature and obtain the temperature after reflection suppression.

[0059] In step S51 above, a reflection weight calculation is constructed based on the solar incident angle, camera observation angle, visible light brightness, and sky visibility factor. The reflection pollution weight of each subdivided region is calculated, and then differentiated processing is implemented according to the reflection pollution weight: For regions with a reflection pollution weight lower than a preset threshold (an example threshold of 0.6), the original infrared observation value is converted into radiance, the environmental reflection equivalent radiance component calculated by the environmental radiation mapping relationship is subtracted, and then the temperature value is converted back to complete the quantitative correction; For high-reflection regions with a reflection pollution weight greater than or equal to the preset threshold, the temperature is not forcibly quantitatively corrected, but a reflection interference confidence attenuation factor is assigned to the pixels in the region (the factor value ranges from 0 to 1, and the higher the reflection pollution weight, the lower the value). The attenuation factor is bound to the temperature data and output, so that the influence weight of the temperature rise data in the region can be reduced according to the factor in the subsequent working condition normalization and multi-scale defect judgment stages, and the temperature after reflection suppression and the full map reflection interference confidence distribution are output.

[0060] Step S52: Select healthy modules in the same row, with the same orientation, no obstructions, and no historical defects, and determine the reference temperature of the modules in the same row using the robust median after removing extreme values.

[0061] In step S52 above, healthy photovoltaic modules that are in the same array row, have the same tilt angle, no shading or dust accumulation in visible light, and have no defect records in historical inspections are selected. The extreme values ​​of the highest and lowest temperatures in this group are removed, and the robust median of the remaining temperatures is taken as the unified temperature benchmark for this row, eliminating the benchmark deviation caused by the slight differences between individual modules.

[0062] Step S53: Combine the reference temperature, real-time irradiance, ambient temperature, wind speed, and inverter load to normalize the operating conditions of the temperature after reflection suppression, and output standardized temperature rise data.

[0063] In step S53 above, multi-dimensional operating condition correction is completed by combining the reference temperature, irradiance, wind speed, and load to eliminate the incomparability of temperature caused by different time periods, weather conditions, and operating conditions, and output a uniform and standardized temperature rise for the entire field.

[0064] Please refer to Figure 6 , Figure 6 The flowchart below shows a multimodal joint thermal defect determination method provided in an embodiment of the present invention; the multimodal joint thermal defect determination method includes: Step S60: Extract multi-scale temperature features at the pixel, substring, and component levels based on standardized temperature rise data.

[0065] In step S60 above, multi-scale features are extracted based on the standardized temperature rise. At the pixel level, the maximum temperature rise, average temperature rise, hot spot edge gradient, and abnormal pixel area of ​​a single point are extracted. At the sub-string level, the temperature rise features of the hot zone along the cell continuity and the bypass diode area are extracted. At the module level, the abnormal area ratio, hot spot aspect ratio, and hot spot distance from the junction box are extracted to form a multi-level temperature feature set.

[0066] Step S61: Identify occlusion-type visual features based on the texture features of the visible light image.

[0067] In step S61 above, the time-registered visible light image is read, and the local texture of the image is extracted using a texture segmentation algorithm. The texture is distinguished by tree shadow strip texture, bird droppings spot texture, bottom dust accumulation band texture, and temporary occlusion block texture. All non-fault occlusion areas are marked for subsequent differentiation between pseudo temperature rise and real electrical thermal defects.

[0068] Step S62: Integrate multi-scale temperature features, occlusion visual features, and electrical monitoring data to perform multi-scale joint thermal defect classification, identify different types of component thermal anomalies, and obtain defect types and defect judgment confidence levels.

[0069] In step S62 above, a rule-based judgment model and a machine learning model are used for collaborative classification. The rule-based judgment model is used for rapid identification of high-confidence faults, and directly outputs the judgment result for faults with clear characteristics such as concentrated high temperature in junction boxes and uniform temperature rise of the entire string. The machine learning model is used to distinguish difficult samples such as hidden cracks, dust accumulation, and string mismatch with blurred boundaries and mixed temperature and shading characteristics. At the same time, the complete electrical topology table of the components to the inverter is connected, and the classification result is corrected by combining string current and MPPT power data. The defect type and judgment confidence level corresponding to each abnormality are output. When the confidence level is lower than the preset threshold, a manual review mark is added.

[0070] Please refer to Figure 7 , Figure 7 This is a flowchart of a diagnostic result generation method provided in an embodiment of the present invention; the diagnostic result generation method includes: Step S70: Assess the power generation loss caused by the defect by combining the defect type with standardized temperature rise data.

[0071] In step S70 above, values ​​in the range of 1.0 to 8.0 are set according to the component local, substring, whole component, string, and inverter levels; combined with the string current consistency correction defect judgment results, a power generation loss model is constructed based on defect type, standardized temperature rise, abnormal area ratio, and electrical level influence coefficient: P loss To estimate power generation losses due to defects; P nom Rated power of the component; Standardized temperature rise; S abn C represents the relative area percentage of the abnormal region. elec This is the electrical fault propagation coefficient; it is used to calculate the estimated power generation loss at a single defect.

[0072] Step S71: Based on the comprehensive information on power generation loss, safety impact, and operation and maintenance costs, obtain maintenance priorities; match handling recommendations according to maintenance priorities.

[0073] In step S71 above, the overall power generation loss and fire risk are considered. fire Risk of defect spread spread The maintenance cost calculation prioritizes maintenance, dividing it into four priority levels based on the score range: P1 Immediate Handling, P2 Planned Inspection, P3 Retesting and Observation, and P4 Temporary No Handling. Standardized operation and maintenance handling suggestions are matched for each level.

[0074] Step S72: Integrate component location, visible light and infrared images, standardized temperature rise data, defect type, power generation loss and remediation suggestions to generate standardized photovoltaic array diagnostic results.

[0075] In step S72 above, the component's three-dimensional spatial coordinates, visible light / infrared screenshots corresponding to defects, standardized temperature rise curves, defect classification, estimated annual power generation loss, and graded disposal suggestions are all integrated to generate a structured, standardized diagnostic report that can be directly imported into the operation and maintenance platform, fully recording the criteria for judging a single defect from all dimensions.

[0076] Please refer to Figure 8 , Figure 8 The flowchart illustrates a method for constructing a three-dimensional health map according to an embodiment of the present invention; the method includes: Step S80: Unify and store the diagnostic results, multimodal inspection data, component 3D geometric model, visible light and infrared images corresponding to defects, and power generation loss to construct a 3D health map of the photovoltaic array.

[0077] In step S80 above, an integrated associated storage database is established, and diagnostic reports, original multimodal inspection data packages, component 3D geometric models, defect evidence images, and power generation losses are bound and associated according to the unique component codes. This allows the construction of a PV-HealthMap 3D health map, which supports the retrieval of historical defects, temperature rise data, and maintenance records by site, array, and component level, enabling full lifecycle traceability of defects.

[0078] Step S81: Output an operation and maintenance reference list and a periodic retesting plan according to the type of photovoltaic power station.

[0079] In step S81 above, differentiated supporting data is output for four types of sites: centralized ground-mounted photovoltaic, mountain photovoltaic, fishery-solar hybrid, and distributed rooftop photovoltaic. Ground / mountain sites output annual loss analysis reports and quarterly defect retest cycles. Rooftop distributed photovoltaic outputs component cleaning routes and roof obstacle removal lists, forming a long-term periodic operation and maintenance guidance plan adapted to different scenarios.

[0080] Please refer to Figure 9 , Figure 9 This is a schematic diagram of the health diagnosis system based on multimodal UAV inspection provided in an embodiment of the present invention. The system includes a data acquisition and registration module 10, a topology reconstruction module 20, an infrared mapping temperature measurement module 30, a temperature correction module 40, and a defect diagnosis output module 50. The data acquisition and registration module 10 is used to acquire multimodal inspection data and perform temporal registration on the multimodal images in the multimodal inspection data. The topology reconstruction module 20 is used to extract visible light images from the multimodal images, construct the photovoltaic array topology, and perform three-dimensional reconstruction based on the geometric constraints of the photovoltaic array topology. The system obtains a three-dimensional geometric model of the component; the infrared mapping temperature measurement module 30 is used to extract infrared image pixels from the multimodal image and map the infrared image pixels to the three-dimensional geometric model of the component to obtain the original temperature information of the component's subdivided areas; the temperature correction module 40 is used to perform reflection suppression and operating condition normalization processing on the original temperature information based on the environmental observation data in the multimodal inspection data to obtain standardized temperature rise data; the defect diagnosis output module 50 is used to combine the standardized temperature rise data, the visible light image in the multimodal image, and the electrical monitoring data in the multimodal inspection data to determine thermal defects and output the diagnostic results.

[0081] The above description is merely an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A health diagnosis method based on multimodal unmanned aerial vehicle (UAV) inspection, characterized in that, include: Multimodal inspection data is collected, and the multimodal images in the multimodal inspection data are time-series registered; wherein, the multimodal inspection data includes visible light images, infrared images, electrical operation monitoring data, and environmental observation data; The visible light image is extracted from the multimodal image to construct the photovoltaic array topology; the three-dimensional reconstruction is performed based on the geometric constraints of the photovoltaic array topology to obtain the three-dimensional geometric model of the component; Infrared images are extracted from the multimodal images, and the infrared image pixels in the infrared images are mapped to the three-dimensional geometric model of the component to obtain the original temperature information of the component's subdivided regions; Based on the environmental observation data in the multimodal inspection data, the original temperature information is subjected to reflection suppression and operating condition normalization processing to obtain standardized temperature rise data. By combining the standardized temperature rise data, visible light images, and electrical monitoring data from the multimodal inspection data, thermal defects are determined and diagnostic results are output.

2. The method according to claim 1, characterized in that, The process of acquiring multimodal inspection data and performing temporal registration on the multimodal images within the multimodal inspection data includes: Based on the planning of the photovoltaic field layout boundary, two types of inspection routes are included: the orthogonal route and the low-angle oblique route. A drone patrol route plan is then output. According to the cruise route plan, the UAV is controlled to simultaneously collect visible light image sequences, infrared image sequences, environmental observation data, attitude data, and electrical monitoring data, and the data are summarized to obtain multimodal inspection data; The visible light image sequence and infrared image sequence are used as multimodal images. Frame matching is completed through hardware synchronization triggering or timestamp alignment to perform temporal registration of the multimodal images. The visible light image sequence consists of multiple visible light images, and the infrared image sequence contains multiple infrared images, each of which contains a number of infrared image pixels.

3. The method according to claim 1, characterized in that, The visible light image is extracted from the multimodal image to construct the photovoltaic array topology; a three-dimensional reconstruction is performed based on the geometric constraints of the photovoltaic array topology to obtain a three-dimensional geometric model of the component, including: The visible light images after time-registration are read one by one, and the layout features are extracted; wherein, the layout features include the straight lines of the photovoltaic module frame, the corner points of the module, the inter-row channels in the field area and the end features of the support; Based on the extracted layout features, a photovoltaic array topology structure is established, which includes unique identifiers at multiple levels: site, array, row, column, and component. Extract geometric constraints such as component frame parallelism, component spacing consistency, and coplanarity of components in the same row from the photovoltaic array topology; The geometric constraints are introduced into the motion-restoring structure 3D reconstruction operation, and the photovoltaic module plane is solved by bundle adjustment to generate the 3D geometric model of the module.

4. The method according to claim 1, characterized in that, The step of extracting the infrared image from the multimodal image, mapping the infrared image pixels within the infrared image to the three-dimensional geometric model of the component, and obtaining the original temperature information of the component's subdivided regions includes: Read the time-registered infrared image sequence and extract all infrared image pixels that carry temperature values; Traverse all infrared image pixels, construct a spatial observation ray in reverse for a single infrared image pixel, and perform an intersection operation between the spatial observation ray and the component plane in the component's three-dimensional geometric model to obtain the three-dimensional intersection point. The three-dimensional intersection points are transformed into the local normalized coordinate system of a single photovoltaic module, and the subdivided regions to which the three-dimensional intersection points belong are divided according to the preset coordinate interval. Each of the subdivided regions is configured with a unique identifier, and the unique identifier of the subdivided region of the infrared image pixel is bound to the original temperature measurement value of the infrared image pixel, so as to obtain the original temperature information of the component subdivided region.

5. The method according to claim 1, characterized in that, The environmental observation data from the multimodal inspection data is used to perform reflection suppression and operating condition normalization processing on the original temperature information to obtain standardized temperature rise data, including: The environmental observation data, including imaging observation geometric parameters, visible light pixel brightness, real-time irradiance, ambient temperature, and wind speed, are read from the multimodal inspection data, and the inverter load in the electrical monitoring data is also read. The infrared thermometry reflection contamination weight is determined by combining the imaging observation geometric parameters and the visible light pixel brightness; the original temperature information is then subjected to reflection suppression processing based on the reflection contamination weight to remove artificially inflated temperatures and obtain the temperature after reflection suppression. Healthy modules in the same row, with the same orientation, no obstruction, and no historical defects were selected. The robust median after removing extreme values ​​was used to determine the reference temperature of the modules in the same row. The operating conditions are normalized by combining the reference temperature, real-time irradiance, ambient temperature, wind speed, and inverter load to output standardized temperature rise data.

6. The method according to claim 1, characterized in that, The process of determining thermal defects by combining the standardized temperature rise data, visible light images, and electrical monitoring data from the multimodal inspection data includes: Based on the standardized temperature rise data, multi-scale temperature features at the pixel, substring, and component levels are extracted. Based on the texture features of the visible light image, identify occlusion-type visual features; By integrating the multi-scale temperature features, occlusion visual features, and electrical monitoring data, multi-scale joint thermal defect classification is performed to identify different types of component thermal anomalies, thereby obtaining the defect type and defect judgment confidence level.

7. The method according to claim 6, characterized in that, The multi-scale joint thermal defect classification, which integrates the multi-scale temperature features, occlusion visual features, and electrical monitoring data, includes: A combination of rule-based decision-making and machine learning models is used for typing. Among them, the rule-based judgment model is used for rapid judgment of high-confidence faults, and the machine learning model is used to distinguish defect samples with ambiguous boundaries and mixed features.

8. The method according to claim 6, characterized in that, The output diagnostic results include: The power generation loss caused by the defects is assessed by combining the aforementioned defect types with standardized temperature rise data; The maintenance priority is determined by combining the information on power generation loss, safety impact, and operation and maintenance costs; treatment recommendations are then matched according to the maintenance priority. By integrating the component locations, visible light and infrared images, standardized temperature rise data, defect types, power generation losses, and remediation suggestions, a standardized photovoltaic array diagnostic result is generated and output.

9. The method according to claim 8, characterized in that, Following the output of the diagnostic results, the system also includes: The diagnostic results, multimodal inspection data, component three-dimensional geometric models, visible light and infrared images corresponding to defects, and power generation losses are uniformly bound and stored to construct a three-dimensional health map of the photovoltaic array. A maintenance reference list and periodic retesting plan will be provided according to the type of photovoltaic power station.

10. A health diagnosis system based on multimodal unmanned aerial vehicle (UAV) inspection, characterized in that, include: The data acquisition and registration module is used to acquire multimodal inspection data and perform temporal registration on the multimodal images in the multimodal inspection data. The topology reconstruction module is used to extract the visible light image from the multimodal image, construct the photovoltaic array topology, and perform three-dimensional reconstruction based on the geometric constraints of the photovoltaic array topology to obtain the three-dimensional geometric model of the component. The infrared mapping temperature measurement module is used to extract the infrared image from the multimodal image, map the infrared image pixels in the infrared image to the three-dimensional geometric model of the component, and obtain the original temperature information of the component's subdivided regions. The temperature correction module is used to perform reflection suppression and operating condition normalization processing on the original temperature information based on the environmental observation data in the multimodal inspection data to obtain standardized temperature rise data. The defect diagnosis output module is used to combine the standardized temperature rise data, the visible light image in the multimodal image, and the electrical monitoring data in the multimodal inspection data to determine thermal defects and output the diagnostic results.

Images