A method for detecting photovoltaic hot spot of unmanned aerial vehicle across modalities
By establishing a component topology diagram and a cross-modal residual deformation field, the problem of the correspondence between infrared abnormal regions and visible light structures in UAV photovoltaic hot spot detection was solved, and accurate photovoltaic hot spot detection was achieved.
Patent Information
- Application Number
- CN202611363839.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-04
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-09-04
AI Technical Summary
In existing methods for detecting photovoltaic hot spots on drones, it is difficult to accurately correspond infrared anomalies to visible light structures. Environmental and operating conditions cause temperature drift, cross-modal feature mismatch, and local abnormal temperature rises are easily missed.
A component topology graph is established. The cross-modal residual deformation field is obtained through initial geometric transformation and constraints on the topology nodes and edges of the components. A registration uncertainty graph is generated. The desired temperature field is estimated by combining environmental quantities and component operating quantities. The temperature residual characteristics are gated, and the photovoltaic hot spot detection results are output.
It achieves the correspondence between visible light structure and infrared temperature position at different resolutions and shooting times, separates common temperature drift, highlights local abnormal temperature rise, and reduces the propagation of registration error to fusion features and detection results.
Smart Images

Figure CN122863462B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent inspection of photovoltaic power generation equipment by drones and multimodal image processing, and particularly to a method for cross-modal detection of photovoltaic hot spots by drones. Background Technology
[0002] Photovoltaic power plants typically use drones equipped with visible light cameras and infrared radiation thermography cameras to inspect solar modules. Visible light images can reflect appearance information such as module boundaries, cell structure, shading, dust accumulation, and surface reflections, while infrared thermal images can reflect the surface temperature distribution of the modules. Due to differences in resolution, field of view, imaging model, and shooting time between the two types of cameras, coupled with variations in drone pose, non-rigid misalignments can easily occur between the module boundaries and cell positions in the two types of images, making it difficult to accurately correlate infrared anomalies with visible light structures.
[0003] Existing detection methods based on a single infrared temperature threshold are easily affected by a combination of factors, including irradiance, ambient temperature, wind speed, cloud cover, module load, and solar incidence angle. Normal temperature rises may be misidentified as hot spots, and localized hot spots may be masked by overall temperature drift. If infrared features with registration errors are directly fused with visible light features, the mismatch will continue to propagate to the temperature residual and the final confidence level. Furthermore, without module topology constraints, feature correspondences between cross-modules or non-adjacent cells can introduce false anomalies.
[0004] Therefore, there is a need for a cross-modal detection method for photovoltaic hot spots using unmanned aerial vehicles (UAVs) that can address the shortcomings of the existing technologies. Summary of the Invention
[0005] One objective of this invention is to propose a cross-modal detection method for photovoltaic hot spots using unmanned aerial vehicles (UAVs). Addressing the challenges of accurate alignment between visible light images and infrared thermal maps, temperature drift caused by environmental and operational conditions, cross-modal feature mismatch, and missed detection of localized abnormal temperature rises, this method establishes a component topology map including the adjacency relationships of strings, modules, and cells. This map is then reused for solving cross-modal residual deformation, selecting historical reference observations, and defining the fusion neighborhood. Registration uncertainty is formed based on bidirectional closed-loop error, related distribution entropy, and multi-scale deformation divergence. The desired temperature field and temperature residual confidence are estimated by combining environmental operational parameters. The infrared temperature residual features are then gated using the uncertainty and residual confidence, and the final confidence level is calibrated. This suppresses environmental temperature drift and mismatch pseudo-anomalies, resulting in photovoltaic hot spot detection results with module and cell locations, abnormal temperature rise residuals, and cause categories.
[0006] This invention provides a method for cross-modal detection of photovoltaic hot spots using an unmanned aerial vehicle (UAV), comprising the following steps:
[0007] S1. Acquire visible light images, radiation thermography infrared thermal images, camera parameters, pose parameters, environmental parameters, and photovoltaic module operating parameters generated by drone inspections. Extract the boundary features of modules and cells from the visible light images and establish a module topology map that includes string relationships, module adjacency relationships, and cell adjacency relationships.
[0008] S2. Determine the initial geometric transformation based on camera parameters and pose parameters, use the node constraints and edge constraints of the component topology graph to obtain the cross-modal residual deformation field, transform the radiometric infrared thermal image to the component specification coordinates, and generate the specification component infrared image and registration uncertainty map.
[0009] S3. Based on the standard component infrared image, registration uncertainty diagram, environmental quantities, photovoltaic module operating quantities and component topology diagram, select reference observations and estimate the desired temperature field to generate temperature residual diagram and residual confidence diagram;
[0010] S4. Extract visible surface features, temperature residual features, and module boundary features. Establish cross-modal correspondence only for features of the same module or adjacent cells. Gated temperature residual features based on registration uncertainty map and residual confidence map to obtain cross-modal fusion features and gating weights.
[0011] S5. Input the cross-modal fusion features into the hot spot detection network to obtain the hot spot location, abnormal temperature rise residual, cause category and initial confidence level. Calibrate the initial confidence level according to the gating weight and output the photovoltaic hot spot detection results.
[0012] Optionally, S1 includes: establishing an acquisition pair between the visible light image and the radiometric infrared thermogram according to the timestamp;
[0013] Component instance segmentation and line segment intersection detection are performed on visible light images to obtain component boundaries, component corner points, and cell row and column intersection points;
[0014] Using components and cells as nodes, and the relationships between the same string, adjacent components, and four adjacent cells as edges, and recording the sequential position of the components in the string as a grid, the component topology graph is formed.
[0015] Irradiance, ambient temperature, wind speed, cloud cover variation, string current, load, solar incidence angle, and acquisition time difference are written into the multimodal inspection data as the environmental parameters and the photovoltaic module operating parameters.
[0016] Optionally, S2 includes: determining the initial geometric transformation based on camera intrinsic parameters, camera extrinsic parameters, UAV pose, and shooting altitude;
[0017] Within the search area defined by the initial geometric transformation, the corner points of the components and the intersection points of the rows and columns of the cells are taken as candidate topology nodes. The weights of the nodes are determined based on the correlation between the visible light boundary features and the infrared gradient features. The weights of the candidate topology nodes located in the local extreme value region of infrared temperature, the visible light reflection region, or the motion occlusion region are multiplied by a preset attenuation coefficient of less than one.
[0018] The cross-modal residual deformation field is obtained under the joint constraints of node correspondence error and relative position error of adjacent nodes;
[0019] Furthermore, the generation of the registration uncertainty map includes: using the cross-modal residual deformation field to perform a forward mapping from the visible light image to the radiometric infrared thermogram and a reverse mapping from the radiometric infrared thermogram to the visible light image, to obtain a two-way closed-loop error;
[0020] Calculate the normalized distribution entropy of the relevance of candidate topological nodes within the search region;
[0021] The residual deformation field is obtained at at least two spatial scales and the deformation vector bifurcation at the corresponding locations is calculated.
[0022] The bidirectional closed-loop error, the normalized distribution entropy, and the deformation vector divergence are normalized and then weighted and synthesized so that the synthesized value does not decrease as any input quantity increases, thereby obtaining the registration uncertainty value of the standard coordinate position of each component and forming the registration uncertainty map.
[0023] Optionally, S3 includes: selecting observations with standardized component infrared images from multiple historical acquisition pairs of the component under test within a preset acquisition time window as candidate reference observations, based on the string identifier and position of the component under test in the component topology diagram.
[0024] The median of the temperature at the location corresponding to the candidate reference observation is used as the central quantity of the location temperature, and the median absolute deviation of the temperature at the location corresponding to the candidate reference observation relative to the central quantity of the location temperature is used as the temperature discrete quantity.
[0025] The desired temperature field is obtained by regression correction of the location temperature center quantity based on irradiance, ambient temperature, wind speed, cloud cover variation, string current, load, solar incidence angle, and acquisition time difference.
[0026] The registration uncertainty values in the registration uncertainty map of each candidate reference observation are normalized to obtain a normalized registration uncertainty value in the range of zero to one. The candidate reference observations are weighted by subtracting the normalized registration uncertainty value from one, and candidate reference observations with temperature deviations exceeding the removal threshold determined by the temperature discrete quantity are iteratively removed.
[0027] The difference between the infrared image of the standard component and the desired temperature field is used as the temperature residual. The effective number of candidate reference observations, the temperature discreteness, and the normalized registration uncertainty value are combined and mapped. The combined mapping result is used as the residual confidence. The residual confidence does not decrease as the effective number of candidate reference observations increases, nor does it increase as the temperature discreteness or the normalized registration uncertainty value increases.
[0028] Optionally, S4 includes: transforming the visible light image to the component canonical coordinates, and constructing a visible surface feature branch, a temperature residual feature branch, and a component boundary feature branch using a shared component boundary index, respectively;
[0029] In each feature pyramid layer, a cross-modal attention mask is generated based on the component identifier and the adjacent edge of the cell, so that the attention calculation range is limited to the same component or adjacent cells.
[0030] The residual confidence in the residual confidence map is normalized to obtain a normalized residual confidence value in the range of zero to one. The product of one minus the normalized registration uncertainty value and the normalized residual confidence value is used as the basic gating weight. The contribution of the temperature residual feature to the cross-modal fusion feature is adjusted by the basic gating weight, and the basic gating weight is retained as the gating weight in step S5.
[0031] Furthermore, the component identifiers, grids, and adjacent edges of the same component topology graph established in step S1 are used for node correspondence constraints and edge relationship constraints in step S2, candidate reference observation selection in step S3, and cross-modal attention mask generation in step S4, respectively.
[0032] The component topology node is mapped to the component canonical coordinate region using the cross-modal residual deformation field. The mean value of the registration uncertainty value within the component canonical coordinate region is used as the registration uncertainty value of the component topology node. The reference position in step S3 and the temperature residual feature position in step S4 are made to use the component canonical coordinate region.
[0033] When the registration uncertainty of the component topology node increases, the reference weight of the corresponding reference position in step S3 decreases, the gating weight of the corresponding temperature residual feature in step S4 decreases, and the post-calibration confidence of the hot spot region covering the component specification coordinate area in step S5 is not greater than the initial confidence.
[0034] Optionally, S5 includes: outputting a hot spot mask within the cell area by the segmentation branch of the hot spot detection network, outputting a hot spot detection frame, component identification and cell row and column positions by the detection branch, outputting abnormal temperature rise residuals by the regression branch, and outputting the categories of shading, dust accumulation, wiring abnormality, surface reflection or pending verification and their category probabilities by the classification branch based on visible surface features, hot spot spatial morphology and temperature residual distribution.
[0035] The mean of the gating weights within the hot spot region is multiplied by the initial confidence level, and the resulting product is limited to the interval between zero and one, which is used as the calibrated confidence level.
[0036] Furthermore, the training of the hot spot detection network includes: constructing a supervision signal using training samples labeled with component boundaries, cell row and column positions, hot spot masks, abnormal temperature rise residuals, and cause categories;
[0037] Calculate the area ratio of the hot spot labeled region to the area of its component region, and make the segmentation loss weight of the hot spot labeled region whose area ratio is less than the median area ratio of the training samples a preset multiple of the segmentation loss weight of the other hot spot labeled regions, wherein the preset multiple is greater than one.
[0038] The registration uncertainty map and residual confidence map are added to the training samples, and a monotonicity constraint loss is applied to the training output when the registration uncertainty value increases but the basic gating weight does not decrease.
[0039] Furthermore, the lower quartile of the residual confidence value within the component to be detected is taken as the component residual confidence, and the upper quartile of the registration uncertainty value within the component to be detected is taken as the component registration uncertainty.
[0040] When the number of valid candidate reference observations of the component to be detected is less than a preset threshold, the confidence level of the component residual is less than the confidence level threshold determined by the verification sample, or the registration uncertainty of the component is greater than the uncertainty threshold determined by the verification sample, the cause category of the corresponding component is set as the category to be verified, and the detection branch result formed by visible surface features and component boundary features is used as the hot spot candidate result.
[0041] When the number of valid candidate reference observations for subsequent acquisition pairs is not less than the number threshold, the component residual confidence is not less than the confidence threshold, and the component registration uncertainty is not greater than the uncertainty threshold, steps S3 to S5 are re-executed and the photovoltaic hot spot detection results are updated.
[0042] The beneficial effects of this invention are:
[0043] 1. The cross-modal residual deformation field is obtained by combining the initial geometric transformation with the component topology node and edge relationship constraints, and a registration uncertainty map is generated to ensure that the visible light structure and infrared temperature position under different resolutions, fields of view and shooting times can correspond within the component specification coordinates, while quantifying the residual mismatch risk.
[0044] 2. By estimating the desired temperature field through environmental parameters, component operating parameters, and historical reference observations of the same physical component, and by using registration uncertainty weighting and iterative elimination of abnormal observations, common temperature drift can be separated from measured temperatures, highlighting local abnormal temperature rises.
[0045] 3. By reusing the component topology diagram to limit the cross-modal attention range and making the temperature residual feature weight and final confidence change with registration uncertainty and residual confidence, the propagation of registration error to fusion features and detection results can be reduced, while retaining the component-level and cell-level outputs of hot spot location, temperature rise residual and cause category. Attached Figure Description
[0046] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0047] Figure 1 This is a flowchart of a method for cross-modal detection of photovoltaic hot spots using an unmanned aerial vehicle (UAV) according to the present invention.
[0048] Figure 2 This is a flowchart of the S2 cross-modal registration and registration uncertainty generation of the present invention. Detailed Implementation
[0049] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0050] refer to Figures 1-2 A method for cross-modal detection of photovoltaic hot spots using an unmanned aerial vehicle (UAV) includes the following steps:
[0051] S1. Acquire visible light images, radiation thermography infrared thermal images, camera parameters, pose parameters, environmental parameters, and photovoltaic module operating parameters generated by drone inspections. Extract the boundary features of modules and cells from the visible light images and establish a module topology map that includes string relationships, module adjacency relationships, and cell adjacency relationships.
[0052] S2. Determine the initial geometric transformation based on camera parameters and pose parameters, use the node constraints and edge constraints of the component topology graph to obtain the cross-modal residual deformation field, transform the radiometric infrared thermal image to the component specification coordinates, and generate the specification component infrared image and registration uncertainty map.
[0053] S3. Based on the standard component infrared image, registration uncertainty diagram, environmental quantities, photovoltaic module operating quantities and component topology diagram, select reference observations and estimate the desired temperature field to generate temperature residual diagram and residual confidence diagram;
[0054] S4. Extract visible surface features, temperature residual features, and module boundary features. Establish cross-modal correspondence only for features of the same module or adjacent cells. Gated temperature residual features based on registration uncertainty map and residual confidence map to obtain cross-modal fusion features and gating weights.
[0055] S5. Input the cross-modal fusion features into the hot spot detection network to obtain the hot spot location, abnormal temperature rise residual, cause category and initial confidence level. Calibrate the initial confidence level according to the gating weight and output the photovoltaic hot spot detection results.
[0056] In this specific embodiment, S1 includes:
[0057] In this specific embodiment, the data processing device reads data generated by the visible light camera, the radiometric infrared camera, the integrated navigation module, and the flight control module on the same UAV within the same inspection route. The resolution of the visible light image is set to 4096×3072 pixels, and the resolution of the radiometric infrared thermal image is set to 640×512 pixels. The integrated navigation module outputs the position and attitude at 100 Hz, and the flight control module synchronously writes the shooting altitude and trigger sequence number. The device uses the trigger sequence number as the primary key and selects the visible light image with the smallest absolute value of the timestamp difference and the radiometric infrared thermal image within an 80 ms time tolerance to establish an acquisition pair. Images exceeding this tolerance are not included in the current detection queue.
[0058] Collection based on index Sure, Indicates the relationship with the first A visible light image index for matching infrared thermal images. Indicates the first Hardware timestamps of visible light images, in seconds. Indicates the first The hardware timestamp of the infrared thermal image is in seconds. Indicates the infrared thermal image index. Represents the candidate index for visible light images. This represents the operation of selecting the index that minimizes the subsequent timestamp difference from all candidate indices. This indicates that the absolute value operation is performed on the values within the parentheses, and the matching result must also satisfy the condition that the absolute value of the timestamp difference is not greater than 1. , This indicates that the acquisition is time-tolerant and the unit is seconds, with the camera trigger delay calibrated to 0.08 seconds. When multiple visible light candidates have the same timestamp difference, the trigger sequence number that is the same as the infrared thermal image is selected.
[0059] Before takeoff, the infrared radiation thermometry camera performs two-point temperature calibration using a blackbody source, converts the pixel values of the thermal image to degrees Celsius, and saves the original radiation value, converted temperature, camera serial number, and calibration version in the acquisition pair record. The visible light camera saves the exposure time, focal length, distortion parameters, and image timestamp. When there are invalid radiation values in the thermal image, the corresponding pixel is marked with an invalid mask and the pixel is not replaced by the temperature of the neighboring area. When the acquisition pair lacks camera parameters, pose parameters, or shooting altitude, it is marked as having incomplete geometric information and sent to the state processing entry of step S2 along with the visible light image, component boundary record, topology version number, and visible light rollback command, without performing geometric registration solution.
[0060] The component parsing module inputs the visible light image into the component instance segmentation network based on the Transformer encoder and feature pyramid decoder by slicing the visible light image into overlapping 1024×1024 pixel slices. The network outputs the component mask, component boundary probability and instance confidence. The training samples are formed by component outline annotation in the inspection images of the same type of photovoltaic power station. The training set, validation set and test set are divided according to the power station to avoid the same string being repeated across sets. During inference, instances with an instance confidence of not less than 0.70 and a mask area within the range of 0.2% to 35% of the image area are retained, and mask merging is performed according to the intersection-union ratio of the overlapping area of 0.60.
[0061] The instance segmentation network is jointly trained using the binary cross-entropy of the component mask and the boundary distance loss. The validation set is hierarchically calculated according to different shooting heights, solar incident angles, and reflection ratios to determine the component recall rate and average boundary distance. The instance confidence threshold is selected as the maximum threshold that ensures the component recall rate is not lower than 0.98. The model file records the training sample version, input slice size, and threshold configuration. When two consecutive overlapping slices provide inconsistent masks for the same component, the instances are sorted from largest to smallest confidence and the mask with the largest intersection-union ratio with the boundary of the retained component is selected. If there is still no consistent result, the structural state of the component is marked as pending review.
[0062] For each retained component instance, the component parsing module performs sub-pixel line segment detection on its boundary probability map, merges collinear segments with a direction difference of less than 3° and an endpoint spacing of less than 8 pixels, takes the intersection of adjacent horizontal and vertical edges as the component corner point, and then performs peak detection on the line response of the busbar and cell gap within the component mask, generating cell row and column intersection points at the intersection of row and column lines. When the peak value of the line response is lower than the 0.55 threshold determined by the validation set, the component-level boundary is retained without generating unconfirmed cell intersection points, and the cell structure state of the component is written as pending completion.
[0063] Example—The ledger binding module utilizes the current UAV pose, camera intrinsic and extrinsic parameters, and the power plant mapping plane to project the projected centroid of each component instance and the camera rays corresponding to its four corner points onto the mounting plane. The matching cost is calculated by weighting the geometric center distance of the ledger component by its long side, the intersection-union ratio (I / O) of the projected quadrilateral and the ledger component's planar outline, and the average normalized distance of the corner points after clockwise sorting, using 0.35, 0.40, and 0.25 respectively. Only candidate components with a center distance not exceeding 20% of the component's long side, a projection I / O ratio not less than 0.60, and an average corner point distance not exceeding 15% of the component's short side are retained. Within the same installation plane, the Hungarian algorithm is used to find the global one-to-one minimum cost match between instances and ledger records. When the difference between the total costs of two candidates is less than 0.02, the one with the larger projection intersection-union ratio is selected. If they are still the same, the one with the larger boundary confidence is selected. When the same ledger record is contested by multiple instances, only the global matching result is retained. Instances with no matching, no candidate instances, or optimal costs greater than 0.35 are written as identity binding pending review and are prohibited from entering historical reference retrieval. The power station identifier, string identifier, and position grid are obtained from the ledger record through the bound instance, and the component identifier is generated by combining the string identifier and the position grid.
[0064] The topology construction module writes the power station identifier, string identifier, component identifier, cell corresponding to the sequential position within the string, four component corner points, component mask, acquisition pair identifier, and boundary confidence for each component node. It writes the component identifier, row number, column number, standard coordinate cell, four corner coordinates, and structural status for each cell node. When the joint identifier of two instances is the same in the same inspection, the instance corresponding to the global matching result of the instance-ledger is retained, and the other instance is registered as a duplicate observation.
[0065] according to Create a component topology diagram. This indicates that the data collection involved the corresponding component topology diagram and was dimensionless. Represents a collection of component nodes. Represents the set of battery cell nodes. Represents the set of edges that are related to the same string. This represents the set of edges representing the adjacency relationships between components. This represents the set of edges representing the four adjacent edges of a battery cell. This indicates that a union operation is performed on the sets on both sides of the current collection pair, where the sets are derived from the instance segmentation results, row and column intersections, and power plant ledgers.
[0066] Within the same string group, establish adjacent relationships between components based on a one-position difference, and establish relationships within the same string group according to the electrical connection order in the ledger. For cell nodes, follow... Establish four adjacent edges. Indicates the first The row number of each battery cell node is a positive integer. Indicates the first The row number of each battery cell node is a positive integer. Indicates the first The column number of each cell node is a positive integer. Indicates the first The column number of each cell node is a positive integer. and This represents the index of two different cell nodes within the same component. This indicates that the absolute value operation is performed on the difference in row number or column number within the parentheses. The 1 on the right side of the equation means that only a difference of one cell in row or column is allowed. This rule prevents cross-component nodes and non-adjacent cell nodes from forming cell adjacency edges.
[0067] The topology building module simultaneously generates a topology entity-geometric anchor mapping table: each component node corresponds to four component corner points numbered clockwise from the top left corner according to the standard coordinates, and each cell node corresponds to four row and column intersection points determined by its row number and column number; adjacent row and column intersection points within the same component form undirected geometric constraint edges in the order of row first and column second index; cell four-adjacent relationship edges only generate geometric constraint edges between the shared boundary endpoints of two adjacent cells; component adjacent relationship edges only generate geometric constraint edges between facing corner points when the two components belong to the same mounting plane, the boundary distance is not greater than 10% of the component's shorter side, and both corner points are valid; same-group relationship edges are only used as search tags and do not generate geometric constraint edges when the component adjacent conditions are not met at the same time; geometric constraint edges are stored in ascending order of the two endpoint indices, self-loops and duplicate edges are deleted, and cross-component geometric constraint edges are prohibited from being generated when the mounting planes are different, component identifiers conflict, or either endpoint has low confidence; the objective function of step S2 only reads the geometric constraint edge table output by this mapping table;
[0068] The environmental operation acquisition module reads data from the weather station and string monitors centered on the acquisition timestamp. It writes the irradiance, ambient temperature, wind speed, cloud cover change, string current, load, solar incidence angle, and acquisition time difference into the multimodal inspection data record in W / m², degrees Celsius, m / s, dimensionless ratio, A, rated load ratio, degree, and ms, respectively. Each field also writes the sensor identifier, sampling time, effective range, and quality flag. The median of continuous measurements within a 10-second window is taken. If a measurement is missing, cross-string interpolation is not performed, and the quality flag is set to invalid.
[0069] The topology construction module writes the component identifier, grid, component adjacency edges, cell four-adjacency edges, identity binding status, component structure status, and cell structure status of the same component's topology graph into a read-only topology version object. The status is uniquely determined by priority: identity binding pending review, component structure pending review, cell structure pending completion, and structure valid. When identity binding is pending review or component structure is pending review, geometric constraint edges and historical retrieval keys for that component are not generated. Step S2 only outputs structural anomaly result packets with visible light rollback instructions. Step S3 does not retrieve historical references and does not generate temperature residuals. Step S4 only generates visible surface and component boundary features based on trusted component masks. Step S5 outputs component-level pending review status. Select and write the cell row and column positions as indeterminate; when the cell structure is to be completed, retain the reliable component nodes, component adjacent relationship edges and component-level masks, and set the cell node set and cell four-adjacent relationship edges to empty. Step S4 prohibits the generation of cell adjacent masks. Step S5 still writes the cell row and column positions as indeterminate and makes the cause category conform to the category to be reviewed. Only when the structure is valid, step S2 uses the object to constrain the node correspondence and adjacent node relationship. Step S3 uses the string identifier and grid to retrieve the historical reference observation of the same physical component. Step S4 uses the component identifier and cell four-adjacent relationship edges to generate a cross-modal attention mask. No subsequent steps may reconstruct the component number that is inconsistent with the topology version.
[0070] Step S1 outputs a multimodal inspection data packet containing acquisition pair identifiers, visible light images, radiometric infrared thermal images, camera parameters, pose parameters, environmental operation fields, invalid masks, component and cell boundary records, and topology version numbers. The component topology map is then associated with the data packet using the acquisition pair identifiers and written into the inspection object library. Step S2 reads the data packet and component topology map of the same version from the object library.
[0071] In this specific embodiment, S2 includes:
[0072] Step S2 reads the multimodal inspection data packet and read-only topology version object written in step S1. Based on the visible light camera intrinsic parameters, infrared camera intrinsic parameters, two camera extrinsic parameters, UAV pose, shooting altitude, and power plant mapping parameters of the component installation plane, establish the intersection relationship between the camera ray and the component plane. Before takeoff, the extrinsic parameters of the two cameras are obtained using the calibration board. When the root mean square error of the extrinsic parameter reprojection is greater than 1.5 pixels, the status is written as calibration out of tolerance and the cross-modal geometric solution for this sortie is stopped.
[0073] Step S2 generates a registration result package based on seven mutually exclusive master states: identity binding pending verification, component structure pending verification, incomplete geometric information, calibration out of tolerance, cell structure pending completion, low registration confidence, and valid. The priority of the master states is as follows: identity binding pending verification, component structure pending verification, incomplete geometric information, calibration out of tolerance, cell structure pending completion, low registration confidence, and valid. When the same acquisition pair hits multiple abnormal conditions simultaneously, a unique master state is determined according to this priority, and the remaining hit conditions are retained as additional reason codes. The first six abnormal states are each written with at least the acquisition pair identifier, the available component identifier, the topology version number, the status code, the reason code, the visible light image, the reliable component boundary index, the visible light backoff instruction, and the specification infrared image covered by the invalid mask, the registration uncertainty map, and the residual deformation field. Among these, the identity binding pending verification, component structure pending verification, and cell structure pending completion are all specified in the order of mutual exclusion master states. When the structure is to be verified, the geometric information is incomplete, or the calibration is out of tolerance, the subsequent geometric solution is skipped and all invalid placeholder layers are output; only when the camera parameters, pose parameters, shooting height and mounting plane parameters are complete and the root mean square error of the extrinsic reprojection is no more than 1.5 pixels, is it allowed to enter the module-level processing branch where the cell structure needs to be completed. This branch only retains the module-level nodes and module-level masks and prohibits the generation of cell nodes, adjacent edges and row and column positions; when the registration is low confidence, the diagnostic quantity is retained but the valid flag of the temperature path is set to no; when any abnormal state is read in step S3, the expected temperature and temperature residual are not calculated; step S4 only constructs the combination feature of visible surface and reliable module boundary; step S5 uses the combination feature to generate the module-level visible light candidate result with the cause category to be verified, writes the cell row and column position as undetermined and writes the status code into the result library;
[0074] According to the homogeneous pixel coordinates of visible light Find the initial position in the infrared image. Indicates the first The initial two-dimensional pixel coordinates of the candidate topology nodes in the infrared thermal image, in pixels. The superscript 0 indicates the initial state before residual deformation correction. Let represent a dimensionless third-order initial geometric transformation matrix determined by camera intrinsic and extrinsic parameters, UAV pose, shooting altitude, and component planar parameters. Indicates the first The homogeneous pixel coordinates of candidate topological nodes in the visible light image, with units homogeneously normalized. This represents the projection function that divides homogeneous coordinates by the last component to obtain two-dimensional pixel coordinates. Indicates the index of the candidate topology node;
[0075] The registration module establishes a square search area with a side length of 8% of the short side length of the component, centered on each initial position. The corner points of the component and the intersection points of the cell rows and columns obtained in step S1 are used as candidate topological nodes. Zero-mean normalized correlation is calculated between the visible light boundary distance transformation map and the infrared temperature gradient map. The infrared temperature gradient is obtained by the 3×3 Sobel operator and does not participate in the correlation statistics within the invalid mask. When the effective pixel ratio of the search area is less than 70%, the candidate node is marked as having no effective correspondence.
[0076] Node corresponding weights Sure, Indicates the first Each candidate topology node has a corresponding weight, with values ranging from zero to one. This represents the dimensionless, zero-mean normalized correlation coefficient of the node within the search region, truncated to zero or one. The value represents the attenuation coefficient of the abnormal region and is dimensionless. It is set to 0.35 when the candidate position falls into the infrared temperature local extreme value region, the visible light reflection region, or the motion occlusion region, and 1 for other positions. The infrared temperature local extreme value region is determined within the 9×9 effective temperature neighborhood. When the difference between the center temperature and the temperature of all neighboring temperatures is not less than the robust median of the neighborhood, and is not less than the maximum of 3 degrees Celsius of the verification sample and 2.5 times the absolute deviation of the neighborhood median, it is recorded as a local maximum. When the difference between the center temperature and the temperature of all neighboring temperatures is not greater than the robust median of the neighborhood and meets the same threshold, it is recorded as a local minimum. A connected plateau with a temperature difference of no more than 0.2 degrees Celsius between adjacent pixels is recorded as an extreme value plateau as long as it contains the above extreme points and expands outward by 2 pixels. The quartiles of the local temperature gradient are only used to expand the confirmed extreme value region by 1 pixel along the temperature boundary and are not used as an extreme value criterion alone. The visible light reflection region and the motion occlusion region are determined by the visible light saturation pixel ratio of 15% and the optical flow residual of adjacent frames of 3 pixels, respectively.
[0077] The deformed node position is denoted as... , Indicates the first The infrared 2D pixel coordinates of the candidate topology nodes after residual deformation correction, with units in pixels. This represents a two-dimensional transmodal residual deformation field defined in the visible light coordinate domain, with components in pixels. Indicates the first The visible light two-dimensional pixel coordinates of each candidate topology node, in pixels. The aforementioned first The infrared two-dimensional pixel coordinates of each candidate topology node are obtained by initial geometric transformation and are in pixels. The superscript 0 indicates that the residual deformation correction is performed before the complete definition of the initial position formula in this step is used. The residual deformation field is represented by a bicubic B-spline control grid with 16-pixel intervals within the component boundary.
[0078] By minimizing Find the cross-modal residual deformation field. This represents the registration target function, with units of pixel squares. This represents the aforementioned two-dimensional cross-modal residual deformation field. The aforementioned first The weights corresponding to each candidate topology node. and They represent the first The and the first The infrared 2D pixel coordinates of the candidate topology nodes after residual deformation correction, with units in pixels. Indicates the first The optimal corresponding coordinates of each candidate topology node in the infrared search area are given, in pixels. Indicates the first The optimal corresponding coordinates of each candidate topology node in the infrared search area are given, in pixels. This represents the set of geometric constraint edges output by the topological entity-geometric anchor point mapping table in step S1. Indicates the index of all valid candidate topology nodes. Summation, This represents the summation over all endpoint pairs in the set of geometrically constrained edges. This indicates all normalized sampling locations within the component boundary. Summation, This represents the dimensionless weight of the edge relation constraint, and is chosen to be 0.6 on the validation set. This represents the weight of the deformation smoothing term, expressed in squared pixels, and a value of 0.08 is selected on the validation set. Indicates position The first-order spatial gradient of the residual deformation field is dimensionless. Indicates the normalized sampling position index within the component boundary. Represents the L2 norm, Denotes the Frobenius norm. Indicates the relationship with the first Each node is indexed by its adjacent nodes through geometrically constrained edges;
[0079] The optimizer uses a three-layer control grid from coarse to fine to perform iterative reweighted least squares. At each layer, it removes nodes whose residuals exceed three times the median of the residuals of that layer and recalculates the control points. It stops when the relative change of the objective function between two consecutive iterations is less than 0.001 or when it reaches 50 iterations. If fewer than six nodes are retained or the effective nodes cover less than 40% of the component area, it does not output the unconstrained local deformation, but sets the residual deformation to zero, marks the registration status of the component as low confidence, sets the temperature path validity flag to no, and forces step S3 to skip the temperature residual calculation.
[0080] The registration module determines a component specification coordinate grid with a width of 1024 pixels and a height of 512 pixels by projecting the four corners of the component. It performs inverse bilinear sampling on the radiometric infrared thermogram using the composite mapping of the initial geometric transformation and the residual deformation field. The temperature value is still stored in degrees Celsius. The invalid mask is sampled along with the sample and interpolation across the invalid region is prohibited. The specification component infrared image is obtained. At the same time, the component topology node in step S1 is mapped to a node record with component identification, row and column position and specification coordinate region.
[0081] For each standard coordinate position, Calculate the two-way closed-loop error. Indicates position The normalized two-way closed-loop error is dimensionless. This represents the initial geometric transformation and residual deformation synthesis mapping from visible light coordinates to infrared coordinates. This represents the inverse synthetic mapping obtained by resolving the source and target images. This represents the diagonal length of the component in its canonical coordinates, expressed in pixels, and serves as the normalization reference. The second norm is represented and the definition in the aforementioned objective function is used, wherein the error is truncated in the range of zero to one;
[0082] For the The nth valid candidate topology node, and its search area's nth... The relevance of each discrete displacement candidate is normalized using Softmax to be: Press back Calculate the normalized distribution entropy of the nodes. Indicates the first The normalized distribution entropy of 1 / 2 valid candidate topological nodes, with values ranging from zero to one. Indicates the first The first valid candidate topology node The normalized probability of a discrete displacement candidate is dimensionless. Indicates the index of valid candidate topology nodes. Represents the candidate index for discrete displacement. This represents the total number of discrete displacement candidates within the search region, and is an integer greater than one. Indicates the range from 1 to All discrete displacement candidate indices Summation, Indicates the total number of candidates The correlation coefficient is a logarithmic function with a base of 0.10. The softmax temperature is calibrated to 0.10 by the validation set, and the probability sum of each node is one. The node coordinates are written into a 1024×512 canonical coordinate grid along with the synthesis mapping. For any canonical coordinate position... Select at most four valid candidate topology nodes within the same component that are closest in Euclidean distance, and assign weights to the corresponding nodes based on the reciprocal of the square of the normalized coordinate distance. Normalized weighted arithmetic mean is obtained When the position coincides with a node, directly take that node. When nodes are equidistant, they are selected in ascending order of their indices; if a node is outside the convex hull of a valid node, the nearest valid node is selected. When there are no valid candidate topology nodes for the same component, Set it to 1 and mark the component as low-confidence for registration; This represents the position obtained by the aforementioned unique mapping rule. The normalized distribution entropy has a value range of zero to one;
[0083] Two residual deformation fields were obtained using 8-pixel and 16-pixel control grids respectively, and then... Calculate the deformation vector bifurcation. Indicates position The normalized multiscale deformation vector is divergent and dimensionless. Indicates the position of the 8-pixel control grid. The two-dimensional deformation vector at the location, with units in pixels. Indicates the position of the 16-pixel control grid. The two-dimensional deformation vector at the location, with units in pixels. This represents the diagonal length of the aforementioned component's specification coordinates, in pixels. The L2 norm is expressed using the definition in the aforementioned objective function, and the result is truncated in the range of zero to one.
[0084] Registration uncertainty according to synthesis, Indicates position The normalized registration uncertainty has a value range of zero to one. Indicates the aforementioned position Normalized bidirectional closed-loop error, Indicates the aforementioned position Normalized distribution entropy, Indicates the aforementioned position The divergence of the normalized multiscale deformation vector. This represents a function that takes the minimum value among its parameters. This represents a function that takes the maximum value among its parameters. This represents the dimensionless weight of the two-way closed-loop error, taken as 0.45. This represents the distribution entropy weight, which is dimensionless and takes the value 0.30. The dimensionless weights of the deformation vector divergence are represented by a weight of 0.25. The sum of the three weights is one and is determined by the verification and acquisition pairs with manual corresponding points to minimize the registration error sorting loss. The uncertainty does not decrease when any input increases while the others remain unchanged.
[0085] For each component topology node, first, the pixel indices within the component's canonical coordinate region that simultaneously belong to the effective mask and have valid registration uncertainty values are used to form the node's effective pixel set. , Indicates the first The set of valid pixel indices for registration uncertainty of each component topology node. Represents the component topology node index, and is validated before performing any mean or node-level decay formulas. Not empty; when When empty, the arithmetic mean of the node registration uncertainty is not calculated and the following node-level decay formula is not executed, thus reducing the node registration uncertainty. Write it as an invalid value, and set the node-level attenuation factor. According to the conservative convention, the effective pixel ratio is directly set to zero and a low-confidence flag is written. Steps S3 and S4 process the temperature residual branch that depends on this node factor along the existing invalid path and allow the visible surface and component boundary to backtrack. It is forbidden to replace it with the empty set mean or zero mean; only when When the set is not empty, the arithmetic mean of the effective values of the registration uncertainty within the set is used as the node registration uncertainty value, and then... Generate node-level decay factors. Indicates the first The node-level decay factor of each component topology node, with a value ranging from zero to one. Indicates the first The node registration uncertainty value of each component topology node is dimensionless; the node-level attenuation factor monotonically decreases as the node registration uncertainty increases; the selection rule from the normalized coordinate position to the node index is uniquely determined as follows: when the structure is valid and the position falls inside the cell, the unique cell node to which the position belongs is selected; when located at the common boundary or common corner of adjacent cells, the unique cell node is selected in ascending order of row number and column number; when the structure is valid but the position does not belong to any valid cell, the component node to which it belongs is selected; when the cell structure needs to be completed and meets the aforementioned component-level processing prerequisites, the component node to which it belongs is selected; the component node factor and cell node factor are not aggregated at any position; the node record simultaneously writes the node registration uncertainty value, the node-level attenuation factor, the effective pixel ratio, and the low confidence flag. Step S2 writes the selection rule version, the normalized component infrared image, the registration uncertainty image, the cross-modal residual deformation field, the normalized coordinate node record, and the registration status into the registration result package. Steps S3 and S4 read data only according to the same component identifier, normalized coordinate area, and selection rule version in the result package.
[0086] In this specific embodiment, S3 includes:
[0087] Step S3 takes the registration result package from step S2, the environmental operation fields from step S1, and the read-only topology version object as input. It first verifies the validity flag of the temperature path in the registration result package and the quality flags of eight fields for the current acquisition pair: irradiance, ambient temperature, wind speed, cloud cover change, string current, load, solar incidence angle, and acquisition time difference. The current environmental operation vector is formed only when all eight fields are valid and the registration status is valid. If any field is invalid, an insufficient environmental data status is written. The expected temperature of environmental regression, temperature residual, and residual confidence are not calculated. The minimum temperature residual result package is formed with invalid masks and invalid placeholder values, and the visible surface and component boundary back-off branch in step S4 is triggered. It is prohibited to replace invalid fields with zero values, adjacent string data, or field deletion. After the input verification, the reference observation management module locks the same physical component according to the string identifier and position. It retrieves historical acquisition pairs with valid infrared images, registration uncertainty maps, and environmental operation fields for the standard component from the inspection object library 30 days before the detection time. It is prohibited to replace the historical observations of the component under test with the temperature of adjacent components.
[0088] Candidate reference observation records include acquisition pair identifier, component identifier, normalized coordinate temperature matrix, registration uncertainty matrix, irradiance, ambient temperature, wind speed, cloud cover change, string current, load, solar incidence angle, and acquisition time difference. After retrieval, observations with irradiance below 200 W / m², effective temperature pixel ratio below 80%, or registration status of low confidence are first removed. Then, they are sorted by absolute difference from the current solar incidence angle from smallest to largest. If they are equal, they are sorted by acquisition time from most recent to oldest. A maximum of 60 candidate reference observations are retained.
[0089] The reference observation health screening is based on inspection work orders, maintenance records, and on-site verification records. Only observations that are confirmed as normal by the work order, are accepted as normal after maintenance, or are before the first confirmed abnormality of the same component and do not have an abnormality marker are considered as health candidates. A historical abnormality occupancy mask is established for each standard coordinate position. If any historical observation is covered by a confirmed hot spot mask, located as an abnormal cell by on-site records, or located in an abnormal area that was not closed before maintenance, its temperature value and position reference weight are set to invalid and cannot be included in the weighted median, discrete quantity, or environmental regression of that position. When there are fewer than eight valid healthy observations at a certain position, the expected temperature, temperature residual, and residual confidence of that position are set to invalid. When the proportion of valid healthy positions in the component is less than 80% or there are fewer than eight healthy observations at the component level, a reference insufficiency status, a completely invalid layer, and a visible light rollback instruction are output, and the valid residual is not calculated based on the contaminated historical center quantity.
[0090] Historical acquisition pairs are filtered according to string identifier, position, topology version compatibility flag, and 30-day time window. Records with inconsistent string identifiers or position are directly deleted. Topology version resampling is performed only when boundary pixel refinement occurs and the component identifier and cell row and column number remain unchanged. Records with changes in row and column structure are not included in the candidate set. When the filtered record is empty, the number of valid candidate reference observations is set to zero. The expected temperature field, temperature residual map, and residual confidence map are set as invalid placeholders and reads are blocked with a full invalid mask. At the same time, the reason for insufficient reference and the visible light backoff instruction are written. Central quantity, discrete quantity, or regression calculations are no longer performed.
[0091] For the Candidate reference observations Determine the observation-level pre-screening weights, and according to... Determine the location reference weight. Indicates the first The observation-level pre-screening weights for each candidate reference observation range from zero to one. This represents the dimensionless arithmetic mean of the registration uncertainty values of the candidate reference observations within the canonical coordinate region of the component to be detected. Indicates the first Candidate reference observations at location The reference weights have values ranging from zero to one. Indicates the first Candidate reference observations at location The uncertainty of normalized registration Indicates the first Each historical observation strictly follows the selection rules and their version described in step S2 to determine the standard coordinate position. The uniquely selected cell node or module node index. This represents the node-level attenuation factor generated by the uncertainty of its own node registration according to the same rule in step S2 for the unique node corresponding to the historical observation. Indicates the candidate reference observation index. The position reference weights represent the component's standard coordinate positions. Observation-level weights are only used to remove low-quality observations from the entire image, while position reference weights are used for subsequent position statistics and regression. Position reference weights monotonically decrease as the registration uncertainty at any position level or the registration uncertainty at the uniquely selected node increases while the remaining variables remain constant. For each position... First, start with the current set of valid candidate reference observation indices. Indices with a position reference weight greater than zero and a valid temperature value are selected and formed into a positive weight set. Then calculate the total position weight. Only when The number of observations is no less than eight and Only when the value is greater than zero is it allowed to calculate the center quantity and discrete quantity of the position and form a regression index set; otherwise, the center quantity, discrete quantity, expected temperature, temperature residual and residual confidence of the position are all set to invalid, the effective position ratio of the component is updated, and when the effective position ratio of the component is less than 80%, the reference insufficient state, the completely invalid layer and the visible light back-off instruction are output according to the existing state machine.
[0092] At each specified coordinate position that satisfies the aforementioned positive weight access control, and Calculate the positional temperature center quantity and the temperature dispersion quantity. Indicates position The temperature is measured centrally and the unit is degrees Celsius. Indicates position The robust temperature discrete quantity is expressed in degrees Celsius. This function calculates the weighted median based on the reference weights after the semicolon. Specifically, it accumulates the weights only for temperature values within the set of positive weights in ascending order, and only takes the first time the accumulated weights reach a positive total weight. The temperature value at half its original value. Indicates the aforementioned position The set of positive weighted observation indices Indicates the first Candidate reference observations at location The radiation temperature is expressed in degrees Celsius. The aforementioned first Candidate reference observations at location Reference weights, This indicates absolute value operation. The coefficient 1.4826 is used to convert the weighted median absolute deviation into a robust standard deviation estimate under a normal distribution, and it is dimensionless.
[0093] The reference observation management module performs iterative elimination using the location temperature center value as the initial value. For each candidate reference observation, it accumulates the location reference weights of its abnormal pixels. When the proportion of abnormal pixel weights to the total effective pixel weights of that observation exceeds 60%, and the temperature deviation at the corresponding location is greater than 2.5 times the temperature dispersion, and the temperature dispersion is not less than 0.3 degrees Celsius, the observation is deleted from the effective set. For locations with a temperature dispersion less than 0.3 degrees Celsius, 0.3 degrees Celsius is used as the lower limit in the same unit. After each round of deletion, the center value and dispersion are recalculated based on the remaining location reference weights until the effective set no longer changes, five rounds are completed, or the number of effective candidates drops to less than eight. The reason for deletion and the iteration round are written into the reference observation record. When the number of effective candidates is less than eight after iteration, a reference insufficiency status, invalid layer, and visible light rollback command are immediately output, and regression is no longer executed.
[0094] The eight environmental operation fields of each valid candidate reference observation are dimensionlessly standardized according to the median and interquartile range of the training samples to form an environmental operation vector. Fields with an interquartile range of zero are scaled using the minimum scale of the same unit in the deployment configuration. Subsequently, for each valid canonical coordinate position... Independent weighted ridge regression, position The response quantity is only for each historical observation at the same location. The radiation temperature is obtained by not using grid block shared parameters, intra-block temperature aggregation, or cross-location observations, so that the observation set, design matrix, and regression parameters at each location have a unique correspondence.
[0095] In position Before performing regression fitting, only the positive weight set is considered. Observations within the area are based on location reference weights. Sort from largest to smallest, and select the cumulative position reference weights in order to achieve the positive total weight of the position. 90% of the observations were used to form a location regression index set. Observations with equal weight are selected from most recent to oldest acquisition time; forming Then, it was verified again that the number of observations was no less than eight and If the value is greater than zero, and any condition is not met, the regression success flag for that location is set to negative, and the expected temperature, temperature residual, and residual confidence remain invalid; the regression objective function is not established or solved. Create an index set of activity environment fields The initial value is a complete set of eight fields arranged in a fixed order: irradiance, ambient temperature, wind speed, cloud cover variation, string current, load, solar incidence angle, and acquisition time difference. This complete set is used in conjunction with the data acquired in step S1 and the quality verification in step S3 to form the historical environmental operation vector. and the current environment runtime vector The eight fields with the same name are identical item by item; when the collection When the corresponding environment runtime vector matrix condition number is greater than 1000, from Delete in collection The environmental field with the smallest internal variance is refitted. When deleting a field, the relative order of the remaining fields remains unchanged. A maximum of two fields can be deleted. If the condition number is still greater than 1000 after deleting two fields, the regression success flag at that position is set to no. The median and interquartile range of the corresponding fields in the training set used for each field are saved together with the location regression model version. Historical observations and current data collection pairs are all stored using the same data. The same field order and the same standardized parameter projection are required. It is forbidden to introduce components that do not belong to the entire set of the aforementioned eight fields, and it is also forbidden to delete fields only for the fitting end or the prediction end. When the proportion of regression failure locations in any component exceeds 20%, the component's temperature evidence status is set to "Environment Regression Unavailable", and subsequent parameter solving for that component is stopped. The expected temperature under the current environmental conditions is not replaced by the weighted location temperature center quantity, and the fully invalid expected temperature field, temperature residual map, residual confidence map, and visible light backoff instruction are output. The regression success flag for the corresponding location is set to "Yes" only when the number of conditions at each valid location is not greater than 1000 and the parameter solution converges and the predicted value is finite.
[0096] Regression parameters according to Fitting, Indicates position The temperature intercept, in degrees Celsius. Indicates position The environmental regression coefficient vector has a dimension equal to the number of active fields. Each component is in degrees Celsius and is arranged according to... The fixed order of the corresponding fields This indicates the temperature intercept to be optimized, and the unit is Celsius. This represents the vector of regression coefficients of the environment to be optimized, which is of the same dimension and order as the activity field subspace, and each component is in degrees Celsius. The aforementioned first Candidate reference observations at location The radiation temperature, The aforementioned first Candidate reference observations at location Reference weights, Indicates from the first Eight-field dimensionless environment run vector of candidate reference observations Chinese Press The active sub-vectors are selected in a fixed order and formed using the normalized parameters of the saved fields with the same name. This indicates that the vector component selection operation is performed according to the given set of field indices and their order. Indicates the aforementioned position The collection of activity environment field indexes. This indicates the number of fields in the collection, assuming no fields have been deleted. For the complete set of eight fields, Indicates the aforementioned position The set of positional regression indexes determined according to the 90% cumulative positional reference weight rule. Indicates in the parameter and Choose the parameter combination that minimizes the subsequent objective function from the given values. Represents the set of location regression indexes All of them Summation, This represents the ridge penalty coefficient, which is dimensionless and selected as 0.05 based on historically verified observations. This represents the vector transpose operation. Describe the L2 norm and use the definition from the objective function in step S2;
[0097] Current desired temperature field according to calculate, Indicates the current position The desired temperature, in degrees Celsius. and The aforementioned positions are respectively indicated. Temperature intercept and corresponding active field set only The environmental regression coefficient vector, This represents the vector transpose operation and follows the previous definition. This represents the current acquisition vector for the dimensionless environment after eight-field quality verification. In the middle, according to the same as the fitted end The current active subvector is formed by the field order and the standardized parameters of the saved fields with the same name. and All dimensions are And the components correspond one by one; when the regression result exceeds the minimum temperature of the effective candidate reference observation by 5 degrees Celsius or the maximum temperature by 5 degrees Celsius, it is truncated to the extended interval; when there are fewer than eight effective candidate reference observations, the current eight-field environment running vector is invalid, the active field set or model version fails to be read, the dimension of the active sub-vector is inconsistent, or the regression success flag is not, the regression result is not accepted, and the invalid layer and visible light rollback instruction are output according to the aforementioned state machine.
[0098] Temperature residual according to calculate, Indicates position The temperature residual, in degrees Celsius. Indicates the current specification component infrared image location. The measured radiation temperature, in degrees Celsius. Indicates the aforementioned current position The expected temperature is expressed in degrees Celsius and follows the complete definition of the immediately preceding formula. The residual of invalid temperature pixels remains invalid, and the positive temperature residual indicates that the temperature at the current location is higher than the expected temperature under the same environmental operating conditions.
[0099] Residual reliability according to calculate, Indicates position The residual confidence level is given, and the value ranges from zero to one. Indicates position The number of valid candidate reference observations, which is a non-negative integer. The dimensionless smoothing constant representing the saturation term of the number of effective candidates is determined to be 4 in this specific implementation based on the calibration results of the residual confidence and the absolute error of the residual estimation from historical verification observations. Indicates the aforementioned position Robust temperature discreteness It represents the decay scale of temperature discreteness, with units in degrees Celsius, and is calibrated to 3 degrees Celsius by verification observations. Indicates the current location of the data collection pair. The uncertainty of normalized registration This represents a function that takes the minimum value among its parameters. This represents a function that takes the maximum value among its parameters. The natural exponential function is defined, and the confidence level does not decrease with the increase of the number of valid candidates and does not increase with the increase of temperature discreteness or registration uncertainty.
[0100] The expected temperature, temperature residual, residual confidence, number of valid candidates, and invalid states are written into the expected temperature field, temperature residual map, residual confidence map, number of valid candidates map, and valid mask respectively according to the same 1024×512 standard grid. Each layer saves the component identifier, topology version, registration result package version, and regression model version. Step S4 reads the temperature residual features only within the valid mask.
[0101] The lower quartile of the residual confidence of the effective pixels of the component to be detected is used as the component residual confidence, and the upper quartile of the registration uncertainty of the effective pixels is used as the component registration uncertainty. In this specific embodiment, the number threshold is set to eight. The confidence threshold is determined by the Youden index of the receiver operating characteristic curve of the correct residual estimation in the validation sample and is fixed at 0.55. The uncertainty threshold is determined by the condition that the registration error in the validation sample does not exceed 1% of the short side of the component and is fixed at 0.45. The temperature evidence status is uniquely determined according to the priority order of registration anomaly, insufficient environmental data, insufficient reference observation, unavailable environmental regression, substandard component statistics and validity. If any of the preceding anomalies is hit, the subsequent status judgment will stop.
[0102] Only when the registration status in step S2 is valid, all eight current environmental running fields are valid, the number of valid candidate reference observations after iteration is not less than eight, the regression success flag is "yes", the component residual confidence level is not less than 0.55 and the component registration uncertainty is not greater than 0.45, the temperature residual result packet status is written as valid and is used for temperature residual feature encoding in step S4; if any condition is not met, the expected temperature field, temperature residual map and residual confidence level map are written as invalid placeholder values, the reading in step S4 is blocked with a full invalid mask, the cause category of the corresponding component is set as the category to be reviewed, the unique reason to be reviewed and the visible light backoff instruction are recorded, and zero value is prohibited when either of the two component statistics is invalid;
[0103] Step S3 outputs a temperature residual result package containing the desired temperature field, temperature residual map, residual confidence map, effective quantity map, location reference weight map, node-level attenuation factor map, component residual confidence, component registration uncertainty, regression success flag, temperature evidence status, and reference observation list. The location reference weight map is transformed by multiplying the registration uncertainty of the standard coordinate position of each historical observation with the attenuation factor of the corresponding unique node point by point. The node-level attenuation factor map is selected according to the selection rule of step S2, where only the attenuation factor of the uniquely selected node is written for each standard coordinate position of the current acquisition pair. When the registration uncertainty of the current or any historical uniquely selected node increases while the remaining quantities remain unchanged, the reference weight of the corresponding standard coordinate position decreases point by point or remains unchanged. Step S4 reads the temperature residual result package only in the same standard coordinate region where the status is valid, the regression success flag is yes, and the effective mask is true. The status to be verified is simultaneously passed to step S5 to trigger the candidate result path that does not depend on the temperature residual branch, thus forming a temperature residual processing data chain in a UAV photovoltaic hot spot cross-modal detection method.
[0104] In this specific embodiment, S4 includes:
[0105] Step S4 reads the visible light image and read-only topology version object from step S1, the cross-modal residual deformation field and registration uncertainty map from step S2, and the temperature residual map and residual confidence map from step S3. Only when the registration result package and the temperature residual result package are both valid, the visible light image is backsampled to the 1024×512 component specification coordinates according to the synthesis mapping saved in step S2 and the temperature residual branch is enabled. When any result package has a visible light backtracking instruction, the visible light image is directly normalized to the specification coordinates according to the component boundary, the temperature residual branch is disabled, and only the visible surface and component boundary branch is retained. The sampling results share the component identifier, specification coordinate area, and effective mask with the specification component infrared image.
[0106] The feature extraction module constructs three parallel branches based on a shared component boundary index. The visible surface feature branch reads the three-channel standard visible light map and uses a four-layer convolutional encoder to extract color, texture, and occlusion representations. The temperature residual feature branch reads the temperature residual map, residual confidence map, and effective mask and uses a three-layer convolutional encoder to extract local residual morphology. The component boundary feature branch encodes the component boundary distance map, cell row and column intersection heat map, and four adjacent edge raster map into boundary representations. Each branch outputs 128-channel features at the one-quarter, one-eighth, and one-sixteenth scales.
[0107] Temperature residual is pressed before entering the encoder. Normalization Indicates position The normalized temperature residuals take values ranging from -1 to 1. This indicates that step S3 is at position The output temperature residual is in degrees Celsius. This represents the normalized scale of the temperature residuals, with units in degrees Celsius, and is calibrated to 12 degrees Celsius based on the 95th quantile of the absolute value of the temperature residuals in the training set. This represents a function that takes the minimum value among its parameters. This represents a function that takes the maximum value among its parameters. The normalized temperature residual at invalid locations is set to zero and blocked from participating in attention by an effective mask.
[0108] In each feature pyramid layer or Generate a cross-modal attention mask, with the only allowed predicate being the query position. and bond position They must belong to the same component, and both must belong to the same cell or be adjacent cells connected by the four adjacent edges in step S1. If the conjunctive predicate is satisfied, the value is zero; otherwise, the value is negative infinity. Indicates the query position AND key position The attention mask values between them are dimensionless. This represents the query location index for visible surface or component boundary branches. The key position index representing the temperature residual branch is negative infinity, which is implemented before Softmax as the minimum value that can be represented by a floating-point type, so that the attention weights across components and non-adjacent cells within the same component are all zero.
[0109] The cross-modal attention module generates a query vector by summing visible surface features and component boundary features, and generates key and value vectors by temperature residual features. After adding the cross-modal attention mask, it performs scaled dot product attention according to the channel dimension. The attention output retains the component identifier, cell row and column position and feature pyramid layer number. When there is no allowed valid key position at the query position, the attention output is set to zero and only visible surface features and component boundary features are retained.
[0110] For each query location, the allowed temperature residual key locations are first strictly filtered according to the conjunctive predicate of the same component and the same cell or four adjacent cells. Then, only within this allowed set, the query vector and key vector are sorted from largest to smallest by scaling dot product and the top 16 candidates are retained. When the scores are the same, the one with the smallest Euclidean distance in normalized coordinates is selected. If the distance is still the same, the selection is made in ascending order of cell row number and column number. When there are fewer than 16 allowed candidates, all allowed candidates are used. Softmax also only normalizes the final allowed candidate set, so that the cross-modal correspondence selection has a definite candidate set, sorting key and parallel processing rules.
[0111] Attention weighting calculate, Indicates the query position Assigned to key position The attention weights are set and their values range from zero to one. Indicates the query position The 128-dimensional query vector is dimensionless after layer normalization. Indicates key position The 128-dimensional bond vector, after layer normalization, is dimensionless. Indicates the aforementioned query position AND key position Attention mask values between This represents the transpose of a vector. This represents the dimensionless channel dimension of the query vector and key vector. Indicates the index along the key position. A function that is normalized and allows the sum of candidate weights to be one, where the weights at positions with a mask of negative infinity are strictly zero;
[0112] Attention output generate, Indicates the query position The temperature residual attention output vector is dimensionless. Indicates the aforementioned query position Assigned to key position Attention weights Indicates key position The corresponding 128-dimensional temperature residual vector is dimensionless after layer normalization. Represents the index of all key positions that are allowed and valid for the mask. Summation is performed, and when the set of allowed keys is empty, the attention output vector is set to all zeros and a flag indicating no temperature is written into the fusion record;
[0113] Basic gating weights calculate, Indicates position The basic gating weights have values ranging from zero to one. This indicates the normalized registration uncertainty output in step S2 at the same standard coordinate position. This indicates the residual confidence level output in step S3 at the same standard coordinate position. When the position registration uncertainty increases or the residual confidence level decreases, the basic gating weight does not increase. The basic gating weight map, along with the effective mask, is saved independently, scaled to each feature pyramid layer by bilinear interpolation without interpolating across component boundaries, and used as the direct input for the basic confidence calibration in step S5.
[0114] Trainable content adjustment factor generate, Indicates position The trainable content adjustment factor has a value range of zero to one. This represents a gated network consisting of one pointwise convolutional layer and one fully connected layer, with the output being a dimensionless scalar. This represents the dimensionless parameter set consisting of all trainable weights and biases of the gating network, obtained by backpropagation of the total training loss (described later). Indicates the location of visible surface feature branches. Output the 128×1 dimensionless column vector obtained after layer normalization. Indicates the component boundary feature branch at location Output the 128×1 dimensionless column vector obtained after layer normalization. Indicates the location of the temperature residual characteristic branch. The 128×1 dimensionless column vector is output after the aforementioned mask attention processing and layer normalization. This represents the Sigmoid function; the input to the gated network explicitly does not include registration uncertainty. Node-level monotonic factor or basic gating weights Thus, the content adjustment factor does not change with the aforementioned uncertainty while keeping the three features the same;
[0115] Trainable final gating weights generate, Indicates position The trainable final gating weights have values ranging from zero to one. This represents the aforementioned basic gating weights. This indicates that the selection rules and their versions described in step S2 are strictly followed to determine the standard coordinate positions of the current acquisition pair. The uniquely selected cell node or module node index. This indicates that step S2 generates a node-level monotonic factor for the unique node and the current target acquisition, with a value range of zero to one and no dimension. The second subscript of the complete symbol is used in this step. This indicates the current pair to be detected, distinct from the second subscript in step S3. The first A historical reference observation, This refers to the aforementioned trainable content adjustment factor; because the content adjustment factor does not read the location registration uncertainty, node-level monotonicity factor, or basic gating weights, when the visible surface features, component boundary features, temperature residual features, and residual confidence remain unchanged, any increase in the location registration uncertainty or the registration uncertainty of the uniquely selected node will cause the final gating weights to decrease or remain unchanged. This product structure constitutes a hard monotonic constraint in the inference stage; the fusion loss, detection loss, regression loss, and gating monotonicity constraint loss are all obtained through... To parameters Backpropagation;
[0116] Cross-modal fusion features of each layer are as follows: generate, Indicates position The cross-modal fusion feature vector is dimensionless. This represents the visible surface feature vector, which, after layer normalization, is a 128×1 dimensionless column vector. This represents the component boundary feature vector, which, after layer normalization, is a 128×1 dimensionless column vector. Indicates the aforementioned position The trainable final gating weights, This represents the temperature residual feature vector after mask attention processing, which, after layer normalization, is a 128×1 dimensionless column vector. This represents the temperature branch channel projection matrix obtained during training, and is a 128×128 dimensionless real matrix with 128 input and 128 output channels. Left multiplication We obtain a 128×1 column vector and connect it to the array. , Add one channel at a time;
[0117] The fusion module upsamples the 1 / 16 scale features step by step and splices them with the 1 / 8 and 1 / 4 scale features according to the same component region. Then, it compresses them into 256 channels using 3×3 convolution. The convolution sampling on both sides of the boundary is truncated by the component boundary index to prevent the temperature residual of one component from propagating to the adjacent non-target component. For the region excluded by the effective mask, only the fusion features formed by the visible surface and the boundary branch are output.
[0118] Component-based filtering is performed on the fusion features according to the component boundary index. Only the effective area of the component and the feature positions within its cell cells are retained. When the component boundary index is found to be inconsistent with the topology version, the temperature residual contribution of the component is deleted and a version conflict flag is written. When the version is consistent and the proportion of effective positions is not less than 80%, the effective state in the fusion result package is updated. When the proportion is lower than this, the combined features of the visible surface and the boundary branch are selected as the output of the component.
[0119] The component identifier and grid in the shared topology version object continue to serve as the primary key of the fusion feature record. The four-adjacent edges of the battery cell are used to generate the attention mask for each layer. Each canonical coordinate position strictly follows the selection rule in step S2, reading only the node-level monotonic factor of the unique battery cell node or component node, and summarizing the basic gating weight, node-level monotonic factor, trainable content adjustment factor, and trainable final gating weight accordingly. The basic gating weight does not increase with the increase of position registration uncertainty, the node-level monotonic factor does not increase with the increase of registration uncertainty of the uniquely selected node, and the final gating weight is multiplied by the two and the content adjustment factor that does not read the uncertainty, thus maintaining the same hard monotonic direction during the inference phase.
[0120] Step S4 outputs multi-scale cross-modal fusion features, cross-modal attention masks corresponding to each scale, basic gating weight map, node-level monotonic factor map, trainable content adjustment factor map, trainable final gating weight map, component boundary index, and effective mask, and forms a fusion result package with component identifier, acquisition pair identifier, and topology version number; Step S5 reads the fusion features to perform hotspot detection, reads the unaggregated basic gating weight map to complete basic confidence calibration, and reads the node-level monotonic factor map and trainable content adjustment factor map to complete additional attenuation calibration. Each layer is named and saved separately without replacing each other.
[0121] In this specific embodiment, S5 includes:
[0122] Step S5 inputs the three-layer cross-modal fusion features from step S4 into a convolutional hotspot detection network with a feature pyramid. The network backbone performs convolutional residual blocks and converges the context at the one-quarter, one-eighth, and one-sixteenth scales. The segmentation branch outputs the pixel-by-pixel hotspot probability within the component's standard coordinates, the detection branch outputs the hotspot detection box, component identifier, and cell row and column positions, the regression branch outputs the abnormal temperature rise residual, and the classification branch outputs five types of probabilities: occlusion, dust accumulation, wiring abnormality, surface reflection, and pending verification. When the fusion result includes a visible light backoff instruction, the network only reads the combined features of the visible surface and component boundary. The regression branch does not output the abnormal temperature rise residual, and the classification branch forces the candidate cause category to be written as pending verification and saves the upstream status code.
[0123] The segmentation branch generates candidate hot spot masks with a probability threshold of 0.45 pixels and removes isolated regions with an area less than 0.01% of the area of their respective components. The detection branch performs non-maximum suppression on the candidate boxes with an intersection-union ratio of 0.50. The regression branch predicts the abnormal temperature rise residuals relative to the expected temperature field in step S3 within the candidate masks, in degrees Celsius. The classification branch stitches together visible surface features, hot spot aspect ratio, mask area ratio, boundary distance, and temperature residual distribution statistics to form the causal classification input.
[0124] Step S5 first reads the identity binding status, component structure status, and cell structure status from the topology version object. Only when all three are valid and the cell node set is not empty, the cell to which the candidate hot spot mask belongs is selected according to the overlap area between the candidate hot spot mask and the cell specification coordinate area. The cell with the largest overlap area ratio is selected and written to the row and column position. When the overlap area ratios are the same, they are sorted by the distance from the mask centroid to the cell center in ascending order. When the maximum overlap area ratio is less than 50%, the component identifier is retained and the cell row and column position is written as a cross-cell candidate. When the identity binding is pending verification, the component structure is pending verification, the cell structure is pending completion, or the cell node set is empty, the overlap sorting is not performed. The cell row and column position is written as uncertain, the cause category is written as pending verification, and the corresponding upstream status code is retained. Among them, uncertain and cross-cell candidate are mutually exclusive states.
[0125] Candidate boxes are filtered based on whether the center of the detection box is located within the effective area of the component. Candidate boxes whose center is outside the component are directly deleted. When the intersection-union ratio of the box and the hot spot mask is less than 0.30, the bounding rectangle of the mask is selected to update the detection box. If the updated box still crosses the boundary of two components, the cause category of the candidate is set to the category to be reviewed.
[0126] The residual of abnormal temperature rise is used as a regression branch to predict the output of candidate hot spot masks. As the sole final value, This represents the anomalous temperature rise residual predicted by the regression branch relative to the expected temperature field in step S3, expressed in degrees Celsius; for each candidate hotspot region First, select the pixels with valid temperature residuals from the pixels with normalized coordinates in step S3 and form a set of valid residual pixels. , and then The number of pixels divided by The number of pixels yields the effective temperature residual pixel ratio; when When the set is empty or the proportion is less than 70%, the candidate is directly set as pending review, and the median of the regional residuals is used. This is marked as invalid and the median, difference, and sign checks are skipped. Participate in consistency verification or classification; only when Only when it is not empty and the stated proportion is not less than 70%, then it is considered as such. Calculate the median of the residuals for the candidate region. This represents the median of the effective temperature residual in step S3 within the candidate hotspot region, expressed in degrees Celsius. This indicates that step S3 is at position The output temperature residual is in degrees Celsius. Represents a set The effective canonical coordinate pixel index within, This function represents the median of a non-empty set; the median is used only as a consistency check and classification input for regression predictions and does not replace the output of the regression branch.
[0127] Statistical analysis was performed on hot spot samples that were correctly detected in the validation set, had a non-empty set of effective residual pixels, and had an effective temperature residual pixel ratio of not less than 70%. And its 95th percentile is set as the consistency threshold, which is 3 degrees Celsius in this specific embodiment; for candidates that pass the aforementioned effective access control, when Greater than 3 degrees Celsius, or If a candidate is given by the visible surface branch that is not greater than zero, then that candidate is set to be reviewed, and no positive abnormal temperature rise residual is written to the result database, and the corresponding reason is recorded; for other candidates that pass the gate, only the regression branch prediction value is recorded. The abnormal temperature rise residual is written into the result database; the candidates that are short-circuited due to empty set or insufficient effective proportion are kept pending review and are not subject to this judgment.
[0128] The network outputs an initial confidence level for each hotspot candidate. , This represents the geometric mean of the probability of the detected branch target in the fused feature and the average probability of the segmented branch mask, and the value ranges from zero to one. Both probabilities are obtained from the Sigmoid output of the training network. When the detection box crosses the component boundary or the effective pixel ratio of the candidate mask is less than 70%, the candidate is marked as pending verification without supplementing the probability with cross-component pixels.
[0129] Candidates for hot spots within the same component are sorted from largest to smallest based on their initial confidence scores. If the cross-union ratio of the masks of adjacent candidates is not less than 0.50, the candidate with the largest initial confidence score is selected and the remaining duplicate candidates are deleted. If the initial confidence scores are the same, the candidate with the largest proportion of effective temperature residual pixels is selected. If both are still the same, the candidate with the smallest row number and column number in the normalized coordinates is selected.
[0130] The training samples are constructed from historical data collection pairs that have completed radiometric calibration and component standard coordinate registration. Each sample includes a standard visible light image, a standard component infrared image, a registration uncertainty image, a temperature residual image, a residual confidence image, a component boundary, cell row and column positions, hot spot mask, abnormal temperature rise residual, and cause category. The component boundary and cell position are double-checked and labeled. The hot spot mask and abnormal temperature rise are jointly labeled by infrared radiation temperature and on-site retest records. The cause category is determined by the closed-loop results of the inspection work order.
[0131] The samples were divided into a 70% training set, a 15% validation set, and a 15% test set based on the power station. Observations of the same component on different dates were only included in the same set. The training input was cropped to 512×512 pixels while keeping the component identification and boundary index consistent. The brightness perturbation was applied only to the visible light branch. The temperature offset perturbation was applied to both the current temperature and the expected temperature to keep the temperature residual unchanged. The geometric enhancement used the same transformation on all layers and annotations.
[0132] Total training loss by calculate, This represents the dimensionless total training loss. This represents the sum of the weighted binary cross-entropy and Dice loss of the hotspot mask, and is dimensionless. This indicates that the loss between the detection box classification and the generalized intersection and union ratio is dimensionless. This represents the smoothed absolute value loss of the abnormal temperature rise residual, which is dimensionless after normalization to 12 degrees Celsius. This represents the cross-entropy loss of the causative category, which is dimensionless. This represents the dimensionless loss due to the gated monotonicity constraint. , , , and These represent the corresponding loss weights, all of which are dimensionless. In this specific implementation, they are taken as 1.0, 1.0, 0.5, 0.5, and 0.2, respectively.
[0133] For each hotspot labeled region, the ratio of its pixel area to the effective pixel area of its component is calculated. The median of this area ratio in the training set is used as the small hotspot segmentation threshold. Labeled regions with an area ratio less than this median are given 2.0 times the weight in the segmentation loss, while the remaining labeled regions are given 1.0 times the weight. The region weights only apply to the corresponding hotspot pixels and are normalized according to the total weight of the effective pixels in the batch, so as to avoid the change in the number of small hotspots from changing the magnitude of the total loss.
[0134] hot spot area ratio according to calculate, This indicates the area of the hotspot marking region relative to the area of its component, and is dimensionless. This indicates the number of valid pixels in the hotspot annotation area, expressed in pixels. The median represents the number of pixels in the effective area of the component and is expressed in pixels. When the number of pixels in the effective area of the component is zero, the sample is deleted and not included in the training. The median is calculated only from the proportion of the effective area of the training set.
[0135] For surfaces with the same visible surface features and residual confidence, the registration uncertainty satisfies The training sample pairs, according to Apply auxiliary monotonicity constraints. This represents the dimensionless loss due to the gated monotonicity constraint. Indicates sample In position The registration is uncertain and dimensionless. Indicates sample In position The registration is uncertain and dimensionless. and This represents two uncertainty perturbation version indices constructed from the same base sample. Indicates sample Through parameters The trainable final gating weights output by the controlled gating network. Indicates sample The trainable final gating weights output by the same gating network, This indicates a monotonic interval that is dimensionless and takes the value 0.01. This represents a function that takes the maximum value among its parameters. The loss is only included in the batch mean when the uncertainty difference is not less than 0.05, and is applied to the parameters through two trainable final gating weights. Backpropagation; the training loss is used to enhance the numerical margin, and the monotonicity during inference is directly guaranteed by the product structure described in step S4, without replacing the hard monotonic structure with this soft loss.
[0136] Training uses the AdamW optimizer with a batch size of 8 and an initial learning rate of 0.0001, and is conducted for a maximum of 120 epochs. The training stops when the total loss on the validation set does not decrease for 12 consecutive epochs, and the model version with the largest component-level detection comprehensive index value on the validation set is retained. The detection comprehensive index is the harmonic mean of component-level precision and recall. The segmentation probability threshold, component residual confidence threshold, and component registration uncertainty threshold are determined only on the validation set. The test set is only used for the final performance verification after the model is frozen. The model file saves the training data version, the topology generation version, and the threshold configuration version.
[0137] Match the inference environment according to the model version, topology version and threshold configuration version. If any of the three versions are inconsistent, stop the temperature branch inference and select the visible surface and boundary branches to generate candidates to be reviewed. If the versions are consistent, update the inference session state and lock the current threshold configuration. Do not switch the model file before a single acquisition pair is processed.
[0138] For each candidate hot spot region First, basic gating weights are formed within the region. Effective base-gated pixel set and additional attenuation factor A valid set of additional attenuation pixels is required, and both must be verified to be non-empty before any mean or confidence formula is executed; if either set is empty, the calculation of the two region means is prohibited, and the base calibration and additional attenuation calibration formulas are skipped, directly applying the confidence score after base calibration. and the confidence level after final calibration Set both sets to zero and mark the candidates as pending review; only when both sets are not empty is the process performed according to the rules. Complete the basic confidence level calibration. This represents the confidence level of the hotspot after basic calibration, and its value ranges from zero to one. This represents the initial confidence level of the aforementioned network for the same hotspot candidate output. This represents the set of canonical coordinate pixels covered by the candidate hotspot mask. This indicates the basic gating weights that are stored independently in step S4. The arithmetic mean of the aforementioned non-empty gated pixel set, with values ranging from zero to one. This represents a function that takes the minimum value among its parameters. This indicates that the function takes the maximum value among its parameters; thus, the basic gating weights are directly retained and used as the gating weights in step S5, and the confidence level after basic calibration is no greater than the initial confidence level;
[0139] Under the condition that both valid pixel sets are not empty and basic calibration has been completed, then press Complete additional attenuation calibration. This represents the final calibrated hotspot confidence level, with a value ranging from zero to one. This indicates the confidence level of the hot spot after the aforementioned basic calibration. Indicates the additional attenuation factor The arithmetic mean of the aforementioned non-empty set of additional attenuated pixels, with a value ranging from zero to one. Indicates position Additional attenuation factor, and These represent the node-level monotonicity factor and the trainable content adjustment factor output in step S4, respectively; the final calibrated confidence level is no greater than the basic calibrated confidence level and no greater than the initial confidence level.
[0140] The cause classification branch uses the category corresponding to the maximum probability of the five categories as the initial cause category. When the maximum category probability is less than 0.60 or the difference between the probabilities of the first two categories is less than 0.10, the category is set to be reviewed. The final photovoltaic hot spot detection result record includes the collection of the tag identifier, component identifier, grid, cell row and column position, standard coordinate hot spot mask, original image coordinate detection box, abnormal temperature rise residual, cause category, category probability, initial confidence, post-calibration confidence, component residual confidence, component registration uncertainty, model version and topology version.
[0141] Before calculating component statistics, first determine the confidence level of the residuals within the component to be tested. Registration uncertainty The intersection of canonical coordinate pixels that are both valid and have consistent component identifiers is defined as... And check that it is not empty; when When empty, skip the two quantile formulas and apply the component residual confidence level. and component registration uncertainty All values should be written as invalid, substitution with zero is prohibited, and the rollback path to be reviewed and the visible surface and component boundary should be directly selected; only when Press only when not empty. and Calculate component statistics. This represents the lower quartile of the residual confidence value within the component to be tested and is used as the component residual confidence level; it is dimensionless. This represents the upper quartile of the registration uncertainty value within the component to be detected and is used as the component registration uncertainty. It is dimensionless. This indicates that step S3 is at position The reliability of the output residuals. This indicates that step S2 is at position The uncertainty of normalized registration of the output. Represents the intersection of non-empty sets The canonical coordinate pixel position index within, The function that takes the quartiles of a non-empty set. The function represents the upper quartile of a non-empty set. Indicates the component to be detected The aforementioned two layers both have valid normalized coordinate pixel intersections. Indicates the component index;
[0142] The validity of temperature evidence is screened based on the number of valid candidate reference observations, the confidence level of component residuals, and the uncertainty of component registration. When all three conditions are met, the cross-modal detection results are selected and written into the current version. When any one condition is not met, the detection branch results formed by visible surface features and component boundary features are selected as hot spot candidate results, and the cause category of the corresponding component is set as the category to be reviewed.
[0143] When the number of valid candidate reference observations given in step S3 is less than eight, the cause category of the corresponding component is set to the category to be reviewed. When the component residual confidence is less than 0.55, the temperature residual is prohibited from participating in the cause determination. When the component registration uncertainty is greater than 0.45, the post-calibration confidence is limited to no more than the visible surface branch confidence.
[0144] When the number of valid candidate reference observations for subsequent acquisition pairs is not less than eight, a two-stage validity review is initiated: In the first stage, the current component residual confidence level that has not yet been generated is not read. The scheduling module triggers step S3 recalculation with the same component identifier, the current step S2 registration result package, and the newly added reference observation. Step S3 generates a temperature residual result package with the current model version, component temperature evidence status, component residual confidence level, and component registration uncertainty obtained by aggregating from the current step S2 result package. In the second stage, steps S4 and S5 are only executed when the current component temperature evidence status is valid, the current component residual confidence level is not less than 0.55, and the current component registration uncertainty is not greater than 0.45. The photovoltaic hot spot detection results are updated according to the acquisition time. If the first stage outputs an invalid status or the second stage threshold is not met simultaneously, steps S4 and S5 are not initiated. The original version to be reviewed is retained, and the current temperature evidence status and the reason for the threshold failure are recorded. It is prohibited to use the old component residual confidence level in the original version to be reviewed as a precondition for triggering the recalculation of step S3.
[0145] The results library matches historical results according to component identifier and collection pair identifier, selects the result with the latest collection time and the model version that has been validated as the current result, updates the confidence level and abnormal temperature rise residual when the new result obtained by re-execution is in the same category as the old result, and retains two versions and writes the category change flag when the categories are different.
[0146] Step S5 outputs the photovoltaic hot spot detection results of a UAV photovoltaic hot spot cross-modal detection method. The results, together with the standard coordinate region, topology version, registration result package version, temperature residual result package version and model version, are written into the inspection result database. Old results retain the version number, reason for pending review and update time for traceability.
[0147] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
[0148] This invention integrates component topology relationships, cross-modal registration uncertainties, environmental correction temperature residuals, and gated fusion into a single processing chain, enabling visible light structural information and infrared anomalous temperature rise information to participate in detection within a unified component specification coordinate system. The reuse of the component topology map in registration, reference observation selection, and attention neighborhood ensures consistent spatial relationships between components and cells at each stage.
[0149] This invention also improves the stability of hot spot detection results under complex environments and changing operating conditions by reducing the weight of abnormal anchor points, forming uncertainty based on multiple types of registration error evidence, weighting historical reference observations with this uncertainty and gating temperature residual characteristics, and finally using gating weights to calibrate hot spot confidence. This restricts the continued propagation of registration errors from the processing structure.
Claims
1. A method for cross-modal detection of photovoltaic hot spots using an unmanned aerial vehicle (UAV), characterized in that, include: S1. Acquire visible light images, infrared thermal images from radiation thermometry, camera parameters, pose parameters, environmental parameters, and photovoltaic module operating parameters generated by the UAV inspection. Perform module instance segmentation and line segment intersection detection on the visible light images to obtain module boundaries, module corners, and cell row and column intersections. Using modules and cells as nodes, and using the same string relationship, module adjacency relationship, and cell four-adjacency relationship as edges, and recording the sequential position of the module in the string as a grid, form the module topology graph. S2. Determine the initial geometric transformation based on camera parameters and pose parameters. Utilize the node and edge constraints of the component topology graph. Within the search area defined by the initial geometric transformation, select component corner points and cell row / column intersections as candidate topology nodes. Determine the node weights based on the correlation between visible light boundary features and infrared gradient features. Multiply the node weights of candidate topology nodes located in infrared temperature local extreme regions, visible light reflection regions, or motion-blocked regions by a preset attenuation coefficient less than one. Calculate the cross-modal residual deformation field under the joint constraints of node correspondence error and relative position error between adjacent nodes. Transform the radiometric infrared thermogram to the component's standard coordinates to generate a standard component infrared image and a registration uncertainty map. Use the cross-modal residual deformation field to perform a forward mapping from the visible light image to the radiometric infrared thermogram and a reverse mapping from the radiometric infrared thermogram to the visible light image to obtain a bidirectional closed-loop error. Calculate the normalized distribution entropy of the candidate topology node correlation within the search area. The residual deformation field is obtained at at least two spatial scales and the deformation vector divergence at the corresponding positions is calculated. The bidirectional closed-loop error, the normalized distribution entropy and the deformation vector divergence are normalized and then weighted and synthesized so that the synthesized value does not decrease as any input increases, thereby obtaining the registration uncertainty value of the standard coordinate position of each component and forming the registration uncertainty map. S3. Select reference observations and estimate the desired temperature field based on the standard component infrared image, registration uncertainty map, environmental parameters, photovoltaic module operating parameters, and module topology map, and generate a temperature residual map and a residual confidence map; select observations with standard component infrared images from multiple historical acquisition pairs of the component under test in the module topology map as candidate reference observations based on the string identifier and grid of the component under test in the preset acquisition time window. The median of the temperature at the corresponding location of the candidate reference observation is used as the central temperature quantity, and the median absolute deviation of the temperature at the corresponding location of the candidate reference observation relative to the central temperature quantity is used as the temperature dispersion quantity. The registration uncertainty values in the registration uncertainty map of each candidate reference observation are normalized to obtain normalized registration uncertainty values in the range of zero to one. The difference between the infrared image of the specification component and the desired temperature field is used as the temperature residual. The effective number of candidate reference observations, the temperature dispersion quantity, and the normalized registration uncertainty value are combined and mapped. The combined mapping result is used as the residual confidence, and the residual confidence is ensured to remain unchanged as the effective number of candidate reference observations increases and to remain unchanged as the temperature dispersion quantity or the normalized registration uncertainty value increases. S4. Extract visible surface features, temperature residual features, and module boundary features. Establish cross-modal correspondence only for features of the same module or adjacent cells. Gated temperature residual features based on registration uncertainty map and residual confidence map to obtain cross-modal fusion features and gating weights. S5. Input the cross-modal fusion features into the hot spot detection network to obtain the hot spot location, abnormal temperature rise residual, cause category and initial confidence level. Calibrate the initial confidence level according to the gating weight and output the photovoltaic hot spot detection results.
2. The method for cross-modal detection of photovoltaic hot spots by unmanned aerial vehicles according to claim 1, characterized in that, S1 includes: The visible light image and the infrared thermographic image are established as acquisition pairs according to the timestamp; the irradiance, ambient temperature, wind speed, cloud cover change, string current, load, solar incidence angle and acquisition time difference are written into the multimodal inspection data as the environmental quantity and the photovoltaic module operation quantity.
3. The method for cross-modal detection of photovoltaic hot spots by unmanned aerial vehicles according to claim 2, characterized in that, S2 includes: The initial geometric transformation is determined by camera intrinsic parameters, camera extrinsic parameters, UAV pose, and shooting altitude.
4. The method for cross-modal detection of photovoltaic hot spots by unmanned aerial vehicles according to claim 3, characterized in that, S3 includes: performing regression correction on the location temperature center quantity based on irradiance, ambient temperature, wind speed, cloud cover variation, string current, load, solar incidence angle, and acquisition time difference to obtain the desired temperature field; using one minus the normalized registration uncertainty value as a reference weight, weighting the candidate reference observations, and iteratively eliminating candidate reference observations whose temperature deviation exceeds the elimination threshold determined by the temperature discrete quantity.
5. The method for cross-modal detection of photovoltaic hot spots by unmanned aerial vehicles according to claim 4, characterized in that, S4 includes: The visible light image is transformed to the component's canonical coordinates, and a visible surface feature branch, a temperature residual feature branch, and a component boundary feature branch are constructed using the shared component boundary index; in Each feature pyramid layer generates a cross-modal attention mask based on the component identifier and the adjacent edge of the cell, so that the attention calculation range is limited to the same component or adjacent cells. The residual confidence in the residual confidence map is normalized to obtain a normalized residual confidence value in the range of zero to one. The product of one minus the normalized registration uncertainty value and the normalized residual confidence value is used as the basic gating weight. The contribution of the temperature residual feature to the cross-modal fusion feature is adjusted by the basic gating weight, and the basic gating weight is retained as the gating weight in step S5.
6. The method for cross-modal detection of photovoltaic hot spots by unmanned aerial vehicles according to claim 5, characterized in that, S5 include: The segmentation branch of the hot spot detection network outputs the hot spot mask within the cell area, the detection branch outputs the hot spot detection frame, component identification, and cell row and column positions, the regression branch outputs the abnormal temperature rise residual, and the classification branch outputs the categories of shading, dust accumulation, wiring abnormality, surface reflection, or pending verification and their probabilities based on visible surface features, hot spot spatial morphology, and temperature residual distribution. The mean of the gating weights within the hot spot area is multiplied by the initial confidence level, and the resulting product is limited to the interval between zero and one, which is used as the calibrated confidence level.
7. The method for cross-modal detection of photovoltaic hot spots by unmanned aerial vehicles according to claim 6, characterized in that, The component identifier, grid, and adjacent edge of the same component topology graph established in step S1 are used for the node correspondence constraint and edge relationship constraint in step S2, the candidate reference observation selection in step S3, and the cross-modal attention mask generation in step S4, respectively. The component topology node is mapped to the component canonical coordinate region using the cross-modal residual deformation field. The mean value of the registration uncertainty value within the component canonical coordinate region is used as the registration uncertainty value of the component topology node. The reference position in step S3 and the temperature residual feature position in step S4 are both based on the component canonical coordinate region. When the registration uncertainty value of the component topology node increases, the reference weight of the corresponding reference position in step S3 decreases, the gating weight of the corresponding temperature residual feature in step S4 decreases, and the post-calibration confidence of the hot spot region covering the component canonical coordinate region in step S5 is not greater than the initial confidence value.
8. The method for cross-modal detection of photovoltaic hot spots by unmanned aerial vehicles according to claim 6, characterized in that, The training of the hot spot detection network includes: constructing a supervision signal using training samples labeled with component boundaries, cell row and column positions, hot spot masks, abnormal temperature rise residuals, and cause categories; calculating the area ratio of the hot spot labeled region to its corresponding component region, such that the segmentation loss weight of the hot spot labeled region is less than the median area ratio of the training samples is a preset multiple of the segmentation loss weight of the other hot spot labeled regions, wherein the preset multiple is greater than one; and adding the registration uncertainty map and the residual confidence map to the training samples. For surfaces with the same visible surface features and residual confidence, the registration uncertainty satisfies The training sample pairs, according to Apply auxiliary monotonicity constraints. This represents the dimensionless loss due to the gated monotonicity constraint. Indicates sample In position The registration is uncertain and dimensionless. Indicates sample In position The registration is uncertain and dimensionless. and This represents two uncertainty perturbation version indices constructed from the same base sample. Indicates sample Through parameters The trainable final gating weights output by the controlled gating network. Indicates sample The trainable final gating weights output by the same gating network, This indicates a monotonic interval that is dimensionless and takes the value 0.
01. This represents a function that takes the maximum value among its parameters. The loss is only included in the batch mean when the uncertainty difference is not less than 0.05, and is applied to the parameters through two trainable final gating weights. Backpropagation.
9. The method for cross-modal detection of photovoltaic hot spots by unmanned aerial vehicles according to claim 6, characterized in that, The lower quartile of the residual confidence value within the component to be detected is taken as the component residual confidence value, and the upper quartile of the registration uncertainty value within the component to be detected is taken as the component registration uncertainty value. When the number of valid candidate reference observations of the component to be detected is less than a preset number threshold, the component residual confidence value is less than the confidence threshold determined by the verification sample, or the component registration uncertainty is greater than the uncertainty threshold determined by the verification sample, the cause category of the corresponding component is set as the category to be verified, and the detection branch result formed by visible surface features and component boundary features is taken as the hot spot candidate result. When the number of valid candidate reference observations of the subsequent acquisition pair is not less than the number threshold, the component residual confidence value is not less than the confidence threshold, and the component registration uncertainty is not greater than the uncertainty threshold, steps S3 to S5 are re-executed and the photovoltaic hot spot detection result is updated.
Citation Information
Patent Citations
Photovoltaic module hot spot detection system and method
CN120433720A
Visible light-infrared bimodal image registration method based on deep learning
CN120525935A
