New energy equipment intelligent inspection method based on laser radar and infrared imaging fusion

Abstract

The invention relates to the technical field of new energy equipment intelligent detection, in particular to a new energy equipment intelligent inspection method based on laser radar and infrared imaging fusion, and the method comprises the steps: building a unified space-time coordinate system, and completing the external parameter calibration and radiometric calibration of two sensors. Through point cloud registration, voxel division and infrared data back projection, a hot geometric voxel graph is generated in combination with weighted average of multi-view projection weights. And dynamically correcting the voxel temperature by adopting a joint compensation model, and constructing a super voxel graph structure after extracting fusion features. And performing feature weighted fusion and abnormal probability calculation by using a graph attention network, and generating a time sequence change graph in combination with historical baseline data to identify an abnormal type. And finally, the severity is calculated through the anomaly probability, the temperature trend and the neighborhood difference, the anomaly position, the class, the grade, the reinspection track and the inspection period suggestion are output, and accurate anomaly detection and intelligent inspection decision making are achieved.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection technology for new energy equipment, specifically to an intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging. Background Technology

[0002] With the large-scale deployment of new energy power generation devices, wind turbines, photovoltaic modules, and energy storage devices operate under complex environments such as high temperature, high humidity, sandstorms, and salt spray for extended periods. This makes them prone to latent defects such as structural fatigue, loose connections, poor conductivity, and insulation aging. These defects are often difficult to detect visually in their early stages, but they can cause energy loss, heat buildup, and equipment failure, and in severe cases, even shutdowns and safety accidents. Therefore, how to conduct high-precision, non-contact inspections of the operating status of new energy equipment has become a key issue in the field of equipment maintenance.

[0003] Current inspection technologies mainly rely on manual infrared detection, visible light image recognition, or single-sensor lidar scanning. Manual infrared inspection is limited by operational experience, resulting in low detection accuracy and poor repeatability; visible light image recognition is easily affected by illumination, dust, and reflection, and is difficult to maintain stability in strong radiation environments; while lidar can provide accurate three-dimensional structural information, it cannot reflect the distribution characteristics of thermal anomalies. Some studies have attempted to post-process and register infrared thermal images with point cloud data, but problems such as large calibration errors between sensors, inconsistent viewing angles, severe temperature drift, and difficulty in compensating for environmental interference exist, making it difficult to meet the inspection requirements of new energy equipment in terms of both spatial and thermal accuracy of the detection results.

[0004] Against this backdrop, existing technologies for the inspection of new energy equipment still suffer from major problems such as the difficulty in synchronously integrating thermal and geometric information, the significant impact of environmental factors on thermal measurement results, and insufficient accuracy in anomaly location. There is an urgent need for an intelligent inspection method that can achieve accurate fusion of multi-source sensor data, has environmental adaptive compensation capabilities, and supports three-dimensional thermal anomaly identification, in order to improve the accuracy and reliability of new energy equipment condition detection. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide an intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging, which addresses the shortcomings of the prior art. This method solves the technical problems in the inspection of new energy equipment, such as the difficulty in synchronously fusing thermal and geometric information, the significant impact of environmental factors on thermal measurement results, and the insufficient accuracy of anomaly positioning.

[0006] The objective of this invention is achieved through the following technical solutions: In a first aspect, the present invention provides an intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging, comprising: The system utilizes several lidar sensors to acquire spatiotemporally aligned point cloud data of the device under inspection, and an infrared thermal imager to acquire infrared thermal imaging data of the device under inspection. The lidar sensors and the infrared thermal imager are calibrated by external parameters and radiometric calibration before acquiring the data. The system also acquires environmental monitoring data and the pose information of the device under inspection. The point cloud data is divided into voxels to obtain a voxel space. The radiometrically calibrated infrared thermal image data is back-projected onto the voxel space to obtain the temperature attributes of each voxel in the voxel space. Weights are assigned to the projection differences from different viewpoints in voxel space, and the temperature attributes are weighted and averaged based on the weights to generate a thermal geometry voxel map. Based on environmental monitoring data and the pose information of the tested equipment, the temperature attributes in the thermal geometry pixel map are dynamically corrected using a joint compensation model, and fusion features are extracted from the corrected thermal geometry pixel map. A graph attention network is used to perform weighted fusion and anomaly probability calculation on the fused features, and a time-series change map is generated by combining historical baseline voxel data. The thermal anomaly areas and their anomaly types on the surface of the inspected equipment are identified based on the time-series change map. The severity of the anomaly is calculated based on the thermal anomaly area and its anomaly type on the surface of the inspected equipment, and a re-inspection decision and maintenance suggestion are generated based on the severity of the anomaly.

[0007] As a further improvement of the present invention, external parameter calibration and radiometric calibration are performed on the lidar and infrared thermal imager, including: External parameter calibration and radiometric calibration of lidar and infrared thermal imager, including: The external parameter calibration is achieved through a thermal geometric dual-visible target, which includes an aluminum-based checkerboard grid and a blackbody film with a set emissivity, used to locally heat the device under test to form a hot corner area. The radiation calibration was achieved using a two-point blackbody source.

[0008] As a further improvement of the present invention, the lidar includes a main lidar and an auxiliary lidar; The main lidar is used to scan the overall structure of the new energy equipment; the auxiliary lidar is used for scanning local details; the main lidar, auxiliary lidar and infrared thermal imager are synchronized in time through a unified trigger signal.

[0009] As a further improvement of the present invention, weights are set according to the projection differences from different viewpoints, and a weighted average is performed on the temperature samples, including:

[0010] In the formula, This represents the final weighted average temperature of the voxels corresponding to the device under test in voxel space. For the i-th perspective on the th The weighting coefficients of each target voxel. The temperature sample value is the value back-projected onto the target voxel by the infrared thermal imager from the i-th viewpoint.

[0011] As a further improvement of the present invention, the thermal geometry voxel diagram includes average temperature, maximum temperature, temperature variance, neighborhood temperature difference, curvature, and reflection intensity.

[0012] As a further improvement of the present invention, dynamic correction is performed on the infrared thermal image data corresponding to voxels in the thermal geometry voxel map based on a joint compensation model, including: Based on environmental monitoring data and the position and orientation information of the inspected equipment, a joint compensation model is used to dynamically correct the temperature attributes in the thermal geometry graph, including: The joint compensation model includes radiation compensation, convection compensation, and self-shading compensation. The radiation compensation is used to determine the radiation reflection compensation amount and reflection suppression amount based on the voxel surface normal angle of solar irradiance and the pose information of the tested equipment in the environmental monitoring data. The convection compensation is calculated based on wind speed and ambient temperature from environmental monitoring data and an empirical convection heat transfer model. Based on point cloud visibility analysis, a self-occlusion compensation weight attenuation is applied to voxels that are occluded by other parts. The self-occlusion compensation is based on the corrected thermal geometry voxel map, and the temperature values ​​of the occluded voxels are weighted down or removed according to their reliability.

[0013] As a further improvement of the present invention, the time-series variation map is aligned with the thermal geometry voxel map and the historical thermal geometry voxel map, and the temperature change trend and time drift information of the voxels are calculated based on the aligned voxel map to realize the analysis of abnormal evolution trends.

[0014] As a further improvement of the present invention, the severity of the anomaly is calculated based on the thermal anomaly area and corresponding anomaly type on the surface of the inspected equipment, and a re-inspection decision and maintenance suggestion are generated based on the severity of the anomaly, including: Based on the temporal variation map, the temperature drift and geometric deformation trends in the thermal anomaly area on the surface of the inspected equipment are identified; the anomaly type is determined based on the relationship between the temperature gradient direction and curvature change in the temperature drift and geometric deformation trends; the severity score is calculated based on the anomaly type and anomaly node; based on the severity score, the anomaly location, category and level are output, and a re-inspection track and inspection cycle suggestion are generated.

[0015] As a further improvement of the present invention, the severity score include:

[0016] In the formula, For neighborhood temperature difference, For the rate of temperature change, To accumulate temperature difference, For curvature, Reflection intensity, For each weight coefficient, For the standardized function, , , , and These are the weights.

[0017] As a further improvement of the present invention, the anomaly types include: hot spot anomaly, crack thermal coupling anomaly, and poor contact anomaly.

[0018] Secondly, the present invention provides an intelligent inspection system for new energy equipment based on the fusion of lidar and infrared imaging, comprising: The data acquisition module uses several lidars to acquire point cloud data and an infrared thermal imager to acquire infrared thermal image data. The lidars and infrared thermal imager are calibrated by external parameters and radiation before acquiring data. It also acquires environmental monitoring data and the pose information of the device under test. The thermal geometry voxel construction module is used to divide the point cloud data into voxels to obtain a voxel space, back-project the radiometrically calibrated infrared thermal image data into the voxel space to obtain the temperature attributes of each voxel in the voxel space, and set weights for the projection differences of different viewpoints in the voxel space. Based on the weights, the temperature attributes are weighted and averaged to generate a thermal geometry voxel map. The environmental compensation and feature extraction module is used to dynamically correct the temperature attributes in the thermal geometry pixel map based on environmental monitoring data and the pose information of the tested equipment, and extract fusion features from the corrected thermal geometry pixel map. The multimodal feature fusion and anomaly detection module is used to perform weighted fusion and anomaly probability calculation on the fused features using a graph attention network, and generate a time-series change map by combining historical baseline voxel data. Based on the time-series change map, the thermal anomaly region and its anomaly type on the surface of the inspected equipment are identified. The time-series change analysis and re-inspection decision module is used to calculate the severity of the anomaly based on the thermal anomaly area and its anomaly type on the surface of the inspected equipment, and to generate re-inspection decisions and maintenance suggestions based on the severity of the anomaly.

[0019] As a further improvement of the present invention, the data acquisition module includes: The main lidar unit is used to generate cloud data of the entire field of view of the new energy equipment, covering the overall structural outline of the inspected equipment; A secondary lidar unit is used to perform fine scanning of a local area of ​​the device under inspection, obtain local detail point clouds, and supplement the set curvature parts and occluded areas. Infrared thermal imaging unit is used to acquire thermal radiation images and record temperature field information; An environmental monitoring unit is used to acquire environmental monitoring data, including wind speed, ambient temperature, and solar irradiance. The positioning and orientation unit is used to acquire the position and orientation information of the device under test.

[0020] As a further improvement of the present invention, the time series change analysis and re-examination decision module includes: Based on the temporal variation map, the temperature drift and geometric deformation trends in the thermal anomaly area on the surface of the inspected equipment are identified; the anomaly type is determined based on the relationship between the temperature gradient direction and curvature change in the temperature drift and geometric deformation trends; the severity score is calculated based on the anomaly type and anomaly node; based on the severity score, the anomaly location, category and level are output, and a re-inspection track and inspection cycle suggestion are generated. The severity score includes:

[0021] In the formula, For neighborhood temperature difference, For the rate of temperature change, To accumulate temperature difference, For curvature, Reflection intensity, For each weight coefficient, For the standardized function, , , , and These are the weights.

[0022] Thirdly, the present invention provides a computer-readable storage medium for storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform the above-described intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging.

[0023] Fourthly, the present invention provides a computing device, comprising: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include steps for performing the above-described intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging.

[0024] The beneficial effects of this invention are as follows: This invention provides an intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging. Through external parameter calibration and radiometric calibration, it achieves spatiotemporal alignment and radiometric consistency between lidar point cloud data and infrared thermal image data, ensuring the physical reliability of multi-source data fusion. After dividing the point cloud into voxels, it back-projects the infrared thermal image data into voxel space, assigning temperature attributes to each voxel, thus completing the direct mapping of the three-dimensional spatial temperature field. It sets weights for multi-view projection differences and performs weighted averaging, reducing the interference of viewpoint occlusion and measurement noise on temperature attributes and improving the signal-to-noise ratio of the thermal geometry voxel map. Based on environmental monitoring data and equipment... Pose information is dynamically corrected for temperature attributes through a joint compensation model, eliminating radiation errors in thermal imaging caused by ambient temperature and humidity and equipment motion status, thus enhancing the dynamic accuracy of temperature attributes. A graph attention network is used to weight and fuse the fusion features and calculate the anomaly probability. Combined with historical baseline voxel data, a time-series change map is generated to achieve spatial localization and time-series evolution analysis of thermal anomaly areas and types. Finally, based on the severity of the anomaly, re-inspection decisions and maintenance suggestions are generated. The synergistic effect of various technical features achieves a leapfrog improvement in thermal anomaly detection from static localization to dynamic time-series analysis, and from single judgment to multi-dimensional decision-making, significantly improving the accuracy of equipment thermal defect detection and maintenance efficiency. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating an intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a module of an intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging according to an embodiment of the present invention; Figure 3 This is an internal structural diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives and technical solutions of this invention clearer and easier to understand, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. The specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0028] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings and specific embodiments. The described embodiments are only some embodiments of the present invention, and not all embodiments.

[0029] Example 1 This embodiment provides an intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging, including the following steps: Several lidar sensors are used to acquire spatiotemporally aligned point cloud data of the equipment under inspection (EDI), and an infrared thermal imager is used to acquire infrared thermal imaging data of the EDI. The lidar sensors and the infrared thermal imager undergo extrinsic parameter calibration and radiometric calibration before data acquisition. Environmental monitoring data and the EDI's pose information are also acquired. The point cloud data is divided into voxels to obtain a voxel space. The radiometrically calibrated infrared thermal imaging data is back-projected into the voxel space to obtain the temperature attributes of each voxel. Weights are assigned to the projection differences from different viewpoints in the voxel space, and a weighted average of the temperature attributes is performed based on these weights to generate thermal geometry. Based on environmental monitoring data and the pose information of the inspected equipment, a joint compensation model is used to dynamically correct the temperature attributes in the thermal geometry voxel map, and fusion features are extracted from the corrected thermal geometry voxel map. A graph attention network is used to perform weighted fusion and anomaly probability calculation on the fusion features, and a time-series change map is generated by combining historical baseline voxel data. The thermal anomaly areas and their anomaly types on the surface of the inspected equipment are identified based on the time-series change map. The severity of the anomaly is calculated based on the thermal anomaly areas and their anomaly types on the surface of the inspected equipment, and a re-inspection decision and maintenance suggestion are generated based on the severity of the anomaly.

[0030] The working principle of this embodiment is as follows: Spatiotemporal alignment and radiometric consistency of lidar point cloud data and infrared thermal imaging data are achieved through extrinsic parameter calibration and radiometric calibration, ensuring the physical reliability of multi-source data fusion; after dividing the point cloud into voxels, the infrared thermal imaging data is back-projected into voxel space, assigning temperature attributes to each voxel, completing the direct mapping of the three-dimensional spatial temperature field; weights are set for multi-view projection differences and weighted averaging is performed to reduce the interference of viewpoint occlusion and measurement noise on temperature attributes, improving the signal-to-noise ratio of the thermal geometry voxel map; based on environmental monitoring data and equipment pose information, temperature attributes are dynamically corrected through a joint compensation model, eliminating the radiometric errors of environmental temperature and humidity and equipment motion state on thermal imaging, enhancing the dynamic accuracy of temperature attributes; a graph attention network is used to weightedly fuse the fusion features and calculate the anomaly probability, combined with historical baseline voxel data to generate a temporal change map, realizing the spatial location and temporal evolution analysis of thermal anomaly areas and types; finally, re-inspection decisions and maintenance suggestions are generated based on the severity of the anomalies.

[0031] In this embodiment, sensor calibration is completed before inspection. A dual-visible thermal geometric target is used to calibrate the external parameters of the lidar and the infrared thermal imager. This target includes an aluminum-based checkerboard pattern (for the lidar to identify geometric corners) and a blackbody film with a set emissivity (for the infrared thermal imager to identify hot corners). By locally heating the inspected equipment, a clear hot corner area is formed, achieving precise alignment of their spatial coordinate systems. Subsequently, a dual-point blackbody source is used to perform radiometric calibration on the infrared thermal imager to establish an accurate correspondence between radiation signals and temperature values.

[0032] The data acquisition system includes a main lidar, an auxiliary lidar, an infrared thermal imager, an environmental monitoring unit (including an anemometer, ambient temperature sensor, and solar irradiance sensor), and a positioning and orientation unit (such as GNSS / IMU). During inspection, the main lidar acquires the overall structural point cloud of new energy equipment (such as photovoltaic panels and wind turbine blades); the auxiliary lidar performs detailed scanning of key local details of the equipment (such as junction boxes, bolt connections, high curvature, or easily obstructed areas). The main / auxiliary lidar and the infrared thermal imager achieve time-synchronized acquisition through a unified hardware trigger signal, ensuring that each frame of point cloud data and infrared thermal image data is strictly aligned in time. Simultaneously, the environmental monitoring unit records wind speed, ambient temperature, and solar irradiance data in real time at the time of acquisition, and the positioning and orientation unit provides the absolute / relative pose information of the inspected equipment.

[0033] Voxel partitioning was performed on point cloud data acquired from multiple perspectives and spatiotemporally aligned to establish a unified 3D voxel space. Radiometrically calibrated infrared thermal images were then back-projected into this voxel space based on the calibrated extrinsic parameter matrix. Since a voxel may be observed from multiple infrared images, differences in projected temperature samples may exist at different perspectives due to factors such as occlusion, distance, and incident angle.

[0034] To obtain more reliable voxel temperature properties, this invention performs a weighted average of projection samples from different viewpoints, including:

[0035] In the formula, This represents the final weighted average temperature of the voxels corresponding to the device under test in voxel space. For the i-th perspective on the th The weighting coefficient of each target voxel can be set comprehensively based on factors such as the angle between the projection ray and the voxel surface normal, the observation distance at this viewpoint, and the image signal-to-noise ratio. The smaller the angle, the closer the distance, and the higher the signal-to-noise ratio, the greater the weight. The temperature sample value is the value back-projected onto the target voxel by the infrared thermal imager from the i-th viewpoint.

[0036] Through the above process, each effective voxel in the voxel space is assigned a temperature attribute. The geometric information provided by the lidar (such as voxel position, surface normal, curvature, and reflection intensity) is fused with the fused temperature information to generate a thermal geometry voxel map. This map includes the following attributes: average temperature, maximum temperature, temperature variance (reflecting temperature stability), neighborhood temperature difference (reflecting local thermal gradient), curvature (reflecting surface geometric features), and reflection intensity (reflecting material properties).

[0037] Environmental factors (such as solar radiation and wind cooling) and the device's own geometric obstruction can affect the measured value of the voxel apparent temperature. Therefore, this embodiment introduces a joint compensation model to dynamically correct the temperature attributes in the thermal geometry voxel map. This model mainly includes: Radiation compensation is used to determine the amount of radiation reflection compensation and reflection suppression based on the voxel surface normal angle of solar irradiance and the pose information of the tested equipment in environmental monitoring data. Convection compensation is calculated based on wind speed and ambient temperature from environmental monitoring data and an empirical convection heat transfer model. Self-occlusion compensation weight attenuation is applied to voxels that are occluded by other parts based on point cloud visibility analysis. Self-occlusion compensation involves reducing the confidence weight or removing the temperature values ​​of occluded voxels based on the corrected thermal geometry voxel map.

[0038] The extracted fused features are input into a Graph Attention Network (GAT). In this network, each voxel is treated as a node in the graph, its features are treated as node features, and the spatial adjacency relationships between voxels are treated as edges. GAT learns the importance weights between the features of adjacent voxels through an attention mechanism, achieving weighted fusion of features and thus enhancing the ability to perceive features of abnormal regions. The network ultimately outputs the probability that each voxel belongs to an abnormal state.

[0039] To identify the types of anomalies (such as dirt, obstruction, electrical connection failures, structural cracks, etc.) and analyze their evolution trends, this invention introduces time-series analysis. The current thermal geometry voxel map is precisely registered and aligned with historical baseline voxel data, and the temperature change trend (such as rate of change and cumulative change) and possible geometric deformation information for each voxel are calculated. Based on this, a time-series variation map is generated. By combining the anomaly probability output by GAT with the time-series variation map (e.g., analyzing the correlation between the direction of the temperature gradient and curvature abrupt change regions), thermal anomaly areas on the surface of the inspected equipment can be identified more accurately, and their anomaly types can be determined.

[0040] The identified thermal anomaly areas are quantitatively assessed. The severity S of the anomaly is calculated using a comprehensive evaluation model, as shown in the following formula:

[0041] In the formula, For neighborhood temperature difference, For the rate of temperature change, To accumulate temperature difference, For curvature, Reflection intensity, For each weight coefficient, For the standardized function, , , , and These are the weights.

[0042] Based on the calculated severity score, anomalies are classified into different levels (such as warning, severe, and critical). The system automatically generates re-inspection decisions and maintenance suggestions, including: marking the location of the anomaly, explaining the anomaly category and level, planning the optimal re-inspection trajectory for the drone or robot, and recommending subsequent inspection cycles based on the anomaly's development trend.

[0043] Example 2 Based on the intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging in Example 1, this example provides a specific embodiment of the method to illustrate an intelligent inspection method for wind turbines, photovoltaic modules, energy storage systems and substation facilities. This method is based on the multimodal fusion of lidar point cloud and infrared thermal imaging data. Through environmental adaptive compensation, voxel modeling and graph attention reasoning, it realizes the three-dimensional spatial positioning of equipment surface anomalies, thermal anomaly identification and temporal change analysis, thereby improving the accuracy and reliability of operation and maintenance inspection of new energy equipment.

[0044] like Figure 1 As shown, this method includes the following main steps: 1. Multi-source synchronous acquisition and external parameter calibration: The inspection system is installed on a drone or ground mobile platform, equipped with a main lidar, an auxiliary lidar, and an infrared thermal imager. The main lidar is used for macroscopic scanning of the equipment's shape, while the auxiliary lidar is used for detailed supplementation; the infrared thermal imager acquires thermal radiation distribution maps. Simultaneously, the environmental monitoring unit records wind speed, ambient temperature, and solar irradiance data in real time at the time of acquisition, and the positioning and orientation unit provides the absolute / relative pose information of the inspected equipment.

[0045] Each sensor performs synchronous sampling via a unified clock signal or hardware trigger, with time errors controlled within 5 milliseconds. During the external parameter calibration phase, a dual-visible thermal geometric target is deployed near the device under test. This target consists of an aluminum-based checkerboard grid and a high-emissivity blackbody film, with localized heating creating hot corner regions. The lidar acquires the center of reflection intensity at the target's corner points, and the infrared thermal imager extracts the locations of these hot corner points. A nonlinear least squares method is used to jointly solve for the rotation matrix and translation vector, establishing the spatial mapping relationship between the lidar and the infrared thermal imager.

[0046] Radiometric calibration was achieved using a dual-point blackbody source, obtaining the response curves of the infrared thermal imager's grayscale values ​​and actual temperature. A lens temperature drift model was established and stored during the calibration phase for temperature correction during inspections.

[0047] 2. Thermal geometry voxel image construction and multi-view weighted fusion: The registered primary and secondary lidar point clouds are filtered and registered to form a 3D surface model in a unified coordinate system. An appropriate voxel resolution is selected based on the equipment type; for example, the voxel side length for wind turbine blades is 5-7 cm, for photovoltaic modules it is 2-3 cm, and for energy storage cabinets it is 2-3 cm.

[0048] Using calibrated extrinsic parameters, infrared thermal image data is back-projected onto the point cloud surface, and the ray landing point corresponding to each pixel determines the voxel's temperature sampling value. For multiple frames of temperature samples obtained from different viewpoints for the same voxel, the reciprocal of the variance of the viewpoint difference is used as a weighting coefficient. For each temperature value Perform a weighted average:

[0049] In the formula, This represents the final weighted average temperature of the voxels corresponding to the device under test in voxel space. For the i-th perspective on the th The weighting coefficients of each target voxel. The temperature sample value is projected onto the target voxel by an infrared thermal imager from the i-th viewpoint. A multi-view weighted thermal geometry voxel map is generated based on the final weighted average temperature, achieving precise binding of temperature information with 3D geometric information.

[0050] Each voxel records parameters such as average temperature, maximum temperature, temperature variance, neighborhood temperature difference, curvature, normal and reflection intensity, forming a basic feature set.

[0051] 3. Environmental Adaptive Compensation and Feature Extraction: To reduce the impact of the natural environment on infrared temperature measurement, the inspection system in this embodiment also includes an inertial measurement unit, a positioning module, and an environmental sensing unit. The inertial measurement unit, positioning module, and environmental sensing unit acquire solar irradiance, wind speed, and ambient temperature, respectively. The positioning module acquires the voxel surface normal angle, and the environmental sensing unit acquires wind speed and ambient temperature. Based on the acquired data, a joint compensation model for radiation, convection, and self-shading is established.

[0052] Radiation compensation: based on on-site solar irradiance Angle between the voxel normal and the incident light direction Calculate the reflection inhibition factor And the specular reflection area is reduced in weight.

[0053] Convection compensation: based on wind speed With ambient temperature The surface temperature is corrected using an empirical heat transfer model:

[0054] in, . To correct the temperature difference value, For velocity-dependent heat transfer coefficients, Voxel surface temperature The ambient temperature. Based on the basic heat transfer coefficient, For cumulative temperature difference.

[0055] Self-occlusion compensation: A point cloud visibility detection algorithm is used to determine whether a voxel is occluded, and a weight decay factor is set for invisible voxels. .

[0056] After compensation, feature extraction is performed on the corrected voxel temperature. The feature vector for each voxel... It consists of average temperature, maximum temperature, temperature variance, neighborhood temperature difference, curvature, and reflection intensity, providing input for subsequent graph network inference.

[0057] 4. Multimodal Feature Fusion and Anomaly Detection: Based on spatial adjacency and attribute similarity, adjacent voxels are aggregated into supervoxel nodes. Edge connections are established between nodes, with edge weights determined by spatial distance and temperature difference. The constructed graph structure is input into the Graph Attention Network (GAT) model, which executes a multi-head attention mechanism between nodes to weighted aggregate features from different neighborhoods.

[0058] The output is the anomaly probability of each voxel node. And anomaly category labels. By combining baseline voxel plots from historical inspections with the current temperature distribution and past records, a time-series variation map is generated to identify temperature drift. The system determines the anomaly type based on the relationship between the temperature gradient direction and curvature abrupt change, including: hotspot anomalies; crack-thermal coupling anomalies; and contact fault anomalies.

[0059] False anomalies with a detection confidence level below the threshold are removed through posterior consistency verification.

[0060] 5. Anomaly Scoring and Re-inspection Decision Generation: For identified anomaly nodes, calculate the comprehensive severity score S:

[0061] in, For neighborhood temperature difference, For the rate of temperature change, To accumulate temperature difference, For curvature, Reflection intensity, For each weight coefficient, This is a standardized function. The scoring results are used to grade and assess the severity of the anomalies.

[0062] When S is higher than the preset threshold The time mark is the target that needs to be re-inspected; when S is below the threshold The time marker is designated as a safety target. A re-inspection track is generated based on the anomaly coordinates, surface normals, and equipment structure.

[0063] The re-inspection trajectory planning is based on visibility constraints and safe distance constraints to ensure that the re-inspection perspective covers abnormal areas. The system automatically provides inspection cycle suggestions based on the severity level, realizing intelligent closed-loop operation and maintenance.

[0064] In summary, the intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging provided in this embodiment forms a complete technical process around key aspects such as multi-source synchronous acquisition, thermal geometry voxel construction, environmental adaptive compensation, multimodal fusion analysis, and re-inspection decision generation. This method can achieve the fusion representation of temperature and geometric information under a unified temporal and spatial coordinate system, and uses graph attention networks for feature weighted analysis, thereby supporting the three-dimensional localization and temporal change analysis of surface anomalies in new energy equipment.

[0065] The technical approach presented here is clearly feasible, with well-defined logical relationships between each step, making it suitable for condition monitoring scenarios in various new energy equipment, including wind power, photovoltaics, energy storage, and substations. By organically combining lidar data with infrared thermal imaging data, it can provide technical reference and a foundation for subsequent engineering applications.

[0066] Example 3 This embodiment provides an intelligent inspection system that can execute the method described in Embodiment 2, used to automatically detect, locate, and generate re-inspection tasks for surface defects in new energy equipment. The system adopts a "edge acquisition—cloud analysis—closed-loop re-inspection" technical model in its overall architecture, integrating LiDAR measurement, infrared thermal imaging sensing, multi-modal fusion processing, and intelligent decision-making modules to achieve full automation from data acquisition to anomaly re-inspection.

[0067] like Figure 2As shown, the system mainly includes the following functional modules: data acquisition module; external parameter calibration and time synchronization module; thermal geometry voxel construction module; environmental compensation and feature extraction module; multimodal feature fusion and anomaly detection module; time series change analysis and re-inspection decision module; edge-cloud collaboration and task management module.

[0068] The modules are interconnected via a high-speed data bus and a unified time base, and a cloud database is used to store multi-period voxel maps and inspection records; details are as follows: The data acquisition module includes: The main lidar unit is used to generate cloud data of the entire equipment's viewpoint, covering the overall structural outline of the equipment. A secondary lidar unit is used for scanning detailed areas, supplementing high-curvature areas and obstructed areas; Infrared thermal imaging unit is used to acquire thermal radiation images and record temperature field information; The environmental monitoring unit is used to measure wind speed, ambient temperature, and solar irradiance. The positioning and orientation unit, including an inertial measurement unit and a high-precision differential positioning module, is used to record the carrier's attitude and position information.

[0069] The aforementioned units are installed on drones or ground mobile platforms to form a multi-source synchronous acquisition array.

[0070] The external parameter calibration and time synchronization module is used to ensure spatial correspondence and temporal consistency among various sensors.

[0071] Before the inspection task begins, the lidar and infrared thermal imager are synchronized for sampling through a unified hardware trigger signal. Extrinsic parameter calibration was performed using a dual-visible thermal geometry target, which has both a reflection corner point and a thermal corner point. The rotation matrix and translation vector are solved by calibration algorithm to establish the mapping matrix from the lidar coordinate system to the thermal image coordinate system; Simultaneously, infrared radiation response calibration was performed using a dual-point blackbody source to correct the grayscale-temperature conversion function of the thermal imager.

[0072] The thermal geometry voxel construction module is used to achieve spatial fusion of lidar point cloud and thermal image data. Its main functions include: point cloud registration and noise removal; 3D voxel segmentation and index construction; thermal image back-projection mapping and multi-view temperature weighted calculation.

[0073] The module incorporates a multi-view weighted algorithm, which determines the projection weights based on the inverse of the variance of the viewpoint differences and calculates the average voxel temperature, achieving high-precision binding of thermal and geometric information. The output is a 3D thermal geometry voxel map, providing a unified data structure for subsequent analysis.

[0074] The environmental compensation and feature extraction module enables dynamic correction of voxel temperature and generation of multidimensional features.

[0075] Compensation section: A joint compensation model for radiation, convection, and self-shading is established based on wind speed, irradiance, and voxel normal angle; Feature extraction section: Extract multi-dimensional fusion features such as average temperature, maximum temperature, temperature variance, neighborhood temperature difference, curvature, and reflection intensity from the corrected voxel map; Output standardized feature vectors for subsequent graph structure modeling and anomaly analysis.

[0076] This module provides reliable input to the algorithm processing part, ensuring the physical consistency of features and environmental robustness.

[0077] The multimodal feature fusion and anomaly detection module executes the core algorithm for anomaly recognition.

[0078] First, voxels are aggregated into a super-voxel set based on spatial adjacency, forming a graph node set. Second, an edge set of the graph is constructed based on spatial distance and temperature difference, and edge weights are defined accordingly. Then, node features are input into a graph attention network, and feature weighted fusion is performed through a multi-head attention mechanism to output node anomaly probabilities. Finally, hotspot-type, crack-thermal coupling-type, and poor contact-type anomalies are identified based on the anomaly probability distribution and preset thresholds. This module outputs a preliminary anomaly list and generates a voxel-level thermal anomaly distribution map.

[0079] The time-series change analysis and re-examination decision module is responsible for historical comparison and task planning.

[0080] Perform point cloud registration and voxel-level comparison between the current voxel map and the historical baseline voxel map; Calculate the voxel temperature drift ΔThist and its rate of change r; According to the scoring model:

[0081] In the formula, For neighborhood temperature difference, For the rate of temperature change, To accumulate temperature difference, For curvature, Reflection intensity, For each weight coefficient, For the standardized function, , , , and These are the corresponding weights.

[0082] Severity assessment is performed for each anomaly, and a rating label is generated; Based on the anomaly center, normal direction, and confidence level, a re-inspection track is generated to ensure multi-angle re-inspection of the anomaly area; It also outputs inspection cycle suggestions to achieve task closure.

[0083] The edge-cloud collaboration and task management module enables the system to operate in a distributed manner.

[0084] Edge nodes: perform data acquisition, voxel construction, environmental compensation, and preliminary anomaly detection, enabling real-time feedback; Cloud platform: Performs historical data alignment, time series analysis, model updates, and re-examination task orchestration; Task Management: Responsible for flight path scheduling, safety policy constraints, and data archiving management.

[0085] Data exchange between the cloud and edge nodes is conducted via an encrypted communication protocol to ensure the security and integrity of inspection data.

[0086] During system operation, the data acquisition module first collects lidar point clouds, infrared thermal images, and environmental parameters. After alignment by the synchronization module, these are input into the voxel construction module. The voxel construction module outputs a thermal geometry voxel map, which is then subjected to temperature correction and feature extraction by the environmental compensation module. Subsequently, the feature fusion module runs a graph attention network on the super-voxel map for anomaly inference. The anomaly detection results are transmitted to the temporal change analysis module for historical comparison and scoring evaluation, generating a re-inspection plan. Finally, the task management module sends the re-inspection track to the execution platform, achieving closed-loop inspection.

[0087] In summary, this system achieves high-precision three-dimensional thermal geometry fusion detection of the surface of new energy equipment, supports simultaneous acquisition and online compensation from multiple sensors, and has the following advantages: High fusion accuracy: The error between the main and auxiliary lidar data and the infrared thermal image data is within the millimeter level; High adaptability: The environmental compensation model suppresses wind speed and irradiance interference, maintaining high detection stability; Strong recognition capability: Intelligent identification of complex defects based on graph attention mechanism; Complete time-series analysis: Long-term health monitoring is achieved by combining digital twin baselines; Closed-loop management: Supports edge-cloud collaboration and automatically generates re-inspection tracks and inspection plans.

[0088] The system architecture is versatile and can be expanded with different sensor configurations or algorithm modules according to the application scenario. For example, it can be used to deploy a multi-drone collaborative mode in a large wind farm or to conduct intensive inspections using a ground mobile platform in an energy storage station scenario.

[0089] Example 4 In another embodiment of the present invention, a computer-readable storage medium is provided as a storage component within a terminal device, the function of which is to store programs and data. It should be noted that the computer-readable storage medium here encompasses not only the built-in storage components of the terminal device but also extended storage components supported by the device. Essentially, it is a tangible medium capable of containing or storing programs that can be invoked by or in conjunction with an instruction execution system, device, or apparatus. This storage medium provides storage areas for the terminal's operating system and stores one or more instructions suitable for processor loading and execution, which can constitute one or more computer programs containing program code.

[0090] Specifically, examples of computer-readable storage media (a non-exclusive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable optical disc read-only memory, optical storage devices, magnetic storage devices, or any reasonable combination of the above types.

[0091] The storage medium may also include data signals propagated as part of a baseband portion or a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any reasonable combination of both. Furthermore, computer-readable storage medium may also refer to other readable media besides conventional readable storage media, capable of sending, propagating, or transmitting programs for use or operation by an instruction execution system, apparatus, or device. Program code on the storage medium can be transmitted via any suitable medium, including but not limited to wireless, wired, optical fiber, or any reasonable combination thereof.

[0092] The program code used to implement the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C. The execution modes of the program code include: running entirely on the user's computing device, running partially on the user's device as a standalone software package, running partially in a distributed manner on both the user's device and a remote computing device, or running entirely on a remote computing device or server. When a remote computing device is involved, the device can be connected to the user's computing device via any type of network such as a local area network (LAN) or a wide area network (WAN), or connected to an external computing device via the Internet through an Internet service provider.

[0093] The processor is capable of loading and executing one or more instructions stored in a computer-readable storage medium to implement the corresponding steps of the intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging described in Example 1.

[0094] Example 5 Figure 3This is a schematic diagram of a computer device provided according to an embodiment of the present invention.

[0095] Please see Figure 3 The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging as described in this embodiment. To avoid repetition, details are omitted here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the computing system constituting the intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging as described in this embodiment. To avoid repetition, details are omitted here.

[0096] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 3 This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0097] The processor 61 may be a central processing unit (CPU), or other general-purpose processors, CPUs, graphics processing units (GPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, quantum computing-based data processing logic units, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0098] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or RAM of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the computer device 60.

[0099] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0100] Any references to memory, databases, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (Read-Only Memory). Memory includes ROM, magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0101] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

Claims

1. A method for intelligent inspection of new energy equipment based on the fusion of lidar and infrared imaging, characterized in that, include: Several lidars are used to acquire spatiotemporally aligned point cloud data of the device under inspection, and an infrared thermal imager is used to acquire infrared thermal imaging data of the device under inspection. The lidars and the infrared thermal imager are obtained by external parameter calibration and radiometric calibration before acquiring the data. It also acquires environmental monitoring data and the position and orientation information of the equipment under inspection; The point cloud data is divided into voxels to obtain a voxel space. The radiometrically calibrated infrared thermal image data is back-projected onto the voxel space to obtain the temperature attributes of each voxel in the voxel space. Weights are assigned to the projection differences from different viewpoints in voxel space, and the temperature attributes are weighted and averaged based on the weights to generate a thermal geometry voxel map. Based on environmental monitoring data and the pose information of the tested equipment, the temperature attributes in the thermal geometry pixel map are dynamically corrected using a joint compensation model, and fusion features are extracted from the corrected thermal geometry pixel map. A graph attention network is used to perform weighted fusion and anomaly probability calculation on the fused features, and a time-series change map is generated by combining historical baseline voxel data. The thermal anomaly area and anomaly type on the surface of the inspected equipment are identified based on the time-series change map. The severity of the anomaly is calculated based on the thermal anomaly area and corresponding anomaly type on the surface of the inspected equipment, and a re-inspection decision and maintenance suggestion are generated based on the severity of the anomaly.

2. The intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging as described in claim 1, characterized in that, External parameter calibration and radiometric calibration of lidar and infrared thermal imager, including: The external parameter calibration is achieved through a thermal geometric dual-visible target, which includes an aluminum-based checkerboard grid and a blackbody film with a set emissivity, used to locally heat the device under test to form a hot corner area. The radiation calibration was achieved using a two-point blackbody source.

3. The intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging according to claim 2, characterized in that, LiDAR includes a main lidar and an auxiliary lidar; The main lidar is used to scan the overall structure of the new energy equipment; the auxiliary lidar is used for scanning local details; the main lidar, auxiliary lidar and infrared thermal imager are synchronized in time through a unified trigger signal.

4. The intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging according to claim 3, characterized in that, Weights are assigned based on the projection differences from different viewpoints, and a weighted average is calculated for the temperature samples, including: In the formula, This represents the final weighted average temperature of the voxels corresponding to the device under test in voxel space. For the i-th perspective on the th The weighting coefficients of each target voxel. The temperature sample value is the value back-projected onto the target voxel by the infrared thermal imager from the i-th viewpoint.

5. The intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging according to claim 1, characterized in that, The thermal geometry voxel graph includes average temperature, maximum temperature, temperature variance, neighborhood temperature difference, curvature, and reflection intensity.

6. The intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging according to claim 5, characterized in that, Based on environmental monitoring data and the position and orientation information of the inspected equipment, a joint compensation model is used to dynamically correct the temperature attributes in the thermal geometry graph, including: The joint compensation model includes radiation compensation, convection compensation, and self-shading compensation. The radiation compensation is used to determine the amount of radiation reflection compensation based on the solar irradiance in the environmental monitoring data and the voxel surface normal angle in the pose information of the tested equipment. The convection compensation is calculated based on wind speed and ambient temperature from environmental monitoring data and an empirical convection heat transfer model. Based on point cloud visibility analysis, a self-occlusion compensation weight attenuation is applied to voxels that are occluded by other parts. The self-occlusion compensation is based on the corrected thermal geometry voxel map, and the temperature values ​​of the occluded voxels are weighted down or removed according to their reliability.

7. The intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging according to claim 1, characterized in that, The time-series variation map is obtained by registering and aligning the thermal geometry voxel map with the historical thermal geometry voxel map, and calculating the temperature change trend and time drift information of the voxels based on the aligned voxel map, so as to realize the analysis of abnormal evolution trend.

8. The intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging according to claim 7, characterized in that, The severity of the anomaly is calculated based on the thermal anomaly area and corresponding anomaly type on the surface of the inspected equipment, and a re-inspection decision and maintenance suggestion are generated based on the severity of the anomaly, including: Based on the temporal variation map, the temperature drift and geometric deformation trends in the thermal anomaly area on the surface of the inspected equipment are identified; the anomaly type is determined based on the relationship between the temperature gradient direction and curvature change in the temperature drift and geometric deformation trends; the severity score is calculated based on the anomaly type and anomaly node; based on the severity score, the anomaly location, category and level are output, and a re-inspection track and inspection cycle suggestion are generated.

9. The intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging according to claim 8, characterized in that, The severity score include: In the formula, For neighborhood temperature difference, For the rate of temperature change, To accumulate temperature difference, For curvature, Reflection intensity, For each weight coefficient, For the standardized function, , , , and These are the weights.

10. The intelligent inspection method for new energy equipment based on the fusion of lidar and infrared imaging according to any one of claims 1, characterized in that, The anomaly types include: hot spot anomaly, crack thermal coupling anomaly, and poor contact anomaly.

Images