Adaptive fusion perception system based on electric field perception and multi-modal space-time registration

Abstract

This invention discloses an adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration, comprising: an electric field sensing module, used to synchronously acquire the three-axis electric field components of each measuring point through a three-axis electric field sensor array deployed on the robot body, and generate a dynamic electric field interference map centered on the robot based on the three-axis electric field components; a sensor adaptive module, used to hierarchically and dynamically adjust the working parameters of the multi-source sensors on the robot body to obtain calibrated working parameters, and the multi-source sensors acquire multi-source data according to the calibrated working parameters; a spatiotemporal registration fusion module, used to perform time synchronization and spatial registration of the multi-source data to generate spatiotemporally consistent multi-source data; and a defect identification module, used to identify target defects based on the spatiotemporally consistent multi-source data and the dynamic electric field interference map, using an improved YOLOv8 model to obtain identification results, and perform joint verification to obtain the final target defect identification result, and perform three-dimensional localization based on the final target defect identification result.

Description

Technical Field

[0001] This invention relates to the field of power detection technology, and in particular to an adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration. Background Technology

[0002] Currently, using robots equipped with multiple sensors (such as visual cameras, LiDAR, and infrared thermal imagers) to inspect high-voltage transmission lines has become an important technical approach to improve the level of automation in operation and maintenance. Under normal conditions, multi-sensor fusion methods based on extended Kalman filtering and deep learning have achieved mature applications in localization, target recognition, and defect detection. However, near 500kV and above ultra-high-voltage lines, strong electric fields reaching up to 45kV / m create extreme electromagnetic interference environments, posing serious challenges to existing technical solutions. Their current development status and limitations can be summarized in the following three aspects: Firstly, while existing technologies have explored direct strategies to address strong electromagnetic interference, their depth is insufficient. For example, CN121564313A attempts to improve the efficiency of locking onto charged targets by introducing an electric field sensor to provide orientation guidance for vision and lidar. However, this type of solution only uses electric field information for initial coarse target localization and fails to deeply integrate it into core aspects such as subsequent adaptive adjustment of sensor parameters, spatiotemporal registration constraints in the data fusion process, and target recognition feature enhancement. The potential of electric field sensing has not been fully realized.

[0003] Secondly, in terms of general methods for improving system environmental adaptability, existing technologies lack specific designs for electromagnetic interference. CN112257522B switches sensor combinations by detecting natural features such as rain and light to improve robustness under different natural environments; while CN111812649A and CN120652981A focus on fusing data from cameras, radar, etc., through filtering algorithms to improve perception accuracy or achieve reliable obstacle avoidance. Although these methods are effective in their respective target scenarios, their initial design purpose is not to deal with strong electromagnetic interference, and therefore they cannot solve the fundamental problems of sensor data distortion (such as image noise and point cloud drift) and spatiotemporal misalignment unique to this environment.

[0004] Finally, existing technologies have limitations in meeting the high performance requirements of power line inspection. The precise inspection of ultra-high voltage (UHV) lines requires robotic systems to possess both millimeter-level positioning accuracy and millisecond-level response speed. However, the aforementioned existing solutions either lack specific quantitative indicators for strong electromagnetic environments (such as CN121564313A and CN120652981A), or their technical approaches themselves do not include electromagnetic interference suppression mechanisms. This results in a difficulty in simultaneously achieving accuracy and real-time performance under extreme electric fields, failing to meet the stringent engineering application requirements.

[0005] In summary, while existing technologies have made progress in the general aspects of multi-sensor fusion and in adapting to specific environments, they are still insufficient to simultaneously guarantee the required detection accuracy for high-voltage transmission lines in the special and demanding application scenario of ultra-high voltage strong electromagnetic environment. Summary of the Invention

[0006] In view of the above-mentioned prior art, the present invention provides an adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration, which mainly solves the technical problems existing in the above-mentioned background art.

[0007] To achieve the above objectives, the technical solution of the present invention is implemented as follows: An adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration includes: The electric field sensing module is used to synchronously collect the triaxial electric field components at each measuring point through a triaxial electric field sensor array deployed on the robot body, and generate a dynamic electric field interference map centered on the robot based on the triaxial electric field components. The dynamic electric field interference map includes: electric field intensity distribution field and electric field gradient field. The sensor adaptive module is used to dynamically adjust the working parameters of the multi-source sensors on the robot body according to the electric field intensity distribution field to obtain the corrected working parameters. The multi-source sensors collect multi-source data according to the corrected working parameters. The multi-source data includes: laser point cloud, visual image and temperature distribution map. The spatiotemporal registration and fusion module is used to perform time synchronization and spatial registration on the multi-source data to generate spatiotemporally consistent multi-source data; The defect identification module is used to identify target defects based on the spatiotemporally consistent multi-source data and the dynamic electric field interference spectrum, using an improved YOLOv8 model to obtain identification results, perform joint verification on the identification results to obtain the final target defect identification result, and perform three-dimensional positioning based on the final target defect identification result. Optionally, the step of synchronously acquiring the triaxial electric field components at each measuring point through a triaxial electric field sensor array deployed on the robot body, and generating a dynamic electric field interference spectrum centered on the robot based on the triaxial electric field components, includes: Step S11: Collect the triaxial electric field intensity components of each triaxial electric field sensor on the robot body, calculate the original synthetic electric field intensity vector of each measuring point, and obtain a discrete multi-measuring-point electric field dataset; Step S12: Filter the discrete multi-point electric field dataset time-by-time using Kalman filtering to obtain a multi-point electric field intensity vector dataset. Step S13: Obtain the robot's current position information in real time, and use the Kriging interpolation method to perform spatial interpolation on the multi-point electric field intensity vector dataset to obtain a three-dimensional electric field vector grid that is continuously distributed in the space around the robot. Step S14: Calculate the electric field intensity distribution field and electric field gradient field based on the three-dimensional electric field vector grid, and fuse them to form a dynamic electric field interference map.

[0008] Optionally, the step of dynamically adjusting the operating parameters of the multi-source sensors on the robot body according to the electric field intensity distribution to obtain calibrated operating parameters, and the multi-source sensors collecting multi-source data according to the calibrated operating parameters, includes: The multi-source sensors include: a lidar mounted on the top of the robot, a vision camera mounted on the front of the robot, and an infrared thermal imager mounted on the side of the robot. Step S21: Set up a three-level hierarchical strategy; Step S22: Obtain the real-time synthesized electric field intensity based on the electric field intensity distribution field, and adjust the working parameters of the multi-source sensors on the robot body based on the real-time synthesized electric field intensity and the three-level hierarchical strategy to obtain the corrected working parameters; Step S23: The multi-source sensor is calibrated according to the calibration working parameters, and multi-source data is collected after calibration.

[0009] Optionally, the three-level hierarchical strategy in step S21 includes: Set a first threshold and a second threshold; When the electric field strength is greater than the first threshold, the lidar is controlled to switch to the first wavelength and the scanning frequency is increased, the vision camera is controlled to enable polarization filtering and high dynamic range mode, and the gain of the infrared thermal imager is reduced. When the electric field strength is between the first threshold and the second threshold, the lidar is controlled to use the first wavelength and a scanning frequency between the standard value and the boost value, the vision camera is controlled to enable the high dynamic range mode, and the gain of the infrared thermal imager is partially reduced. When the electric field strength is less than the second threshold, the lidar, the vision camera, and the infrared thermal imager are controlled to use standard operating parameters.

[0010] Optionally, the step of performing time synchronization and spatial registration on the multi-source data to generate spatiotemporally consistent multi-source data includes: Step S31: Based on the electric field vector constraint, the extended Kalman filter is used to correct the timestamp deviation of the lidar, visual camera and infrared thermal imager to obtain time-aligned laser point cloud, visual image and temperature distribution map. Step S32: Perform semantic recognition on the temporally aligned visual image and output a visual semantic image with pixel-level semantic labels; Step S33: Using the ICP algorithm constrained by the electric field spatial gradient, perform joint spatial registration on the time-aligned laser point cloud and the visual image to obtain the first spatial registration extrinsic parameters; Step S34: Based on the first spatial registration extrinsic parameters, the time-aligned laser point cloud, and the robot's real-time pose, the laser point cloud is uniformly converted to the visual camera coordinate system to construct a three-dimensional point cloud map aligned with the pixel space of the visual image. Step S35: Solve the second spatial registration extrinsic parameters of the infrared thermal imager relative to the visual camera using the checkerboard offline calibration method. Based on the second spatial registration extrinsic parameters, map and align the time-aligned temperature distribution map to the visual camera coordinate system to obtain a temperature distribution map aligned with the visual image space.

[0011] Optionally, step S31 includes: Define a state vector, the expression of which is:

[0012] in, For the robot's three-dimensional position coordinates, The robot's heading attitude angle, For the robot's movement speed, These are the timestamp deviations of the lidar, visual camera, and infrared thermal imager, respectively. An electric field constraint term is constructed by introducing the electric field gradient field from the dynamic electric field interference spectrum. Based on this electric field constraint term, a nonlinear observation equation is established. The expression of the observation equation is as follows:

[0013]

[0014] in, For the observation equation, The coordinates of the feature point cloud observed by the lidar. These are the pixel coordinates of the feature points observed by the visual camera. The target temperature and location as observed by the infrared thermal imager. For the electric field gradient constraint term, For the electric field gradient, This represents the displacement of the triaxial electric field sensor array between two observations. This represents the measured change in electric field intensity at the positions before and after displacement of the triaxial electric field sensor array. Here, v is the nonlinear observation function, and v is the observation noise. The extended Kalman filter prediction and update iteration are performed, and the optimal timestamp deviation of the lidar, vision camera and infrared thermal imager is solved by electric field gradient constraint residual optimization. The output is time-synchronized laser point cloud, visual image and infrared temperature distribution map.

[0015] Optionally, the step of performing extended Kalman filter prediction and update iteration, using electric field gradient constrained residual optimization to solve for the optimal timestamp deviation of the lidar, visual camera, and infrared thermal imager, and outputting time-synchronized laser point cloud, visual image, and infrared temperature distribution map includes: Based on the optimal state estimate and posterior covariance matrix of the previous period as input, a state transition function is constructed to complete the prior prediction of the state vector of the current period and obtain the prior state estimate. The state transition Jacobian matrix is ​​solved synchronously, and the prior covariance matrix is ​​obtained by updating the superimposed process noise covariance matrix. Read the electric field intensity in the electric field intensity distribution field, and adaptively adjust the weight of the electric field constraint term in the extended Kalman filter covariance matrix according to the electric field intensity; Substitute the observation vector containing the electric field gradient constraint term in the current period into the nonlinear observation mapping function, calculate the observation prediction value and the observation residual, solve the observation Jacobian matrix and calculate the Kalman gain. The prior state estimate is corrected using the Kalman gain and the observation residual. The three timestamp deviation variables in the prior state estimate are iteratively optimized and updated to obtain the optimal posterior state estimate and optimal timestamp deviation for the current period. The posterior covariance matrix is ​​updated synchronously. The original timestamps of the lidar, visual camera, and infrared thermal imager are compensated frame by frame using the optimal timestamp deviation, and time-synchronized laser point cloud, visual image, and infrared temperature distribution map are output.

[0016] Optionally, step S33 includes: By introducing an electric field gradient consistency constraint, an objective function is constructed. The spatial registration extrinsic parameters are then obtained by solving the objective function. The expression for the objective function is as follows:

[0017]

[0018] in, Let be the objective function. For laser point clouds, For visual feature points projected onto a point in three-dimensional space, Let be a rotation matrix. It is a translation vector. and These are the electric field gradients at the locations of the lidar and the vision camera, respectively. These are the weighting coefficients. This is the standard ICP geometric registration loss term.

[0019] Optionally, the identification result obtained by using an improved YOLOv8 model to identify the target defect based on the spatiotemporally consistent multi-source data and the dynamic electric field interference spectrum includes: The improved YOLOv8 model includes an input layer, a backbone layer, a neck layer, and a head layer connected in sequence. The input layer is used to input dimensions of... The four-channel fused image tensor has the first three channels being the RGB image of the visual image, and the fourth channel being the grayscale image of the electric field intensity generated by perspective interpolation based on the laser point cloud and the electric field intensity distribution field. The Backbone layer is a CSPDarknet backbone network adapted to four-channel input. It receives the four-channel fused image tensor and performs convolution downsampling layer by layer to obtain multi-scale feature maps. The Neck layer includes a feature pyramid structure and an electric field attention module. The electric field attention module calculates electric field attention weights based on the electric field intensity grayscale image, multiplies the electric field attention weights element-wise with the multi-scale feature map, and adaptively enhances the features of the strong electric field region to obtain an enhanced electric field region feature map. The feature pyramid structure fuses the enhanced electric field region feature map to obtain a multi-scale fused feature map. The Head layer includes two output branches. One output branch outputs the target defect category, confidence level, and candidate defect bounding box of the transmission line based on the multi-scale fused feature map. The other output branch outputs the electric field interference level based on the multi-scale fused feature map. Optionally, the step of performing joint verification on the identification results to obtain the final target defect identification result includes: Based on the visual semantic image, semantic verification is performed, the pixel ratio of power components in each candidate defect bounding box is counted, and candidate defect bounding boxes with a pixel ratio of power components lower than a preset threshold are removed to obtain intermediate candidate defect bounding boxes. The intermediate candidate defect bounding boxes are classified and determined. If the target defect category of the intermediate candidate defect bounding box is a heat-generating defect, temperature verification is performed. The temperature of the corresponding region is extracted based on the temperature distribution map. The region temperature is compared with a preset temperature threshold. If the region temperature reaches the preset temperature threshold, it is retained; otherwise, it is discarded. If the target defect category of the intermediate candidate defect bounding box is a non-heating defect, then temperature verification is not performed, and the intermediate candidate defect bounding box is retained. Based on all retained intermediate candidate defect bounding boxes, the final target defect identification result is output.

[0020] The beneficial effects of the present invention are as follows: The adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration provided by the present invention can be applied in strong electric field environments, and has millimeter-level positioning accuracy and faster response speed in strong electric field environments, which can meet the stringent requirements of power line inspection.

[0021] Traditional methods can cause timestamp drift of several milliseconds under strong electric fields. However, by using extended Kalman filtering with electric field vector constraints, timestamps from multiple sensors can be effectively corrected, resulting in time-synchronized data from lidar, vision cameras, and infrared thermal imagers with time alignment. The timestamp correction accuracy is improved to the sub-millisecond level. Furthermore, by introducing ICP point cloud registration with electric field gradient constraints, spatial registration errors are reduced, effectively improving the accuracy of spatial registration, which is significantly better than traditional ICP registration methods.

[0022] By adding electric field feature channels to improve YOLOv8, an improved YOLOv8 model is obtained. Using the improved YOLOv8 model for defect identification in high-voltage transmission lines, the accuracy of defect detection is effectively improved, high-accuracy defect classification is achieved, and the three-dimensional coordinates of defects are effectively extracted, enabling better localization of target defects. By dynamically and adaptively adjusting the operating parameters of multi-source sensors on the robot body in a tiered manner, the system maintains effective recognition capabilities even in degraded modes and remains usable in extreme interference environments, thus enhancing the reliability of engineering applications. Furthermore, the system is applicable to various scenarios with strong electromagnetic interference, such as high-voltage transmission line inspection, substation inspection, and converter station inspection, making it more versatile and possessing significant engineering application value. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the structure of an adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the electric field sensing module provided in an embodiment of the present invention. Figure 3 This is a flowchart illustrating the sensor adaptive module provided in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the spatiotemporal registration and fusion module provided in this embodiment of the invention. Detailed Implementation

[0024] To fully understand this invention, a detailed structure will be presented in the following description to illustrate the technical solution proposed by this invention. Optional embodiments of the invention are described in detail below; however, in addition to these detailed descriptions, the invention may have other embodiments.

[0025] Example 1 Please refer to the attached document. Figure 1 This application provides an adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration, including: The electric field sensing module is used to synchronously collect the triaxial electric field components at each measuring point through a triaxial electric field sensor array deployed on the robot body, and generate a dynamic electric field interference map centered on the robot based on the triaxial electric field components. The dynamic electric field interference map includes: electric field intensity distribution field and electric field gradient field. The sensor adaptive module is used to dynamically adjust the working parameters of the multi-source sensors on the robot body according to the electric field intensity distribution field to obtain the corrected working parameters, and the multi-source sensors collect multi-source data according to the corrected working parameters. The spatiotemporal registration and fusion module is used to perform time synchronization and spatial registration on the multi-source data to generate spatiotemporally consistent multi-source data; The defect identification module is used to identify target defects based on the spatiotemporally consistent multi-source data and the dynamic electric field interference spectrum, using an improved YOLOv8 model to obtain identification results, perform joint verification on the identification results to obtain the final target defect identification result, and perform three-dimensional positioning based on the final target defect identification result.

[0026] Specifically, the triaxial electric field components at each measuring point on the robot are acquired through a triaxial electric field sensor array in the electric field sensing module. Based on these components, the electric field intensity distribution field and electric field gradient field are calculated, thus obtaining the robot's dynamic electric field interference spectrum. A multi-source sensor array, including a lidar, a vision camera, and an infrared thermal imager, is also installed on the robot. A sensor adaptive module adjusts the parameters of the lidar, vision camera, and infrared thermal imager according to the electric field intensity distribution to obtain calibration parameters. These calibration parameters correct the raw data obtained by the lidar, vision camera, and infrared thermal imager, significantly improving key indicators such as lidar point cloud density, visual image signal-to-noise ratio, and infrared image noise ratio, effectively suppressing strong electromagnetic interference. The impact of sensor data: After adjusting the LiDAR, visual camera, and infrared thermal imager according to the calibration parameters, the LiDAR, visual camera, and infrared thermal imager re-acquire multi-source data, corresponding to obtain laser point clouds, visual images, and temperature distribution maps. Then, using the spatiotemporal registration and fusion module, to ensure the accuracy of subsequent defect identification, the multi-source data, namely laser point clouds, visual images, and temperature distribution maps, are synchronized in time and registered in space to ensure the temporal consistency and coordinate system uniformity of the multi-source data, generating spatiotemporally consistent multi-source data. The defect identification module constructs an improved YOLOv8 model. Based on the spatiotemporally consistent multi-source data and dynamic electric field interference spectrum, the improved YOLOv8 model is used to identify target defects, and the three-dimensional positioning is performed based on the identification results to determine the specific location of the defect.

[0027] In power scenarios, the working environment of high-voltage transmission lines contains a large number of high-voltage conductors, fittings, and insulators. Existing technologies mostly rely on RGB visual images for image recognition to obtain cable defect identification results. However, due to the presence of a large number of high-voltage conductors and other high-voltage live equipment in the working environment of high-voltage transmission lines, there is a strong spatial electric field around the high-voltage live equipment, which is prone to strong electric field interference. This results in image quality degradation such as noise and local whitening and blurring. Consequently, it is difficult to clearly identify small damage, cracks, broken strands, and other defects using only RGB images, making it very easy to miss detections. Therefore, in this technical solution, by generating a dynamic electric field interference map, which includes pixel-level aligned three-dimensional electric field intensity distribution, electric field gradient, and electric field vector, the dynamic electric field map is introduced as a second mode. The electric field information is not affected by optical imaging, so as to form a dual-mode joint discrimination of visual texture and electric field physical quantity. The recognition accuracy is greatly improved in strong light and high voltage distortion scenarios, the recognition robustness is stronger, and the scene applicability is stronger.

[0028] As an optional implementation method, please refer to the appendix. Figure 2 The process of synchronously acquiring triaxial electric field components at each measuring point through a triaxial electric field sensor array deployed on the robot body, and generating a dynamic electric field interference spectrum centered on the robot based on the triaxial electric field components, includes: Step S11: Collect the triaxial electric field intensity components of each triaxial electric field sensor on the robot body, calculate the original synthetic electric field intensity vector of each measuring point, and obtain a discrete multi-measuring-point electric field dataset; Step S12: Filter the discrete multi-point electric field dataset time-by-time using Kalman filtering to obtain a multi-point electric field intensity vector dataset. Step S13: Obtain the robot's current position information in real time, and use the Kriging interpolation method to perform spatial interpolation on the multi-point electric field intensity vector dataset to obtain a three-dimensional electric field vector grid that is continuously distributed in the space around the robot. Step S14: Calculate the electric field intensity distribution field and electric field gradient field based on the three-dimensional electric field vector grid, and fuse them to form a dynamic electric field interference map.

[0029] Specifically, a triaxial electric field sensor array is formed by deploying multiple triaxial electric field sensors on the robot body. The triaxial electric field intensity components of the X, Y, and Z axes at the measurement points of each triaxial electric field sensor are collected synchronously. For each triaxial electric field sensor measurement point, the collected triaxial electric field intensity components are vector synthesized to obtain the original synthesized electric field intensity vector of the corresponding measurement point. The original synthesized electric field intensity vectors of all triaxial electric field sensor measurement points together constitute the discrete multi-measurement point electric field dataset around the robot body. The Kalman filter algorithm is used to perform time-series noise reduction on each original synthetic electric field intensity vector in the discrete multi-point electric field dataset. The electric field disturbance introduced by sensor measurement noise and robot motion jitter is used as the filtering observation error. The electric field vector state transition equation and observation equation are constructed, and the electric field intensity vector values ​​at each measurement point are iteratively corrected. After filtering out random interference fluctuations, a time-stable multi-point electric field intensity vector dataset is output. The robot's current position information is acquired in real time. Using the robot's body coordinate system as the spatial reference center, Kriging interpolation is employed to perform spatial interpolation calculations on the multi-point electric field intensity vector dataset. Based on discrete measurement point vector fitting, the three-dimensional electric field vector at any spatial coordinate point within the robot's surrounding monitoring area is solved, generating a continuously distributed three-dimensional electric field vector grid covering the robot's surrounding monitoring range. Based on this continuous three-dimensional electric field vector grid, two types of field data (electric field intensity distribution field and electric field gradient field) are derived and calculated, and then fused into a dynamic electric field interference map. Specifically, for each spatial coordinate point within the grid, the magnitude of the three-dimensional electric field vector at that point is calculated, and the magnitudes of all grid points are then combined. The values ​​are arranged according to spatial coordinates to form a spatial scalar cloud map, that is, to obtain the electric field intensity distribution field; the electric field intensity scalar of each spatial coordinate point in the grid is discretely differentially calculated along the three spatial dimensions of X, Y, and Z to obtain the electric field spatial gradient vector of each grid point, and the set of all gradient vectors forms the electric field gradient field; during the entire robot operation, the three-axis electric field sensor continuously and synchronously samples, the robot pose data is updated in real time, and steps S11 to S14 are executed cyclically to complete the interpolation and iterative recalculation of the two types of field information, and finally a dynamic electric field interference map centered on the robot is generated, which can be continuously updated with the robot's movement and the time-varying state of external electric field interference.

[0030] It should be noted that the electric field intensity distribution field can show the strength of the electric field around the robot and quickly locate areas of strong interference; the electric field gradient field can characterize the rate and direction of change of the electric field strength, thereby locating the source of interference. The electric field intensity changes most drastically near the source of interference, and the degree of electric field distortion can also be quantified. The larger the gradient, the more severe the electric field abrupt change in the area, and the higher the degree of danger.

[0031] As an optional implementation method, please refer to the appendix. Figure 3 The step involves dynamically adjusting the operating parameters of the multi-source sensors on the robot body according to the electric field intensity distribution to obtain calibrated operating parameters. The multi-source sensors then collect multi-source data based on these calibrated operating parameters, including: The multi-source sensors include: a lidar mounted on the top of the robot, a vision camera mounted on the front of the robot, and an infrared thermal imager mounted on the side of the robot. Step S21: Set up a three-level hierarchical strategy; Step S22: Obtain the real-time synthesized electric field intensity based on the electric field intensity distribution field, and adjust the working parameters of the multi-source sensors on the robot body based on the real-time synthesized electric field intensity and the three-level hierarchical strategy to obtain the corrected working parameters; Step S23: The multi-source sensor is calibrated according to the calibration working parameters, and multi-source data is collected after calibration.

[0032] The three-level hierarchical strategy in step S21 includes: Set a first threshold and a second threshold; When the electric field strength is greater than the first threshold, the lidar is controlled to switch to the first wavelength and the scanning frequency is increased, the vision camera is controlled to enable polarization filtering and high dynamic range mode, and the gain of the infrared thermal imager is reduced. When the electric field strength is between the first threshold and the second threshold, the lidar is controlled to use the first wavelength and a scanning frequency between the standard value and the boost value, the vision camera is controlled to enable the high dynamic range mode, and the gain of the infrared thermal imager is partially reduced. When the electric field strength is less than the second threshold, the lidar, the vision camera, and the infrared thermal imager are controlled to use standard operating parameters.

[0033] For example, a first threshold and a second threshold are preset, with the first threshold being 30kV / m and the second threshold being 10kV / m. The three-level classification strategy is as follows: when the electric field strength is greater than the first threshold (electric field strength > 30kV / m), the lidar switches to a 905nm wavelength (compared to the commonly used 1550nm wavelength, 905nm has a 40% lower sensitivity to electromagnetic interference), the scanning frequency is increased to 20Hz (the standard mode is 10Hz), and the point cloud density is doubled; the vision camera enables a polarization filter mode to filter out electric field-induced polarization light interference and enables HDR (high dynamic range) mode; the infrared thermal imager gain is reduced by 30% (from the standard gain G to 0.7G) to suppress electromagnetic noise amplification and increase the integration time to 1.5 times the standard value; When the electric field strength is between the first and second thresholds (10-30 kV / m), the lidar uses a 905 nm wavelength and a scanning frequency of 15 Hz; the vision camera is set to HDR mode; and the infrared thermal imager gain is reduced by 15%. When the electric field strength is less than the second threshold (electric field strength <10kV / m), the lidar uses a wavelength of 1550nm and a scanning frequency of 10Hz; the vision camera uses a standard exposure mode; and the infrared thermal imager uses a standard gain.

[0034] The robot's real-time 3D spatial position is obtained. Spatial interpolation is performed in the electric field intensity distribution field of the dynamic electric field interference spectrum to solve the real-time synthetic electric field intensity corresponding to the robot's current spatial position. This real-time synthetic electric field intensity is then mapped to the electric field intensity in the three-level hierarchical strategy. The real-time synthetic electric field intensity is then compared with the first threshold and the second threshold to determine the specific adjustment strategy. The operating parameters of the multi-source sensors are corrected according to the corresponding adjustment strategy. After the correction is completed, the multi-source sensors collect data again to obtain laser point clouds, visual images, and temperature distribution maps.

[0035] As an optional implementation method, please refer to the appendix. Figure 4 The step of performing time synchronization and spatial registration on the multi-source data to generate spatiotemporally consistent multi-source data includes: Step S31: Based on the electric field vector constraint, the extended Kalman filter is used to correct the timestamp deviation of the lidar, visual camera and infrared thermal imager to obtain time-aligned laser point cloud, visual image and temperature distribution map. Step S32: Perform semantic recognition on the temporally aligned visual image and output a visual semantic image with pixel-level semantic labels; Step S33: Using the ICP algorithm constrained by the electric field spatial gradient, perform joint spatial registration on the temporally aligned laser point cloud and the visual semantic image to solve for the first spatial registration extrinsic parameters; Step S34: Based on the first spatial registration extrinsic parameters, the time-aligned laser point cloud, and the robot's real-time pose, the laser point cloud is uniformly converted to the visual camera coordinate system to construct a three-dimensional point cloud map aligned with the pixel space of the visual image. Step S35: Solve the second spatial registration extrinsic parameters of the infrared thermal imager relative to the visual camera using the checkerboard offline calibration method. Based on the second spatial registration extrinsic parameters, map and align the time-aligned temperature distribution map to the visual camera coordinate system to obtain a temperature distribution map aligned with the visual image space.

[0036] Specifically, the data acquisition activities of the LiDAR, visual camera, and infrared thermal imager in the multi-source sensors operate continuously and independently. Therefore, the multi-source sensors continuously acquire data natively. Each frame of data is generated with a native original timestamp in the hardware. That is, the multi-source sensors in the sensor adaptation module continuously acquire data using updated calibration parameters, outputting multi-source data carrying the original hardware timestamp. In the spatiotemporal registration and fusion module, the EKF based on electric field constraints is used to solve the hardware clock deviation of each sensor, and numerical compensation is performed on the original frame timestamps of the multi-source data to complete the time alignment (time synchronization) of the multi-source data. Semantic recognition is performed on the visual images to obtain visual semantic images, realizing the identification of transmission line targets, backgrounds, and defects. Pixel-level classification and labeling of the defect region; for example, DeepLabV3+ is used to process the visual image, assigning a category label (such as wire, insulator, tower, background, etc.) to each pixel, providing contextual information for the subsequent YOLOv8 model and assisting in defect identification; since the laser point cloud is located in the coordinate system of the lidar itself, which is a three-dimensional point coordinate, while the visual image is two-dimensional image data, in order to facilitate subsequent target defect identification using the improved YOLOv8 (input data is two-dimensional image data), it is necessary to transform the coordinate system of the laser point cloud. Therefore, joint spatial registration is performed on the temporally aligned laser point cloud and the visual semantic image to obtain the first spatial registration extrinsic parameters. The extrinsic parameters include rotation matrix and translation vector. Based on the first spatial registration extrinsic parameters, the temporally aligned laser point cloud, and the robot's real-time pose, the laser point cloud is uniformly transformed to the visual camera coordinate system. Through multi-frame point cloud stitching, distortion correction, filtering, noise reduction, and downsampling, a 3D point cloud map aligned with the pixel space of the visual image is constructed. The 3D point cloud map is formed by fusing the LiDAR point cloud and visual depth information, providing accurate spatial geometric information for subsequent 3D coordinate extraction. The second spatial registration extrinsic parameters of the infrared thermal imager relative to the visual camera are solved using a checkerboard offline calibration method. A checkerboard calibration plate with infrared temperature difference characteristics is selected, and the visual camera and infrared thermal imager are rigidly fixed, synchronously triggering the acquisition of different poses. Visible light and infrared images of a checkerboard pattern at different states and distances are used to form multiple sets of paired images. Corner points within the checkerboard pattern are detected in both the visible light and infrared images, and the pixel coordinates of the corresponding corner points in the two sets of images are obtained. Dual-camera stereo calibration is performed using multiple sets of matched corner points to solve for the rotation matrix and translation vector of the infrared thermal imager relative to the visual camera, forming the second spatial registration extrinsic parameters. Based on the second spatial registration extrinsic parameters and the infrared thermal imager intrinsic parameters, a pixel projection mapping relationship is constructed. The time-aligned infrared temperature distribution map is then projected pixel by pixel, mapped and aligned to the visual image pixel coordinate system, ensuring that the temperature pixels corresponding to the same spatial target match the positions of the visual image pixels. The temperature distribution map facilitates subsequent identification of heat-generating defects. Since the visual semantic image is in the same coordinate system as the visual image, there is no need to perform coordinate system transformation on the visual semantic image. Therefore, according to steps S31-S35, the final result is a temporally aligned and spatially registered 3D point cloud map, visual image, infrared temperature distribution map, and visual semantic image.

[0037] As an optional implementation, step S31 includes: Define a state vector, the expression of which is:

[0038] in, For the robot's three-dimensional position coordinates, The robot's heading attitude angle, For the robot's movement speed, These are the timestamp deviations of the lidar, visual camera, and infrared thermal imager, respectively. An electric field constraint term is constructed by introducing the electric field gradient field from the dynamic electric field interference spectrum. Based on this electric field constraint term, a nonlinear observation equation is established. The expression of the observation equation is as follows:

[0039]

[0040] in, For the observation equation, The coordinates of the feature point cloud observed by the lidar. These are the pixel coordinates of the feature points observed by the visual camera. The target temperature and location as observed by the infrared thermal imager. For the electric field gradient constraint term, The electric field gradient is read from the electric field gradient field of the dynamic electric field interference pattern. This represents the displacement of the triaxial electric field sensor array between two observations. This represents the measured change in electric field intensity at the positions before and after displacement of the triaxial electric field sensor array. Let X be a nonlinear observation function that maps the state vector X to the observation space, and v be the observation noise. The extended Kalman filter prediction and update iteration are performed, and the optimal timestamp deviation of the lidar, vision camera and infrared thermal imager is solved by electric field gradient constraint residual optimization. The output is time-synchronized laser point cloud, visual image and infrared temperature distribution map.

[0041] Specifically, the state vector contains the robot's three-dimensional position, attitude angle, and velocity, as well as the timestamp deviations of the multi-source sensors. The observation information corresponding to the robot's three-dimensional position, heading attitude angle, and velocity is output by the robot's onboard motion sensing unit. This motion sensing unit includes, but is not limited to, inertial measurement units, wheeled odometry, and laser SLAM localization modules. The relevant sensing and localization methods are existing technologies and will not be elaborated upon here. Based on the current state vector, the displacement of the three-axis electric field sensor array between two adjacent observations is calculated recursively. ; Displace Substituting the electric field gradient constraint term, we obtain the residual between the theoretical change in the electric field and the measured change in the electric field. residual As an independent observation component, it is incorporated into the nonlinear observation equation and participates in the EKF update iteration to achieve joint estimation of robot motion state and multi-sensor timestamp deviation.

[0042] As an optional implementation, the step of performing extended Kalman filter prediction and update iteration, using electric field gradient constrained residual optimization to solve for the optimal timestamp deviation of the lidar, visual camera, and infrared thermal imager, and outputting time-synchronized laser point cloud, visual image, and infrared temperature distribution map includes: Based on the optimal state estimate and posterior covariance matrix of the previous period as input, a state transition function is constructed to complete the prior prediction of the state vector of the current period and obtain the prior state estimate. The state transition Jacobian matrix is ​​solved synchronously, and the prior covariance matrix is ​​obtained by updating the superimposed process noise covariance matrix. Read the electric field intensity in the electric field intensity distribution field, and adaptively adjust the weight of the electric field constraint term in the extended Kalman filter covariance matrix according to the electric field intensity; Substitute the observation vector containing the electric field gradient constraint term in the current period into the nonlinear observation mapping function, calculate the observation prediction value and the observation residual, solve the observation Jacobian matrix and calculate the Kalman gain. The prior state estimate is corrected using the Kalman gain and the observation residual. The three timestamp deviation variables in the prior state estimate are iteratively optimized and updated to obtain the optimal posterior state estimate and optimal timestamp deviation for the current period. The posterior covariance matrix is ​​updated synchronously. The original timestamps of the lidar, visual camera, and infrared thermal imager are compensated frame by frame using the optimal timestamp deviation, and time-synchronized laser point cloud, visual image, and infrared temperature distribution map are output.

[0043] Specifically, the weight of the electric field constraint term in the extended Kalman filter covariance matrix is ​​adaptively adjusted according to the electric field strength. For example, in the strong interference region, i.e., when the electric field strength is >30kV / m, the weight of the electric field constraint term is 0.4; in the medium interference region, i.e., when the electric field strength is 10-30kV / m, the weight of the electric field constraint term is 0.25; and in the weak interference region, i.e., when the electric field strength is <100kV / m, the weight of the electric field constraint term is 0.1.

[0044] In the state prediction phase, the optimal state estimate of the previous period, the posterior covariance matrix of the previous period, and the state vector at the current moment are obtained. Prior prediction is then performed based on the state vector to obtain the prior state estimate, expressed as:

[0045] in, For prior state estimation, This is an estimate of the optimal state for the previous period. This is the state transition function; Next, the state transition Jacobian matrix is ​​calculated, and the process noise covariance matrix is ​​introduced to update the prior covariance matrix. The prior covariance matrix can characterize the estimation uncertainty of all variables (including the three timestamp biases) in the current prior state, serving as the basis for subsequent observation updates. The expression for the prior covariance matrix is:

[0046] in, The prior covariance matrix, Let Jacobian be the state transition matrix. The posterior covariance matrix of the previous period. The process noise covariance matrix; Next, in the observation update phase, the electric field strength in the dynamic electric field interference spectrum is read first, and an adaptive mapping relationship between the noise variance of the electric field constraint term and the electric field strength is pre-established. If the electric field strength value is large, it indicates that the electric field is distorted and the interference is strong, so the weight of the electric field constraint term in the observation covariance is increased; if the electric field strength value is close to 0, it indicates that the electric field is stable and distorted, so the weight of the electric field constraint term is reduced. Based on the noise variance corresponding to the observations of lidar, visual camera, and infrared thermal imager and the noise variance of the electric field constraint term after adaptive update of the electric field strength, the observation noise covariance matrix is ​​obtained. Substituting the prior state estimate into the nonlinear observation function, we obtain the predicted observation vector, expressed as:

[0047] in, To predict the observation vector; The observation residual is calculated based on the predicted observation vector and the measured observation vector for the current period, and the expression is as follows:

[0048]

[0049] in, To observe the residuals, This represents the measured observation vector for the current period. Next, the observation Jacobian matrix is ​​solved to establish a linear mapping relationship between the state space and the observation space. Then, the Kalman gain is calculated based on the prior covariance matrix and the observation Jacobian matrix, expressed as:

[0050] in, For Kalman gain, To observe the Jacobian matrix, To observe the noise covariance matrix; Using Kalman gain and observation residuals, the prior state estimate is corrected to obtain the posterior optimal state estimate, expressed as:

[0051] in, For posterior optimal state estimation; During the process of correcting the prior state estimate, the observation residuals will drive... The three variables converge iteratively, continuously reducing the residual of the electric field gradient constraint until the estimated value of the timestamp deviation tends to stabilize; The posterior covariance matrix is ​​updated synchronously, and the expression is:

[0052] in, The posterior covariance matrix is... It is the identity matrix; The prior state estimate is corrected using the Kalman gain and the observation residual. The three timestamp bias variables in the state vector are updated iteratively and the posterior covariance matrix is ​​updated synchronously. After iterative optimization, the posterior state estimate is finally output, which is the optimal state vector. The optimal timestamp bias is extracted from the optimal state vector. Because lidar, vision cameras, and infrared thermal imagers have inherent clock offsets, their original timestamps will have discrepancies at the same physical moment. Therefore, by solving for the optimal timestamp deviation, frame-by-frame compensation is performed on the original timestamps, mapping the output data of lidar, vision cameras, and infrared thermal imagers to a unified global time reference. This results in synchronized timestamps aligned to the unified reference. These synchronized timestamps are then bound to the corresponding frame data, forming synchronized lidar, vision camera, and infrared thermal imager time data with unified time sequence markings, thus achieving time sequence alignment of the laser point cloud, visual image, and infrared temperature distribution map. By utilizing extended Kalman filtering based on electric field vector constraints, the timestamp deviations of lidar, vision cameras, and infrared thermal imagers can be effectively corrected, achieving a timestamp correction accuracy of ±0.5ms. In strong electric field environments, the timestamp drift of traditional methods can reach ±5ms.

[0053] As an optional implementation, step S33 includes: By introducing an electric field gradient consistency constraint, an objective function is constructed. The spatial registration extrinsic parameters are then obtained by solving the objective function. The expression for the objective function is as follows:

[0054]

[0055] in, Let be the objective function. For laser point clouds, For visual feature points projected onto a point in three-dimensional space, Let be a rotation matrix. It is a translation vector. and These are the electric field gradients at the locations of the lidar and the vision camera, respectively. These are the weighting coefficients. This is the standard ICP geometric registration loss term.

[0056] Specifically, since the laser point cloud and the visual semantic image are in different coordinate systems, they need to be transformed into a unified coordinate system to facilitate subsequent defect identification. Therefore, by introducing electric field spatial gradient constraints, the objective function is solved using the ICP algorithm with electric field spatial gradient constraints. For example, the iteration threshold of the ICP algorithm is set to 0.02 mm, and the maximum number of iterations is 50, thereby iteratively finding the optimal rotation matrix and translation vector. Based on the found optimal rotation matrix and translation vector, spatial registration is achieved. It should be noted that due to the presence of time-varying electric field distortions such as high voltage and leakage current in the robot's operating environment, laser ranging and visual imaging reference offsets can occur independently. Since the electric field gradient is determined solely by spatial position and is unaffected by sensor measurement offsets, this technical solution introduces an electric field gradient consistency constraint into the objective function. This constraint ensures the consistency between the spatially registered laser point cloud and the visual image in the electric field space; that is, the electric field gradient should be continuously changing in space, thereby improving the accuracy of spatial registration. Furthermore, the electric field gradient consistency constraint can counteract registration drift caused by electric field disturbances, ensuring stable registration results in strong electric field scenarios. Simultaneously, by transforming the laser point cloud into the visual camera coordinate system, the L2 norm of the electric field gradient vectors at the positions of the LiDAR and the visual camera is used as the consistency loss. This allows for the simultaneous minimization of geometric errors and gradient vector errors during the iterative solution of the objective function, forcing the rotation matrix and translation vector of the LiDAR and visual camera coordinate systems to converge to the true transformation values, thus reducing spatial registration offset errors.

[0057] Step S31 ensures that the data from the three types of sensors—LiDAR, visual camera, and infrared thermal imager—are aligned in time. Step S32 yields a visual semantic image. Steps S33-S35 transform the LiDAR point cloud into the visual camera coordinate system based on the time-aligned data, so that the LiDAR point cloud and the visual semantic image are in the same coordinate system. They also transform the temperature distribution map into the visual camera coordinate system, thereby achieving joint spatial registration of the three types of sensor data.

[0058] As an optional implementation, the identification result obtained by using an improved YOLOv8 model to identify target defects based on the spatiotemporally consistent multi-source data and the dynamic electric field interference spectrum includes: The improved YOLOv8 model includes an input layer, a backbone layer, a neck layer, and a head layer connected in sequence. The input layer is used to input dimensions of... The four-channel fused image tensor has the first three channels being the RGB image of the visual image, and the fourth channel being the grayscale image of the electric field intensity generated by perspective interpolation based on the laser point cloud and the electric field intensity distribution field. The Backbone layer is a CSPDarknet backbone network adapted to four 4-channel inputs. It receives the four-channel fused image tensor and performs convolution downsampling layer by layer to obtain multi-scale feature maps. The Neck layer includes a feature pyramid structure and an electric field attention module. The electric field attention module calculates electric field attention weights based on the electric field intensity grayscale image, and multiplies these weights element-wise with the multi-scale feature map to adaptively enhance features in strong electric field regions, resulting in an enhanced electric field region feature map. The feature pyramid structure fuses the enhanced electric field region feature map to obtain a multi-scale fused feature map. The formula for calculating the electric field attention weights is:

[0059] in, For electric field attention weights, and For learnable parameters, It is the Sigmoid activation function. This is a grayscale image of the electric field intensity. The Head layer includes two output branches. One output branch outputs the target defect category, confidence level, and candidate defect bounding box of the transmission line based on the multi-scale fusion feature map. The other output branch outputs the electric field interference level based on the multi-scale fusion feature map.

[0060] Specifically, the process involves reading the spatiotemporally consistent multi-source data obtained after confidence-weighted fusion in step S35, extracting visual images and laser point cloud maps from the spatiotemporally consistent multi-source data; acquiring the extrinsic parameters of the visual camera obtained in step S33 through electric field gradient constraint ICP spatial registration, where the extrinsic parameters include rotation matrices and translation vectors, and simultaneously reading the intrinsic parameters of the visual camera obtained from offline calibration; reading the electric field intensity distribution field in the dynamic electric field interference spectrum output by the electric field sensing module; back-projecting the 3D point cloud map onto the imaging plane of the visual camera to determine the 3D spatial depth coordinates corresponding to each pixel of the image; using the camera intrinsic parameters and the rotation matrix and translation vector obtained from registration in S33, performing a perspective projection transformation on the electric field intensity distribution field to map the electric field grid points in the robot coordinate system to the 2D image pixel plane; performing spatial interpolation on pixels without direct electric field sampling points after projection, and normalizing the electric field intensity corresponding to each pixel to the 0~1 interval, generating a RGB image. A single-channel electric field intensity grayscale image with perfectly aligned image resolution and pixel coordinates is generated. The three-channel image of the visual image and the electric field intensity grayscale image are stacked and stitched along the channel dimension to generate a four-channel fused image tensor with size H×W×4, which is then input to the YOLOV8 network input. By processing the input data using an improved YOLOv8, the final identification result of the power transmission cable is obtained. Specifically, the input channels are improved from the traditional three channels to four channels, and a new electric field physical field channel matching the RGB pixels is added. This channel is not a simple superposition of grayscale images, but is generated by perspective projection interpolation based on spatial registration intrinsic and extrinsic parameters, 3D point cloud map, and electric field intensity distribution. This ensures that the grayscale electric field intensity images of the four channels are accurately aligned with the pixels of the visual image. It also allows the network to achieve cross-modal alignment without additional learning data, reducing the difficulty of model training. At the same time, by using both the RGB image and the electric field intensity grayscale image as input features into the improved YOLOv8, the model can simultaneously learn the target area and the electric field distortion features at the target area, providing a basis for subsequent defect identification. By improving the Neck layer and setting an electric field attention module in the Neck layer, the electric field attention weights are generated using only the electric field attention module. This can amplify the image features of areas with strong electric field distortion, thereby adaptively suppressing irrelevant backgrounds and enhancing dangerous defect features. At the same time, strong electric fields can cause problems such as image glare and texture blurring, which can easily reduce the accuracy of pure vision models in defect detection and recognition. Based on the electric field attention weights, feature enhancement can be performed independently of visual features, without being affected by visual features. By improving the output branches of the Head and adding an output branch for the electric field interference level, the model can not only identify defects, but also simultaneously determine the electric field interference hazard level corresponding to the defect, thus quantifying the risk.

[0061] This also includes: training the improved YOLOv8 model by constructing a strong electric field environment dataset containing defect samples under different electric field strengths (10-50 kV / m), totaling 50,000 images covering 15 typical defect types; and optimizing the YOLOv8 model using a multi-task learning loss function, the expression of which is:

[0062] in, For multi-task learning loss function, For classifying losses, For bounding box regression loss, To predict losses based on electric field strength, This is the weighting coefficient (value 0.3). The training process employs data augmentation strategies, including random cropping, rotation, brightness adjustment, and electric field intensity perturbation (adding ±10% random noise to the original electric field value) to improve the network's robustness to changes in the electric field.

[0063] The step of performing joint verification on the identification results to obtain the final target defect identification result includes: Based on the visual semantic image, semantic verification is performed, the pixel ratio of power components in each candidate defect bounding box is counted, and candidate defect bounding boxes with a pixel ratio of power components lower than a preset threshold are removed to obtain intermediate candidate defect bounding boxes. The intermediate candidate defect bounding boxes are classified and determined. If the target defect category of the intermediate candidate defect bounding box is a heat-generating defect, temperature verification is performed. The temperature of the corresponding region is extracted based on the temperature distribution map. The region temperature is compared with a preset temperature threshold. If the region temperature reaches the preset temperature threshold, it is retained; otherwise, it is discarded. If the target defect category of the intermediate candidate defect bounding box is a non-heating defect, then temperature verification is not performed, and the intermediate candidate defect bounding box is retained. Based on all retained intermediate candidate defect bounding boxes, the final target defect identification result is output.

[0064] Specifically, when using the YOLOv8 model for defect detection, the input is a visual image and a grayscale image of the electric field intensity. This means that the YOLOv8 model considers both visual and electric field features simultaneously to make a judgment. However, in some cases, it cannot accurately identify whether an object within the electric field range is a power component. For example, the electric field of high-voltage equipment has a radiation range, and distant rocks, metal debris, and concrete structures of towers can be sensed by a weak electric field. At the same time, the texture and shape may resemble a defect, so a high-confidence defect candidate box will be output. However, in reality, rocks are not power components and are unlikely to be faulty in power transmission equipment, which can easily lead to identification errors.

[0065] Therefore, to improve the accuracy of defect identification, semantic verification is performed using visual semantic images. The coordinates of the candidate defect bounding boxes output by the Head are mapped to a visual semantic image of the same resolution. All pixels within the rectangle enclosed by the candidate defect bounding boxes are traversed, and the number of effective pixels belonging to electrical components (wires, insulators, hardware, etc.) is counted. The total number of pixels within the defect bounding box is also counted. The ratio of these two values ​​is calculated to obtain the pixel proportion of the electrical component. This proportion is compared with a preset threshold. If the proportion is less than the threshold, the candidate box is determined to be located in the background area, which may be a misjudged defect. Therefore, the candidate defect bounding box is removed. Intermediate candidate defect bounding boxes are obtained based on the results of this sequential traversal. Using the second... Spatial registration extrinsic parameters are used to project the intermediate candidate defect bounding boxes onto the pixel-aligned temperature distribution map. At this point, the defects corresponding to the intermediate candidate defect bounding boxes may be either heat-generating or non-heat-generating. Therefore, for targets with heat-generating defects, temperature verification is performed. If the verification passes, it means that the defect category is indeed heat-generating, and the intermediate candidate defect bounding box is retained; otherwise, it is discarded. If the target defect category of the intermediate candidate defect bounding box is a non-heat-generating defect, temperature verification is not performed. Based on semantic and temperature verification, the remaining defect bounding boxes are obtained, and their corresponding target defect category, confidence level, and electric field interference level are output together to obtain the final target defect identification result.

[0066] It should be noted that although both the improved YOLOv8 model and the semantic segmentation network utilize visual features in their recognition results and visual semantic images, their roles and learning objectives differ significantly. The improved YOLOv8 model focuses on identifying local abnormal defects on the surface of components, lacking global scene perception capabilities, and is prone to misclassifying background objects with similar textures within the electric field range as defects. The semantic segmentation network achieves pixel-level segmentation of electric components and background regions, distinguishing targets from the global contour dimension. In other words, using the semantic segmentation results to verify defect candidate boxes essentially introduces prior constraints on component space, filtering out candidate defect bounding boxes that fall into non-component regions, rather than simply using similar features for self-verification. On this basis, temperature, as an independent physical mode, is further introduced for secondary verification of heat-generating defects, achieving joint constraints of multi-dimensional information and effectively improving the accuracy of defect recognition.

[0067] The three-dimensional coordinates of the defect are extracted based on the final defect identification result for three-dimensional localization. Specifically, the three-dimensional localization is performed based on the defect bounding box obtained after semantic and temperature verification and the three-dimensional point cloud map obtained in step S34. The three-dimensional coordinates of the defect are extracted. The specific steps are as follows: (1) Back projection: Based on the camera intrinsic parameters, each pixel in the defect bounding box is back projected onto a ray of the three-dimensional point cloud map; (2) Point cloud matching: Using the nearest neighbor search algorithm, the point cloud cluster closest to the ray is found in the three-dimensional point cloud map; (3) Geometric feature extraction: The geometric features of the normal vector and curvature of the matched point cloud cluster are extracted, and a curvature threshold is set. The high curvature point cloud belonging to the defect edge is filtered out by the curvature threshold; (4) PCA fitting: Principal component analysis (PCA) is performed on the filtered high curvature point cloud to fit the main direction and size of the defect. Finally, the three-dimensional coordinates of the center point of the defect are calculated. The error of the three-dimensional coordinates of the center point is less than 3mm, that is, the three-dimensional coordinates of the target defect are obtained.

[0068] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration, characterized in that, include: The electric field sensing module is used to synchronously collect the triaxial electric field components at each measuring point through a triaxial electric field sensor array deployed on the robot body, and generate a dynamic electric field interference map centered on the robot based on the triaxial electric field components. The dynamic electric field interference map includes: electric field intensity distribution field and electric field gradient field. The sensor adaptive module is used to dynamically adjust the working parameters of the multi-source sensors on the robot body according to the electric field intensity distribution field to obtain the corrected working parameters. The multi-source sensors collect multi-source data according to the corrected working parameters. The multi-source data includes: laser point cloud, visual image and temperature distribution map. The spatiotemporal registration and fusion module is used to perform time synchronization and spatial registration on the multi-source data to generate spatiotemporally consistent multi-source data; The defect identification module is used to identify target defects based on the spatiotemporally consistent multi-source data and the dynamic electric field interference spectrum, using an improved YOLOv8 model to obtain identification results, perform joint verification on the identification results to obtain the final target defect identification result, and perform three-dimensional positioning based on the final target defect identification result.

2. The adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration according to claim 1, characterized in that, The process involves synchronously acquiring the triaxial electric field components at each measuring point using a triaxial electric field sensor array deployed on the robot body, and generating a dynamic electric field interference map centered on the robot based on the triaxial electric field components, including: Step S11: Collect the triaxial electric field intensity components of each triaxial electric field sensor on the robot body, calculate the original synthetic electric field intensity vector of each measuring point, and obtain a discrete multi-measuring-point electric field dataset; Step S12: Filter the discrete multi-point electric field dataset time-by-time using Kalman filtering to obtain a multi-point electric field intensity vector dataset. Step S13: Obtain the robot's current position information in real time, and use the Kriging interpolation method to perform spatial interpolation on the multi-point electric field intensity vector dataset to obtain a three-dimensional electric field vector grid that is continuously distributed in the space around the robot. Step S14: Calculate the electric field intensity distribution field and electric field gradient field based on the three-dimensional electric field vector grid, and fuse them to form a dynamic electric field interference map.

3. The adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration according to claim 1, characterized in that, The process involves dynamically adjusting the operating parameters of the multi-source sensors on the robot body according to the electric field intensity distribution to obtain calibrated operating parameters. The multi-source sensors then collect multi-source data based on these calibrated operating parameters, including: The multi-source sensors include: a lidar mounted on the top of the robot, a vision camera mounted on the front of the robot, and an infrared thermal imager mounted on the side of the robot. Step S21: Set up a three-level hierarchical strategy; Step S22: Obtain the real-time synthesized electric field intensity based on the electric field intensity distribution field, and adjust the working parameters of the multi-source sensors on the robot body based on the real-time synthesized electric field intensity and the three-level hierarchical strategy to obtain the corrected working parameters; Step S23: The multi-source sensor is calibrated according to the calibration working parameters, and multi-source data is collected after calibration.

4. The adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration according to claim 3, characterized in that, The three-level hierarchical strategy in step S21 includes: Set a first threshold and a second threshold; When the electric field strength is greater than the first threshold, the lidar is controlled to switch to the first wavelength and the scanning frequency is increased, the vision camera is controlled to enable polarization filtering and high dynamic range mode, and the gain of the infrared thermal imager is reduced. When the electric field strength is between the first threshold and the second threshold, the lidar is controlled to use the first wavelength and a scanning frequency between the standard value and the boost value, the vision camera is controlled to enable the high dynamic range mode, and the gain of the infrared thermal imager is partially reduced. When the electric field strength is less than the second threshold, the lidar, the vision camera, and the infrared thermal imager are controlled to use standard operating parameters.

5. The adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration according to claim 1, characterized in that, The step of performing time synchronization and spatial registration on the multi-source data to generate spatiotemporally consistent multi-source data includes: Step S31: Based on the electric field vector constraint, the extended Kalman filter is used to correct the timestamp deviation of the lidar, visual camera and infrared thermal imager to obtain time-aligned laser point cloud, visual image and temperature distribution map. Step S32: Perform semantic recognition on the temporally aligned visual image and output a visual semantic image with pixel-level semantic labels; Step S33: Using the ICP algorithm constrained by the electric field spatial gradient, perform joint spatial registration on the time-aligned laser point cloud and the visual image to obtain the first spatial registration extrinsic parameters; Step S34: Based on the first spatial registration extrinsic parameters, the time-aligned laser point cloud, and the robot's real-time pose, the laser point cloud is uniformly converted to the visual camera coordinate system to construct a three-dimensional point cloud map aligned with the pixel space of the visual image. Step S35: Solve the second spatial registration extrinsic parameters of the infrared thermal imager relative to the visual camera using the checkerboard offline calibration method. Based on the second spatial registration extrinsic parameters, map and align the time-aligned temperature distribution map to the visual camera coordinate system to obtain a temperature distribution map aligned with the visual image space.

6. The adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration according to claim 5, characterized in that, Step S31 includes: Define a state vector, the expression of which is: in, For the robot's three-dimensional position coordinates, The robot's heading attitude angle, For the robot's movement speed, These are the timestamp deviations of the lidar, visual camera, and infrared thermal imager, respectively. An electric field constraint term is constructed by introducing the electric field gradient field from the dynamic electric field interference spectrum. Based on this electric field constraint term, a nonlinear observation equation is established. The expression of the observation equation is as follows: in, For the observation equation, The coordinates of the feature point cloud observed by the lidar. These are the pixel coordinates of the feature points observed by the visual camera. The target temperature and location as observed by the infrared thermal imager. For the electric field gradient constraint term, For the electric field gradient, This represents the displacement of the triaxial electric field sensor array between two observations. This represents the measured change in electric field intensity at the positions before and after displacement of the triaxial electric field sensor array. Here, v is the nonlinear observation function, and v is the observation noise. The extended Kalman filter prediction and update iteration are performed, and the optimal timestamp deviation of the lidar, vision camera and infrared thermal imager is solved by electric field gradient constraint residual optimization. The output is time-synchronized laser point cloud, visual image and infrared temperature distribution map.

7. The adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration according to claim 6, characterized in that, The process involves performing extended Kalman filter prediction and update iterations, utilizing electric field gradient-constrained residual optimization to solve for the optimal timestamp deviation of the lidar, visual camera, and infrared thermal imager, and outputting time-synchronized laser point clouds, visual images, and infrared temperature distribution maps, including: Based on the optimal state estimate and posterior covariance matrix of the previous period as input, a state transition function is constructed to complete the prior prediction of the state vector of the current period and obtain the prior state estimate. The state transition Jacobian matrix is ​​solved synchronously, and the prior covariance matrix is ​​obtained by updating the superimposed process noise covariance matrix. Read the electric field intensity in the electric field intensity distribution field, and adaptively adjust the weight of the electric field constraint term in the extended Kalman filter covariance matrix according to the electric field intensity; Substitute the observation vector containing the electric field gradient constraint term in the current period into the nonlinear observation mapping function, calculate the observation prediction value and the observation residual, solve the observation Jacobian matrix and calculate the Kalman gain. The prior state estimate is corrected using the Kalman gain and the observation residual. The three timestamp deviation variables in the prior state estimate are iteratively optimized and updated to obtain the optimal posterior state estimate and optimal timestamp deviation for the current period. The posterior covariance matrix is ​​updated synchronously. The original timestamps of the lidar, visual camera, and infrared thermal imager are compensated frame by frame using the optimal timestamp deviation, and time-synchronized laser point cloud, visual image, and infrared temperature distribution map are output.

8. The adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration according to claim 5, characterized in that, Step S33 includes: By introducing an electric field gradient consistency constraint, an objective function is constructed. The spatial registration extrinsic parameters are then obtained by solving the objective function. The expression for the objective function is as follows: in, Let be the objective function. For laser point clouds, For visual feature points projected onto a point in three-dimensional space, Let be a rotation matrix. It is a translation vector. and These are the electric field gradients at the locations of the lidar and the vision camera, respectively. These are the weighting coefficients. This is the standard ICP geometric registration loss term.

9. The adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration according to claim 5, characterized in that, The target defect identification result, obtained by using the improved YOLOv8 model based on the spatiotemporally consistent multi-source data and the dynamic electric field interference spectrum, includes: The improved YOLOv8 model includes an input layer, a backbone layer, a neck layer, and a head layer connected in sequence. The input layer is used to input dimensions of... The four-channel fused image tensor has the first three channels being the RGB image of the visual image, and the fourth channel being the grayscale image of the electric field intensity generated by perspective interpolation based on the laser point cloud and the electric field intensity distribution field. The Backbone layer is a CSPDarknet backbone network adapted to four-channel input. It receives the four-channel fused image tensor and performs convolution downsampling layer by layer to obtain multi-scale feature maps. The Neck layer includes a feature pyramid structure and an electric field attention module. The electric field attention module calculates electric field attention weights based on the electric field intensity grayscale image, multiplies the electric field attention weights element-wise with the multi-scale feature map, and adaptively enhances the features of the strong electric field region to obtain an enhanced electric field region feature map. The feature pyramid structure fuses the enhanced electric field region feature map to obtain a multi-scale fused feature map. The Head layer includes two output branches. One output branch outputs the target defect category, confidence level, and candidate defect bounding box of the transmission line based on the multi-scale fusion feature map. The other output branch outputs the electric field interference level based on the multi-scale fusion feature map.

10. The adaptive fusion sensing system based on electric field sensing and multimodal spatiotemporal registration according to claim 9, characterized in that, The step of performing joint verification on the identification results to obtain the final target defect identification result includes: Based on the visual semantic image, semantic verification is performed, the pixel ratio of power components in each candidate defect bounding box is counted, and candidate defect bounding boxes with a pixel ratio of power components lower than a preset threshold are removed to obtain intermediate candidate defect bounding boxes. The intermediate candidate defect bounding boxes are classified and determined. If the target defect category of the intermediate candidate defect bounding box is a heat-generating defect, temperature verification is performed. The temperature of the corresponding region is extracted based on the temperature distribution map. The region temperature is compared with a preset temperature threshold. If the region temperature reaches the preset temperature threshold, it is retained; otherwise, it is discarded. If the target defect category of the intermediate candidate defect bounding box is a non-heating defect, then temperature verification is not performed, and the intermediate candidate defect bounding box is retained. Based on all retained intermediate candidate defect bounding boxes, the final target defect identification result is output.

Images