Live working method and system based on fusion of laser radar and visual recognition

Abstract

This invention discloses a method and system for live-line working based on the fusion of lidar and visual recognition. The method includes: transforming lidar point cloud data into coordinates and calculating the weighted height difference using an extrinsic parameter optimization strategy, then performing adaptive weighted fusion with switch images through different feature extraction branches; registering the power equipment point cloud and environmental point cloud in the lidar point cloud data using an IMLS-ICP algorithm based on improved intensity consistency, and using a lidar-visual inertial odometry fusion algorithm for pose estimation and optimization to obtain the globally optimal pose and construct a three-dimensional environment model based on it; using the globally optimal pose as the starting point, the switch position as the ending point, and the three-dimensional environment model as the map, within the safe area defined by electric field simulation on the map, planning an obstacle avoidance path using an improved A* algorithm including electric field strength constraints, and controlling a robotic arm to perform gear shifting operations according to the obstacle avoidance path. This invention improves the safety and efficiency of live-line working.

Description

Technical Field

[0001] This invention relates to the control technology of live-line working robots for power equipment, specifically to a live-line working method and system based on the fusion of lidar and visual recognition. Background Technology

[0002] Live-line working is a crucial means of ensuring power supply reliability in power system operation and maintenance. Traditional live-line working mainly relies on manual labor, which presents problems such as high operational risks, high labor intensity, and low efficiency. With the development of robotics technology, live-line working robots are gradually being applied in power maintenance scenarios, but current live-line working robots still have significant shortcomings in environmental perception:

[0003] 1. Limited perception capability of a single sensor: Traditional live-line working robots mostly use a single vision sensor or lidar. The recognition accuracy of vision sensors decreases under strong light, shadow or complex lighting conditions, while lidar lacks the ability to recognize the color and status of equipment, and cannot meet the needs of judging the status of equipment in live-line work.

[0004] 2. Insufficient environmental modeling and positioning accuracy: The live-line working environment contains complex structures such as high-voltage equipment, insulators, and cables. A single sensor is insufficient to construct an accurate three-dimensional environmental model, resulting in large positioning errors for the live-line working robot and making it unable to complete precise operations.

[0005] 3. Weak resistance to electromagnetic interference: Strong electromagnetic interference exists in high-voltage energized environments. Traditional sensor systems are easily affected by interference during data acquisition and processing, leading to distorted sensing data and affecting operational safety.

[0006] 4. Poor adaptability to operation: Existing systems are difficult to adapt to complex scenarios (such as crossing lines), which limits the application scope of live-line working robots. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a method and system for live-line work based on the fusion of lidar and visual recognition, which improves the environmental perception capability of live-line work robots in live-line environments through multi-sensor data fusion, realizes high-precision positioning, obstacle detection and equipment status recognition, and improves the safety and efficiency of live-line work.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] A live-line working method based on the fusion of lidar and visual recognition, the method being applied to a live-line working robot, the live-line working robot being equipped with a lidar unit, a camera unit, an IMU unit, and a robotic arm, the method comprising the following steps:

[0010] S1. Real-time acquisition of lidar point cloud data, power equipment image data, and IMU data, and performance of data preprocessing;

[0011] S2. The preprocessed LiDAR point cloud data is transformed from the LiDAR coordinate system to the camera coordinate system through an extrinsic parameter optimization strategy, thereby projecting the point cloud data onto the image plane. Then, the weighted height difference is calculated for each pixel in the projected image to obtain the LiDAR weighted height difference map. Finally, the LiDAR weighted height difference map and the preprocessed power equipment image are respectively extracted through the corresponding feature extraction branches to obtain corresponding feature maps. The feature maps are then fused through adaptive weights to obtain a fused feature map.

[0012] S3. The IMLS-ICP algorithm based on intensity consistency improvement is used to register the power equipment point cloud and the environmental point cloud in the preprocessed lidar point cloud data to obtain the point cloud registration result. At the same time, the lidar-visual inertial odometry fusion algorithm is used for pose estimation and optimization. Specifically, it includes: first, the pose prediction of the live-line working robot is performed based on the preprocessed IMU data; then, the pose prediction result is corrected for ESKF error based on the power equipment features of the fused feature map. The state vector during ESKF error correction includes the IMU zero bias error term caused by electromagnetic interference; then, the ESKF error correction result and the point cloud registration result are fused and optimized to obtain the global optimal pose of the live-line working robot, and a three-dimensional environment model is constructed based on the global optimal pose.

[0013] S4. Using the global optimal pose of the live-line working robot as the starting point, the location of the target power equipment as the ending point, and the three-dimensional environment model as the map, within the safe area divided according to the electric field simulation in the map, an improved A* algorithm with an electric field strength constraint term in the cost function is used to plan the obstacle avoidance path, and the robotic arm is controlled to perform the gear shifting operation according to the planned obstacle avoidance path.

[0014] Furthermore, in step S2, when transforming the preprocessed LiDAR point cloud data from the LiDAR coordinate system to the camera coordinate system using an extrinsic parameter optimization strategy, a loss function containing a double-constraint matching of the power equipment feature center (such as the switch center) and the slot edge is constructed. Then, the rotation matrix and translation vector in the loss function are optimized to achieve extrinsic parameter calibration. Finally, based on the calibrated extrinsic parameters, the preprocessed LiDAR point cloud data is transformed from the LiDAR coordinate system to the camera coordinate system. The mathematical expression of the loss function is as follows:

[0015]

[0016] in, It is a rotation matrix; It is a translation vector; For the first extracted from the image The pixel coordinates of the center of the switch circle; For the corresponding point cloud of lidar Three-dimensional coordinates of the center of the switch; For the first extracted from the image The pixel coordinates of the edge points of each slot; For the corresponding point cloud of lidar Three-dimensional coordinates of the edge points of each slot; Represents the camera projection function; These are the weighting coefficients; These represent the number of matching power equipment feature centers and edge points, respectively.

[0017] Furthermore, in step S2, when calculating the weighted height difference for each pixel in the projected image, the mathematical expression is as follows:

[0018]

[0019] in, For pixels ( x,y The weighted height difference; This represents the height value corresponding to the target pixel. For the neighboring region The height value corresponding to each point; This represents the number of neighboring points; These are adaptive weighting coefficients; Let be the coordinates of the neighboring points; the mathematical expression for the adaptive weighting coefficients is as follows:

[0020]

[0021] Among them, Sigmoid ( ) is the Sigmoid activation function.

[0022] Further, in step S2, when extracting features from the lidar weighted height difference map and the preprocessed power equipment image through their respective feature extraction branches to obtain corresponding feature maps, and then fusing the feature maps using adaptive weights to obtain a fused feature map, the process includes:

[0023] The weighted height difference map from the LiDAR is input into the geometry branch for edge enhancement and contour extraction to obtain the geometric feature map. The edge enhancement and contour extraction specifically include: calculating the gradient magnitude of the lidar weighted height difference map. Binary contour maps are obtained by nonmaximum suppression and double thresholding. Generate geometric feature map ,in This is the contour enhancement coefficient.

[0024] The preprocessed power equipment image data is input into the semantic branch for convolutional feature extraction to obtain a semantic feature map. The semantic branch is a lightweight convolutional neural network. The input is an RGB image, which is sequentially passed through a convolutional layer with a 3×3 kernel and a ReLU activation function, and a 2×2 max pooling layer for feature extraction, finally outputting a semantic feature map.

[0025] For the geometric feature map Perform a linear transformation to make it compatible with the semantic feature map. The feature space alignment is expressed mathematically as follows:

[0026]

[0027] in, The transformed geometric features; The original geometric features of the weighted height difference map for lidar; The linear transformation parameters learned by the neural network are expressed mathematically as follows:

[0028]

[0029] in, It is a convolutional network; For network weights, Semantic features of power equipment images;

[0030] The transformed geometric features and semantic feature maps are fused using adaptive weights to obtain a fused feature map, as shown in the following mathematical expression:

[0031]

[0032] In the above formula, The mathematical expression for the dynamically fused weighting coefficients is as follows:

[0033]

[0034] in, For the Sigmoid function; These are network parameters.

[0035] Furthermore, in step S2, after obtaining the fused feature map, there is also a step of anti-interference processing of the fused feature map. Specifically, the median filtering window is dynamically adjusted according to the real-time electromagnetic interference spectrum to filter the fused feature map, and then a Kalman filter is used for state estimation, and its observation noise covariance matrix is ​​updated in real time.

[0036] Furthermore, step S2 also includes online accuracy evaluation and local recalibration steps, specifically including: calculating the average rotation error and average translation error based on the rotation matrix and translation vector of the current extrinsic parameters respectively, and calculating the average rotation error and average translation error of the current extrinsic parameters in real time; if the error exceeds a preset threshold, the operation is paused, and lidar point cloud data and power equipment image data of the local area of ​​the current operation are collected and resubmitted into the loss function for extrinsic parameter optimization to achieve local rapid recalibration.

[0037] Furthermore, in step S3, when registering the power equipment point cloud with the environment point cloud in the preprocessed lidar point cloud data using the IMLS-ICP algorithm based on improved intensity consistency, specifically, the IMLS-ICP algorithm is used to register the power equipment point cloud with the environment point cloud in consecutive frames. Simultaneously, point cloud clustering of the slot region is extracted in real time, and the corresponding center point coordinates are calculated. When the center point displacement between adjacent frames is less than a set threshold, the current registration iteration is terminated. The mathematical expression of the weight function of the IMLS-ICP algorithm based on improved intensity consistency is as follows:

[0038]

[0039] in, , This represents a point in a point cloud surface. The height parameter from the point cloud to the surface. Point strength, The mean local intensity The standard deviation of strength.

[0040] Furthermore, in step S3, when predicting the pose of the live-line working robot based on the preprocessed IMU data and correcting the ESKF error based on the fused feature map, the specific steps include:

[0041] The preprocessed IMU data is sequentially updated using position, velocity, and attitude equations to predict pose values.

[0042] Using the three-dimensional coordinates of the power equipment feature center (such as the center of a switch) extracted from the fused feature map, along with the corresponding local point cloud average reflection intensity and visual semantic confidence, as joint observation values, an observation equation is established:

[0043]

[0044] in, The three-dimensional coordinates of the switch center are... Its local average reflection intensity, The semantic confidence score for the corresponding image location. To observe noise.

[0045] The ESKF state vector is extended to include position error, velocity error, rotation error, IMU bias error, and bias error term caused by electromagnetic interference. , construct the state equation. It is obtained by weighting the amplitude of the power frequency and its harmonic interference components in the spectrum analysis of IMU data.

[0046] ESKF update is performed based on the state equation and observation equation, and the error state correction is calculated to correct the pose prediction value. During the update process, the Jacobian matrix... The observation equations need to be calculated for all state variables (including...) The partial derivatives of ) and the observation noise covariance matrix It is a diagonal matrix containing pixel error, intensity noise, and semantic confidence variance.

[0047] Furthermore, in step S4, when planning obstacle avoidance paths using the improved A* algorithm within the safe areas defined by electric field simulation in the map, the specific steps are as follows: Obstacles are identified based on the 3D environment of the map, and electric field simulation is performed on the map to divide non-obstacle areas where the electric field strength is not greater than the safety threshold into safe areas; the starting point is used as the initial path point for iterative expansion, and during iterative expansion, the comprehensive actual path cost is calculated for each path point's neighboring nodes in the safe area. Target heuristic estimation With electric field strength constraint Cost function Among them, the electric field strength constraint term , The electric field strength at the node is... As a safety threshold, The penalty coefficient is used to select the neighboring node with the smallest cost function as the next path point until the destination is reached, thus obtaining the obstacle avoidance path.

[0048] This invention also proposes a live-line working system based on the fusion of lidar and visual recognition, including a live-line working robot equipped with a lidar unit, a camera unit, an IMU unit, and a robotic arm. The live-line working robot is programmed or configured to execute the live-line working method based on the fusion of lidar and visual recognition, including:

[0049] The data processing and fusion module is used to perform data preprocessing, external parameter optimization, LiDAR weighted height difference map calculation and feature fusion of the live-line working method based on the fusion of LiDAR and visual recognition, to obtain a fused feature map;

[0050] The localization and environment modeling module is used to perform point cloud registration, lidar-visual inertial odometry fusion, and pose graph optimization of the live-line working method based on the fusion of lidar and visual recognition, so as to obtain the global optimal pose and construct a three-dimensional environment model.

[0051] The path planning and control module is used to perform path planning based on the improved A* algorithm for the live-line working method based on the fusion of lidar and visual recognition, and to control the movement of the robotic arm.

[0052] The data processing and fusion module, the positioning and environment modeling module, and the path planning and control module are programmed or configured in the control unit of the live-line working robot.

[0053] Compared with the prior art, the advantages of the present invention are as follows:

[0054] This invention achieves high-precision environmental perception and operation control through deep fusion of multiple sensors: LiDAR and visual extrinsic parameter optimization ensure pixel-level alignment between point cloud and image; weighted height difference conversion and adaptive feature fusion take into account both the geometric contour of insulators and the visual semantics of power equipment, improving equipment recognition rate under complex lighting conditions; IMLS-ICP registration, ESKF error correction, and pose graph optimization form a tightly coupled positioning architecture of "front-end prediction - back-end optimization", maintaining centimeter-level positioning accuracy even under strong electromagnetic interference; electric field constraint path planning rigidly embeds high-voltage safety distance into the operation trajectory generation, enabling the robotic arm to autonomously avoid obstacles and accurately adjust gears in energized environments, systematically improving operation safety and efficiency. Attached Figure Description

[0055] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0056] The present invention will be further described below with reference to the accompanying drawings and specific preferred embodiments, but this does not limit the scope of protection of the present invention.

[0057] Example 1

[0058] This embodiment proposes a live-line working method based on the fusion of LiDAR and visual recognition. This method is applied to a live-line working robot equipped with a LiDAR unit, camera unit, IMU unit, and robotic arm. Combined with high-precision positioning and anti-interference technology, it significantly improves the robot's perception capability, positioning accuracy, and operational safety in complex high-voltage environments. This embodiment uses live-line tap changing of a de-energized tap changer as a typical application scenario to illustrate the implementation process of this method. The "feature center of the power equipment" specifically refers to the rotation center of the switch body. This center is represented as the geometric center of a circular marker in the image and can be obtained by fitting a cylindrical or disk model in the point cloud. For other types of equipment (such as insulators), it corresponds to key geometric feature points such as the center of its axis or the center of its mounting flange. Figure 1 As shown, the method includes the following steps:

[0059] S1: Sensor Data Acquisition and Preprocessing. Specifically, this involves real-time acquisition of LiDAR point cloud data, power equipment image data, and IMU data, followed by data preprocessing. This includes:

[0060] S1.1: LiDAR point cloud data acquisition and preprocessing. Specifically, environmental point clouds are acquired through multi-line LiDAR, and pass-through filtering and statistical filtering are performed to remove noise points. The filtered point cloud data is then used as the preprocessed LiDAR point cloud data.

[0061] S1.2: Power equipment image data acquisition and preprocessing. Specifically, images of power equipment are acquired through high-definition color cameras and depth cameras, and image correction and coarse component positioning are performed to obtain corrected color-depth image pairs. The ROI regions of components such as switches and insulators are extracted, and these images and ROI coordinates are used as preprocessed power equipment image data.

[0062] In this embodiment, image correction is specifically performed using a calibration method, establishing the transformation relationship between the world coordinate system and the pixel coordinate system based on the pinhole imaging principle:

[0063]

[0064] in the formula The camera intrinsic parameter matrix obtained by Zhang Zhengyou's calibration method; Scale factor; These are pixel coordinates; The coordinates of a three-dimensional point in the world coordinate system; Focal length (in pixels); These are the coordinates of the optical center.

[0065] Considering both radial and tangential distortion of the lens, the corrected coordinate expression is:

[0066]

[0067] in the formula These are the normalized coordinates after distortion correction.

[0068] Considering the prevalent strong light interference and equipment surface reflection issues in high-voltage environments, this embodiment adds a narrow-band filter to the camera before calibration and uses a low-reflectivity coating to treat the calibration board surface. This ensures that the calibration images maintain high contrast and low noise even in complex field environments, effectively improving the robustness of intrinsic parameter calibration under harsh lighting conditions and preventing image quality issues from affecting the accuracy of subsequent extrinsic parameter calibration.

[0069] In this embodiment, when performing coarse component localization, a pre-trained detection model (such as YOLO) is used to extract the ROI region of components such as switches and insulators.

[0070] S1.3: IMU data acquisition and preprocessing. Specifically, after acquiring IMU data, timestamp alignment and outlier removal are performed. When performing timestamp alignment, the IMU data can be linearly interpolated based on the lidar timestamp. When removing outliers, a sliding window of a specified size can be used to detect and remove outliers from the interpolated IMU data, thereby obtaining synchronized IMU data without outliers as preprocessed IMU data.

[0071] S2: Multi-sensor data fusion. Specifically, an extrinsic parameter optimization strategy is used to transform the preprocessed LiDAR point cloud data from the LiDAR coordinate system to the camera coordinate system, thereby projecting the point cloud data onto the image plane. Then, a weighted height difference is calculated for each pixel in the projected image to obtain a LiDAR weighted height difference map. Finally, the LiDAR weighted height difference map and the preprocessed power equipment image are respectively used for feature extraction through corresponding feature extraction branches to obtain corresponding feature maps. These feature maps are then fused using adaptive weights to obtain a fused feature map. Specifically, this includes:

[0072] S2.1: Spatiotemporal calibration. Specifically, when transforming the preprocessed LiDAR point cloud data from the LiDAR coordinate system to the camera coordinate system using an extrinsic parameter optimization strategy, a loss function is first constructed that incorporates a double-constraint matching between the feature center of the power equipment (the switch center in this embodiment) and the edge of the slot. Then, the rotation matrix and translation vector in the loss function are optimized to achieve extrinsic parameter calibration. Finally, based on the calibrated extrinsic parameters, the preprocessed LiDAR point cloud data is transformed from the LiDAR coordinate system to the camera coordinate system.

[0073] In this embodiment, for the no-excitation tap changer adjustment scenario, a cross-modal feature alignment extrinsic parameter optimization strategy based on LiDAR-vision is proposed. Unlike traditional general calibration methods that rely solely on calibration board features, this strategy incorporates dual-constraint matching between the switch center and the slot edge. This design can accurately solve the problems of inaccurate feature matching and easy drift in calibration accuracy caused by equipment reflection under high-voltage environments. The mathematical expression of the loss function is as follows:

[0074]

[0075] in, It is a rotation matrix; It is a translation vector; Extracting the first from the image of power equipment The pixel coordinates of the center of each power equipment feature For the corresponding point cloud of lidar Three-dimensional coordinates of the feature center of each power equipment Extracting the first from the image of power equipment pixel coordinates of the slot edge points For the corresponding point cloud of lidar Three-dimensional coordinates of the edge points of the slot. For camera projection function, These are the weighting coefficients. These represent the number of matching power equipment feature centers and slot edge points, respectively.

[0076] The Levenberg-Marquardt algorithm was used to optimize the solution of the rotation matrix R and translation vector t, ultimately controlling the rotation error to within 0.1° and the translation error to within 2mm. This level of accuracy fully meets the requirements of precision operations such as adjusting the tap changer without excitation, avoiding deviations in the operation of the robotic arm due to calibration errors.

[0077] After calculating the extrinsic parameters, the following lidar coordinate system is established. With camera coordinate system The transformation relationship is used to achieve coordinate transformation of the preprocessed lidar point cloud data:

[0078]

[0079] In the formula, These are the pixel coordinates of the image; These are the three-dimensional point coordinates in the lidar coordinate system. It is a rotation matrix; It is a translation vector; The projection matrix; Pixel scale factor; These are the coordinates of the center point of the image plane.

[0080] Compared to the traditional Zhang's calibration method, under a 110kV electromagnetic environment, the external parameters calculated in this step can effectively reduce rotational and translational errors, significantly improve calibration accuracy, and effectively enhance edge extraction accuracy.

[0081] S2.2: Feature layer fusion.

[0082] First, a weighted height difference map of the LiDAR is generated. In this embodiment, to address the sparsity problem of the LiDAR point cloud, a weighted height difference transformation method adapted to the geometric features of the power equipment is designed. When calculating the weighted height difference for each pixel in the projected image, the mathematical expression is as follows:

[0083]

[0084] in, For pixels ( x,y The weighted height difference; This represents the height value corresponding to the target pixel. For the neighboring region The height value corresponding to each point; This represents the number of neighboring points; These are adaptive weighting coefficients; These are the coordinates of a neighboring point.

[0085] Traditional point cloud conversion methods often employ fixed weight calculations, which cannot accurately match the structural characteristics of different power components. This embodiment uses the Sigmoid function to dynamically calculate adaptive weight coefficients. For slender cylindrical structures like insulators, the weights are linearly decreased with increasing radial distance to enhance axial features. For circular disk structures like switches, the weight distribution is adjusted to highlight the height difference between the center and the slot edge. The mathematical expression for the adaptive weight coefficients is as follows:

[0086]

[0087] Among them, Sigmoid ( The Sigmoid activation function dynamically adjusts the weight ratio based on the positional relationship between neighboring points and the target pixel. This allows the converted LiDAR weighted height difference map to more clearly present the key geometric features of power equipment, providing high-quality geometric information input for subsequent feature fusion. In practical applications, this improvement increased the accuracy of insulator edge extraction to 96% and the switch slot recognition rate to 95%, a 23% improvement compared to the fixed-weight method.

[0088] Then, geometric features and visual semantic features are extracted, and a fused feature map is generated by adaptive weight fusion.

[0089] This embodiment sets up a dual-branch feature extraction module before fusion: a geometric branch and a semantic branch. The geometric branch performs edge enhancement and contour extraction on the weighted height difference map of the LiDAR, outputting a geometric feature map; the semantic branch performs convolutional feature extraction on the visual image, outputting a semantic feature map. The outputs of the two branches are concatenated along the channel dimension and used as input for subsequent adaptive fusion.

[0090] This embodiment constructs a two-layer architecture of "linear transformation alignment + dynamic weight fusion" for adaptive fusion, overcoming the limitation of traditional fixed weight fusion that cannot adapt to the feature requirements of different power components. First, a convolutional network is used to learn the linear transformation parameters α and β to achieve accurate alignment between LiDAR features and visual features; then, a lightweight network is used to calculate the dynamic weight λ in real time, adaptively adjusting the fusion ratio of the two features according to the component type.

[0091] Specifically, when extracting features from the lidar weighted height difference map and the preprocessed power equipment image through their respective feature extraction branches to obtain corresponding feature maps, and then fusing these feature maps using adaptive weights to obtain a fused feature map, the process includes:

[0092] S2.2.1: Input the weighted height difference map from the LiDAR into the geometry branch for edge enhancement and contour extraction to obtain the geometric feature map. The edge enhancement and contour extraction process is as follows:

[0093] Calculate the gradient magnitude of the lidar weighted height difference map:

[0094]

[0095] in The partial derivatives of the aforementioned lidar weighted height difference map are calculated using the Sobel operator.

[0096] Non-maximum suppression (NMS): Comparison along the gradient direction Only local maxima are retained to obtain refined edges. .

[0097] Contour connection and filling: using a double thresholding method (high threshold) low threshold Connect the edges and perform a morphological closing operation (with a 3×3 rectangle as the structuring element) to fill the closed contour, resulting in a binary contour map. .

[0098] Generating a geometric feature map: The original height difference map is overlaid with the contour map to obtain the geometric feature map. :

[0099]

[0100] in This is the profile enhancement factor (usually taken as 0.5).

[0101] S2.2.2: Input the preprocessed power equipment image into the semantic branch for convolutional feature extraction to obtain the semantic feature map. The semantic branch is a lightweight convolutional neural network, with the following structure:

[0102] Input: Preprocessed RGB power equipment image, resized to 320×240.

[0103] Convolutional layer 1: Convolutional kernel 64×3×3, stride 1, padding 1, followed by ReLU activation and 2×2 max pooling.

[0104] Convolutional layer 2: Convolutional kernel 128×3×3, stride 1, padding 1, followed by ReLU activation and 2×2 max pooling.

[0105] Convolutional layer 3: Convolutional kernel 256×3×3, stride 1, padding 1, followed by ReLU activation.

[0106] Output layer: 1×1 convolution reduces the number of channels to 32, outputting semantic feature maps. .

[0107] The network is pre-trained on a publicly available power equipment dataset to extract visual semantic features such as equipment color, texture, and nameplate markings.

[0108] S2.2.3: Perform a linear transformation on the geometric feature map to align the features of the geometric feature map with those of the semantic feature map. The mathematical expression is as follows:

[0109]

[0110] in, Geometric features after transformation of the weighted height difference map from lidar; The original geometric features of the weighted height difference map for lidar; The linear transformation parameters learned by the neural network are expressed mathematically as follows:

[0111]

[0112] in, It is a convolutional network; For network weights, Semantic features of power equipment images;

[0113] S2.2.4: The transformed geometric features and semantic feature maps are fused using adaptive weights to obtain a fused feature map. The mathematical expression is as follows:

[0114]

[0115] In the above formula, The weighting coefficients are dynamically fused and adaptively adjusted according to the component type. They are calculated in real time using a lightweight network, and the mathematical expression is as follows:

[0116]

[0117] in, For the Sigmoid function; These are network parameters.

[0118] For components such as insulators and clamps that depend on geometry, the feature weight λ of the lidar will be adaptively increased, for example, approaching 0.8, so as to better highlight their three-dimensional contours and surface undulation features.

[0119] For components that rely on visual information, such as switch indicators and color codes, the visual feature weight is 1. λ will increase accordingly, for example, λ will approach 0.3, to ensure that the semantic information of color and texture can be fully utilized;

[0120] For key operational features such as the center of the switch and the edge of the slot, a balanced fusion of geometric and visual features will be adopted, with λ being approximately 0.5, balancing positioning accuracy and the reliability of status recognition.

[0121] The network responsible for calculating dynamic weights has been thoroughly trained on a high-voltage equipment dataset, enabling it to accurately and adaptively distinguish different component types and automatically select the optimal fusion strategy. Testing shows that the output fused feature map achieves a 25% improvement in recognition accuracy under complex high-voltage environments compared to traditional fixed-weight fusion.

[0122] S2.3: Anti-interference processing. Specifically, the median filter window is dynamically adjusted based on the real-time electromagnetic interference spectrum to filter the fused feature map, and then a Kalman filter is used for state estimation, while its observation noise covariance matrix is ​​updated in real time, thereby reducing noise in the fused feature map.

[0123] To address the issue of sudden noise in sensor data caused by electromagnetic pulses in high-voltage environments, this embodiment proposes a hardware-software co-operated anti-interference scheme of "adaptive parameter filtering + double-layer shielding." First, median filtering is used to quickly suppress sudden noise caused by electromagnetic pulses. Then, the processed data is fed into a Kalman filter for accurate state estimation. Crucially, the filtering parameters can be optimized in real time according to the spectral characteristics of the electromagnetic interference in the field—when the interference frequency is between 50-100Hz, the median filter window is automatically adjusted to 7; when the interference frequency is between 100-500Hz, the window is adjusted to 5. Simultaneously, the observation noise covariance matrix R of the Kalman filter is updated in real time through maximum likelihood estimation. k.in:

[0124] The spectral characteristics of electromagnetic interference in the field can be obtained based on the voltage level and prior electromagnetic characteristics of the working environment, or by analyzing the electromagnetic interference spectrum through short-time FFT before IMU data preprocessing.

[0125] The Kalman filter update formula is:

[0126]

[0127] in the formula This is the state estimate at time k; The predicted state value at time k; These are the observed values; The observation matrix; Kalman gain; To predict the covariance matrix; To observe the noise covariance; This is the updated covariance matrix.

[0128] The median filtering formula is:

[0129]

[0130] in the formula This is the filtered data; This is for median operations.

[0131] Furthermore, the live-line working robot in this embodiment can also employ a double-layer copper mesh shielding structure and common-mode filter circuit design for the lidar unit, camera unit, and IMU unit, achieving a shielding effectiveness of over 80dB. Testing showed that, across a full voltage range of 10kV-500kV, the sensor data noise amplitude was reduced by 85% compared to designs without anti-interference measures, effectively ensuring data reliability.

[0132] This step allows the filtering parameters to be adjusted in real time according to the spectral characteristics of electromagnetic interference on site, achieving hardware and software collaborative anti-interference and ensuring stable operation in electromagnetic environments at different voltage levels.

[0133] S2.4: Online accuracy assessment and feedback.

[0134] First, calibration accuracy evaluation indicators:

[0135] Rotational error assessment uses Euler angle error:

[0136]

[0137] in the formula These represent the rotational errors about the X, Y, and Z axes, respectively. To estimate Euler angles; It is a true value.

[0138] The average angle error is:

[0139]

[0140] in the formula This represents the average rotational error.

[0141] Translation error is assessed using Euclidean distance difference:

[0142]

[0143] in the formula These represent the translation errors in the X, Y, and Z directions, respectively. This is an estimated value; It is a true value.

[0144] The average translation error is:

[0145]

[0146] in the formula This represents the average translation error.

[0147] Then, based on the above accuracy evaluation indicators, the average rotation error R and average translation error t of the current external parameters are calculated in real time according to the rotation matrix and translation vector of the current external parameters. If the average rotation error R and average translation error t are greater than the recalibration trigger threshold, the subsequent operation is triggered; otherwise, step 3 is skipped. Through the online accuracy evaluation and recalibration mechanism, the high-precision sensing capability can be maintained throughout the long-term operation.

[0148] Specifically, the subsequent operations triggered when the rotation error is >0.2° or the translation error is >3mm include:

[0149] (1) Record the current operating conditions (light intensity, humidity, electromagnetic field strength);

[0150] (2) Suspend the operation and start the online recalibration process;

[0151] (3) The human-computer interaction module prompts maintenance personnel to check the sensor protection status.

[0152] It should be noted that the recalibration trigger threshold (rotation error > 0.2° or translation error > 3mm) is not set arbitrarily, but determined through a large number of experiments. When the error reaches this threshold, the failure rate of the switch adjustment operation will exceed 1%.

[0153] Furthermore, to improve practicality, this embodiment optimizes the online recalibration algorithm, employing local calibration instead of traditional global calibration. During local calibration, LiDAR point cloud data and power equipment image data from a specified area are collected and substituted into the loss function for solving extrinsic parameters. Then, the rotation matrix and translation vector in the loss function are optimized to achieve extrinsic parameter recalibration. Then, step 2.1 is executed again to transform the preprocessed LiDAR point cloud data from the LiDAR coordinate system to the camera coordinate system based on the calibrated extrinsic parameters. This reduces the recalibration time to less than 3 minutes, a 60% reduction compared to global calibration. Simultaneously, during recalibration, the robotic arm is automatically locked at a safe position more than 1 meter away from the high-voltage equipment, pausing all operations until recalibration is completed and verified, ensuring the safety of the recalibration process. The specific local calibration process is as follows:

[0154] Triggering condition: When the average rotation error is detected during online evaluation. or average translation error When this happens, the system pauses operation and enters local recalibration mode.

[0155] Data acquisition: Control the movement of the robotic arm so that the camera and lidar can jointly observe the switch to be operated and its adjacent area (about 1m×1m), and acquire high-density point cloud and high-definition image of this local area.

[0156] Feature extraction and matching: Using only this local data, the features of the switch center and slot edge are re-extracted, and image-point cloud matching pairs are established.

[0157] Optimization Solution: Substitute the matching pairs into the aforementioned double-constraint loss function, and use the Levenberg-Marquardt algorithm to optimize and solve for the new rotation matrix. With translation vector .

[0158] Verification and Update: Calculate the error of the new extrinsic parameters. If the accuracy requirements are met, update the system extrinsic parameters; otherwise, expand the acquisition area or trigger an alarm for manual intervention.

[0159] S3: Environmental Modeling and Localization. Specifically, the IMLS-ICP algorithm, improved based on intensity consistency, is used to register the power equipment point cloud and the environmental point cloud in the preprocessed LiDAR point cloud data to obtain the point cloud registration result. Simultaneously, the LiDAR-Visual Inertial Odometry (LVIO-SAM) fusion algorithm is used for pose estimation and optimization: First, the pose of the live-line working robot is predicted based on the preprocessed IMU data. Then, the pose prediction result is corrected using ESKF error correction based on the power equipment features in the fused feature map. Next, the ESKF error correction result and the point cloud registration result are fused and optimized to obtain the globally optimal pose of the live-line working robot. Finally, a 3D environment model is constructed based on the globally optimal pose. Thus, the globally optimal pose is solved through pose graph optimization, while simultaneously constructing and dynamically updating the 3D environment model. This includes:

[0160] S3.1: Point cloud registration. Specifically, the IMLS-ICP algorithm based on intensity consistency improvement is used to register the point clouds of power equipment in consecutive frames with the environmental point clouds, and slot center point displacement monitoring is added as the iteration termination condition.

[0161] In this embodiment, to address the issue of uneven laser point cloud intensity caused by the prevalence of metal materials in power equipment, an intensity consistency term is introduced into the weight function of the IMLS-ICP algorithm. This allows the weights to consider both point cloud distance and intensity information, effectively solving the problem of decreased registration accuracy on rusted or contaminated equipment surfaces. Furthermore, an iterative termination mechanism for slot center point displacement monitoring is added. When the slot center point displacement between adjacent frames is less than 2mm, the registration iteration is immediately terminated, avoiding the accuracy impact of overfitting and environmental disturbances. Therefore, when using the IMLS-ICP algorithm to register the power equipment point cloud with the environmental point cloud in the preprocessed LiDAR point cloud data, specifically, the IMLS-ICP algorithm is used to register the switch point cloud with the environmental point cloud in consecutive frames. Simultaneously, the point cloud clusters in the slot region are extracted in real time, and the corresponding center point coordinates are calculated. When the center point displacement between adjacent frames is less than the corresponding threshold, the current registration iteration is terminated.

[0162] When using the IMLS-ICP algorithm for point cloud registration, the implicit surface reconstruction model is as follows:

[0163]

[0164] in the formula For point Point cloud surface The distance; For point The normal vector; Here is the weighting function, which is:

[0165]

[0166] in the formula The height parameter is the distance from the point cloud to the surface.

[0167] In this embodiment, to address the issue of uneven laser point cloud intensity caused by the predominantly metallic surface of power equipment, an intensity consistency term is introduced into the IMLS-ICP weighting function. The mathematical expression for the weighting function of the IMLS-ICP algorithm based on improved intensity consistency is as follows:

[0168]

[0169] in Point strength, The mean local intensity The standard deviation is for strength. This improvement enhances registration accuracy on rusted and contaminated equipment surfaces.

[0170] The optimization goal of point cloud registration is:

[0171]

[0172] in the formula The current frame point; For the projection points of the curved surface; It is the normal vector; Let be the rotation matrix and translation vector to be solved.

[0173] During point cloud registration, point cloud clustering of the slot region is extracted in real time, and the coordinates of its axis center point are extracted by fitting a cylindrical model using RANSAC. If the center point shifts between adjacent frames Less than the threshold If the error is within 2mm, the slot position is considered stable, and the current registration iteration is terminated to avoid overfitting and the influence of environmental disturbances. This design is particularly suitable for power equipment with obvious geometric structures, such as non-excited tap changers, and can effectively avoid the impact of overfitting and environmental disturbances on registration accuracy. It also improves registration efficiency by 30% while maintaining high accuracy at the ±2mm level.

[0174] S3.2: LiDAR-Visual Inertial Odometry Fusion (LVIO-SAM). Specifically, pose is predicted using an IMU measurement model. The 3D coordinates of the extracted power equipment feature centers from the fused feature map, along with the corresponding local point cloud average reflection intensity and visual semantic confidence, are used as joint observations. Errors are corrected using ESKF (Electromagnetic Skeptic Error Correction), and the state vector during ESKF error correction includes an IMU zero-bias error term caused by electromagnetic interference. Finally, the global optimal pose is solved through pose graph optimization, thereby constructing and dynamically updating the 3D environment model. This includes:

[0175] S3.2.1: Predict the pose of the live-line working robot based on the preprocessed IMU data and correct the ESKF error based on the fused feature map. High-voltage electromagnetic interference is one of the core challenges in live-line working environments, which can easily cause abnormal fluctuations in the IMU output. This embodiment not only adds a sliding window anomaly detection step in the IMU data preprocessing stage, which can effectively remove outliers caused by interference, but also makes two key improvements based on the open-source LVIO-SAM framework for high-voltage and strong electromagnetic environments: one is to add a zero-bias error term δb caused by electromagnetic interference to the ESKF state vector. em First, it solves the problem of IMU data distortion caused by electromagnetic interference in traditional frameworks; second, it optimizes the observation equation by introducing joint constraints of lidar intensity features and visual semantic features, thereby improving the stability of pose estimation. The specific process is as follows:

[0176] S3.2.1.1: Apply the position update equation, velocity update equation, and attitude update equation sequentially to the preprocessed IMU data to perform pose prediction, obtaining the predicted pose values:

[0177] The state vector is defined as:

[0178]

[0179] in the formula It is a rotation matrix; It is a position vector; It is the velocity vector; This is the IMU bias vector.

[0180] The IMU measurement model is:

[0181]

[0182] in the formula Angular velocity and acceleration measured by the IMU; These are the actual angular velocities and accelerations; For bias; For measuring noise; The rotation matrix from the world frame to the coordinate system of the live-lined robot; This is the acceleration due to gravity.

[0183] Integral process for IMU pose prediction:

[0184] Based on the IMU measurement model, the predicted robot pose is obtained by integrating the angular velocity and acceleration over discrete time intervals.

[0185]

[0186] in the formula They are respectively Position, velocity, and rotation matrix at any given moment; These are measured values ​​for acceleration and angular velocity; This corresponds to zero bias; This is the vector of gravitational acceleration; The sampling time interval; This is the exponential mapping from Lie algebra to rotation matrices.

[0187] S3.2.1.2: Using the three-dimensional coordinates of the switch center extracted from the fused feature map, the corresponding local point cloud average reflection intensity, and visual semantic confidence as joint observation values, an observation equation is established that incorporates joint constraints from lidar intensity features and visual semantic features. The ESKF state vector is then extended to include position error, velocity error, rotation error, IMU zero-bias error, and zero-bias error term caused by electromagnetic interference, thus constructing the ESKF state equation:

[0188] The state equation of the error Kalman filter (ESKF) is:

[0189]

[0190] in the formula For displacement error; For speed error; This is rotational error; For zero bias error in acceleration and angular velocity; This is due to gravity error; This refers to the zero-bias error term of the accelerometer and gyroscope caused by electromagnetic interference. The online estimation method is used, specifically by weighting the amplitudes of the power frequency and harmonic interference components in the IMU data spectrum analysis: This is achieved by analyzing the raw acceleration and angular velocity data of the IMU. Perform a short-time Fourier transform (STFT) to analyze its spectrum. Identify abnormal spectral peaks caused by electromagnetic interference near the power frequency (50 / 60Hz) and its harmonics. Calculate the electromagnetic interference error term.

[0191]

[0192] in For spectral amplitude, The weight matrix is ​​calibrated for the experiment and is used to map the spectral energy to zero bias error.

[0193] The joint constraint observation equation is:

[0194]

[0195] in the formula The three-dimensional coordinates of the switch center extracted from the fused feature map (LiDAR observation); The projection function projects the three-dimensional coordinates of the circle's center onto the image plane to obtain the predicted pixel coordinates. In point clouds The average reflection intensity within a sphere centered at a radius of 10cm; For semantic branches at image pixels The semantic confidence of the switch category output at (corresponding to the center projection position) (obtained through the Softmax layer); For observation noise, its covariance matrix It includes pixel error, intensity noise, and semantic confidence variance.

[0196] S3.2.1.3: Based on the stated state equation and observation equation, perform ESKF updates, calculate the error state correction amount, and correct the pose prediction value. The state update process is as follows:

[0197]

[0198] in the formula Kalman gain; To predict the covariance matrix; To observe the Jacobian matrix; Observation noise covariance matrix; These are observations from external sensors; This is the error state correction amount; This is the updated covariance matrix.

[0199] Observation noise covariance Let it be a diagonal matrix. , in the formula Pixel projection noise variance (related to camera intrinsics and feature localization accuracy); The point cloud intensity noise variance (affected by ambient lighting and equipment surface material); This represents the semantic confidence noise variance (which depends on the real-time performance of the visual recognition model). During the update process, the Jacobian matrix... The observation equations need to be calculated for all state variables (including...) The partial derivatives of ) and the observation noise covariance matrix It is a diagonal matrix containing pixel error, intensity noise, and semantic confidence variance.

[0200] The above steps specifically extend the zero-bias error term δb caused by electromagnetic interference in the ESKF state vector. em By updating the estimate in real time through observation and correcting this error, the stability of attitude estimation in strong electromagnetic environments is greatly improved.

[0201] S3.2.2: Based on the ESKF error correction results and point cloud registration results, pose graph fusion optimization is performed to obtain the globally optimal pose of the live-line working robot, and a 3D environment model is constructed based on the globally optimal pose. Specifically, this includes:

[0202] S3.2.2.1: Using the point cloud registration results, ESKF error correction results, and the switch center observations in the fused features as constraints, a pose graph optimization problem is constructed. The global optimal pose of the live-line working robot is obtained by solving the problem through maximum likelihood estimation. The process is as follows:

[0203] (1) The pose output by ESKF is used as the initial node, and the IMLS-ICP registration result is used as the edge constraint between adjacent frames.

[0204] (2) Use the coordinates of the switch center of the visual observation as the absolute observation edge and add them to the pose graph;

[0205] (3) Construct an optimization problem based on maximum likelihood estimation:

[0206]

[0207] In the formula, z lvio z i c p z visual These correspond to the measurements from LVIO, point cloud registration, and visual observation, respectively. Ω represents the information matrix of each sensor, primarily reflecting the confidence weights of different sensors under high-pressure conditions. Solving this optimization problem yields a globally consistent, high-precision pose sequence, providing a reliable positional basis for subsequent operational planning.

[0208] S3.2.2.2: Transform the point clouds of each keyframe to the global coordinate system based on the global optimal pose and stitch them together to construct a three-dimensional environment model.

[0209] S4: Job Planning and Execution. Specifically, this involves using the globally optimal pose of the live-line working robot as the starting point, the target switch position as the ending point, and a 3D environment model as the map. Within the safe area defined by electric field simulation on this map, an improved A* algorithm is used to plan an obstacle avoidance path, and the robotic arm is controlled to execute the gear shifting operation according to the planned obstacle avoidance path. This includes:

[0210] S4.1: Robotic Arm Path Planning. Specifically, based on the 3D environment model and the real-time pose of the live-lined robot, an improved A* algorithm is used for obstacle avoidance path planning.

[0211] In this embodiment, electric field strength constraints are incorporated into the path planning. A safe zone is defined using electric field simulation, with an electric field strength ≤10kV / m as the core constraint, ensuring that the minimum safe distance between the generated path and high-voltage equipment is no less than 50cm. The planned path is then broken down into a sequence of joint movements for the robotic arm, providing a foundation for precise operation. This design effectively avoids the risk of electrostatic induction caused by the robotic arm entering a high electric field region, thus improving operational safety.

[0212] Therefore, in this embodiment, when planning obstacle avoidance paths using the improved A* algorithm within the safe area defined by electric field simulation in the map, obstacles are identified based on the three-dimensional environment of the map, and electric field simulation is performed on the map. When performing electric field simulation, an electric field intensity distribution model can be constructed offline through finite element simulation based on the three-dimensional model of the substation and the voltage level of the high-voltage equipment, and a mapping relationship between position coordinates and electric field intensity can be established. During operation, the electric field intensity value at any position can be obtained by looking up a table or spatial interpolation. Alternatively, an electric field sensor can be integrated into the live-line working robot body or the end of the robotic arm to measure the electric field intensity at the working position in real time for safe area verification and dynamic adjustment of path planning.

[0213] After performing electric field simulation, non-obstacle areas with electric field strength not exceeding the safety threshold are designated as safe zones. Then, the starting point is used as the initial path point for iterative expansion. During iterative expansion, the cost of the path, including the comprehensive actual path cost, is calculated for each path point's neighboring nodes within the safe zone. Target heuristic estimation With electric field strength constraint Cost function Among them, the electric field strength constraint term , The electric field strength at the node is... As a safety threshold, The penalty coefficient is used to select the neighboring node with the smallest cost function as the next path point until the destination is reached, thus obtaining the obstacle avoidance path.

[0214] S4.2: Switch Gear Adjustment Execution and Verification. The robotic arm performs gear adjustment operations according to the obstacle avoidance path disassembly sequence. During this process, a combined vision and LiDAR measurement method can be used to monitor the gear angle. The LiDAR acquires the three-dimensional posture of the gear lever, while the vision sensor (camera unit) assists in texture matching. The gear angle measurement accuracy can reach ±0.1°. If the detected gear angle deviation exceeds 0.5°, the joint angle is adjusted through the PID controller—with the proportional coefficient Kp set to 5.0, the integral coefficient Ki set to 0.1, and the derivative coefficient Kd set to 0.5—for precise fine-tuning. After gear adjustment, the gear angle is verified again to ensure operational accuracy. Simultaneously, the robotic arm's end effector uses an epoxy resin insulation layer with an insulation resistance of not less than 10¹. 0Ω, with a withstand voltage rating of over 500kV, further ensuring operational safety.

[0215] Example 2

[0216] This embodiment proposes a live-line working method based on the fusion of lidar and visual recognition. It is essentially the same as Embodiment 1, but differs in that, in addition to using shielding materials and protective circuits to suppress electromagnetic interference for hardware anti-interference and anomaly detection and composite filtering to ensure data stability for software anti-interference, this embodiment also constructs a full-link safety verification mechanism during the safety verification process. Besides real-time monitoring of the distance between the robotic arm and the high-voltage equipment and the consistency of the verification angle, it also supplements fault emergency handling and real-time insulation performance monitoring functions. When a sensor fails, redundancy switching is automatically initiated—for example, if lidar fails, vision + IMU fusion positioning is used; if vision fails, lidar + IMU fusion positioning is used. If communication is interrupted, automatic shutdown logic is triggered within 2 seconds, and the robotic arm quickly retracts to a safe position and locks. Simultaneously, an insulation resistance sensor is integrated at the end of the robotic arm, automatically verifying insulation performance every 5 minutes. When the insulation resistance is below 10¹... 0 When the voltage reaches Ω, immediately stop the machine and sound an alarm. These safety mechanisms ensure absolute safety during high-voltage operations and prevent accidents caused by equipment malfunction or performance degradation.

[0217] Example 3

[0218] This embodiment proposes a live-line working method based on the fusion of lidar and visual recognition. It is essentially the same as Embodiment 1, but differs in that it fully considers the special requirements of live-line working scenarios such as high voltage, strong electromagnetic fields, complex weather conditions, and irregularly shaped equipment at multiple levels, including sensor selection, algorithm parameter settings, and fusion strategy design. Furthermore, this embodiment designs customized adaptation solutions for three common harsh scenarios in live-line working: rainy days, foggy days, and equipment corrosion, overcoming the limitations of poor scenario adaptability in existing systems. For rainy days, an anomaly detection algorithm based on point cloud density thresholds is used to remove raindrop noise, achieving a noise removal rate of over 98% while ensuring the integrity of the equipment point cloud is no less than 95%. For foggy days, an improved dark channel prior defogging algorithm is used, introducing an adaptive atmospheric light intensity correction factor, improving the image contrast by 40% after defogging while maintaining an equipment feature recognition rate of over 92%. For corroded equipment, an edge-preserving filtering + texture reconstruction feature enhancement scheme is adopted. First, bilateral filtering smooths the noise, then Gabor texture extraction enhances the features of the corroded area, ultimately achieving a recognition accuracy of over 93% for corroded equipment. These designs enable live-line working robots to operate stably under complex weather and equipment conditions, significantly expanding their application scope.

[0219] Through these customized designs, this embodiment can be precisely adapted to special power operation environments, fully demonstrating the scenario-specificity and innovative value of the solution, and also greatly improving the operational stability and reliability of the live-line working robot.

[0220] Example 4

[0221] This embodiment proposes a live-line working system based on the fusion of lidar and visual recognition, including a live-line working robot equipped with a lidar unit, a camera unit, an IMU unit, and a robotic arm. The live-line working robot is programmed or configured to execute the live-line working method based on the fusion of lidar and visual recognition in Embodiment 1.

[0222] Specifically, the live-line working system of this embodiment includes the following functional modules:

[0223] 1. Multi-sensor sensing module:

[0224] LiDAR Unit: A multi-line LiDAR with 32 lines or more is selected, with a 64-line model recommended. The scanning frequency is set between 10-20Hz, and the ranging range covers 0.5-20m, achieving a ranging accuracy of ±2cm at a distance of 10m. The LiDAR is installed on the top or front of the live-line working robot, primarily used for real-time acquisition of 3D point cloud data of the working environment, accurately obtaining geometric information such as the distance and shape of obstacles. Considering the harsh environment of live-line work, the LiDAR selected in this embodiment has a wide operating temperature range of -20℃ to 60℃, an IP67 protection rating, and an electromagnetic interference resistance level that meets the EN55022 Class A standard, enabling stable operation in complex field environments.

[0225] The camera unit consists of a high-definition color camera and a depth camera. The color camera has a resolution of at least 4K and a frame rate of 25fps, while the depth camera achieves a ranging accuracy of ±1cm at a distance of 2m. To reduce interference from strong light, this embodiment uses a narrowband filter for the camera, with the filtered wavelength controlled within the range of 550nm ±10nm. The visual recognition unit is mainly responsible for collecting visual features such as color, texture, and markings of the equipment, while simultaneously acquiring depth information of some scenes through the depth camera. Similar to LiDAR, the visual sensor also has an operating temperature range of -20℃ to 60℃ and an IP67 protection rating, with a strong light suppression capability exceeding 100,000 lux, enabling it to adapt to the strong light and complex lighting environments of live-line work sites.

[0226] Inertial Measurement Unit (IMU): Used to acquire motion posture data of live-lined robots, assisting in time synchronization and motion compensation of sensor data.

[0227] 2. Data Fusion Processing Module:

[0228] Spatiotemporal calibration unit: Through a joint calibration algorithm, it achieves spatial coordinate alignment and timestamp synchronization between the lidar and the vision sensor, and establishes a unified spatiotemporal coordinate system.

[0229] Feature fusion unit: Employing weighted height difference transformation and feature adaptive fusion algorithms, the LiDAR point cloud data is converted into a two-dimensional LiDAR weighted height difference map, and then fused with visual image features at multiple levels to extract key features from the environment.

[0230] Anti-interference processing unit: integrates electromagnetic shielding circuit and filtering algorithm to reduce noise in sensor data and improve data reliability.

[0231] 3. Motion control module:

[0232] Path planning unit: Based on the fused 3D environment model and combined with the safety distance requirements for live-line work, it generates obstacle avoidance paths and work trajectories.

[0233] Precise positioning unit: Utilizing a fusion algorithm of lidar and visual inertial odometry, combined with an error Kalman filter, it achieves real-time pose estimation of the live-line working robot, with positioning accuracy reaching the centimeter level.

[0234] 4. Job Execution Module:

[0235] Robotic arm and end effector: Equipped with a multi-degree-of-freedom robotic arm and specialized tools, it performs precise operations according to control commands.

[0236] 5. Human-computer interaction module:

[0237] The remote monitoring terminal can display the fused environmental model, the operating status of the live-line working robot, and the work screen in real time, supporting remote control and switching between automatic operation modes. To ensure the safety of remote control, this embodiment adopts a dual-mode communication scheme of 5G + fiber optic, with 5G communication latency controlled within 50ms and fiber optic communication latency even lower than 10ms. A latency compensation mechanism is also designed, using a predictive attitude compensation algorithm to predict the next posture based on the historical motion data of the live-line working robot, offsetting the control deviation caused by communication latency. When switching between automatic and manual modes, the system first pauses the robotic arm's movement, with a response time of no more than 0.5 seconds, and then verifies whether the current position of the live-line working robot is safe—ensuring that the distance to the high-voltage equipment is no less than 50cm. Only after passing the verification will the mode switch be completed. During the switching process, any work actions are strictly prohibited to avoid the risk of collision caused by mode switching.

[0238] In summary, this invention proposes a live-line working method and system based on the fusion of lidar and visual recognition. This method can be widely applied to the control of live-line working robots in scenarios such as high-voltage transmission lines and substations, effectively improving the safety and reliability of power system operation. Actual testing has verified that this invention possesses excellent performance.

[0239] In indoor simulation experiments, the positioning accuracy reached ±1.5cm with a 98% device recognition rate in the absence of electromagnetic interference; in an electromagnetic environment of 50kV / m, the positioning accuracy was ±2cm with a 96% recognition rate; even in a strong electromagnetic environment of 100kV / m, the positioning accuracy remained at ±2.5cm with a 94% recognition rate. Under strong light and weak light conditions, the recognition rates were 95% and 93%, respectively. In field experiments, single-switch adjustment operations at a 110kV substation were completed in less than 5 minutes with a safe distance control accuracy of ±3cm; single-switch adjustment at a 220kV substation took no more than 6 minutes with a safe distance control accuracy of ±3.5cm, and the failure rate was 0% in both experiments. Comparative experiments show that compared to a single vision sensor solution, this invention improves positioning accuracy by 70% and operational efficiency by 50%; compared to existing fusion solutions for live-line work robots in power operations, positioning accuracy is improved by 60% in strong electromagnetic environments, and device recognition rate in complex scenarios is improved by 25%.

[0240] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0241] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A live-line working method based on the fusion of lidar and visual recognition, characterized in that, The method is applied to a live-line working robot, which is equipped with a lidar unit, a camera unit, an IMU unit, and a robotic arm. The method includes the following steps: S1. Real-time acquisition of lidar point cloud data, power equipment image data, and IMU data, and performance of data preprocessing; S2. The preprocessed LiDAR point cloud data is transformed from the LiDAR coordinate system to the camera coordinate system through an extrinsic parameter optimization strategy. Specifically, a loss function containing dual constraint matching of the power equipment feature center and the slot edge is constructed. Then, the rotation matrix and translation vector in the loss function are optimized to achieve extrinsic parameter calibration. Finally, based on the calibrated extrinsic parameters, the preprocessed LiDAR point cloud data is transformed from the LiDAR coordinate system to the camera coordinate system, thereby projecting the point cloud data onto the image plane. Then, the weighted height difference is calculated for each pixel in the projected image to obtain the LiDAR weighted height difference map. Finally, the LiDAR weighted height difference map and the preprocessed power equipment image are respectively extracted through the corresponding feature extraction branches to obtain corresponding feature maps. The feature maps are then fused using adaptive weights to obtain a fused feature map. S3. Using the IMLS-ICP algorithm based on improved intensity consistency, the power equipment point cloud in the preprocessed lidar point cloud data is registered with the environmental point cloud to obtain the point cloud registration result; simultaneously, a lidar-visual inertial odometry fusion algorithm is used for pose estimation and optimization, specifically including: Pose prediction of a live-line working robot based on preprocessed IMU data; Based on the power equipment features in the fused feature map, the pose prediction result is corrected for ESKF error. The state vector during ESKF error correction includes an IMU zero bias error term caused by electromagnetic interference. The error correction results and point cloud registration results are fused and optimized to obtain the global optimal pose of the live-line working robot, and a three-dimensional environment model is constructed based on the global optimal pose. S4. Using the global optimal pose as the starting point, the target power equipment location as the ending point, and the three-dimensional environment model as the map, within the safe area divided according to the electric field simulation in the map, an improved A* algorithm containing an electric field strength constraint term in the cost function is used to plan an obstacle avoidance path, and the robotic arm is controlled to perform a gear shifting operation according to the planned obstacle avoidance path.

2. The live-line working method based on the fusion of lidar and visual recognition according to claim 1, characterized in that, In step S2, the mathematical expression of the loss function is: in, Let be a rotation matrix. It is a translation vector. Extracting the first from the image The pixel coordinates of the center of each power equipment feature For the corresponding point cloud Three-dimensional coordinates of the feature center of each power equipment Extracting the first from the image pixel coordinates of the slot edge points For the corresponding point cloud Three-dimensional coordinates of the edge points of the slot. For camera projection function, These are the weighting coefficients. These represent the number of matching power equipment feature centers and slot edge points, respectively.

3. The live-line working method based on the fusion of lidar and visual recognition according to claim 1, characterized in that, In step S2, when calculating the weighted height difference for each pixel in the projected image, the mathematical expression is: in, For pixels ( x,y The weighted height difference; This represents the height value corresponding to the target pixel. For the neighboring region The height value corresponding to each point; This represents the number of neighboring points; These are adaptive weighting coefficients; Let be the coordinates of the neighboring points; the mathematical expression for the adaptive weighting coefficients is as follows: Among them, Sigmoid ( ) is the Sigmoid activation function.

4. The live-line working method based on the fusion of lidar and visual recognition according to claim 1, characterized in that, In step S2, when the weighted height difference map of the lidar and the preprocessed power equipment image are respectively extracted through the corresponding feature extraction branches to obtain corresponding feature maps, and the feature maps are fused through adaptive weights to obtain a fused feature map, the process includes: The weighted height difference map from the LiDAR is input into the geometric branch for edge enhancement and contour extraction to obtain a geometric feature map. The preprocessed power equipment image is input into the semantic branch for convolutional feature extraction to obtain a semantic feature map. The geometric feature map is then subjected to a linear transformation to align its features with those of the semantic feature map. The mathematical expression is as follows: in, Geometric features after transformation of the weighted height difference map from lidar; The original geometric features of the weighted height difference map for lidar; The linear transformation parameters learned by the neural network are expressed mathematically as follows: in, It is a convolutional network; For network weights, Semantic features of power equipment images; The transformed geometric features and semantic feature maps are fused using adaptive weights to obtain a fused feature map, as shown in the following mathematical expression: In the above formula, For dynamic fusion of weighting coefficients, , For the Sigmoid function, These are network parameters.

5. The live-line working method based on the fusion of lidar and visual recognition according to claim 1, characterized in that, After obtaining the fused feature map in step S2, the method further includes an anti-interference processing step for the fused feature map, specifically: dynamically adjusting the median filtering window according to the real-time electromagnetic interference spectrum to filter the fused feature map, then using a Kalman filter for state estimation, and updating its observation noise covariance matrix in real time.

6. The live-line working method based on the fusion of lidar and visual recognition according to claim 2, characterized in that, Step S2 also includes an online accuracy assessment and recalibration step, specifically including: real-time calculation of the average rotation error and average translation error of the current extrinsic parameters; if the error exceeds a preset threshold, the operation is paused, and lidar point cloud data and power equipment image data of the local area of ​​the current operation are collected and resubmitted into the loss function for extrinsic parameter optimization to achieve local rapid recalibration.

7. The live-line working method based on the fusion of lidar and visual recognition according to claim 1, characterized in that, In step S3, when registering the power equipment point cloud and the environment point cloud in the preprocessed lidar point cloud data using the IMLS-ICP algorithm based on improved intensity consistency to obtain the point cloud registration result, specifically, the IMLS-ICP algorithm is used to register the power equipment point cloud and the environment point cloud in consecutive frames. Simultaneously, point cloud clustering of the slot region is extracted in real time, and the corresponding center point coordinates are calculated. When the center point displacement between adjacent frames is less than the corresponding threshold, the current registration iteration is terminated. The mathematical expression of the weight function of the IMLS-ICP algorithm based on improved intensity consistency is as follows: in, , This represents a point in a point cloud surface. The height parameter from the point cloud to the surface. Point strength, The mean local intensity The standard deviation of strength.

8. The live-line working method based on the fusion of lidar and visual recognition according to claim 4, characterized in that, In step S3, when performing ESKF error correction on the pose prediction result based on the power equipment features in the fused feature map, the three-dimensional coordinates of the power equipment feature center extracted from the fused feature map, the corresponding local point cloud average reflection intensity, and visual semantic confidence are used as joint observation values, and error correction is performed through the error state Kalman filter. The joint observation equation for ESKF error correction is specifically as follows: in, The three-dimensional coordinates of the feature center of the power equipment extracted from the fused feature map. This is a projection function that projects the three-dimensional coordinates of the circle's center onto the image plane to obtain the predicted pixel coordinates. In point clouds The average reflection intensity within a sphere of a specified radius at the center; The image pixel corresponding to the projection position of the semantic branch at the center of the circle. The semantic confidence of the power equipment output at the point; To observe the noise, This is the covariance matrix corresponding to the observation noise, which includes pixel error, intensity noise, and semantic confidence variance.

9. The live-line working method based on the fusion of lidar and visual recognition according to claim 1, characterized in that, In step S4, the cost function of the improved A* algorithm is: in, From the starting point to the node The actual path cost, For the node Heuristic cost estimation to the destination The electric field strength constraint term is calculated as follows: ,in For nodes Electric field strength at that location, As a safety threshold, This is the penalty coefficient.

10. A live-line working system based on the fusion of lidar and visual recognition, characterized in that, include: A live-line working robot is provided with a lidar unit, a camera unit, an IMU unit, and a robotic arm. The live-line working robot is programmed or configured to perform the live-line working method based on lidar and visual recognition fusion as described in any one of claims 1 to 9.

Images