Electric power facility detection method and system

Abstract

The invention discloses an electric power facility detection method and system, and the method comprises the steps: carrying out the feature plane fitting of each single-frame point cloud in laser radar point clouds, and extracting virtual corner data; performing single-mode registration processing on the laser radar point cloud and the camera image based on the virtual angular point data, and performing cross-mode registration processing on the obtained laser radar global point cloud and camera global feature point cloud to obtain a global extrinsic parameter matrix; projecting each three-dimensional coordinate in the laser radar global point cloud to a two-dimensional pixel coordinate of the camera image with the corresponding timestamp to generate a colored laser radar global point cloud; reversely projecting a two-dimensional semantic result obtained by performing semantic segmentation on the camera image to a three-dimensional point cloud space of the corresponding colored laser radar global point cloud to obtain a three-dimensional global point cloud of semantic annotation; and generating a detection result of the target electric power facility according to the three-dimensional global point cloud. According to the method, accurate and stable equipment-level segmentation of the panoramic scene of the transformer substation is realized.

Description

Technical Field

[0001] This invention relates to the field of power system operation and maintenance testing technology, and in particular to a method and system for testing power facilities. Background Technology

[0002] In the field of power system operation and maintenance, accurate inspection of power facilities (such as cabinets, supports, hardware, insulator strings, etc. in substations) is a core link in ensuring the safe and stable operation of the power grid.

[0003] Existing methods for power facility inspection acquire images using monocular or panoramic cameras and learn the appearance and contextual features of equipment using models such as convolutional neural networks and Transformers to output 2D bounding boxes, instance masks, and category labels. However, while these methods can identify equipment at the pixel level, they inherently lack true spatial scale and three-dimensional connectivity. Specifically, the non-uniformity of projection in panoramic images leads to pixel compression of distant small targets and slender components (such as drain plates and positioning pins). Mask quality fluctuates drastically with the field of view, and cross-view occlusion or dynamic interference can easily cause problems such as instance ID drift and topological breaks, making it difficult to support equipment-level measurement and refined operation and maintenance requirements. Summary of the Invention

[0004] This invention provides a method and system for detecting power facilities, in order to solve the technical problem of how to improve existing power facility detection methods and achieve the effect of full-scene, high-precision, and intelligent detection of power facilities.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a method for detecting power facilities, comprising: In response to power equipment detection signals, acquire lidar point cloud and camera images of the target power facility; For each single frame of the lidar point cloud, feature plane fitting processing is performed, and virtual corner point data is extracted based on the result of the feature plane fitting processing. Based on the virtual corner data corresponding to each single-frame point cloud, single-modal registration processing is performed on the lidar point cloud and the camera image respectively to obtain the lidar global point cloud and the camera global feature point cloud; wherein, the single-modal registration processing is designed to optimize and offset the multi-frame registration error accumulated by the feature alignment of adjacent frames based on loop closure detection. Cross-modal registration processing is performed on the global point cloud of the lidar and the global feature point cloud of the camera to obtain the global extrinsic parameter matrix; Based on the global extrinsic matrix and the pre-calibrated camera intrinsic matrix, each three-dimensional coordinate in the global point cloud of the lidar is projected to the two-dimensional pixel coordinates of the corresponding timestamp camera image to generate a colored lidar global point cloud. The camera images are semantically segmented to obtain two-dimensional semantic results for each frame; and the two-dimensional semantic results are back-projected onto the three-dimensional point cloud space of the corresponding colored lidar global point cloud based on the global extrinsic matrix and the camera intrinsic matrix to obtain a semantically annotated three-dimensional global point cloud. The detection results of the target power facility are generated based on the three-dimensional global point cloud.

[0006] As one preferred embodiment, the step of performing feature plane fitting processing on each single frame of the lidar point cloud and extracting virtual corner data based on the result of the feature plane fitting processing includes: Based on the random sampling consensus algorithm, non-planar noise points in each single frame of the lidar point cloud are removed, and valid point clouds that satisfy the planar features are retained. Based on the three-axis least squares method, plane fitting calculations are performed on the effective point cloud in the x-axis, y-axis, and z-axis directions respectively to obtain the plane fitting error corresponding to each axis direction. The plane parameter with the smallest fitting error is selected as the feature plane fitting result of the single frame point cloud. The feature planes generated by fitting the feature planes are extended along the three-dimensional space dimension to obtain the extended planes corresponding to each feature plane. Calculate the spatial intersection points of the extension lines of adjacent feature planes in the extended plane, and use the spatial intersection points as the virtual corner point data corresponding to the single frame point cloud.

[0007] As one preferred embodiment, the step of performing single-modal registration processing on the lidar point cloud and the camera image based on the virtual corner data corresponding to each single-frame point cloud to obtain the lidar global point cloud and the camera global feature point cloud includes: Using the first frame of the lidar point cloud as the reference frame, the virtual corner data of the reference frame is extracted as reference feature points; and the virtual corner data corresponding to each of the remaining single frames of the lidar point cloud are extracted as feature points to be registered. Based on the iterative nearest point algorithm, the reference feature points and the feature points to be registered in each two adjacent frames are registered to obtain the spatial transformation matrix between adjacent frames. Based on the spatial transformation matrix, the remaining single-frame point clouds in the lidar point cloud are aligned to the first coordinate system of the reference frame to obtain the initial lidar global point cloud. Based on the initial global point cloud of the lidar, iterative nearest point registration is performed again on the virtual corner data of each single frame lidar point cloud to obtain the first error optimization matrix of each single frame lidar point cloud. The initial global point cloud of the lidar is corrected based on the first error optimization matrix to obtain the optimized global point cloud of the lidar.

[0008] As one preferred embodiment, the step of performing single-modal registration processing on the lidar point cloud and the camera image based on the virtual corner data corresponding to each single-frame point cloud to obtain the lidar global point cloud and the camera global feature point cloud further includes: Based on the timestamp of the LiDAR point cloud, the camera image is subjected to frame interpolation processing to obtain a camera frame that is consistent with the timestamp of each single frame of the LiDAR point cloud. Based on the pre-calibrated camera intrinsic parameter matrix, the virtual corner data of each single frame of LiDAR point cloud is projected onto the camera frame with the corresponding timestamp to generate virtual corner projection feature points on the camera side. The camera frame whose timestamp matches that of the lidar reference frame in the camera image is taken as the camera reference frame. The corresponding virtual corner projection feature points are extracted as camera reference feature points, and the virtual corner projection feature points of the remaining camera frames are extracted as camera registration feature points. Based on the iterative nearest point algorithm, the camera reference feature points and the camera feature points to be registered are registered in each two adjacent frames to obtain the spatial transformation matrix between adjacent frames of the camera. Based on the spatial transformation matrix, all other camera frames are aligned to the second coordinate system of the camera reference frame to obtain the initial global feature point cloud of the camera. Based on the initial global feature point cloud of the camera, iterative nearest point registration is performed again on the virtual corner projection feature points of each camera frame to obtain the second error optimization matrix of each camera frame; The initial camera global feature point cloud is corrected based on the second error optimization matrix to obtain the optimized camera global feature point cloud.

[0009] As one preferred embodiment, the step of projecting each three-dimensional coordinate in the global point cloud of the lidar based on the global extrinsic matrix and the pre-calibrated camera intrinsic matrix to the two-dimensional pixel coordinates of the corresponding timestamp camera image to generate a colored lidar global point cloud includes: Based on the global extrinsic matrix, each three-dimensional coordinate in the global point cloud of the lidar is transformed from the lidar coordinate system to the camera coordinate system to obtain the camera's three-dimensional coordinates in the camera coordinate system. Based on the pre-calibrated camera intrinsic parameter matrix, the three-dimensional coordinates of the camera are projected onto a camera frame that is consistent with the timestamp of the lidar point cloud, and the corresponding two-dimensional pixel coordinates are calculated. The target color value corresponding to the two-dimensional pixel coordinates is obtained by performing bilinear interpolation on the two-dimensional pixel coordinates. Based on the mapping relationship between each three-dimensional coordinate in the global point cloud of the lidar and the corresponding target color value, an initial colored lidar point cloud is generated. The initial colored lidar point cloud is subjected to voxelization filtering to obtain the global point cloud of the colored lidar.

[0010] Another embodiment of the present invention provides a power equipment testing system, comprising: The acquisition module is used to acquire lidar point cloud and camera images of the target power facility in response to power equipment detection signals; The extraction module is used to perform feature plane fitting processing on each single frame of the lidar point cloud, and extract virtual corner point data based on the result of the feature plane fitting processing. The first registration module is used to perform single-modal registration processing on the lidar point cloud and the camera image respectively based on the virtual corner point data corresponding to each single frame point cloud, to obtain the lidar global point cloud and the camera global feature point cloud; wherein, the single-modal registration processing is designed to optimize and offset the multi-frame registration error accumulated by the feature alignment of adjacent frames based on loop closure detection. The second registration module is used to perform cross-modal registration processing on the global point cloud of the lidar and the global feature point cloud of the camera to obtain a global extrinsic parameter matrix. The first projection module is used to project each three-dimensional coordinate in the global point cloud of the lidar to the two-dimensional pixel coordinates of the corresponding timestamp camera image based on the global extrinsic matrix and the pre-calibrated camera intrinsic matrix, so as to generate a colored lidar global point cloud. The second projection module is used to perform semantic segmentation processing on the camera images to obtain two-dimensional semantic results corresponding to each frame; and to back-project the two-dimensional semantic results onto the three-dimensional point cloud space of the corresponding colored lidar global point cloud based on the global extrinsic matrix and the camera intrinsic matrix to obtain a semantically annotated three-dimensional global point cloud. The generation module is used to generate the detection results of the target power facility based on the three-dimensional global point cloud.

[0011] As one preferred embodiment, the extraction module is specifically used for: Based on the random sampling consensus algorithm, non-planar noise points in each single frame of the lidar point cloud are removed, and valid point clouds that satisfy the planar features are retained. Based on the three-axis least squares method, plane fitting calculations are performed on the effective point cloud in the x-axis, y-axis, and z-axis directions respectively to obtain the plane fitting error corresponding to each axis direction. The plane parameter with the smallest fitting error is selected as the feature plane fitting result of the single frame point cloud. The feature planes generated by fitting the feature planes are extended along the three-dimensional space dimension to obtain the extended planes corresponding to each feature plane. Calculate the spatial intersection points of the extension lines of adjacent feature planes in the extended plane, and use the spatial intersection points as the virtual corner point data corresponding to the single frame point cloud.

[0012] As one preferred embodiment, the first registration module is specifically used for: Using the first frame of the lidar point cloud as the reference frame, the virtual corner data of the reference frame is extracted as reference feature points; and the virtual corner data corresponding to each of the remaining single frames of the lidar point cloud are extracted as feature points to be registered. Based on the iterative nearest point algorithm, the reference feature points and the feature points to be registered in each two adjacent frames are registered to obtain the spatial transformation matrix between adjacent frames. Based on the spatial transformation matrix, the remaining single-frame point clouds in the lidar point cloud are aligned to the first coordinate system of the reference frame to obtain the initial lidar global point cloud. Based on the initial global point cloud of the lidar, iterative nearest point registration is performed again on the virtual corner data of each single frame lidar point cloud to obtain the first error optimization matrix of each single frame lidar point cloud. The initial global point cloud of the lidar is corrected based on the first error optimization matrix to obtain the optimized global point cloud of the lidar.

[0013] As one preferred embodiment, the first registration module is further configured to: Based on the timestamp of the LiDAR point cloud, the camera image is subjected to frame interpolation processing to obtain a camera frame that is consistent with the timestamp of each single frame of the LiDAR point cloud. Based on the pre-calibrated camera intrinsic parameter matrix, the virtual corner data of each single frame of LiDAR point cloud is projected onto the camera frame with the corresponding timestamp to generate virtual corner projection feature points on the camera side. The camera frame whose timestamp matches that of the lidar reference frame in the camera image is taken as the camera reference frame. The corresponding virtual corner projection feature points are extracted as camera reference feature points, and the virtual corner projection feature points of the remaining camera frames are extracted as camera registration feature points. Based on the iterative nearest point algorithm, the camera reference feature points and the camera feature points to be registered are registered in each two adjacent frames to obtain the spatial transformation matrix between adjacent frames of the camera. Based on the spatial transformation matrix, all other camera frames are aligned to the second coordinate system of the camera reference frame to obtain the initial global feature point cloud of the camera. Based on the initial global feature point cloud of the camera, iterative nearest point registration is performed again on the virtual corner projection feature points of each camera frame to obtain the second error optimization matrix of each camera frame; The initial camera global feature point cloud is corrected based on the second error optimization matrix to obtain the optimized camera global feature point cloud.

[0014] As one preferred embodiment, the first projection module is specifically used for: Based on the global extrinsic matrix, each three-dimensional coordinate in the global point cloud of the lidar is transformed from the lidar coordinate system to the camera coordinate system to obtain the camera's three-dimensional coordinates in the camera coordinate system. Based on the pre-calibrated camera intrinsic parameter matrix, the three-dimensional coordinates of the camera are projected onto a camera frame that is consistent with the timestamp of the lidar point cloud, and the corresponding two-dimensional pixel coordinates are calculated. The target color value corresponding to the two-dimensional pixel coordinates is obtained by performing bilinear interpolation on the two-dimensional pixel coordinates. Based on the mapping relationship between each three-dimensional coordinate in the global point cloud of the lidar and the corresponding target color value, an initial colored lidar point cloud is generated. The initial colored lidar point cloud is subjected to voxelization filtering to obtain the global point cloud of the colored lidar.

[0015] Compared with the prior art, the beneficial effects of the embodiments of the present invention are at least one of the following: 1) This invention relies on a single-modal registration mechanism optimized by feature plane fitting and loop closure detection to effectively offset the cumulative error of adjacent frame registration, ensuring the spatial consistency of the global point cloud of the lidar and the global feature point cloud of the camera in a unified coordinate system. The generated 3D model has accurate scale and continuous structure, completely solving the problems of lack of real scale in traditional 2D technology and registration drift in traditional point cloud technology. On the other hand, the camera image provides subpixel-level fine semantic boundaries through YOLO instance segmentation, which significantly improves the recognition recall rate of small parts such as hardware and wire clamps. Then, it is back-projected to the 3D point cloud through the global extrinsic matrix to achieve the synergy of "fine semantics + real 3D scale". Even in the complex environment of dense obstruction of substation equipment and strong metal reflection, it can stably output equipment category, spatial location and geometric size information, making up for the shortcomings of the lack of single semantics or scale in traditional technology.

[0016] 2) This invention possesses outstanding engineering practicality and cost advantages, perfectly adapting to the actual needs of power operation and maintenance scenarios. At the deployment level, semantic discrimination is concentrated in the two-dimensional image domain, with computational load increasing approximately linearly with pixel size. On the three-dimensional side, only environment reconstruction and parameter projection are performed. Compared to point-level / voxel-level networks in point cloud space, this significantly reduces overall power consumption and computing power requirements, enabling real-time detection directly at edge devices. At the data operation and maintenance level, two-dimensional image annotation can be completed using conventional instance masking tools, and some geometric features can be automatically extracted, avoiding the high labor costs of large-scale point-level annotation. Furthermore, LiDAR-camera calibration and point cloud reconstruction share a single calibration system, greatly improving equipment portability and ease of operation. In addition, the generated colored semantic 3D point cloud possesses structural, textural, and semantic information, directly supporting the construction of digital twin models of power facilities. This provides a visual data foundation for equipment status assessment and fault early warning. Simultaneously, the robustness of multimodal fusion allows it to adapt to complex operating conditions such as long-tail equipment categories and long-range data acquisition, significantly promoting the digital and intelligent upgrade of power operation and maintenance. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a power facility testing method in one embodiment of the present invention; Figure 2 This is a schematic diagram of loop closure detection in one embodiment of the present invention; Figure 3 This is a schematic diagram of a loop closure optimization method in one embodiment of the present invention; Figure 4 This is a structural block diagram of a power facility detection system according to one embodiment of the present invention; Figure label: Among them, 11 is the acquisition module; 12 is the extraction module; 13 is the first registration module; 14 is the second registration module; 15 is the first projection module; 16 is the second projection module; and 17 is the generation module. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0020] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0021] One embodiment of the present invention provides a method for testing power facilities. For details, please refer to [link / reference]. Figure 1 , Figure 1 The diagram shown is a flowchart illustrating a power facility testing method according to one embodiment of the present invention, which includes: S1: In response to the power equipment detection signal, acquire the lidar point cloud and camera image of the target power facility; among them, the power equipment detection signal is the core instruction that triggers this step, and its source needs to be clearly defined in combination with the power operation and maintenance scenario: it can be an active detection signal initiated by operation and maintenance personnel according to the regular inspection plan, such as the substation equipment scanning instruction issued by the operation and maintenance terminal, or a passive early warning signal automatically generated by the power grid status monitoring system based on equipment operating parameters (such as abnormal current and temperature) or environmental parameters (such as excessive humidity and dust). Both types of signals must carry target detection area information to ensure that data collection focuses on the target power facility rather than indiscriminate collection across the entire scenario, and avoid redundant data consuming computing power.

[0022] In this embodiment, multi-source data collaborative acquisition is achieved through an integrated LiDAR and camera device. This acquisition device integrates a LiDAR module, an RGB camera module, and a hard-triggered synchronization unit. Hardware circuit design ensures frame-level time consistency between the two; that is, for every frame of point cloud output by the LiDAR, the camera is simultaneously triggered to acquire one frame of image, completely solving the timestamp deviation problem caused by clock drift in traditional independent sensors.

[0023] Equipment deployment needs to be tailored to the characteristics of the substation scenario. Specifically, if the target facility is a static device such as a fixed cabinet or bracket, the integrated device can be fixed to a rotatable gimbal. If it is an outdoor device such as a transmission line tower, it can be mounted on a drone or mobile inspection vehicle to ensure that the acquisition view covers the entire scene of the target facility, including the equipment itself (such as the cabinet shell and insulator strings), connecting components (such as hardware and clamps), and surrounding related structures (such as support brackets), so as to avoid incomplete subsequent 3D reconstruction due to missing viewpoints.

[0024] The lidar scans the target scene by emitting laser beams, receives reflected light signals, and calculates the three-dimensional coordinates and reflection intensity value of each laser point to form a single frame of point cloud data. In this embodiment, the lidar is set to a 10Hz acquisition frame rate to ensure that the spatial overlap of adjacent frame point clouds is not less than 30% (to provide sufficient overlap features for subsequent single-modal registration); the scanning range needs to cover the three-dimensional spatial boundary of the target facility. For example, for a cabinet with a height of 3 meters, the Z-axis range of the point cloud acquisition needs to include a 1-meter redundant area from the ground to the top of the cabinet to avoid missing the hardware components at the top.

[0025] The camera module must be an RGB camera with a resolution of at least 1920×1080. The lens focal length must be compatible with the scanning range of the LiDAR. For example, if the LiDAR has a scanning width of 5 meters at a distance of 10 meters, the camera must use an 8mm focal length lens to ensure that the camera's field of view can completely cover the LiDAR's scanning area at the same acquisition position, avoiding blind spots where point cloud data exists but the image lacks corresponding texture. During acquisition, pay attention to the camera's orientation. Specifically, if the target is a scene around a cabinet, the camera should be oriented to the side (not downwards) to ensure that the image captures the texture details of the equipment surface (such as cabinet markings and color differences in insulator strings). If there are calibration objects, ensure that the calibration objects are within the overlapping area of ​​the camera and LiDAR's fields of view to provide feature basis for subsequent external parameter calibration.

[0026] During data acquisition, the hard trigger unit of the integrated device needs to record the timestamps of the LiDAR frames and camera frames in real time, with the error controlled within 1ms. After acquisition, the data management module needs to perform consistency verification on the timestamps. Specifically, if a LiDAR frame does not have a corresponding camera frame with a specific timestamp, re-acquisition needs to be triggered; if there are image frames with duplicate timestamps, the frame with higher clarity and no motion blur needs to be retained. At the same time, the raw data needs to be standardized in format. Specifically, the LiDAR point cloud is converted to PCD format, and the camera image is saved in JPEG format, with files named according to the timestamp + device number to ensure that subsequent steps can quickly match point clouds and images with the same timestamp through filenames.

[0027] In this embodiment, step S1 outputs a structured multi-source dataset, which must contain three core types of content: first, the original LiDAR point cloud files, each file corresponding to a frame of point cloud, including three-dimensional coordinates and reflection intensity; second, the synchronization camera image files, each file corresponding to a frame of point cloud timestamp, including device texture information; and third, the acquisition metadata files, which record parameters such as LiDAR frame rate, camera resolution, hard trigger synchronization error, acquisition time, and target facility location.

[0028] S2: Perform feature plane fitting processing on each single frame of the lidar point cloud, and extract virtual corner data based on the result of the feature plane fitting processing.

[0029] Preferably, in one embodiment of the present invention, non-planar noise points in each single frame of the lidar point cloud are removed based on the Random Sample Consensus (RANSAC) algorithm, retaining valid point clouds that satisfy planar characteristics. The purpose of removing non-planar noise points is to filter valid point clouds that satisfy planar characteristics from single frame point clouds using the RANSAC algorithm, eliminating interference data. In specific implementation, the number of algorithm iterations (usually 100-200 times, to cover the possible noise proportion in the substation scenario and avoid missing valid planar points) and a distance threshold (usually 2-5 mm, set based on the lidar's ranging accuracy; if the distance from a point to the initially fitted plane exceeds this threshold, it is determined to be a non-planar noise point) need to be set. The specific operation process includes: For each single frame of point cloud, three points are randomly selected using the RANSAC algorithm to fit an initial plane. The number of points in the plane that meet the distance threshold, i.e. the number of interior points, is counted. This process is repeated until the initial plane with the largest number of interior points is found. All interior points in the plane are retained as valid point clouds, and the remaining points are determined as non-planar noise points and removed.

[0030] Furthermore, based on the three-axis least squares method, plane fitting calculations are performed on the effective point cloud along the x-axis, y-axis, and z-axis respectively to obtain the plane fitting error corresponding to each axis direction. The plane parameter with the smallest fitting error is selected as the feature plane fitting result of the single-frame point cloud. The three-axis least squares method refers to performing least squares plane fitting on the effective point cloud along the x-axis, y-axis, and z-axis respectively, and selecting the optimal fitting result by comparing the errors. This method solves the problem that traditional single-axis fitting cannot adapt to the diversity of planes in substation equipment. Its specific implementation process includes: Establish a coordinate system, specifically using the lidar coordinate system as the reference, with the x-axis along the horizontal scanning direction, the y-axis along the vertical scanning direction, and the z-axis along the lidar emission direction. Fit planes along each of the three axes. Specifically, when fitting along the x-axis, keep the x-coordinate constant and perform least-squares plane calculations on the effective point cloud of the yz coordinates to obtain the plane equation in the x-axis direction. Similarly, obtain the plane equations in the y-axis and z-axis directions. Calculate the fitting error. Specifically, for the plane equation in each direction, calculate the sum of squared residuals from all effective points to that plane (an error quantification index; the smaller the value, the higher the fitting accuracy). Select the plane equation with the smallest sum of squared residuals and its corresponding parameters, and use it as the feature plane fitting result for that single frame of point cloud.

[0031] Furthermore, the feature planes generated from the fitting results of each feature plane are extended along the three-dimensional space dimension to obtain the extended planes corresponding to each feature plane; the spatial intersection points of the extended lines of adjacent feature planes in the extended planes are calculated, and the spatial intersection points are used as the virtual corner point data corresponding to the single frame point cloud. Here, a virtual corner point refers to a virtual feature point with unique spatial coordinates obtained by extending the feature planes along the three-dimensional space and solving for the intersection points of the lines of adjacent extended planes.

[0032] Compared to physical corner points extracted directly from point clouds (such as the edge corners of devices, which are prone to inaccurate coordinates due to low sampling density), virtual corner points are generated by extending a plane, relying on the high-precision fitting results of the feature plane, resulting in stronger coordinate stability and uniqueness. The specific extraction process includes: Plane extension specifically includes extending the feature plane in three-dimensional space in all directions based on the plane equation obtained from the feature plane fitting result (the extension range must cover the boundary of the single-frame point cloud to ensure that the extension lines of adjacent planes can intersect), thus obtaining the extended plane; solving for the intersection line, specifically, for adjacent feature planes in the single-frame point cloud (such as the front and right sides of a cabinet, which are perpendicular in physical space), calculating their spatial intersection line by simultaneously solving the equations of the two extended planes; solving for the virtual corner point, specifically, solving for the spatial intersection point of the two intersecting lines by simultaneously solving the equations of the two lines, which is the virtual corner point.

[0033] In this embodiment, 4-6 virtual corner points need to be extracted for each single frame point cloud (to cover different areas of the point cloud and avoid registration deviation caused by feature concentration). For example, 4 virtual corner points can be extracted based on the four feature planes of the calibration object, and 3 virtual corner points can be extracted based on the three orthogonal planes of the device, to ensure that there are enough feature anchor points for subsequent registration.

[0034] Preferably, in one embodiment of the present invention, the method further includes quality verification of the fitting results to prevent invalid features caused by local noise or algorithm parameter deviations from entering subsequent processes. Specific verification dimensions include: Valid point cloud percentage verification: If the number of valid point clouds on a certain feature plane accounts for less than 10% of the total number of point clouds in a single frame (this needs to be adjusted according to the size of the device plane; for example, the plane percentage of small fittings can be appropriately reduced to 5%), then the fitting of that feature plane is deemed invalid, and the RANSAC distance threshold needs to be readjusted before fitting again. Virtual corner point rationality verification: Calculate the distance between virtual corner points. If the distance between two virtual corner points deviates from the physical dimensions of the equipment (such as the length, width, and height of the cabinet, which can be obtained through pre-entered equipment parameters) by more than 10%, then the virtual corner point is determined to be an abnormal point, and the plane extension direction or intersection calculation process needs to be rechecked. Plane equation consistency check: For two consecutive frames of point cloud from the same device, if the angle between the normal vectors of their feature planes exceeds 5° (representing a sudden change in plane attitude, which may be due to device movement or algorithm error during point cloud acquisition), the abnormal frame needs to be removed or the point cloud of that frame needs to be reacquired.

[0035] The output of this step includes: a feature plane parameter file, stored in TXT format, recording the number of feature planes in each frame of point cloud, the plane equation coefficients of each feature plane, the number of effective point clouds, and the fitting error value; a virtual corner coordinate file, stored in CSV format, recording the number of virtual corners in each frame of point cloud, the 3D coordinates of each virtual corner, and the corresponding two intersecting plane numbers; and a processing log file, recording the RANSAC parameters of the point cloud frame, the error data of the 3A-LSM fitting process, and the quality verification results.

[0036] S3: Based on the virtual corner data corresponding to each single-frame point cloud, single-mode registration processing is performed on the lidar point cloud and the camera image respectively to obtain the lidar global point cloud and the camera global feature point cloud; wherein, the single-mode registration processing is designed to optimize and offset the multi-frame registration error accumulated by the feature alignment of adjacent frames based on loop closure detection.

[0037] Preferably, in one embodiment of the present invention, the registration on the lidar side uses virtual corner points as the core anchor points. Specifically, the first frame in the lidar point cloud is selected as the reference frame (since the first frame has no prior registration error, it can be used as the origin of the global coordinate system), and the virtual corner point data of the frame after processing by S2 is extracted as the reference feature. At the same time, the virtual corner point data of the remaining single frame point clouds are extracted as the feature points to be registered.

[0038] Furthermore, inter-frame registration is performed based on the Iterative Closest Point (ICP) algorithm. The ICP algorithm is a registration algorithm that iteratively finds the closest corresponding points between two sets of point clouds and minimizes the distance error between these corresponding points to solve for the spatial transformation matrix. Specifically, for each adjacent pair of frames, such as frame k and frame k+1, the reference feature points and the feature points to be registered are input into the ICP algorithm. By minimizing the Euclidean distance error between the corresponding points, the spatial transformation matrix between the two frames is solved. This spatial transformation matrix includes a rotation matrix and a translation matrix, which together describe the position and pose changes of frame k+1 relative to frame k.

[0039] Based on the spatial transformation matrix, all single-frame point clouds to be registered are aligned to the coordinate system of the reference frame, i.e., the first coordinate system. All aligned single-frame point clouds are integrated to obtain the initial global point cloud of the lidar.

[0040] The above steps can initially achieve the stitching of multi-frame point clouds. However, because the registration errors between adjacent frames accumulate frame by frame (e.g., the error in the 3rd frame includes the superposition of errors from the 1st-2nd and 2nd-3rd frames), the initial global point cloud may have scale drift or local misalignment. See details. Figure 2 ,in Figure 2 This is a schematic diagram of loop closure detection provided in an embodiment of the present invention. Figure 2 The left and right parts represent the ideal case of loop closure optimization and the phenomenon of cumulative error, respectively.

[0041] To address the issue of accumulated errors in the initial global point cloud, this embodiment optimizes the process through loop closure detection. Loop closure detection refers to a method that uses the global point cloud as a reference to reverse-engineer the pose of each single-frame point cloud, thereby offsetting the accumulation of inter-frame errors.

[0042] See details Figure 3 ,in, Figure 3 This is a schematic diagram of a loop closure optimization method provided in an embodiment of the present invention. The specific implementation process includes: using an initial global point cloud of a LiDAR as a global reference, wherein the initial global point cloud is formed by integrating all single-frame point clouds, and its features can reflect the geometric overall picture of the entire scene, serving as a reference for error correction; for each virtual corner point of the single-frame LiDAR point cloud, ICP registration is performed again with the virtual corner points of the initial global point cloud, and the error optimization matrix of that single frame relative to the global reference is solved. This matrix quantifies the pose deviation of the single frame due to accumulated errors; the error optimization matrix is ​​applied to the corresponding single-frame point cloud to adjust the position and pose of the single frame in the global coordinate system. For example, for the nth frame, which is far from the reference frame, if it deviates from its true position by 5cm due to accumulated errors, the error optimization matrix can pull it back to the correct pose. After correction, all single-frame point clouds are re-integrated to obtain an optimized global LiDAR point cloud. This point cloud completely solves the problem of error accumulation between adjacent frames, has a uniform scale and continuous structure, and can ensure global geometric consistency even for multi-frame data acquired over long distances.

[0043] Preferably, in one embodiment of the present invention, the registration on the camera side needs to first solve the problems of time synchronization and feature association. Since the acquisition frame rates of the camera and the LiDAR may be different (e.g., 10Hz for LiDAR and 20Hz for camera), frame interpolation processing of the camera image needs to be performed based on the timestamp of the LiDAR. The frame interpolation processing refers to generating an intermediate camera frame that is completely consistent with the timestamp of a single LiDAR frame based on the pixel information of adjacent camera frames. Specifically, it includes calculating the difference between the timestamp of the LiDAR frame and the timestamp of the camera frame, and using a bilinear interpolation algorithm to supplement the pixel texture, ensuring that each single frame of LiDAR point cloud has a unique corresponding synchronized camera frame, and avoiding feature mismatch caused by time misalignment.

[0044] Furthermore, by calling the pre-calibrated camera intrinsic parameter matrix (including focal length and principal point coordinates), the virtual corner points of each single-frame LiDAR are converted into two-dimensional virtual corner point projection feature points of the synchronized camera frame using the perspective projection formula. This step transforms the three-dimensional features of the LiDAR into the two-dimensional features of the camera, providing common source anchor points for camera-side registration and solving the problem of the camera lacking direct virtual corner points.

[0045] The logic for constructing the global point cloud on the camera side is the same as that on the lidar side. First, the camera reference frame is used as the anchor point. Then, a synchronous camera frame with the same timestamp as the lidar reference frame is selected as the camera reference frame. The virtual corner projection feature points of this frame are extracted as the camera reference feature points. The projection feature points of the other synchronous camera frames are used as the camera registration feature points.

[0046] ICP registration is performed on adjacent camera frames. Specifically, the camera reference feature points and the feature points to be registered are input into ICP, the spatial transformation matrix of adjacent camera frames is solved, and all camera frames to be registered are aligned to the coordinate system of the camera reference frame, i.e., the second coordinate system, according to the matrix. The initial global feature point cloud of the camera is obtained by integrating the coordinates. The feature point cloud is a point cloud composed of virtual corner projection feature points, which is used to characterize the global feature distribution of the camera image.

[0047] The loop closure detection optimization is performed. Specifically, based on the initial camera global feature point cloud, ICP registration is performed again on the projected feature points of each camera frame to solve the second error optimization matrix, correct the pose deviation of each camera frame, and offset the cumulative inter-frame error on the camera side (such as inter-frame offset caused by camera shake), and finally obtain the optimized camera global feature point cloud.

[0048] S4: Perform cross-modal registration processing on the global point cloud of the lidar and the global feature point cloud of the camera to obtain the global extrinsic parameter matrix.

[0049] Cross-modal registration relies on the iterative nearest-point algorithm, but ICP is prone to getting trapped in local optima. Therefore, it is necessary to pre-estimate the initial extrinsic parameter matrix based on the first frame of data to provide a precise starting point for iteration. This embodiment takes the lidar reference frame and camera reference frame determined in S3 as the object.

[0050] Specifically, this invention first uses an initial extrinsic parameter estimation method based on the least squares method to estimate the point cloud of the first frame of the lidar. The first frame point cloud of the camera with its corresponding timestamp Registration was performed, and initial values ​​for extrinsic parameters were obtained. Then, a global point cloud was constructed using the LiDAR and camera. and The feature points of all frames constitute the global feature point cloud, denoted as . and The two are then globally registered using the ICP algorithm. The specific registration steps are as follows: First, locate the global feature point cloud of the lidar. Each point in Global feature point cloud of the camera nearest neighbor in As the corresponding point.

[0051] Because the global feature point cloud is composed of The feature point cloud is composed of frames, with each frame containing ten feature points. Therefore, the global feature point clouds of the LiDAR and the camera respectively contain... The point has been selected in (1). The correspondence between these points, Substituting the corresponding points into the loss function of the formula, the least squares method is used to register the two global feature point clouds, and the error function is solved. Minimal rotation and translation matrix and This serves as the result of the external parameter matrix for this iteration. The error function is also included. Represented as: The results obtained and The global feature point cloud of the LiDAR applied to the previous iteration This makes it consistent with the camera's global feature point cloud. Align them.

[0052] Repeat the above steps until the preset threshold or maximum number of iterations is reached to obtain the final global extrinsic parameter matrix. and .

[0053] S5: Based on the global extrinsic matrix and the pre-calibrated camera intrinsic matrix, project each three-dimensional coordinate in the global point cloud of the lidar to the two-dimensional pixel coordinates of the corresponding timestamp camera image to generate a colored lidar global point cloud.

[0054] The original coordinates of the LiDAR global point cloud are based on the LiDAR coordinate system, while the pixel coordinates of the camera image are based on the image plane coordinate system. A cross-coordinate system transformation must first be performed using a global extrinsic parameter matrix to map the 3D points from the LiDAR coordinate system to the camera coordinate system. Specifically, the 3D coordinates of each 3D point in the LiDAR global point cloud in the camera coordinate system are calculated based on the global extrinsic parameter matrix output by S4, expressed as: in, The coordinates of the midpoint of the lidar point cloud. These are the coordinates of the point in the image. For the camera intrinsic parameter matrix, and The external parameters of the LiDAR and camera, obtained by solving the above formula after global optimization, are the rotation and translation matrices between the two sensor coordinate systems.

[0055] Because of what is sought Since the coordinates are not integers, interpolation is required to determine the color information of this point. This is achieved by performing linear interpolation in two directions on the RGB three-channel color values ​​to obtain the non-integer coordinates. The color values ​​are used to obtain the color information of the 3D point cloud. The specific steps are as follows: First, leave The four most recent integer coordinates are: , , , ,exist Linear interpolation in the direction: After that Linear interpolation in the direction: in, Represents color value, This is the final bilinear interpolation result.

[0056] Each valid 3D point in the LiDAR global point cloud (after coordinate system transformation and pixel validity determination) is associated with the target color value obtained through bilinear interpolation, constructing a point cloud data structure of 3D coordinates + RGB color to form the initial colored LiDAR point cloud. Each frame of the LiDAR global point cloud must correspond to a camera image in S1 with the same timestamp for that frame to avoid color misalignment caused by cross-frame matching (e.g., a LiDAR point cloud frame corresponding to an incorrect camera frame, resulting in "the device is red but mapped to blue"). Furthermore, the initial point cloud needs to be marked with missing color points; that is, 3D points whose pixel coordinates exceed the image range or are missing from the camera frame are marked as needing to be supplemented. Subsequent filling can be achieved through color interpolation of adjacent frames, ensuring that the initial point cloud has no large areas of color blanks.

[0057] Because the density of the global point cloud of LiDAR is extremely high (each frame contains more than 60,000 points, and the global point cloud reaches millions), the initial colored point cloud data volume is too large, which will occupy a lot of storage space and affect the efficiency of subsequent semantic processing. It needs to be simplified by voxelization filtering. Voxelization filtering refers to dividing the point cloud space into a fixed-size cubic voxel grid, and using the voxel core to represent all points in the grid, thereby realizing the point cloud downsampling method.

[0058] First, based on the maximum and minimum values ​​of all points in the point cloud along the three axes, the point cloud space is divided into voxel grids of fixed size. For each voxel grid, the centroid of all points in the voxel grid is calculated using a formula as the representative of the points in the voxel grid.

[0059] in, Representing coordinates The centroid of all points in the voxel grid, This represents the coordinates of each point in the voxel grid. This represents the total number of points within the voxel grid.

[0060] Finally, all points except the centroid in the voxel are deleted, completing the downsampling of the color point cloud terrain scene reconstruction result. The final colored LiDAR global point cloud is output in a standardized format, including: the initial number of point clouds, the number of point clouds after filtering, the proportion of color missing points, voxel size, average color error, and other information.

[0061] S6: Perform semantic segmentation on the camera images to obtain two-dimensional semantic results corresponding to each frame; and project the two-dimensional semantic results back onto the three-dimensional point cloud space of the corresponding colored lidar global point cloud based on the global extrinsic matrix and the camera intrinsic matrix to obtain a semantically annotated three-dimensional global point cloud. In this embodiment, YOLO instance segmentation with P2 high-resolution branch (Backbone+FPN+Decoupling Head+ProtoNet) is used as the semantic segmentation model. This model is designed specifically for the characteristics of substations with many small components and dense equipment, and can balance segmentation accuracy and inference speed.

[0062] The model backbone uses CSPDarknet53 to enhance feature extraction capabilities; the Feature Pyramid Network (FPN) introduces a P2 high-resolution branch, which retains high-resolution feature maps with 4x downsampling compared to the P3 and above branches in conventional YOLO, and can capture the boundaries of sub-pixel-level fine parts such as fittings and drainage plates; the head is decoupled from the loss calculation of classification and regression tasks to avoid interference between tasks; ProtoNet is responsible for generating prototype feature maps for semantic masks.

[0063] The preprocessed image patches are input into the model, and each patch outputs "class probability", "2D bounding box", and "instance mask prototype coefficients" for inference, as follows: M j =σ(Upsample(∑ α kj ,P k )) Among them, instance mask From prototype feature map With coefficient The linear combination is upsampled and then passed through the Sigmoid function. A binary mask is obtained (1 for the target region and 0 for the background). For the mask of the overlapping area between blocks, a weighted average fusion is used (the mask value of the pixel in the overlapping area is the average of the inference results of the two adjacent blocks) to avoid semantic discontinuity caused by block division.

[0064] The loss functions for model training and inference take into account multi-dimensional accuracy, including classification loss (addressing the long-tail problem of equipment categories, such as the imbalance of having many samples for "insulator strings" and few for "positioning pins"), bounding box regression loss (improving the accuracy of equipment boundary positioning), masking loss (optimizing the overlap between semantic regions and ground truth masks), and boundary quality loss (enhancing the edge clarity of small components such as hardware). Through this model, each frame of camera image can output pixel-by-pixel semantic labels, instance-level masks, and category confidence scores, forming a complete two-dimensional semantic result.

[0065] To facilitate subsequent back projection, the two-dimensional semantic results of each image frame need to be structurally stored according to the "pixel-semantic" correspondence, forming an index table that can be efficiently queried. Specifically, common equipment categories in substations are uniformly coded (e.g., "cabinet" = 1, "insulator string" = 2, "fittings" = 3, "line clamp" = 4, background = 0) to avoid redundancy in category names; for each pixel in each image frame, three core pieces of information are stored: "semantic label code", "instance ID", and "category confidence". For example, the index record for pixel (200, 300) is "label = 3 (fittings), instance ID = 5 (5th fitting instance), confidence = 0.95"; a confidence threshold is set (usually 0.7, which can be adjusted according to the accuracy requirements of operation and maintenance). Pixels with a confidence level below the threshold are marked as "semantic uncertain" and are not included in the back projection for the time being (to avoid erroneous semantics polluting the 3D point cloud). The confidence level can be supplemented later through semantic interpolation of neighboring pixels.

[0066] In this embodiment, each pixel with a semantic label does not correspond to a unique 3D point in the camera coordinate system. Instead, it represents a "projected ray" that originates from the camera's optical center and passes through that pixel. The direction of this ray is determined by the camera's intrinsic parameters, such as the principal point coordinates (image center) and focal length (optical parameters), which together constrain the spatial angle of the ray. The ray extends along 3D space, covering the area where the LiDAR point cloud may exist. Next, this ray in the camera coordinate system needs to be transformed to the LiDAR coordinate system using a global extrinsic parameter matrix. The global extrinsic parameter matrix contains the position offset (translation) and attitude difference (rotation) between the LiDAR and the camera. With its help, the direction and starting point of the camera ray can be synchronously transformed to the LiDAR's spatial coordinate system, placing the ray and the colored LiDAR global point cloud in the same spatial context.

[0067] For each ray transformed into the lidar coordinate system, the 3D point cloud within a certain range (usually set to 3-5 mm, adjusted according to the point cloud density to avoid matching irrelevant points) is traversed. Since the global point cloud of the colored lidar has retained the centroid point through voxelization filtering and includes 3D coordinate information, the distance from each point to the ray can be calculated. The point cloud with the closest distance is selected as the corresponding 3D point for that pixel semantics. For example, for a pixel with the semantics of "fittings", its ray will preferentially match the point cloud of the device connection part in the lidar space, rather than the point cloud of the cabinet at a distance.

[0068] For point clouds where no semantic match is found (such as points obscured on the back of a cabinet), locate the 5-8 nearest labeled semantic point clouds. Analyze the semantic category distribution of these neighboring points and determine the semantic meaning of the point by weighting the category proportions. For example, if 6 out of 8 points around an obscured point have the semantic meaning of "cabinet," then that point is labeled as "cabinet." The completion process must avoid background areas and only operate on point clouds where the device may exist to prevent misassigning background semantics to device point clouds and ensure consistency of semantic labeling.

[0069] In this embodiment, when outputting semantically annotated 3D global point clouds, the extended PCD format, which supports multi-field storage, is adopted. In addition to retaining the original 3D coordinates and RGB color information, new fields such as "semantic label encoding," "instance ID," and "category confidence" are added to facilitate direct reading by subsequent detection algorithms. At the same time, a "semantic projection report" is generated, which includes: the average accuracy of semantic segmentation for each frame of image (compared with manual annotation results), the semantic coverage of the 3D point cloud (the proportion of semantically annotated point clouds to the total point cloud, which usually needs to be ≥90%), and the semantic conflict rate (the number of conflicts between multiple pixels corresponding to the same point cloud, which needs to be ≤1%).

[0070] S7: Generate the detection results of the target power facility based on the three-dimensional global point cloud.

[0071] Based on the semantic label encoding in the point cloud, all point clouds are divided into different semantic groups. The number of device instances in each group is counted to form a basic device list. Then, each semantic group is further split into independent single device point clouds according to the instance ID. At the same time, invalid point clouds with "unknown category" and "semantic uncertainty" (confidence level < 0.7) are removed to ensure that subsequent analysis focuses on device instances with clear semantics.

[0072] Traverse the point cloud coordinates of a single instance, and obtain the geometric dimensions of the device such as length, width, height (or diameter) by calculating the difference between the maximum and minimum coordinate values ​​in the x, y, and z axes. Obtain the centroid coordinates of the device by calculating the average of the three-dimensional coordinates of all points to determine its spatial location. If there are missing parameters in the point cloud due to occlusion, supplement the calculation based on the symmetry attributes of the device (such as the symmetry structure of a rectangular cabinet) to ensure the integrity of the parameters.

[0073] The equipment status is determined based on a pre-built standard equipment parameter library. The actual size of the instance is compared with the standard size to calculate the deviation value. If it exceeds the preset threshold (e.g., ±30mm), it is judged as a size abnormality. The centroid coordinates of the instance are compared with the standard installation coordinates to calculate the spatial distance deviation. If it exceeds the allowable range (e.g., ±20mm), it is judged as a position offset abnormality. At the same time, the continuity of the point cloud is analyzed. If there are blank areas in the equipment (e.g., insulator strings) caused by non-obstruction, it is judged as a structural abnormality, and the abnormal quantitative data is recorded simultaneously.

[0074] Conduct topology analysis between devices, calculate the minimum spatial distance between point clouds of different device instances based on the device connection rule library. If the distance is too far, it is judged as a connection anomaly. At the same time, investigate the linkage risk of related devices. If the position of a certain hardware is offset and causes the insulator string to tilt, the potential anomaly of the insulator string is marked simultaneously.

[0075] Abnormal equipment is sorted by risk level: Level 1 is an emergency anomaly affecting the operation of main equipment (such as busbar fitting detachment); Level 2 is a single-equipment functional anomaly requiring immediate attention (such as fitting size deviation); and Level 3 is a planned appearance anomaly (such as uneven color in non-critical parts). Each anomaly is labeled with its specific location (e.g., "middle section of insulator string"), type (e.g., "fracture"), deviation value, and risk impact (e.g., "may lead to line outage"). A structured inspection report is then generated, including an inspection overview (inspection area, time, total number of equipment, number of anomalies), an equipment list (listing instance IDs and statuses by semantic category), anomaly details (priority, type, deviation value, location), topology analysis (normal / abnormal connection relationships, linkage risks), and visualization elements such as 3D point cloud screenshots of anomaly areas, size comparison charts, and anomaly location distribution maps. Maintenance recommendations are also provided for each anomaly.

[0076] Another embodiment of the present invention provides a power equipment testing system; for details, please refer to [link to relevant documentation]. Figure 4 , Figure 4 The diagram shown is a structural block diagram of a power equipment detection system according to one embodiment of the present invention, which includes: The acquisition module 11 is used to acquire the lidar point cloud and camera image of the target power facility in response to the power equipment detection signal; The extraction module 12 is used to perform feature plane fitting processing on each single frame of the lidar point cloud, and extract virtual corner point data based on the result of the feature plane fitting processing. The first registration module 13 is used to perform single-modal registration processing on the lidar point cloud and the camera image respectively based on the virtual corner point data corresponding to each single frame point cloud, to obtain the lidar global point cloud and the camera global feature point cloud; wherein, the single-modal registration processing is designed to optimize and offset the multi-frame registration error accumulated by the feature alignment of adjacent frames based on loop closure detection. The second registration module 14 is used to perform cross-modal registration processing on the global point cloud of the lidar and the global feature point cloud of the camera to obtain a global extrinsic parameter matrix. The first projection module 15 is used to project each three-dimensional coordinate in the global point cloud of the lidar to the two-dimensional pixel coordinates of the corresponding timestamp camera image based on the global extrinsic matrix and the pre-calibrated camera intrinsic matrix, so as to generate a colored lidar global point cloud. The second projection module 16 is used to perform semantic segmentation processing on the camera image to obtain a two-dimensional semantic result corresponding to each frame image; and to back-project the two-dimensional semantic result to the three-dimensional point cloud space of the corresponding colored lidar global point cloud based on the global extrinsic matrix and the camera intrinsic matrix to obtain a semantically annotated three-dimensional global point cloud. The generation module 17 is used to generate the detection results of the target power facility based on the three-dimensional global point cloud.

[0077] Preferably, in one embodiment of the present invention, the extraction module is specifically used for: Based on the random sampling consensus algorithm, non-planar noise points in each single frame of the lidar point cloud are removed, and valid point clouds that satisfy the planar features are retained. Based on the three-axis least squares method, plane fitting calculations are performed on the effective point cloud in the x-axis, y-axis, and z-axis directions respectively to obtain the plane fitting error corresponding to each axis direction. The plane parameter with the smallest fitting error is selected as the feature plane fitting result of the single frame point cloud. The feature planes generated by fitting the feature planes are extended along the three-dimensional space dimension to obtain the extended planes corresponding to each feature plane. Calculate the spatial intersection points of the extension lines of adjacent feature planes in the extended plane, and use the spatial intersection points as the virtual corner point data corresponding to the single frame point cloud.

[0078] Preferably, in one embodiment of the present invention, the first registration module is specifically used for: Using the first frame of the lidar point cloud as the reference frame, the virtual corner data of the reference frame is extracted as reference feature points; and the virtual corner data corresponding to each of the remaining single frames of the lidar point cloud are extracted as feature points to be registered. Based on the iterative nearest point algorithm, the reference feature points and the feature points to be registered in each two adjacent frames are registered to obtain the spatial transformation matrix between adjacent frames. Based on the spatial transformation matrix, the remaining single-frame point clouds in the lidar point cloud are aligned to the first coordinate system of the reference frame to obtain the initial lidar global point cloud. Based on the initial global point cloud of the lidar, iterative nearest point registration is performed again on the virtual corner data of each single frame lidar point cloud to obtain the first error optimization matrix of each single frame lidar point cloud. The initial global point cloud of the lidar is corrected based on the first error optimization matrix to obtain the optimized global point cloud of the lidar.

[0079] Preferably, in one embodiment of the present invention, the first registration module is further configured to: Based on the timestamp of the LiDAR point cloud, the camera image is subjected to frame interpolation processing to obtain a camera frame that is consistent with the timestamp of each single frame of the LiDAR point cloud. Based on the pre-calibrated camera intrinsic parameter matrix, the virtual corner data of each single frame of LiDAR point cloud is projected onto the camera frame with the corresponding timestamp to generate virtual corner projection feature points on the camera side. The camera frame whose timestamp matches that of the lidar reference frame in the camera image is taken as the camera reference frame. The corresponding virtual corner projection feature points are extracted as camera reference feature points, and the virtual corner projection feature points of the remaining camera frames are extracted as camera registration feature points. Based on the iterative nearest point algorithm, the camera reference feature points and the camera feature points to be registered are registered in each two adjacent frames to obtain the spatial transformation matrix between adjacent frames of the camera. Based on the spatial transformation matrix, all other camera frames are aligned to the second coordinate system of the camera reference frame to obtain the initial global feature point cloud of the camera. Based on the initial global feature point cloud of the camera, iterative nearest point registration is performed again on the virtual corner projection feature points of each camera frame to obtain the second error optimization matrix of each camera frame; The initial camera global feature point cloud is corrected based on the second error optimization matrix to obtain the optimized camera global feature point cloud.

[0080] Preferably, in one embodiment of the present invention, the first projection module is specifically used for: Based on the global extrinsic matrix, each three-dimensional coordinate in the global point cloud of the lidar is transformed from the lidar coordinate system to the camera coordinate system to obtain the camera's three-dimensional coordinates in the camera coordinate system. Based on the pre-calibrated camera intrinsic parameter matrix, the three-dimensional coordinates of the camera are projected onto a camera frame that is consistent with the timestamp of the lidar point cloud, and the corresponding two-dimensional pixel coordinates are calculated. The target color value corresponding to the two-dimensional pixel coordinates is obtained by performing bilinear interpolation on the two-dimensional pixel coordinates. Based on the mapping relationship between each three-dimensional coordinate in the global point cloud of the lidar and the corresponding target color value, an initial colored lidar point cloud is generated. The initial colored lidar point cloud is subjected to voxelization filtering to obtain the global point cloud of the colored lidar.

[0081] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A power facility detection method characterized by, The method comprises the following steps: in response to a power facility detection signal, acquiring a laser radar point cloud and a camera image of a target power facility; performing feature plane fitting processing on each single-frame point cloud in the laser radar point cloud, and extracting virtual corner point data based on the result of the feature plane fitting processing; based on the virtual corner point data corresponding to each single-frame point cloud, performing single-modal registration processing on the laser radar point cloud and the camera image respectively, to obtain a laser radar global point cloud and a camera global feature point cloud; wherein the single-modal registration processing is designed to optimize and offset the multi-frame registration error accumulated by the alignment of adjacent frame features based on loop detection; performing cross-modal registration processing on the laser radar global point cloud and the camera global feature point cloud to obtain a global extrinsic parameter matrix; based on the global extrinsic parameter matrix and a pre-calibrated camera intrinsic parameter matrix, projecting each three-dimensional coordinate in the laser radar global point cloud to a two-dimensional pixel coordinate of a camera image corresponding to a time stamp, to generate a colored laser radar global point cloud; performing semantic segmentation processing on the camera image to obtain a two-dimensional semantic result corresponding to each frame of image; and based on the global extrinsic parameter matrix and the camera intrinsic parameter matrix, inversely projecting the two-dimensional semantic result to a three-dimensional point cloud space of the corresponding colored laser radar global point cloud, to obtain a three-dimensional global point cloud with semantic annotation; generating a detection result of the target power facility according to the three-dimensional global point cloud.

2. The electric facility detection method according to claim 1, wherein The method comprises the following steps: based on a random sample consensus algorithm, removing non-planar noise points in each single-frame point cloud in the laser radar point cloud, and retaining valid point clouds satisfying planar features; based on a three-axis direction least squares method, performing planar fitting calculation on the valid point clouds in x-axis, y-axis and z-axis directions respectively, to obtain planar fitting errors corresponding to each axis direction, and screening out the planar parameters with the smallest fitting error as the feature plane fitting result of the single-frame point cloud; elongating the feature planes generated by each feature plane fitting result along the three-dimensional space dimensions to obtain an elongated plane corresponding to each feature plane; calculating the spatial intersection points of the elongated lines of adjacent feature planes in the elongated plane, and taking the spatial intersection points as the virtual corner point data corresponding to the single-frame point cloud.

3. The electric facility detecting method according to claim 1, wherein The method comprises the following steps: taking a first frame of point cloud in the laser radar point cloud as a reference frame, extracting the virtual corner point data of the reference frame as reference feature points, and respectively extracting the virtual corner point data corresponding to each single-frame point cloud in the laser radar point cloud as to-be-registered feature points; based on an iterative closest point algorithm, performing registration calculation on the reference feature points and the to-be-registered feature points of each adjacent two frames to obtain a spatial transformation matrix between adjacent frames; according to the spatial transformation matrix, aligning each single-frame point cloud in the laser radar point cloud to the first coordinate system of the reference frame to obtain an initial laser radar global point cloud; Performing iterative closest point registration on the virtual corner point data of each single-frame lidar point cloud again based on the initial lidar global point cloud, to obtain a first error optimization matrix of each single-frame lidar point cloud; Based on the first error optimization matrix, the initial lidar global point cloud is corrected to obtain an optimized lidar global point cloud.

4. The electric facility detecting method according to claim 3, wherein The single-modality registration processing of the lidar point cloud and the camera image based on the virtual corner point data corresponding to each single-frame point cloud is to obtain a lidar global point cloud and a camera global feature point cloud, and further comprises: Based on the timestamp of the lidar point cloud, the camera image is subjected to frame interpolation processing to obtain a camera frame consistent with the timestamp of each single-frame lidar point cloud; According to the pre-calibrated camera intrinsic parameter matrix, the virtual corner point data of each single-frame lidar point cloud is projected into the camera frame corresponding to the timestamp to generate virtual corner point projection features on the camera side; Taking the camera frame in the camera image consistent with the timestamp of the lidar reference frame as the camera reference frame, the virtual corner point projection features corresponding thereto are extracted as the camera reference features, and the virtual corner point projection features of the remaining camera frames are extracted as the camera features to be registered; Based on the iterative closest point algorithm, the camera reference features and the camera features to be registered of each adjacent two frames are registered to obtain a spatial transformation matrix between the adjacent camera frames; According to the spatial transformation matrix, the remaining camera frames are all aligned to the second coordinate system of the camera reference frame to obtain an initial camera global feature point cloud; Performing iterative closest point registration on the virtual corner point projection features of each camera frame again based on the initial camera global feature point cloud to obtain a second error optimization matrix of each camera frame; Based on the second error optimization matrix, the initial camera global feature point cloud is corrected to obtain an optimized camera global feature point cloud.

5. The electric facility detecting method according to claim 1, wherein The projection of each three-dimensional coordinate in the lidar global point cloud to a two-dimensional pixel coordinate of the camera image corresponding to the timestamp based on the global extrinsic parameter matrix and the pre-calibrated camera intrinsic parameter matrix to generate a colored lidar global point cloud comprises: Converting each three-dimensional coordinate in the lidar global point cloud from a lidar coordinate system to a camera coordinate system based on the global extrinsic parameter matrix to obtain camera three-dimensional coordinates in the camera coordinate system; Projecting the camera three-dimensional coordinates into the camera frame consistent with the timestamp of the lidar point cloud based on the pre-calibrated camera intrinsic parameter matrix to calculate the corresponding two-dimensional pixel coordinates; Performing bilinear interpolation processing on the two-dimensional pixel coordinates to obtain target color values corresponding to the two-dimensional pixel coordinates; According to the mapping relationship between each three-dimensional coordinate in the lidar global point cloud and the corresponding target color value, an initial colored lidar point cloud is generated; Performing voxelization filtering processing on the initial colored lidar point cloud to obtain a colored lidar global point cloud.

6. A power utility detection system characterised by, The method comprises: The acquisition module is configured to acquire a lidar point cloud and a camera image of a target power facility in response to a power facility detection signal. The extraction module is configured to perform feature plane fitting processing on each single-frame point cloud in the laser radar point cloud, and extract virtual corner point data based on a result of the feature plane fitting processing. The first registration module is configured to perform single-modality registration processing on the laser radar point cloud and the camera image based on the virtual corner point data corresponding to each single-frame point cloud, to obtain a laser radar global point cloud and a camera global feature point cloud. The second registration module is configured to perform cross-modality registration processing on the laser radar global point cloud and the camera global feature point cloud, to obtain a global extrinsic parameter matrix. The first projection module is configured to project each three-dimensional coordinate in the laser radar global point cloud to a two-dimensional pixel coordinate of a camera image corresponding to a time stamp based on the global extrinsic parameter matrix and a camera intrinsic parameter matrix pre-calibrated, to generate a colored laser radar global point cloud. The second projection module is configured to perform semantic segmentation processing on the camera image, to obtain a two-dimensional semantic result corresponding to each frame of image, and to perform inverse projection of the two-dimensional semantic result to a three-dimensional point cloud space of the corresponding colored laser radar global point cloud based on the global extrinsic parameter matrix and the camera intrinsic parameter matrix, to obtain a three-dimensional global point cloud with semantic annotation. The generation module is configured to generate a detection result of the target power facility according to the three-dimensional global point cloud.

7. The power facility detection system of claim 6, wherein, The extraction module is specifically configured to: remove non-planar noise points in each single-frame point cloud in the laser radar point cloud based on a random sample consensus algorithm, and retain valid point clouds satisfying a planar feature; perform planar fitting calculation in x-axis, y-axis and z-axis directions on the valid point clouds based on a three-axis direction least square method, to obtain a planar fitting error corresponding to each axis direction, and screen a planar parameter with a minimum fitting error as a feature plane fitting result of the single-frame point cloud; extend feature planes generated by each feature plane fitting result along three-dimensional space dimensions, to obtain an extended plane corresponding to each feature plane; calculate a spatial intersection point of extended lines of adjacent feature planes in the extended plane, and take the spatial intersection point as virtual corner point data corresponding to the single-frame point cloud.

8. The power facility detection system of claim 6, wherein, The first registration module is specifically configured to: take a first frame point cloud in the laser radar point cloud as a reference frame, extract virtual corner point data of the reference frame as reference feature points, and extract virtual corner point data corresponding to each single-frame point cloud in the laser radar point cloud other than the reference frame as to-be-registered feature points; perform registration calculation on the reference feature points and the to-be-registered feature points of each adjacent two frames based on an iterative closest point algorithm, to obtain a spatial transformation matrix between adjacent frames; align each single-frame point cloud in the laser radar point cloud other than the reference frame to a first coordinate system of the reference frame according to the spatial transformation matrix, to obtain an initial laser radar global point cloud; take the initial laser radar global point cloud as a reference, and perform iterative closest point registration on virtual corner point data of each single-frame laser radar point cloud again, to obtain a first error optimization matrix of each single-frame laser radar point cloud. The initial laser radar global point cloud is corrected based on the first error optimization matrix, to obtain an optimized laser radar global point cloud.

9. The power facility detection system of claim 8, wherein, The first registration module is further configured to: perform frame interpolation processing on the camera image based on the time stamp of the laser radar point cloud, to obtain a camera frame consistent with the time stamp of each single-frame laser radar point cloud; project virtual corner point data of each single-frame laser radar point cloud into a camera frame corresponding to the time stamp according to a camera intrinsic parameter matrix calibrated in advance, to generate virtual corner point projection feature points on the camera side; take the camera frame in the camera image consistent with the time stamp of the laser radar reference frame as a camera reference frame, extract the virtual corner point projection feature points corresponding thereto as camera reference feature points, and extract virtual corner point projection feature points of the remaining camera frames as camera feature points to be registered; perform registration calculation on the camera reference feature points and the camera feature points to be registered of each adjacent two frames based on an iterative closest point algorithm, to obtain a spatial transformation matrix between adjacent camera frames; align each of the remaining camera frames to a second coordinate system of the camera reference frame according to the spatial transformation matrix, to obtain an initial camera global feature point cloud; perform iterative closest point registration on the virtual corner point projection feature points of each camera frame again with the initial camera global feature point cloud as a reference, to obtain a second error optimization matrix of each camera frame; perform correction processing on the initial camera global feature point cloud based on the second error optimization matrix, to obtain an optimized camera global feature point cloud.

10. The power facility detection system of claim 6, wherein, The first projection module is specifically configured to: convert each three-dimensional coordinate in the laser radar global point cloud from a laser radar coordinate system to a camera coordinate system based on the global extrinsic parameter matrix, to obtain a camera three-dimensional coordinate in the camera coordinate system; project the camera three-dimensional coordinate into a camera frame consistent with the time stamp of the laser radar point cloud based on a camera intrinsic parameter matrix calibrated in advance, to calculate a corresponding two-dimensional pixel coordinate; perform bilinear interpolation processing on the two-dimensional pixel coordinate, to obtain a target color value corresponding to the two-dimensional pixel coordinate; generate an initial colored laser radar point cloud according to a mapping relationship between each three-dimensional coordinate in the laser radar global point cloud and a corresponding target color value; perform voxelization filtering processing on the initial colored laser radar point cloud, to obtain a colored laser radar global point cloud.

Images