Electric transmission line external damage space automatic early warning method based on 2D-3D matching

Abstract

The invention discloses a 2D-3D matching-based automatic early warning method for an external damage space of a power transmission line. The early warning method comprises the following steps of obtaining a two-dimensional image and three-dimensional point cloud data of a power line corridor area; identifying an object close to a power line in the two-dimensional image through a YOLOv81 network, and generating a rectangular bounding box on the two-dimensional image; taking four vertex coordinates of the rectangular bounding box as pixel coordinates, and converting the pixel coordinates into three-dimensional coordinates in a world coordinate system; and selecting a power line point set from the three-dimensional point cloud data, and calculating the minimum distance between the top three-dimensional coordinate of the rectangular bounding box and the power line point set.

Description

Technical Field

[0001] The present invention relates to the technical field of power transmission line external damage prevention, and in particular to an automatic early warning method for external damage space of a power transmission line based on 2D-3D matching. Background Art

[0002] Power lines cover a wide area and develop rapidly, which poses a major challenge to the monitoring capabilities of power companies. With the rapid development of large-scale infrastructure construction in various places, the number of accidents involving large mechanical equipment contacting power lines is also increasing year by year. Foreign object intrusion is one of the biggest safety hazards of power lines. For example, vehicles such as cranes and trucks touching power lines is a common fatal electric shock accident that can cause cross-regional power outages and lead to huge economic losses and disasters. In the worst case, it may directly lead to injury or death. For a long time, the main method for power line inspection has been manual visual inspection. It includes on-foot and helicopter surveys with the help of telescopes or cameras. This method is inefficient, labor-intensive, and may even threaten the safety of inspectors. With the rapid development of drones, sensors and vision technology, the detection methods of power lines have been greatly optimized. Different sensors carried by drones, such as light detection and ranging (LiDAR) and cameras, can complete power line inspections for different tasks. Cameras installed on power towers can obtain the status of power lines in real time. At the same time, this also brings a large amount of image and video data that needs to be processed and analyzed. At present, there is a lack of effective data analysis and processing methods to solve the problem of foreign object intrusion, and it still relies heavily on experienced electrical engineers.

[0003] In addition, the above method has a lag phenomenon and cannot provide effective and timely early warning for power line safety accidents. Summary of the invention

[0004] In view of the above-mentioned prior art, the present invention provides an automatic early warning method for external broken space of a power transmission line based on 2D-3D matching, which mainly solves the technical problems existing in the above-mentioned background technology.

[0005] To achieve the above object, the technical solution of the embodiment of the present invention is implemented as follows:

[0006] An automatic early warning method for external broken space of a power transmission line based on 2D-3D matching, the early warning method comprising the following steps:

[0007] Obtain 2D images and 3D point cloud data of the power line corridor area;

[0008] Identify objects close to the power lines in the two-dimensional image through a YOLOv8l network, and generate a rectangular bounding box on the two-dimensional image;

[0009] Taking the coordinates of the four vertices of the rectangular bounding box as pixel coordinates, converting the pixel coordinates into three-dimensional coordinates in a world coordinate system;

[0010] Selecting a set of power line points from the three-dimensional point cloud data, and calculating a minimum distance between the top three-dimensional coordinates of the rectangular bounding box and the set of power line points;

[0011] According to the safety distance regulations of different voltage levels, a corresponding alarm threshold is set, and an alarm signal is generated when the minimum distance is less than the alarm threshold.

[0012] Optionally, a two-dimensional image of the power line corridor is obtained by a monocular camera installed on the pole tower, and three-dimensional point cloud data of the power line corridor is obtained by using a laser radar.

[0013] Optionally, the pixel coordinates are converted into three-dimensional coordinates in a world coordinate system by the following formula:

[0014]

[0015] Among them, (X w , Y w , Z w ) represents any three-dimensional coordinate in the world coordinate system, (u, v, 1) represents any pixel coordinate, Z c Indicates the depth value, R -1 represents the inverse matrix of R, R represents a 3×3 rotation matrix, T is a 3×1 translation vector, M is the internal parameter matrix of the monocular camera, M -1 is the inverse matrix of M.

[0016] Optionally, the internal parameter matrix M of the monocular camera is obtained by Zhang's calibration method, and the external parameters R and T of the monocular camera relative to the lidar are estimated by the EPnP algorithm.

[0017] Optionally, the extrinsic parameters R and T of the monocular camera relative to the lidar are estimated using the EPnP algorithm, specifically including:

[0018] Extract geometric feature points from the 3D point cloud data of the power line corridor area, such as the turning points of the power lines and the corner points of the towers;

[0019] Extracting visual feature points from a two-dimensional image of the power line corridor area;

[0020] Use the nearest neighbor matching algorithm to match feature points;

[0021] Based on the matching results, the EPnP algorithm is used to estimate the external parameters R and T of the monocular camera relative to the lidar.

[0022] Optionally, the specific process of calculating the depth value includes: when an object close to the power lines appears in the power line corridor area, obtaining three-dimensional point cloud data of the object through a lidar, calculating the distance between the three-dimensional point cloud of the object and the three-dimensional point cloud of the monocular camera in the power line corridor area, and using the distance as the depth value.

[0023] Optionally, the minimum distance between the top three-dimensional coordinates of the rectangular bounding box and the set of power line points is calculated by the following formula:

[0024]

[0025] Where (X W0 , Y W0 , Z W0 ) represents any three-dimensional coordinate of the top of the rectangular bounding box, (X W1 , Y W1 , Z W1 ) represents the coordinates of any point in the power line point set, and dt represents the minimum distance.

[0026] The beneficial effects of the present invention are: it can use plane target detection to identify and locate safety hazards under the power line corridor, and output the 2D coordinate information of the object. And through camera calibration and posture estimation, the 2D-3D mapping relationship of the monitoring scene is obtained to calculate the shortest safe distance from the object to the power line. Finally, according to the regulations on the safe distance of power lines of different voltage levels, dangerous objects are warned in time. The optimized model algorithm has high recognition accuracy, low complexity, fast calculation speed, and small distance calculation error, which greatly improves the efficiency and accuracy of power corridor safety distance monitoring and ensures the safety of transmission lines. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of an automatic early warning method for external broken space of a transmission line based on 2D-3D matching in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solution of the present invention is further elaborated in detail below in conjunction with the drawings and specific embodiments of the specification. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. In the following description, the expression "some embodiments" is related to a subset of all possible embodiments, but it should be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0029] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.

[0030] It should be understood that the present invention can be implemented in different forms and should not be interpreted as being limited to the embodiments proposed herein. On the contrary, providing these embodiments will make the disclosure thorough and complete, and the scope of the present invention will be fully conveyed to those skilled in the art. And the purpose of the terms used herein is only to describe specific embodiments and is not intended to be a limitation of the present invention. When used herein, the singular forms of "one", "one" and "said / the" are also intended to include plural forms, unless the context clearly indicates another way. It should also be understood that the terms "compose" and / or "include" when used in this specification determine the presence of the features, integers, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or groups. When used herein, the term "and / or" includes any and all combinations of the relevant listed items.

[0031] It should also be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "inside", "outside", "left", "right" and similar expressions used herein are for illustrative purposes only and are not intended to be the only implementation method.

[0032] In order to fully understand the present invention, a detailed structure will be proposed in the following description to illustrate the technical solution proposed by the present invention. The optional embodiments of the present invention are described in detail as follows, but in addition to these detailed descriptions, the present invention may also have other implementations.

[0033] Please refer to the attached Figure 1 The present application provides an automatic early warning method for a broken space outside a power transmission line based on 2D-3D matching, and the early warning method comprises the following steps:

[0034] S1, obtaining a two-dimensional image and three-dimensional point cloud data of the power line corridor area;

[0035] Specifically, a monocular camera installed on a pole tower is used to obtain a two-dimensional image of the power line corridor, and a laser radar is used to obtain three-dimensional point cloud data of the power line corridor.

[0036] S2. Identify objects close to the power lines in the two-dimensional image through a YOLOv81 network, and generate a rectangular bounding box on the two-dimensional image;

[0037] Specifically, in order to improve the performance of the YOLOv8 model in power line safety detection, the following aspects were optimized: data enhancement and transfer learning, optimized loss function, optimized loss function, and real-time target detection optimization. The traditional YOLOv8 model is mainly trained on standard image datasets, while power line safety detection has specific scene characteristics (such as occlusions in power line corridors, changes in ambient light, etc.). Therefore, the power corridor dataset is first expanded, and data enhancement methods (such as rotation, scaling, cropping, color transformation, etc.) are used to generate more diverse training samples, thereby improving the generalization ability of the model. In addition, the knowledge learned from general target detection tasks is transferred to power line safety detection tasks by using transfer learning, thereby accelerating training and improving accuracy.

[0038] The loss function of YOLOv8 is mainly composed of position loss, confidence loss and classification loss. These loss functions have been proven to be effective in traditional target detection tasks. However, in power line safety detection, the weights of the loss functions are adjusted, especially increasing the sensitivity to small objects (such as branches and wires) to cope with the detection problems of small objects and long-distance objects that are unique to power line safety detection.

[0039] The default network structure of the YOLOv8 model has performed well on multiple datasets, but it has been adapted for the power line environment. By adjusting the number of convolutional layers, the size of the filter, and the network hierarchy, the detection accuracy of the model in complex scenarios can be improved while maintaining efficient reasoning. In addition, depthwise separable convolutions are used to reduce the number of model parameters and increase reasoning speed.

[0040] In view of the real-time requirements in power line safety detection, the reasoning process of YOLOv8 is optimized. Under the premise of ensuring high accuracy, the reasoning process is accelerated by using quantization technology (such as INT8 quantization) and hardware acceleration (such as GPU or TPU), thus achieving real-time detection.

[0041] S3, using the coordinates of the four vertices of the rectangular bounding box as pixel coordinates, and converting the pixel coordinates into three-dimensional coordinates in a world coordinate system;

[0042] The pixel coordinates are converted into three-dimensional coordinates in the world coordinate system by the following formula:

[0043]

[0044] Among them, (X w , Y w , Z w ) represents any three-dimensional coordinate in the world coordinate system, (u, v, 1) represents any pixel coordinate, Z c Indicates the depth value, R -1 represents the inverse matrix of R, R represents a 3×3 rotation matrix, T is a 3×1 translation vector, M is the internal parameter matrix of the monocular camera, M -1 is the inverse matrix of M.

[0045] In one embodiment, the internal parameter matrix M of the monocular camera is obtained by Zhang's calibration method, and the external parameters R and T of the monocular camera relative to the laser radar are estimated by the EPnP algorithm.

[0046] In one embodiment, the extrinsic parameters R and T of the monocular camera relative to the laser radar are estimated by the EPnP algorithm, specifically including:

[0047] Geometric feature points are extracted from the three-dimensional point cloud data of the power line corridor area, such as the turning points of the power lines and the corner points of the poles and towers. Geometric feature-based methods (such as Harris corner point detection and FAST feature detection) can be used to extract key points in the point cloud.

[0048] Extract visual feature points from the two-dimensional image of the power line corridor area, such as SIFT, SURF, ORB and other feature points. These feature points are usually located in the prominent areas of the image, such as the turning points of the power lines, the corners of the towers, etc.;

[0049] Use the nearest neighbor matching algorithm to match feature points;

[0050] The PnP algorithm estimates the external parameters R and T of the monocular camera relative to the lidar through known matching point pairs (i.e., 3D points in the 3D point cloud data of the power line corridor area and 2D points in the 2D image of the power line corridor area).

[0051] In one embodiment, after estimating the external parameters of the camera, a virtual depth map or point cloud projection map is generated from all two-dimensional images of the power line corridor area in the three-dimensional point cloud of the power line corridor area. The virtual point cloud projection map is superimposed with the actual two-dimensional image of the power line corridor area to verify the accuracy of the matching.

[0052] If the matching result is not ideal, you can readjust the external parameters, or add more matching point pairs and re-perform the PnP algorithm and nonlinear optimization.

[0053] In one embodiment, the specific process of calculating the depth value includes: when an object close to the power lines appears in the power line corridor area, the three-dimensional point cloud data of the object is obtained through a lidar, and the distance between the three-dimensional point cloud of the object and the three-dimensional point cloud of the monocular camera in the power line corridor area is calculated, and the distance is used as the depth value.

[0054] S4, selecting a set of power line points from the three-dimensional point cloud data, and calculating the minimum distance between the top three-dimensional coordinates of the rectangular bounding box and the set of power line points;

[0055] Specifically, the minimum distance between the top three-dimensional coordinates of the rectangular bounding box and the set of power line points is calculated by the following formula:

[0056]

[0057] Where (X W0 , Y W0 , Z W0 ) represents any three-dimensional coordinate of the top of the rectangular bounding box, (X W1 , Y W1 , Z W1 ) represents the coordinates of any point in the power line point set, and dt represents the minimum distance.

[0058] S5. According to the safety distance regulations of different voltage levels, a corresponding alarm threshold is set, and when the minimum distance is less than the alarm threshold, an alarm signal is generated.

[0059] The present invention constructs a power line corridor vehicle data set based on the historical monitoring data of the power corridor for model training and verification.

[0060] The power line corridor vehicle dataset contains several types of objects, including dump trucks, excavators, cranes, pump trucks, agricultural machinery tower cranes, etc. There are 1992 images in total.

[0061] Randomly divided into training set and test set, the ratio is 80% and 20% respectively. PCVD comes from two parts, one is the monitoring data of 500kv transmission line of power company. We collected monitoring data of different time periods, with a total of 1192 images. Except for a few video screenshots, the original images have a high resolution of 2560×1440. Objects are annotated using Labelme tool. 800 images are randomly selected from COCO dataset, including two categories of trucks and cars. The models are trained and tested on a workstation equipped with two GPUs (NvidiaGeForce RTX 2080Ti). To ensure fair performance comparison, all methods are carried out under the same experimental conditions.

[0062] The batch size of each training input is set to 2, and a total of 200 epochs are trained. The initial learning rate and learning rate decay factor are 0.00625 and 0.1 respectively. In addition, a series of data enhancement strategies are added.

[0063] Multiple metrics are selected to evaluate the proposed single-stage detector. They include mean average precision (mAP), mean recall (mAR), model complexity (FLOPs), and detection speed (FPS). Comparing the model output with GT, the prediction results can be divided into four cases: true positive samples (TP), false positive samples (FP), false negative samples (FN), and true negative samples (TN). r) and recall(R e ) is defined as:

[0064] AP refers to the average accuracy of a single category of objects, which is obtained by calculating the area composed of the precision and recall (PR) curve. The average of the AP values ​​of all categories is mAP, which is a key comprehensive indicator. mAR is the average of the maximum values ​​of each category, reflecting the recognition performance of the detector for positive samples. FLOPs is the number of floating-point operations, which is used to measure the computational complexity of the algorithm. It reflects the hardware computing unit required by the model. FPS indicates the number of frames that the model can detect per second. The results of the ablation experiment are as follows:

[0065] Model <![CDATA[mAP 50：95 ]]> <![CDATA[mAP 50 ]]> <![CDATA[mAR 50：95 ]]> FLOPs FPS v8n 0.611(+0.050) 0.792 0.703 77.715G 19.188 v8s 0.641(+0.020) 0.824 0.725 34.105G 18.311 v8m 0,645(+0.016) 0.825 0.728 34.105G 18.443 v8l 0,661 0.855 0.743 34.105G 23.471

[0066] The above are several different types of model weight architectures of YOLOv8. The method proposed in this invention improves mAP and mAR. 50:95 It successfully achieved 66.1% on the previous model. Compared with the original traditional model, it achieved a 5% improvement, the computational complexity was reduced by more than two times, and the detection speed was increased by more than 4FPS. The comparison results verified that the optimized model has better performance.

[0067] Use optimized linear calibration pose estimation to achieve 2D-3D visual matching. Then use coordinate backprojection to achieve pose transformation from 2D coordinates to 3D coordinates (lidar points).

[0068] The model takes surveillance video data as input and outputs detection results frame by frame, which are provided in "Result_SuppInfo.mp4". According to the survey, the average height of a car (van) is 1.8m, and the height of a truck (standard dump truck produced in China) is 3.4m. In one set of data, the distances from the van to two different adjacent transmission corridors are 30.33m and 34.80m respectively. Based on the height of the truck, the predicted DLGcar They are 32.13 meters and 36.6 meters respectively. Combined with the actual DLG (The distances from powerlines to ground), the error (Error car ) are 0.54m and 0.38m respectively. Similarly, the DLG of a truck can be calculated truck and Error truck Their prediction results are shown in the following table.

[0069]

[0070]

[0071] The maximum prediction error is within 0.6m. In addition, the larger the object size, the smaller the error, which can be explained by the fact that the object detection algorithm can locate large objects more accurately.

[0072] According to the safety distance requirements of different voltage levels, the alarm threshold is adjusted to achieve effective monitoring and early warning. By dynamically adjusting the alarm threshold, the present invention can flexibly respond to different security threats based on factors such as the distance of objects around the power line, changes in the power corridor environment, and the type of objects. A multi-level, hierarchical response strategy is adopted to ensure real-time safety monitoring of the power line area. At the same time, false alarm suppression and multi-sensor fusion are used to effectively improve the accuracy and reliability of the alarm system.

[0073] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An automatic early warning method for external broken space of power transmission lines based on 2D-3D matching, characterized in that: The early warning method comprises the following steps: Obtain 2D images and 3D point cloud data of the power line corridor area; Identify objects close to the power lines in the two-dimensional image through a YOLOv8 l network, and generate a rectangular bounding box on the two-dimensional image; Taking the coordinates of the four vertices of the rectangular bounding box as pixel coordinates, converting the pixel coordinates into three-dimensional coordinates in a world coordinate system; Selecting a set of power line points from the three-dimensional point cloud data, and calculating a minimum distance between the top three-dimensional coordinates of the rectangular bounding box and the set of power line points; According to the safety distance regulations of different voltage levels, a corresponding alarm threshold is set, and an alarm signal is generated when the minimum distance is less than the alarm threshold.

2. According to claim 1, a method for automatic early warning of broken space outside a power transmission line based on 2D-3D matching is characterized in that: The two-dimensional image of the power line corridor is obtained by a monocular camera installed on the tower, and the three-dimensional point cloud data of the power line corridor is obtained by using a laser radar.

3. According to claim 2, a method for automatic early warning of external broken space of a power transmission line based on 2D-3D matching is characterized in that: The pixel coordinates are converted into three-dimensional coordinates in the world coordinate system by the following formula: Among them, (X w , Y w , Z w ) represents any three-dimensional coordinate in the world coordinate system, (u, v, 1) represents any pixel coordinate, Z c Indicates the depth value, R -1 represents the inverse matrix of R, R represents a 3×3 rotation matrix, T is a 3×1 translation vector, M is the internal parameter matrix of the monocular camera, M -1 is the inverse matrix of M.

4. According to claim 3, a method for automatic early warning of external broken space of a power transmission line based on 2D-3D matching is characterized in that: The internal parameter matrix M of the monocular camera is obtained by Zhang’s calibration method, and the external parameters R and T of the monocular camera relative to the lidar are estimated by the EPnP algorithm.

5. According to claim 4, a method for automatic early warning of external broken space of power transmission line based on 2D-3D matching is characterized in that: The EPnP algorithm is used to estimate the external parameters R and T of the monocular camera relative to the lidar, including: Extract geometric feature points from the 3D point cloud data of the power line corridor area, such as the turning points of the power lines and the corner points of the towers; Extracting visual feature points from a two-dimensional image of the power line corridor area; Use the nearest neighbor matching algorithm to match feature points; Based on the matching results, the EPnP algorithm is used to estimate the external parameters R and T of the monocular camera relative to the lidar.

6. The automatic early warning method for the broken space outside the power transmission line based on 2D-3D matching according to claim 5 is characterized in that: The specific process of calculating the depth value includes: when an object close to the power lines appears in the power line corridor area, the three-dimensional point cloud data of the object is obtained through the lidar, and the distance between the three-dimensional point cloud of the object and the three-dimensional point cloud of the monocular camera in the power line corridor area is calculated, and the distance is used as the depth value.

7. The automatic early warning method for external broken space of a power transmission line based on 2D-3D matching according to claim 6 is characterized in that: The minimum distance between the top three-dimensional coordinates of the rectangular bounding box and the set of power line points is calculated by the following formula: Where (X W0 , Y W0 , Z W0 ) represents any three-dimensional coordinate of the top of the rectangular bounding box, (X W1 , Y W1 , Z W1 ) represents the coordinates of any point in the power line point set, and dt represents the minimum distance.

Images