A parallel line repeat hazard determination method and system based on multi-source data

Abstract

The application provides a kind of parallel line repeat hidden danger judging method and system based on multi-source data, belong to transmission line intelligent monitoring technical field. Including: obtaining the point cloud data and visual monitoring image of parallel line;Determine the initial camera pose of visual monitoring device on each line based on point cloud data, generate depth map in combination with camera internal parameter and point cloud data;With the matching degree of point cloud projection and image scene features as optimization target, optimize camera pose;Through hidden danger detection algorithm, identify the hidden danger target in visual monitoring image, obtain pixel coordinate information;According to pixel coordinate, depth map, camera internal parameter and optimized pose, calculate the three-dimensional positioning coordinates of hidden danger target;Compare the positioning coordinates of hidden danger targets of different lines, if the spatial distance is less than the repeat judgment threshold, it is judged as repeat hidden danger. The application realizes the automatic identification and filtering of parallel line repeat hidden danger, improves the hidden danger positioning accuracy and operation and maintenance efficiency, reduces redundant alarm data.

Description

Technical Field

[0001] This invention belongs to the field of intelligent monitoring technology for power transmission lines, and specifically relates to a method and system for determining repeated hidden dangers in parallel lines based on multi-source data. Background Technology

[0002] With the expansion of the power grid and the increasing scarcity of corridor resources, the use of double-circuit or multi-circuit transmission lines on the same tower, as well as long-distance parallel construction with existing lines, has become a common choice for transmission line construction. However, during operation, parallel lines are repeatedly threatened by external damage hazards such as those caused by mechanical construction. Traditional manual inspection or online monitoring methods, which involve inspecting each line and each section individually, can lead to the same external damage hazard being repeatedly recorded by multiple monitoring devices on different lines at different times, generating a large number of identical alarms, images, and coordinate information within the system.

[0003] These massive amounts of repetitive alarms not only consume a lot of storage and computing resources, but also drown out the real hidden dangers that appear for the first time, causing maintenance personnel to be caught in high-frequency and inefficient confirmation work, seriously affecting the efficiency of closed-loop handling of hidden dangers and resource allocation, and may even lead to missed detections due to alarm fatigue.

[0004] In existing technologies, some solutions have attempted to reconstruct the three-dimensional point cloud of the route through methods such as stereo vision. However, these methods focus more on the reconstruction of the three-dimensional model itself and are not deeply integrated with the actual control needs such as the accurate spatial positioning of hidden dangers and the identification of repetition in parallel route scenarios. Therefore, they cannot effectively solve the problem of filtering out the above-mentioned repeated alarms.

[0005] Therefore, there is an urgent need for a method that can comprehensively utilize multi-source data to accurately locate and repeatedly determine potential mechanical damage hazards in parallel power transmission lines, so as to improve the efficiency and accuracy of intelligent operation and maintenance of power transmission lines. Summary of the Invention

[0006] This invention proposes a method and system for determining the repetitive hidden dangers of parallel lines based on multi-source data, used to achieve: ① Comprehensive application of multi-source data for power transmission lines: It realizes the fusion and application of data such as power transmission channels, point clouds, and images; ②Automated camera pose calculation: Using optimization algorithms such as particle swarm optimization, and taking the positions of objects in the point cloud and image as references, automated pose calculation can be achieved. ③ Hazard duplication determination based on point cloud image fusion: Taking advantage of the high precision of point cloud data, high-precision positioning of hazards detected by different visual monitoring images is achieved. The positioning information is combined to determine whether the hazards are the same. The method is simple and easy to understand.

[0007] The present invention adopts the following technical solution.

[0008] The first aspect of the present invention provides a method for determining the potential for duplicate parallel lines based on multi-source data, comprising: Multi-source data of parallel lines are acquired, including point cloud data of each line and corresponding visual monitoring images. Based on the point cloud data of each line, the initial camera pose of the visual monitoring equipment on each line is determined. Then, combined with the intrinsic parameters of the visual monitoring equipment and the point cloud data, a depth map corresponding to each visual monitoring image is generated. The initial camera pose of each line is optimized with the matching degree between the projection of the point cloud data on the corresponding visual monitoring image and the scene features in the image as the optimization objective. Based on the transmission line hidden danger detection algorithm, hidden danger targets in each visual monitoring image are identified, and the pixel coordinate information of each hidden danger target in the image is obtained. For each hidden danger target, the positioning coordinates of the hidden danger target in three-dimensional space are calculated based on its pixel coordinate information, the corresponding depth map, the camera intrinsic parameters, and the optimized camera pose. For hidden danger targets identified from the visual monitoring images of different lines, their positioning coordinates are compared. If the spatial distance between the positioning coordinates of any two hidden danger targets is less than the repetition judgment threshold, then the two hidden danger targets are determined to be repeated hidden dangers.

[0009] Optionally, determining the initial camera pose of the visual monitoring devices on each line includes: Semantic instance segmentation is performed on the point cloud data, the point cloud of each tower is identified from the segmentation results, and an independent instance identifier is assigned to each tower. Based on the pole instance corresponding to the installation location of the visual monitoring equipment, obtain the geometric center coordinates of the point cloud of that pole, as well as the geometric center coordinates of the point cloud of the next pole adjacent to that pole; The initial translation parameter is obtained by using the geometric center coordinates of the tower where the visual monitoring device is located as the initial translation parameter. The initial rotation matrix is ​​calculated based on the direction vector between the geometric center coordinates of the two towers. The initial camera pose is obtained by combining the initial rotation matrix and the initial translation parameter.

[0010] Optionally, the semantic instance segmentation adopts a point cloud segmentation network model, and the cross-entropy loss function is weighted by category during training, wherein the weight coefficient of the conductor category is greater than the weight coefficient of the ground category, and the weight coefficient of the tower category is greater than the weight coefficient of the conductor category. After training, the point cloud data is input into the point cloud segmentation network model, and the class label and tower instance label of each 3D point are output. In the post-segmentation processing, based on the spatial distribution of the conductor point cloud, conductor point clouds located outside the spatial corridor of the line are reclassified as non-conductors, and point clouds located within the preset neighborhood of the conductor point cloud that have not been classified as conductors are added as conductor point clouds.

[0011] Optionally, generating depth maps corresponding to each visualized surveillance image includes: Each 3D point in the point cloud data after semantic instance segmentation is projected onto the pixel coordinate system of the visualized monitoring image based on the initial camera pose and camera intrinsic parameters to obtain the pixel coordinates corresponding to the 3D point. Obtain the depth value of each 3D point relative to the camera's optical center, and assign this depth value to the pixel position to which the 3D point is projected, thereby generating a depth map with the same resolution as the visualized monitoring image.

[0012] Optionally, optimizing the initial camera pose for each line includes: Scene features are extracted from visual surveillance images, including the pixel positions of line segment endpoints obtained through a line detection algorithm; Project the conductor point cloud and tower point cloud in the point cloud data onto the image plane according to the current candidate camera pose to obtain the projection point set; Calculate the Euclidean distance between the pixels in the projection point set and the pixel positions of the line segment endpoints, and count the number of matching point pairs whose Euclidean distance is less than a preset distance threshold; With the goal of maximizing the number of matching point pairs, the candidate camera pose is iteratively updated using a particle swarm optimization algorithm until a preset iteration stopping condition is met, and the optimized camera pose is output.

[0013] Optionally, the power transmission line hazard detection algorithm is a deep learning target detection model based on the YOLOv8 architecture; The deep learning object detection model is fine-tuned on a training dataset containing images of power transmission line corridors. The proportion of parallel line scene images in the training dataset is not less than a first preset proportion, and images of different spans and different weather conditions are allocated according to a second preset proportion. After training, this deep learning object detection model is used to identify mechanical damage hazards in visualized surveillance images, and outputs the category, bounding box coordinates, and confidence score of each hazard.

[0014] Optionally, after identifying potential hazards in each visualized surveillance image, obtaining the pixel coordinate information of each hazard in the image includes: Extract the pixel coordinates of the lower edge center point from the detected bounding box coordinates, and use them as the projection point of the target's ground contact location in the image; When multiple potential hazards are detected in the same visual monitoring image, the pixel coordinates corresponding to each potential hazard are extracted.

[0015] Optionally, calculating the location coordinates of the potential hazard in three-dimensional space includes: For each potential hazard target, obtain the pixel coordinates from the center point of the lower edge of its detection box; Read the corresponding depth value from the depth map based on the pixel coordinates; Using the camera intrinsic parameter matrix, the pixel coordinates and depth value are converted into three-dimensional coordinates in the camera coordinate system; By using the rotation matrix and translation vector represented by the optimized camera pose, the three-dimensional coordinates in the camera coordinate system are transformed to the world coordinate system to obtain the location coordinates of the potential hazard.

[0016] Optionally, the spatial distance is the Euclidean distance between the location coordinates of the two potential hazards on the horizontal plane; The repetition determination threshold is a length value set in advance based on the statistical value of point cloud positioning accuracy. When the Euclidean distance between two potential hazards is less than the repetition determination threshold, they are determined to be repeated hazards.

[0017] The second aspect of the present invention provides a parallel line duplication hazard determination system based on multi-source data, used to implement the parallel line duplication hazard determination method based on multi-source data described in the first aspect of the present invention, comprising: The module includes a data acquisition module, a point cloud semantic instance segmentation module, an initial camera parameter module, a depth map generation module, a camera pose calculation module, a power transmission line hazard detection module, a hazard location module, and a repeating hazard determination module, among which: The data acquisition module is used to acquire multi-source data of the parallel lines, including point cloud data of each line in the parallel lines and corresponding visual monitoring images, as well as camera intrinsic parameters of each visual monitoring device. The point cloud semantic instance segmentation module is used to perform semantic instance segmentation on the point cloud data of each line in order to classify and identify conductors, towers and ground features; The initial camera parameter module is used to determine the initial camera pose of the visual monitoring equipment on each line based on the identified tower point cloud. The depth map generation module is used to generate depth maps corresponding to each visualized monitoring image based on the initial camera pose, camera intrinsic parameters, and point cloud data. The camera pose calculation module is used to optimize the initial camera pose of each line by using the matching degree between the projection of point cloud data on the corresponding visualized monitoring image and the scene features in the image as the optimization target, so as to obtain the optimized camera pose. The power transmission line hidden danger detection module is used to identify hidden danger targets in each visualized monitoring image based on a pre-trained hidden danger identification model, and to obtain the pixel coordinate information of each hidden danger target in the image; The hazard location module is used to calculate the location coordinates of each hazard target in three-dimensional space through coordinate transformation based on its pixel coordinate information, corresponding depth map, camera intrinsic parameters and optimized camera pose. The duplicate hazard determination module is used to compare the location coordinates of hazard targets identified from visual monitoring images of different lines. If the spatial distance between the location coordinates of any two hazard targets is less than the duplicate determination threshold, then the two hazard targets are determined to be duplicate hazards.

[0018] Compared with the prior art, the beneficial effects of the present invention include at least the following: 1. This invention integrates lidar point cloud data with visual monitoring images, and uses the high-precision spatial information of the point cloud data to perform three-dimensional positioning of potential hazards detected in the images. This solves the problem that single image data is difficult to accurately distinguish hazards in parallel line scenarios due to differences in viewing angle, and achieves accurate judgment and effective filtering of repeated hazards in parallel lines.

[0019] 2. This invention identifies towers from point clouds and automatically estimates the initial camera pose by semantic instance segmentation. Then, using the matching degree between point cloud projection and image scene features as the optimization target, the particle swarm optimization algorithm is used to refine the camera pose. This solves the problem of traditional methods relying on manual calibration or complex control point layout, and realizes efficient and fully automated calculation of camera pose.

[0020] 3. This invention combines the pixel coordinates of the potential hazard in the image with the corresponding depth map, uses camera intrinsic parameters and optimized pose to back-calculate the two-dimensional detection results to three-dimensional world coordinates, and makes repeatability judgment based on the horizontal Euclidean distance between the hazards. This solves the problem of automatic identification and merging of repeated hazards in multi-source monitoring data of parallel lines, and effectively reduces the amount of redundant alarm data in the operation and maintenance system.

[0021] 4. Through modular design, this invention divides functions such as point cloud processing, pose optimization, hidden danger detection and location determination into independent modules. The computational resource consumption of each module is stable and controllable, which solves the problem that complex algorithms are difficult to deploy in actual power transmission line operation and maintenance scenarios, realizes efficient operation on end-side equipment, and improves the practicality and economy of intelligent inspection. Attached Figure Description

[0022] Figure 1 This is a flowchart of a method provided according to an embodiment of the present invention; Figure 2 The data refers to the point cloud data of transmission line 1 and transmission line 2 at a certain span, provided according to the embodiments of the present invention. Figure 3 These are visualization images of transmission line 1 (left) and transmission line 2 (right) provided according to embodiments of the present invention; Figure 4 This is a schematic diagram of point cloud semantic instance segmentation results provided in accordance with embodiments of the present invention; Figure 5This is a visual diagram of the potential hazards detected in transmission line 1 (left) and transmission line 2 (right) according to an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this invention. The described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the spirit of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of this invention.

[0024] In Embodiment 1, this invention provides a method for determining the repetitive hidden dangers of parallel lines based on multi-source data, such as... Figure 1 As shown, it includes the following steps: Step 1: Obtain multi-source data for the parallel lines, including point cloud data of each line in the parallel lines and corresponding visualized monitoring images.

[0025] Preferably, the multi-source data also includes camera intrinsic parameters of each visualization monitoring device. The camera intrinsic parameters are a set of fixed parameters that describe the relationship between the camera's own geometric characteristics and imaging, and are determined by the camera hardware structure.

[0026] Preferably, the point cloud data is acquired by a drone equipped with a lidar, and the coordinate system is based on the line tower as the origin; the visualized monitoring images are captured by visualized monitoring equipment installed on the transmission line tower, and are used to monitor two-dimensional image data of changes in the transmission channel scene.

[0027] Step 2: Based on the point cloud data of each line, determine the initial camera pose of the visualization monitoring device on each line, and then combine the intrinsic parameters of the visualization monitoring device and the point cloud data to generate depth maps corresponding to each visualization monitoring image.

[0028] Preferably, determining the initial camera pose of the visual monitoring devices on each line includes: Semantic instance segmentation is performed on the point cloud data, the point cloud of each tower is identified from the segmentation results, and an independent instance identifier is assigned to each tower. Based on the pole instance corresponding to the installation location of the visual monitoring equipment, obtain the geometric center coordinates of the point cloud of that pole, as well as the geometric center coordinates of the point cloud of the next pole adjacent to that pole; The initial translation parameter is obtained by using the geometric center coordinates of the tower where the visual monitoring device is located as the initial translation parameter. The initial rotation matrix is ​​calculated based on the direction vector between the geometric center coordinates of the two towers. The initial camera pose is obtained by combining the initial rotation matrix and the initial translation parameter.

[0029] Preferably, the semantic instance segmentation uses the PointNet++ network model, and the cross-entropy loss function is weighted by category during training, wherein the weight coefficient of the conductor category is greater than the weight coefficient of the ground category, and the weight coefficient of the tower category is greater than the weight coefficient of the conductor category. After training, the point cloud data is input into the PointNet++ network model, which outputs the category label and tower instance label for each 3D point. In the post-segmentation processing, based on the spatial distribution of the conductor point cloud, conductor point clouds located outside the spatial corridor of the line are reclassified as non-conductors, and point clouds located within the preset neighborhood of the conductor point cloud that have not been classified as conductors are added as conductor point clouds.

[0030] More preferably, the semantic instance segmentation is achieved through a point cloud semantic instance segmentation algorithm. This algorithm can classify point cloud data into multiple preset categories such as conductors, towers, ground, vegetation, and buildings, and segment each tower into an independent instance. Specifically, the PointNet++ model is used as the point cloud semantic instance segmentation algorithm. During model training, the cross-entropy loss function is weighted by category to address the problem of the limited number of point clouds for ontology categories such as conductors and towers, taking into account the characteristics of the power scenario.

[0031] The formula for calculating the weighted loss function is:

[0032] in, C represents the weight coefficient of category y to which the current point belongs (this coefficient is an empirical value set specifically for the scene; for example, the weight of ground is 1, the weight of vegetation is 0.8, the weight of conductor is 1.2, and the weights of main tower and next-level tower are 1.5, all of which are empirical values); C represents the total number of segmentation categories (such as ground, vegetation, conductor, main tower, and next-level tower). Let be the model's predicted score for the i-th class. For the sigmoid function, For category labels (one-hot encoded).

[0033] Meanwhile, during model training, a single tower example was segmented into an independent category for each tower category (generally, there are two towers in the line point cloud, which are labeled as the main tower and the next level tower respectively for training).

[0034] In the post-segmentation processing, the traverse point cloud categories were optimized: points not located within the 3-meter spatial corridor of the line area were removed as erroneously segmented points, and the segmented point clouds within 1 meter of the detected traverse point clouds were supplemented and corrected according to the categories of the traverse point clouds that were missed in segmentation.

[0035] The training data came from point cloud scanning of power scene by drones equipped with lidar, totaling more than 4,000 point cloud data points at various intervals, all of which were labeled.

[0036] Key hyperparameters used during training include: 1024 sampling points, batch size 32, 120 training epochs, learning rate 0.01, and weight decay 1e-4. Using the latitude and longitude recorded by the device, the point cloud of the tower where the target tower is located is identified. The geometric center point of the point cloud of that tower and the geometric center point of the point cloud of the next adjacent tower are calculated. The initial pose is then calculated using the spatial vectors of these two points.

[0037] In the post-segmentation processing, the traverse point cloud categories were optimized: points not located within the 3-meter spatial corridor of the line area were removed as erroneously segmented points, and the segmented point clouds within 1 meter of the detected traverse point clouds were supplemented and corrected according to the categories of the traverse point clouds that were missed in segmentation.

[0038] Specifically, methods for estimating the initial camera pose include: The point cloud of the tower is located using the latitude and longitude recorded by the device. The ensemble center point of the tower's point cloud is calculated, which involves obtaining the coordinate center points cx, cy, and cz of the tower's point cloud in the x, y, and z directions respectively, and then merging them to obtain the ensemble center point of the tower's point cloud. The geometric center point of the point cloud of the next base tower is calculated using the same method, and the result is obtained. Select the center point of the point cloud of the tower where the device is located as the translation amount T in the initial pose.

[0039] The rotation matrix R in the initial pose is calculated through the following steps: 1. Define the world coordinate system: the X-axis points perpendicular to the YZ plane, the Y-axis points in the direction of the next tower, and the Z-axis points vertically upward.

[0040] 2. Calculate the vector: Based on the center points of the two towers, calculate the difference in direction vectors on the horizontal plane:

[0041] 3. Calculate the rotation matrix R:

[0042] Ultimately, the initial camera pose transformation matrix RT is composed of the rotation matrix R and the translation vector T, i.e., RT = [R, T]. Preferably, generating depth maps corresponding to each visualized surveillance image includes: Each 3D point in the point cloud data after semantic instance segmentation is projected onto the pixel coordinate system of the visualized monitoring image based on the initial camera pose and camera intrinsic parameters to obtain the pixel coordinates corresponding to the 3D point. Obtain the depth value of each 3D point relative to the camera's optical center, and assign this depth value to the pixel position to which the 3D point is projected, thereby generating a depth map with the same resolution as the visualized monitoring image.

[0043] Step 3: Using the matching degree between the projection of point cloud data onto the corresponding visualized monitoring image and the scene features in the image as the optimization objective, optimize the initial camera pose of each line to obtain the optimized camera pose.

[0044] Preferably, the optimization of the initial camera pose for each line includes: Scene features are extracted from visual surveillance images, including the pixel positions of line segment endpoints obtained through a line detection algorithm; Project the conductor point cloud and tower point cloud in the point cloud data onto the image plane according to the current candidate camera pose to obtain the projection point set; Calculate the Euclidean distance between the pixels in the projection point set and the pixel positions of the line segment endpoints, and count the number of matching point pairs whose Euclidean distance is less than a preset distance threshold; With the goal of maximizing the number of matching point pairs, the candidate camera pose is iteratively updated using a particle swarm optimization algorithm until a preset iteration stopping condition is met, and the optimized camera pose is output.

[0045] Specifically, a particle swarm optimization algorithm is used to optimize the camera pose. In this algorithm, the position vector X of each particle is 6-dimensional and is used to represent the camera pose.

[0046] Where α, β, and γ are Euler angles, representing rotations about the X, Y, and Z axes, respectively. The search range needs to be limited according to the actual scenario; for example, it can be set to [-π / 2, π / 2] to avoid invalid rotations. tx, ty, and tz are translation values, and their search range needs to be limited according to the actual scale of the point cloud. For example, they can be set to [-2, 75].

[0047] The optimization goal is to ensure that the pixel positions of the classified point cloud (especially conductor and tower points) projected onto the image through the current pose are as consistent as possible with the positions of scene features directly detected from the surveillance image. The fitness function is defined as the maximum total number of matches between the pixel positions in the point cloud-mapped depth map and the pixels obtained from line detection in the image (i.e., maximizing the number of matching points).

[0048]

[0049] in, The point cloud is the set of pixels mapped to the image plane through the camera intrinsic parameter matrix M and the device pose RT; The image line detection algorithm (the algorithm is a probabilistic Hough transform, which is directly called using the cv2.HoughLinesP() function in Python) extracts the pixel position of the line segment in the image; 1(·) is an indicator function, which is 1 when the condition is met, and 0 otherwise; The pixel distance threshold is usually 1 pixel (an empirical value; this can also be understood as a match if the Euclidean distance between two pixel coordinates is less than 1).

[0050] To further improve the optimization effect and matching accuracy, without changing the above optimization framework, this method can also introduce the following parallel enhancement strategies to expand the matching point calculation method in the original fitness function: 1. Category-weighted matching point calculation: When calculating the number of matching points, different weights are assigned based on the importance of different categories of scenery. For example, the weight of the guide wire point cloud is set to 1.5, the weight of the tower point cloud is set to 1.8, and the weight of other scenery point clouds is set to 1.0. The final number of matching points is a weighted sum to highlight the contribution of key structures to pose optimization.

[0051] 2. Introduce structural similarity constraints: Project the point clouds of conductors and towers onto a two-dimensional plane to generate a grayscale structural map of the same size as the original image; simultaneously, extract edge information from the image using an edge detection algorithm to generate a grayscale image of the same size edge. Calculate the structural similarity (SSIM) between the two, with a value ranging from [-1, 1]. Multiply the original fitness function value F by this structural similarity ss to obtain a new fitness function value Fss = F × ss, achieving joint optimization of local matching and global structural consistency.

[0052] 3. Dynamic Threshold Matching Point Filtering: After obtaining the matching point set, the RANSAC algorithm is introduced to filter the matching points, eliminating false matches and improving the quality of the matching point set. Simultaneously, a dynamic threshold strategy is adopted during the iteration process of the particle swarm optimization algorithm: in the early iteration stage (global search phase), the matching threshold is appropriately relaxed to retain more matching points to avoid missed detections; in the later iteration stage (local optimization phase), the matching threshold is tightened to accurately eliminate erroneous matches and improve convergence accuracy.

[0053] The three enhancement strategies described above are independent of each other and can be used individually or in any combination. In practical applications, the appropriate strategy can be flexibly selected based on the characteristics of the scenario and available computing resources. The iteration stopping condition remains the same: reaching the preset maximum number of iterations (e.g., 200 times), or the maximum total number of matches obtained from two consecutive iterations no longer increases (i.e., the fitness function value converges).

[0054] Step 4: Based on the power transmission line hidden danger detection algorithm, identify the hidden danger targets in each visualized monitoring image and obtain the pixel coordinate information of each hidden danger target in the image.

[0055] Preferably, the power transmission line hazard detection algorithm is a deep learning target detection model based on the YOLOv8 architecture; The deep learning object detection model is fine-tuned on a training dataset containing images of power transmission line corridors. The proportion of parallel line scene images in the training dataset is not less than a first preset proportion, and images of different spans and different weather conditions are allocated according to a second preset proportion. After training, this deep learning object detection model is used to identify mechanical damage hazards in visualized surveillance images, and outputs the category, bounding box coordinates, and confidence score of each hazard.

[0056] More preferably, after identifying the potential hazards in each visualized surveillance image, obtaining the pixel coordinate information of each potential hazard in the image includes: Extract the pixel coordinates of the lower edge center point from the detected bounding box coordinates, and use them as the projection point of the target's ground contact location in the image; When multiple potential hazards are detected in the same visual monitoring image, the pixel coordinates corresponding to each potential hazard are extracted.

[0057] It should be noted that the method described in this invention primarily targets the repeated identification of mechanical external damage hazards (such as excavators, pile drivers, cranes, and concrete pump trucks) in parallel power lines. These hazards have clear visual shapes and ground contact locations, making them suitable for processing using a unified image detection algorithm without requiring classification or differentiated parameter settings based on hazard type. Therefore, this method employs a unified deep learning target detection model to identify mechanical hazard targets in images and output their category, bounding box coordinates, and confidence score. Preferably, the power transmission line hazard detection algorithm is a deep learning-based target detection model used to identify power transmission line hazard targets in images and output the hazard target's category, bounding box coordinates, and confidence score.

[0058] Specifically, the power transmission line hazard detection algorithm uses the YOLOv8 model and is fine-tuned on a proprietary power transmission line hazard dataset containing 300,000 images of power transmission line corridors. This dataset is primarily acquired from power transmission line visualization monitoring devices and uses the VOC dataset format. The dataset already covers common mechanical external damage categories such as excavators, pile drivers, cranes, and concrete pump trucks, ensuring the model's ability to identify target hazards.

[0059] To further improve detection performance in parallel line scenarios, this invention introduces the following optimization strategies during the model training and inference stages: 1. Post-processing strategy Improved Non-Maximum Suppression (NMS): To address situations such as dense clustering of similar targets or partial occlusion in parallel line scenarios, the overlap threshold of NMS is adjusted to reduce missed detections. Simultaneously, for mutually exclusive category hazards detected at the same location (e.g., simultaneously predicting "bulldozer" and "excavator"), only the category with higher confidence is retained, contradictory predictions are eliminated, and duplicate alarms are avoided.

[0060] Category association enhancement: By utilizing the spatial correlation between parallel lines, if a certain type of hidden danger is detected in the left line image, the prediction results of the same type of hidden danger at the same spatial location in the right line image are weighted with confidence (e.g., multiplied by a coefficient of 1.2) to improve the recall rate of the right line and reduce false negatives.

[0061] 2. Dataset-related strategies Scene coverage design: To ensure the model's adaptability to parallel line scenarios, special requirements are made during dataset construction: The proportion of images showing double-circuit or multi-circuit parallel lines on the same tower shall not be less than 20%; The samples of potential hazards were evenly distributed across different spans (near tower, middle span, and far tower). Images collected under different weather conditions are distributed proportionally: 50% for sunny days, 30% for cloudy days, and 20% for foggy days.

[0062] 3D Verification Annotation: For parallel line data, a 3D verification mechanism is introduced during the annotation process. The annotated 2D bounding boxes are projected onto the corresponding point cloud data to verify their overlap with the 3D target. If the overlap is less than 70%, it is considered an annotation error and needs to be corrected, thereby improving annotation quality and ensuring the model training effect.

[0063] To further improve detection performance in parallel circuit scenarios, the following strategies can be adopted: Post-processing strategy: Improve the non-maximum suppression (NMS) algorithm of YOLOv8. In parallel line scenarios, the overlap threshold of NMS can be adjusted for specific categories (such as mechanical vehicles) to better handle dense or partially occluded targets.

[0064] Dataset augmentation strategy: Based on the possible distribution characteristics of potential hazards in parallel line scenarios, synthetic or real samples such as "similar machinery appearing in parallel" or "targets linearly distributed in the channel" are added to the dataset to enhance the model's generalization ability for this scenario.

[0065] The output hazard information includes the hazard category (e.g., truck, fireworks), bounding box coordinates (a 1×4 matrix, where the first two elements are the coordinates of the top-left corner and the last two are the coordinates of the bottom-right corner), and confidence score. The confidence score is the raw confidence score output by the model, used to measure the reliability of the detection results. All mechanical external damage hazards use the same confidence threshold (e.g., 0.25) and target box selection rules, requiring no differentiation settings.

[0066] The training hyperparameters included: optimizer Adam, initial learning rate 0.001, learning rate momentum 0.937, number of training epochs 200, and batch size 16. During detection, a confidence threshold of 0.25 (an empirical value) was set, and only detection results with a confidence level higher than this threshold were retained as valid potential threats for subsequent localization and judgment.

[0067] Step 5: For each potential hazard target, based on its pixel coordinate information, corresponding depth map, camera intrinsic parameters, and optimized camera pose, calculate the target's location coordinates in three-dimensional space through coordinate transformation.

[0068] Preferably, calculating the location coordinates of the potential hazard target in three-dimensional space includes: For each potential hazard target, obtain the pixel coordinates from the center point of the lower edge of its detection box; Read the corresponding depth value from the depth map based on the pixel coordinates; Using the camera intrinsic parameter matrix, the pixel coordinates and depth value are converted into three-dimensional coordinates in the camera coordinate system; By using the rotation matrix and translation vector represented by the optimized camera pose, the three-dimensional coordinates in the camera coordinate system are transformed to the world coordinate system to obtain the location coordinates of the potential hazard.

[0069] Specifically, the coordinate transformation process is as follows: From pixel coordinates to camera coordinates: Obtain the pixel coordinates (u,v) of the center point of the lower edge of the hazard bounding box and its corresponding depth value d in the depth map, as well as the camera intrinsic parameter matrix M (3×3 matrix). Calculate the coordinates of the hazard target in the camera coordinate system using the following formula. :

[0070] From camera coordinate system to world coordinate system: Using the optimized camera pose (i.e., the device pose matrix RT, which is a 3×4 matrix), the coordinates in the camera coordinate system are converted to world coordinate system. Transform to the world coordinate system to obtain the final three-dimensional coordinates of the hidden danger. The complete formula for this transformation is:

[0071] in, It includes the camera's rotation and translation parameters. , , This forms a translation vector that describes the camera's position; to A rotation matrix is ​​constructed, describing the camera's attitude. The elements of this rotation matrix can be represented by Euler angles (yaw angle ψ, pitch angle θ, roll angle φ), and their specific correspondences are as follows:

[0072] Wherein, the yaw angle ψ represents the rotation about the Z-axis (horizontal yaw), the pitch angle θ represents the rotation about the new Y-axis (pitch), and the roll angle φ represents the rotation about the new X-axis (roll).

[0073] Step 6: For the hidden danger targets identified from the visual monitoring images of different lines, compare their location coordinates. If the spatial distance between the location coordinates of any two hidden danger targets is less than the repetition judgment threshold, then the two hidden danger targets are determined to be repeated hidden dangers.

[0074] Preferably, the spatial distance is the Euclidean distance between the location coordinates of the two potential hazards on the horizontal plane; The repetition determination threshold is a length value set in advance based on the statistical value of point cloud positioning accuracy. When the Euclidean distance between two potential hazards is less than the repetition determination threshold, they are determined to be repeated hazards.

[0075] Specifically, let the three-dimensional location coordinates of the potential hazard identified from the image of line A be:

[0076] The three-dimensional coordinates of the potential hazard identified from the image of Line B are:

[0077] The formula for calculating the horizontal Euclidean distance is:

[0078] For example, if dist < 1 meter (based on positioning accuracy statistics, the positioning accuracy of hidden dangers within a span of a typical transmission line is around 1 meter), then the two hidden danger targets are determined to be the same duplicate hidden danger.

[0079] In Embodiment 2, this invention provides a parallel line duplication hazard determination system based on multi-source data, used to implement the parallel line duplication hazard determination method based on multi-source data described in Embodiment 1, comprising: The module includes a data acquisition module, a point cloud semantic instance segmentation module, an initial camera parameter module, a depth map generation module, a camera pose calculation module, a power transmission line hazard detection module, a hazard location module, and a repeating hazard determination module, among which: The data acquisition module is used to acquire multi-source data of the parallel lines, including point cloud data of each line in the parallel lines and corresponding visual monitoring images, as well as camera intrinsic parameters of each visual monitoring device.

[0080] The point cloud semantic instance segmentation module is used to perform semantic instance segmentation on the point cloud data of each line in order to classify and identify conductors, towers and ground features.

[0081] The initial camera parameter module is used to determine the initial camera pose of the visual monitoring equipment on each line based on the identified tower point cloud.

[0082] The depth map generation module is used to generate depth maps corresponding to each visualized monitoring image based on the initial camera pose, camera intrinsic parameters, and point cloud data.

[0083] The camera pose calculation module is used to optimize the initial camera pose of each line by using the matching degree between the projection of point cloud data on the corresponding visualized monitoring image and the scene features in the image as the optimization target, so as to obtain the optimized camera pose.

[0084] The power transmission line hazard detection module is used to identify hazard targets in each visualized monitoring image based on a pre-trained hazard identification model, and to obtain the pixel coordinate information of each hazard target in the image.

[0085] The hazard location module is used to calculate the location coordinates of each hazard target in three-dimensional space through coordinate transformation, based on its pixel coordinate information, corresponding depth map, camera intrinsic parameters and optimized camera pose.

[0086] The duplicate hazard determination module is used to compare the location coordinates of hazard targets identified from visual monitoring images of different lines. If the spatial distance between the location coordinates of any two hazard targets is less than the duplicate determination threshold, then the two hazard targets are determined to be duplicate hazards.

[0087] To more clearly illustrate the outstanding substantive features of this invention and the significant progress it brings to the prior art, an application example of implementing this invention is described below.

[0088] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Specific application examples include: This embodiment provides a method for determining the hidden danger of parallel line duplication based on multi-source data, which can be used to accurately and efficiently identify the hidden danger of parallel line duplication.

[0089] Step 1: Obtain point cloud data of transmission line 1 and transmission line 2 running parallel to each other in a certain area at a certain span, such as... Figure 2 As shown; acquire visual surveillance image data of transmission line 1 and transmission line 2 respectively (e.g. Figure 3 (As shown) and the intrinsic parameter matrices of the visualization monitoring image equipment, line1_M and line2_M. Intrinsic parameter matrix of the visualization monitoring image equipment for transmission line 1, line1_M:

[0090] Transmission Line 2 Visual Monitoring Image Equipment Intrinsic Parameter Matrix line2_M:

[0091] It should be noted that the parallel lines described in this invention refer to transmission lines that are double-circuit or multi-circuit on the same tower, or that are erected parallel to existing lines over a long distance along the same corridor. These types of lines are highly prone to recurring safety hazards during operation and maintenance, making them a key application scenario for this invention.

[0092] Step 2: Obtain the classified point cloud using a point cloud classification algorithm, where towers are marked in black, and each tower has a separate instance category, such as... Figure 4 As shown.

[0093] Based on the tower points of transmission line 1, the initial pose of the visualization monitoring image device for transmission line 1 is calculated as line1_RT.

[0094] Based on the tower points of transmission line 2, the initial pose of the visualization monitoring image equipment for transmission line 2 is calculated using cloud computing (line2_RT).

[0095] Step 3: Generate the line1_depth and line2_depth matrices for different categories of scenery for transmission line 1 and transmission line 2, respectively.

[0096] Generation of depth matrices for different types of scenery along transmission line 1:

[0097] Generation of depth matrices for different types of scenery along transmission lines 2:

[0098] Step 4: Optimize the initial poses of transmission line 1 and transmission line 2 using the camera pose calculation module to obtain the optimized precise camera poses line1_RTend and line2_RTend respectively.

[0099]

[0100] Step 5: Detect the coordinates of the hazard box in the visual monitoring image of transmission line 1 using the transmission line hazard detection module. The coordinates of the hidden danger boxes in the visual monitoring image of transmission line 2 were detected. The visualization effect of hidden dangers is as follows: Figure 5 As shown, the confidence levels for the detected trucks were 0.28 ( Figure 5 The hidden danger in the left figure) and 0.29 ( Figure 5 (Hidden danger in the right image).

[0101] Step 6: Using the hazard location module, based on the hazard bounding box coordinates, depth map, camera intrinsic parameters, and optimized pose, calculate the three-dimensional location coordinates of the hazard in transmission line 1. and the three-dimensional location coordinates of the hidden dangers in transmission line 2. .

[0102] Step 7: Calculate the horizontal Euclidean distance dist between line1_danger_3d and line2_danger_3d, which is 0.794 meters. Since this distance is less than the preset 1-meter threshold, the hazard is determined to be a repeating hazard.

[0103] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention.

Claims

1. A method for determining the potential for duplicate tracks in parallel circuits based on multi-source data, characterized in that, include: Acquire multi-source data of parallel lines, including point cloud data of each line in the parallel lines and corresponding visualized monitoring images; Based on the point cloud data of each line, the initial camera pose of the visual monitoring equipment on each line is determined. Then, combined with the intrinsic parameters of the visual monitoring equipment and the point cloud data, a depth map corresponding to each visual monitoring image is generated. The determination of the initial camera pose of the visual monitoring devices on each line includes: Semantic instance segmentation is performed on the point cloud data, the point cloud of each tower is identified from the segmentation results, and an independent instance identifier is assigned to each tower. Based on the pole instance corresponding to the installation location of the visual monitoring equipment, obtain the geometric center coordinates of the point cloud of that pole, as well as the geometric center coordinates of the point cloud of the next pole adjacent to that pole; The geometric center coordinates of the tower where the visual monitoring device is located are used as the initial translation parameters. The initial rotation matrix is ​​calculated based on the direction vector between the geometric center coordinates of the two towers. The initial camera pose is obtained by combining the initial rotation matrix and the initial translation parameters. The initial camera pose of each line is optimized by taking the matching degree between the projection of point cloud data on the corresponding visualized monitoring image and the scene features in the image as the optimization objective. The optimization of the initial camera pose for each line includes: Scene features are extracted from visual surveillance images, including the pixel positions of line segment endpoints obtained through a line detection algorithm; Project the conductor point cloud and tower point cloud in the point cloud data onto the image plane according to the current candidate camera pose to obtain the projection point set; Calculate the Euclidean distance between the pixels in the projection point set and the pixel positions of the line segment endpoints, count the number of matching point pairs whose Euclidean distance is less than a preset distance threshold, and use the number of matching point pairs to characterize the original fitness function value. The conductor point cloud and tower point cloud are projected onto a two-dimensional plane according to the current candidate camera pose to generate a grayscale structure map of the same size as the visualized monitoring image. Edge information in the visualized monitoring image is extracted by an edge detection algorithm to generate an image edge grayscale map of the same size. The structural similarity between the grayscale structure map and the image edge grayscale map is calculated. The original fitness function value is multiplied by the structural similarity to obtain a new fitness function value. With the goal of maximizing the new fitness function value, the candidate camera pose is iteratively updated using a particle swarm optimization algorithm until a preset iteration stopping condition is met, and the optimized camera pose is output. Based on the power transmission line hidden danger detection algorithm, hidden danger targets in each visual monitoring image are identified, and the pixel coordinate information of each hidden danger target in the image is obtained. For each hidden danger target, the positioning coordinates of the hidden danger target in three-dimensional space are calculated based on its pixel coordinate information, the corresponding depth map, camera intrinsic parameters and optimized camera pose. For hidden danger targets identified from visual monitoring images of different lines, their positioning coordinates are compared. If the spatial distance between the positioning coordinates of any two hidden danger targets is less than the repetition judgment threshold, then the two hidden danger targets are judged as repeated hidden dangers.

2. The method for determining the potential repetition risk of parallel lines based on multi-source data according to claim 1, characterized in that: The semantic instance segmentation adopts a point cloud segmentation network model, and the cross-entropy loss function is weighted by category during training, wherein the weight coefficient of the conductor category is greater than the weight coefficient of the ground category, and the weight coefficient of the tower category is greater than the weight coefficient of the conductor category. After training, the point cloud data is input into the point cloud segmentation network model, and the class label and tower instance label of each 3D point are output. In the post-segmentation processing, based on the spatial distribution of the conductor point cloud, conductor point clouds located outside the spatial corridor of the line are reclassified as non-conductors, and point clouds located within the preset neighborhood of the conductor point cloud that have not been classified as conductors are added as conductor point clouds.

3. The method for determining the potential repetition risk of parallel lines based on multi-source data according to claim 1, characterized in that: The process of generating depth maps corresponding to each visualized surveillance image includes: Each 3D point in the point cloud data after semantic instance segmentation is projected onto the pixel coordinate system of the visualized monitoring image based on the initial camera pose and camera intrinsic parameters to obtain the pixel coordinates corresponding to the 3D point. Obtain the depth value of each 3D point relative to the camera's optical center, and assign this depth value to the pixel position to which the 3D point is projected, thereby generating a depth map with the same resolution as the visualized monitoring image.

4. The method for determining the potential for duplicate parallel lines based on multi-source data according to claim 1, characterized in that: The power transmission line hazard detection algorithm is a deep learning target detection model based on the YOLOv8 architecture; The deep learning object detection model is fine-tuned on a training dataset containing images of power transmission line corridors. The proportion of parallel line scene images in the training dataset is not less than a first preset proportion, and images of different spans and different weather conditions are allocated according to a second preset proportion. After training, this deep learning object detection model is used to identify mechanical damage hazards in visualized surveillance images, and outputs the category, bounding box coordinates, and confidence score of each hazard.

5. The method for determining the potential for duplicate parallel lines based on multi-source data according to claim 4, characterized in that: After identifying potential hazards in each visualized surveillance image, the pixel coordinates of each hazard in the image are obtained, including: Extract the pixel coordinates of the lower edge center point from the detected bounding box coordinates, and use them as the projection point of the target's ground contact location in the image; When multiple potential hazards are detected in the same visual monitoring image, the pixel coordinates corresponding to each potential hazard are extracted.

6. The method for determining the potential repetition risk of parallel lines based on multi-source data according to claim 1, characterized in that: Calculating the location coordinates of the potential hazard in three-dimensional space includes: For each potential hazard target, obtain the pixel coordinates from the center point of the lower edge of its detection box; Read the corresponding depth value from the depth map based on the pixel coordinates; Using the camera intrinsic parameter matrix, the pixel coordinates and depth value are converted into three-dimensional coordinates in the camera coordinate system; By using the rotation matrix and translation vector represented by the optimized camera pose, the three-dimensional coordinates in the camera coordinate system are transformed to the world coordinate system to obtain the location coordinates of the potential hazard.

7. The method for determining the potential repetition risk of parallel lines based on multi-source data according to claim 1, characterized in that: The spatial distance is the Euclidean distance between the location coordinates of the two potential hazards on the horizontal plane; The repetition determination threshold is a length value set in advance based on the statistical value of point cloud positioning accuracy. When the Euclidean distance between two potential hazards is less than the repetition determination threshold, they are determined to be repeated hazards.

8. A parallel line duplication hazard determination system based on multi-source data, used to implement the parallel line duplication hazard determination method based on multi-source data as described in any one of claims 1-7, characterized in that, include: The module includes a data acquisition module, a point cloud semantic instance segmentation module, an initial camera parameter module, a depth map generation module, a camera pose calculation module, a power transmission line hazard detection module, a hazard location module, and a repeating hazard determination module, among which: The data acquisition module is used to acquire multi-source data of the parallel lines, including point cloud data of each line in the parallel lines and corresponding visual monitoring images, as well as camera intrinsic parameters of each visual monitoring device. The point cloud semantic instance segmentation module is used to perform semantic instance segmentation on the point cloud data of each line in order to classify and identify conductors, towers and ground features; The initial camera parameter module is used to determine the initial camera pose of the visual monitoring equipment on each line based on the identified tower point cloud. The depth map generation module is used to generate depth maps corresponding to each visualized monitoring image based on the initial camera pose, camera intrinsic parameters, and point cloud data. The camera pose calculation module is used to optimize the initial camera pose of each line by using the matching degree between the projection of point cloud data on the corresponding visualized monitoring image and the scene features in the image as the optimization target, so as to obtain the optimized camera pose. The power transmission line hidden danger detection module is used to identify hidden danger targets in each visualized monitoring image based on a pre-trained hidden danger identification model, and to obtain the pixel coordinate information of each hidden danger target in the image; The hazard location module is used to calculate the location coordinates of each hazard target in three-dimensional space through coordinate transformation, based on its pixel coordinate information, corresponding depth map, camera intrinsic parameters and optimized camera pose. The duplicate hazard determination module is used to compare the location coordinates of hazard targets identified from visual monitoring images of different lines. If the spatial distance between the location coordinates of any two hazard targets is less than the duplicate determination threshold, then the two hazard targets are determined to be duplicate hazards.

Images