A near-electric operation real-time automatic ranging early warning system and method based on laser point cloud and image fusion

Abstract

The application discloses a near-electricity operation real-time automatic ranging early warning system and method based on laser point cloud and image fusion. The method comprises the following steps: fusing laser point cloud data and image data collected in a near-electricity operation site to obtain a mapping relationship between pixels in an image and the laser point cloud data; extracting, according to the mapping relationship, motion monitoring target point cloud data corresponding to a motion monitoring target in the image data and conductor point cloud data corresponding to a static target in the image data from the point cloud data; calculating the nearest distance between the conductor point cloud data and the motion monitoring target point cloud data; and implementing early warning on the near-electricity operation according to the nearest distance. The application realizes the real-time automatic ranging early warning of the near-electricity operation by fusing the laser point cloud and the image data, automatically spreading the conductor and the motion target point cloud, and performing the real-time ranging on the conductor and the motion monitoring target, thereby improving the automatic degree of the extraction of the conductor and the motion monitoring target point cloud.

Description

Technical Field

[0001] This invention relates to the field of near-electric work monitoring and early warning technology, specifically to a real-time automatic ranging and early warning system and method for near-electric work based on laser point cloud and image fusion. Background Technology

[0002] With the continuous advancement of power grid construction, the use of large construction machinery such as cranes for near-power line work is becoming increasingly common. Due to the limited space and complex environment of these work sites, collisions between construction machinery and power lines are prone to occur, leading to accidents. Currently, to avoid collisions, manual monitoring of safety distances is the primary method, which is not only time-consuming and labor-intensive but also lacks timely and comprehensive monitoring, failing to effectively ensure the safety of near-power line work. However, using machines to replace manual labor for real-time monitoring and early warning of near-power line work can significantly reduce the occurrence of accidents.

[0003] Current technical solutions for real-time monitoring and early warning of near-powered work primarily employ methods based on laser point clouds and images. While laser point cloud-based methods can acquire precise 3D spatial information of the work scene, the lack of color and texture information, coupled with the sparsity of point clouds, makes target identification difficult and hinders direct visualization applications. Image-based methods, on the other hand, can acquire rich visual information of the scene, enabling rapid and accurate target detection, but the lack of depth information makes precise 3D target localization challenging. Methods combining laser point cloud and image fusion leverage the advantages of both data, resulting in high accuracy and good visualization of 3D spatial distance monitoring. However, existing solutions have limited application scenarios, weak interactivity, and limited monitoring functions, making it difficult to implement more effective, intuitive, and precise near-powered work risk prevention and control. Summary of the Invention

[0004] This invention provides a real-time automatic ranging and early warning system and method for near-powered work based on laser point cloud and image fusion. By fusing laser point cloud and image data, and utilizing the automatic spread of the target point cloud by the conductor and motion monitoring, real-time ranging and early warning for near-powered work are achieved. The monitoring range is wider, the safety is higher, and the results are more in line with the requirements of actual engineering applications.

[0005] One of the objectives of this invention is a real-time automatic ranging and early warning system for near-electric work based on laser point cloud and image fusion, comprising:

[0006] Data fusion module: used to fuse laser point cloud data and image data collected at the near-electricity work site to obtain the mapping relationship between pixels in the image and point cloud data;

[0007] Point cloud processing module: used to extract, according to the mapping relationship, the motion monitoring target point cloud data corresponding to the motion monitoring target in the image data, and the conductor point cloud data corresponding to the static target from the laser point cloud data; the motion monitoring target refers to mechanical equipment in the near-electricity operation process; the static target refers to static power facilities such as transmission lines;

[0008] Distance measuring module: used to calculate the closest distance between the traverse point cloud data and the motion monitoring target point cloud data;

[0009] Early warning module: used to issue early warnings for near-electric work based on the nearest distance.

[0010] Furthermore, the method for extracting conductor point cloud data in the point cloud processing module includes:

[0011] The target recognition network is used to obtain the detection boxes of all wires in the image data, and the point cloud data of the wires in each detection box is obtained according to the mapping relationship.

[0012] Point cloud clustering: For the conductor point cloud data within each detection box, a density-based clustering algorithm is used to divide the conductor point cloud data into different clusters, and the different clusters constitute the clustering result;

[0013] Construct a sprawling point cloud set and a traverse point cloud set; select a point p from the clustering results. i As the current point of spread of the conductor, it is added to the set of spread points;

[0014] Conductor spread: Calculate the current conductor spread point p i Direction vector α i ; Get at point p i Direction vector α i tangent plane S i ; Obtain the tangent plane S i The set of point clouds whose planar distance is less than a first set distance Obtain point cloud set The radius r of the envelope circle i If r i relative to r i-1 If a mutation occurs: remove point p from the spread point cloud set. i And no longer spreading to point p i Otherwise, point p will be used. i Add to the conductor point cloud set, which is used to store the point cloud data of the conductor; set the current envelope circle radius r. i The value is updated to r i-1 The value; select the next traverse spread point as the current traverse spread point P from the remaining unprocessed point cloud in the clustering results. iAnd add it to the spread point cloud set;

[0015] Repeat the above traverse propagation steps until no new point clouds are added to the propagation point cloud set; the traverse point cloud set is the traverse point cloud data representing all traverses.

[0016] The technical advantages of the above-mentioned method for extracting conductor point cloud data include: obtaining conductor detection boxes through a target recognition network, then obtaining conductor point clouds based on mapping relationships, and combining point cloud clustering and conductor propagation operations, which can accurately extract conductor point cloud data representing all conductors from complex point cloud data, effectively eliminating other interference factors and improving the accuracy and completeness of conductor point cloud extraction; during conductor propagation, by calculating the direction vector and tangent plane of the points, and determining whether to continue propagating based on the abrupt change in the radius of the envelope circle, the system can adaptively handle conductor point clouds of different shapes and distributions, ensuring that the extraction results are more consistent with the actual conductor shape.

[0017] Furthermore, the method for extracting motion monitoring target point cloud data in the point cloud processing module includes:

[0018] Constructing a point cloud map P map =[P D ,P S ] and its corresponding rasterized map P grid , where P D P is the set of dynamic point clouds from the most recent n-1 frames. S It is the set of static point clouds from the most recent n-1 frames;

[0019] The point cloud data acquired by the LiDAR is segmented according to the moving target point cloud segmentation algorithm to obtain the dynamic point cloud of the most recent frame. and static point cloud Save to P respectively D and P S middle;

[0020] Remove point cloud map P map The point cloud of the in-th frame, to ensure P D and P S It contains only the dynamic and static point clouds of the most recent n-1 frames;

[0021] Based on dynamic point cloud and static point cloud In the current raster map P grid The number of points in the corresponding grid position is used to update the attribute of the corresponding grid as either a static grid or a dynamic grid.

[0022] A density-based propagation algorithm is employed, using the dynamic point cloud of the current frame. For the initial point cloud to the point cloud map P mapThe point cloud within is spread out so that the spread point cloud only contains dynamic point clouds, and the spread point cloud constitutes a spread point cloud set.

[0023] Repeat the above process until no new point cloud is spread. The final set of spread point clouds is the motion monitoring target point cloud.

[0024] The technical effects of the above-mentioned method for extracting point cloud data of motion monitoring targets include: constructing a point cloud map and performing rasterization processing, combined with a motion target point cloud segmentation algorithm, which can effectively distinguish between dynamic and static point clouds, realize accurate extraction of point clouds of motion monitoring targets, and provide accurate data for subsequent calculation of the distance between motion monitoring targets and guide wires; by saving the point cloud data of the most recent n-1 frames and continuously updating and removing point clouds of old frames, the system can make full use of the information of multiple frames of data, improve the tracking and monitoring capabilities of moving targets, and adapt to the complex motion state of moving targets in the work scenario.

[0025] Furthermore, the method for updating the attribute of the corresponding raster to a static raster or a dynamic raster includes:

[0026] If the current raster map P grid If the number of dynamic point clouds in any grid cell of the current frame is greater than the number of static point clouds, then the grid cell is a dynamic grid cell; otherwise, it is a static grid cell.

[0027] The technical benefits of updating raster attributes include: determining whether a raster attribute is static or dynamic based on the number of dynamic and static point clouds in the current frame within the raster, providing a clear classification basis for subsequent raster-based point cloud processing and analysis, and helping to more accurately identify and process dynamic point clouds; the method is computationally simple and easy to implement, improving the system's operating efficiency while ensuring accuracy, and enabling rapid updating of raster attributes to meet real-time requirements.

[0028] Further technical solutions include: the method for making the point cloud obtained by the spread contain only dynamic point clouds includes:

[0029] If, during the spread process, the point cloud obtained from the spread of a certain moving target contains point clouds within a static grid, then the portion of the point cloud belonging to the static grid is removed from the point cloud obtained from the spread of the certain moving target.

[0030] If the point cloud obtained by the spread of a certain moving target point is entirely within a static grid, then the spread of that moving target point shall be stopped.

[0031] The technical effects of the above-mentioned solution include: removing point clouds within static grids during the propagation process and stopping the propagation of point clouds that are entirely within static grids, ensuring that the final propagated point cloud contains only dynamic point clouds, improving the purity of the motion monitoring target point cloud, and reducing the interference of static point clouds on motion target monitoring; effectively avoiding the situation of misjudging static point clouds as motion target point clouds, making the system's positioning and tracking of motion monitoring targets more accurate, and thus improving the overall system's monitoring accuracy of moving targets in near-electrical operations.

[0032] Furthermore, the method for calculating the nearest distance includes:

[0033] Constructing a kd-tree for motion monitoring target point cloud; kd-tree (K-Dimensional Tree) is a data structure for organizing data points in k-dimensional space, which can perform nearest neighbor search more efficiently; kd-tree organizes point cloud data according to certain rules, which can quickly eliminate most impossible points when searching for the nearest point, thereby greatly improving search efficiency;

[0034] For each point in the motion monitoring target point cloud data, find the point that is spatially closest to it from the guide wire point cloud data, ultimately forming a point cloud pair set P. pair ,in This represents the pair of points closest to the i-th point in the motion monitoring target point cloud data, and its calculation formula is:

[0035]

[0036] In the formula, For the i-th point in the motion monitoring target point cloud data, Distance in point cloud data for guide lines The nearest point;

[0037] The point cloud pair set P is calculated using the Euclidean distance formula. pairs The minimum distance between any two points is the closest distance between the guide wire and the motion monitoring target.

[0038] The beneficial effects of the above-mentioned technical solution for calculating the nearest distance include: the kd-tree data structure can quickly find the spatially nearest point for each point in the motion monitoring target point cloud data from the traverse point cloud data, greatly improving the efficiency of the nearest point search, and has obvious advantages when processing large-scale point cloud data; the use of the Euclidean distance formula to calculate the distance between every two points in the point cloud pair set can accurately obtain the nearest distance between the traverse and the motion monitoring target, providing accurate distance data support for subsequent early warning, and ensuring the accuracy and reliability of the early warning.

[0039] Furthermore, methods for issuing early warnings for near-powered work include:

[0040] When the nearest distance is less than a first preset threshold, an early warning is triggered to remind that there may be operational risks.

[0041] When the nearest distance is less than the second set threshold, the warning device is triggered to control the over-limit lock controller to automatically lock the mechanical equipment transmission mechanism; the second set threshold is less than the first set threshold.

[0042] The technical effects of the aforementioned early warning system include: the mechanism of setting different thresholds for graded early warning can take different countermeasures according to the degree of danger, providing multi-level safety protection; by triggering the early warning device to control the over-limit lockout controller, proactive protection against the risks of working near power is achieved, enabling timely measures to be taken before danger occurs, effectively reducing the incidence of accidents during working near power, and protecting the lives of workers and the normal operation of equipment.

[0043] Furthermore, it also includes a joint calibration module for jointly calibrating the lidar and the monocular camera to obtain an extrinsic parameter calibration matrix; the extrinsic parameter calibration matrix is ​​used to fuse the lidar point cloud and image data to obtain the mapping relationship between pixels in the image and point cloud data.

[0044] Furthermore, the calibration method includes: using an edge computing processor equipped with a lidar and a camera to collect target feature points; the target feature points are typically easily identifiable markers; and jointly calibrating the lidar and camera by extracting image feature points and point cloud feature points of the target to obtain an extrinsic parameter calibration matrix for transforming the lidar coordinate system and the camera coordinate system and realizing the fusion of point cloud and image.

[0045] A second objective of this invention is a real-time automatic ranging and early warning method for near-electric work based on laser point cloud and image fusion, comprising:

[0046] The mapping relationship between pixels in the image and point cloud data is obtained by fusing laser point cloud data and image data collected at the near-electricity operation site;

[0047] Based on the mapping relationship, the motion monitoring target point cloud data corresponding to the motion monitoring target in the image data and the guide wire point cloud data corresponding to the static target in the image data are extracted from the laser point cloud data;

[0048] Calculate the closest distance between the conductor point cloud data and the motion monitoring target point cloud data;

[0049] Early warning is issued for near-electric work based on the aforementioned closest distance.

[0050] A non-transitory computer-readable storage medium for achieving the third objective of the present invention stores a computer program thereon, wherein when the computer program is executed by a processor, it implements the steps of the real-time automatic ranging and early warning method for near-electric work based on laser point cloud and image fusion.

[0051] A computer program product for achieving the fourth objective of the present invention includes a computer program / instruction that, when executed by a processor, implements the steps of the real-time automatic ranging and early warning method for near-electric work based on laser point cloud and image fusion.

[0052] The beneficial effects of this invention include:

[0053] The technical solution described in this invention encompasses multiple modules, including fusion, extraction, calculation, and early warning, forming a complete near-power operation monitoring and early warning system. This system achieves full-process functionality from data acquisition to final early warning, providing comprehensive protection for near-power operation safety. The data fusion module integrates laser point clouds with image data, fully combining the spatial information of the laser point clouds with visual information such as texture and color of the images. This improves the perception accuracy of the work scene and lays the foundation for subsequent accurate extraction of target point cloud data and distance calculation. By automatically spreading the conductor and moving target point clouds, real-time distance measurement is performed between the conductor and the moving monitoring target, improving the automation level of conductor and moving target point cloud extraction and realizing real-time automatic distance measurement and early warning for near-power operations. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating the method described in this invention;

[0055] Figure 2 System block diagram of the present invention. Detailed Implementation

[0056] The following detailed embodiments are provided to explain the technical solutions of the present invention, so that those skilled in the art can understand the present invention. The scope of protection of the present invention is not limited to the following specific embodiments. Any modifications or improvements made by those skilled in the art that incorporate the technical solutions of the present invention but differ from the following detailed embodiments are also within the scope of protection of the present invention.

[0057] This invention provides a real-time automatic ranging and early warning method for near-electric work based on laser point cloud and image fusion, such as... Figure 1 As shown, it includes:

[0058] S1: Use an edge computing processor equipped with a LiDAR and a camera to collect target feature points; the target feature points are usually easily identifiable markers; the LiDAR and camera are jointly calibrated by extracting the image feature points and point cloud feature points of the target, such as using the EPnP algorithm to solve for 2D feature points and 3D feature points, to obtain the extrinsic parameter calibration matrix used to transform the LiDAR coordinate system and the camera coordinate system and realize the fusion of point cloud and image;

[0059] S2: Acquire real-time laser point cloud and image data from lidar and monocular camera, and fuse the laser point cloud and image data according to the external parameter calibration matrix to obtain the mapping relationship between pixels in the image and point cloud data, thereby realizing real-time automatic association and mapping between pixels and point cloud.

[0060] S3: The point cloud data of conductors (such as power lines, cables, etc.) is automatically extracted using the conductor point cloud automatic propagation method. body Furthermore, methods for extracting point cloud data from all conductors include:

[0061] S3.1 Obtain detection boxes for all wires in the image through the target recognition network, and then obtain the wire point cloud within each detection box according to the mapping relationship; the point cloud data corresponding to each detection box can be stored in different temporary sets for subsequent processing.

[0062] S3.2 For the point cloud of the conductor within each detection box, an automatic spread is performed using a density-based clustering algorithm (such as DBSCAN). This algorithm divides the point cloud into different clusters based on the density relationship between points, and selects a point from the clustering results as the first conductor spread point p1; the selection rule can be random selection or selection of a point located near the center of the detection box; p1 is the initial spread point cloud, and this spread point cloud set will continuously expand as subsequent steps progress;

[0063] S3.3 For the current conductor propagation point p i Calculate point p i Direction vector α i One method for calculating the direction vector is to calculate the principal directions between the point and its neighboring points, for example, using Principal Component Analysis (PCA), thereby obtaining the direction vector at point p. i α i tangent plane S i The equation of the tangent plane can be determined using the point-normal form.

[0064] S3.4 Obtaining the tangent plane S i The planar distance is less than M S Point clouds By traversing all point clouds and calculating the distance from each point to the tangent plane, point clouds with distances less than Ms can be selected. The point cloud set is obtained using a minimum envelope circle algorithm (such as the Welzl algorithm). The radius r of the envelope circle i ,judge Whether it holds true; for the case of i=1, since r0 does not exist, an initial value for r0 can be set (such as 0 or a suitable value based on experience); if r i A mutation occurs, i.e., |r i -r i-1 |>M r At that time, remove point p from the spread point cloud set. i And no longer spreading to point p i If the condition is not met, then point p will be... i Add to the traverse point cloud set;

[0065] S3.5 will change the current r i The value is updated to r i-1 This prepares for the next iteration; from the remaining unprocessed point cloud, the next traverse spread point P is selected according to certain rules (such as the point closest to the current spread point that has not been marked as non-spreading). i+1 And add it to the spread point cloud set;

[0066] S3.6 Repeat S3.2-S3.5 until no new point cloud is spread;

[0067] S3.7 After processing the conductor point cloud corresponding to each detection box, the conductor point cloud set is the conductor point cloud data P of all conductors in the fused image. body .

[0068] S4: The automatic spread method for moving targets (such as large machinery in operation) is used to automatically spread the monitored moving targets to obtain the moving target point cloud P. target Furthermore, it may include:

[0069] S4.1 Constructing a Point Cloud Map P map =[P D ,P S ] and its corresponding rasterized map P grid , where P D Let P be the set of dynamic point clouds from the most recent n-1 frames. S Given a set of static point clouds from the most recent n-1 frames, the rasterized map P... grid Each grid cell has either static or dynamic attributes.

[0070] S4.2 The point cloud data of each frame acquired by the LiDAR is segmented according to the moving target point cloud segmentation algorithm to obtain the dynamic point cloud and static point cloud of each frame. Here, the dynamic point cloud refers to the point cloud data generated by moving targets, while the static point cloud is the point cloud data generated by stationary objects. For example, in a near-electricity operation scenario, the point cloud generated by the large machinery operated by the worker is a dynamic point cloud, while the point cloud generated by the surrounding fixed power facilities, buildings, etc., is a static point cloud. For example, the dynamic point cloud can be obtained by segmenting the point cloud data of the current i-th frame. and static point cloud And save them to P respectively. D and P S Meanwhile, remove P. map The point cloud of the in-th frame, to ensure P D and P S It contains only the dynamic and static point clouds of the most recent n-1 frames. That is:

[0071] P D =[ , , ..., ]

[0072] P S =[ , , ..., ]

[0073] n is a fixed parameter value, representing the number of frames whose points are retained; for example, setting n=5 indicates that P D and P S The internal storage contains dynamic and static point clouds from the most recent 4 frames, where i represents the current frame number, which increases over time and i>n;

[0074] S4.3 Based on dynamic point cloud and static point cloud In the raster map P grid The position in the map is updated, and the raster attributes are updated. If the current raster map P is... grid In a raster map, if the number of dynamic point clouds in the current i-th frame is greater than the number of static point clouds, then that raster is a dynamic raster; otherwise, it is a static raster. In this way, as the point cloud data is updated with each frame, the rasterized map P... grid The properties of each grid cell will also be dynamically updated accordingly, thus accurately reflecting the distribution of dynamic and static objects in the current scene;

[0075] S4.4 employs a density-based propagation algorithm, using the dynamic point cloud of the current frame. For the initial point cloud pair P mapInternal point clouds spread outwards. Density-based spreading algorithms (such as extensions of the DBSCAN algorithm) expand from the initial point cloud according to the density relationships between points, grouping densely connected point clouds together. For example, for dynamic point clouds... For each point in the cloud, the algorithm searches for points in its neighborhood whose density meets certain conditions and adds these points to the spreading point cloud set; during the spreading process, if a moving target spreads to point p... j If the point cloud obtained from the spread contains point clouds within static grids, then the spread point p from the moving target... j The point cloud obtained through propagation is processed by removing the portion belonging to the static grid to ensure that the resulting point cloud contains only dynamic point clouds relevant to the moving target. This allows for more accurate monitoring and analysis of the moving target in subsequent operations. If point p... j If the point cloud obtained from the spread is entirely within a static raster, it indicates that the spread point has reached the static region, so stop p. j Point spread is used to avoid mistakenly including the point cloud of stationary objects in the point cloud of moving targets;

[0076] S4.5 Repeat the above propagation and processing process until no new points can be propagated. The final point cloud set is the motion monitoring target point cloud P. target .

[0077] S5: Calculate the closest distance between the guide wire and the motion monitoring target, which may further include:

[0078] S5.1: Constructing a motion monitoring target point cloud P target The kd-tree (K-Dimensional Tree, a data structure used to organize data points in k-dimensional space, enabling more efficient nearest neighbor search. The kd-tree organizes point cloud data according to certain rules, allowing for the rapid elimination of most impossible points when searching for the nearest point, thus greatly improving search efficiency), is used for P... target Each point in the point cloud P is a traverse point. body Find the point that is spatially closest to it, and finally form a set P of point cloud pairs. pair ,in Represents the distance to the traverse point cloud P body The closest pair of points to the i-th point in the array is calculated as follows:

[0079]

[0080] In the formula, For motion monitoring target point cloud P target The i-th point in the middle, For the point cloud P of the guide wire bodymedium distance the closest point;

[0081] S5.2: Calculate the distance between every two points in the point cloud pair set P using the Euclidean distance formula pairs and find the minimum distance, and denote the minimum distance as d min , which is the closest distance between the wire and the motion monitoring target.

[0082] S6: Judge in real time whether the closest distance d min reaches the set early warning threshold, and trigger the anti-collision acousto-optic alarm in time. Further, it may include:

[0083] S6.1: Judge in real time whether the closest distance d between the wire and the motion monitoring target min is less than a first set threshold M1; M1 is set according to actual demands, which is not limited in the present invention;

[0084] S6.2: If the closest distance d between the wire and the motion monitoring target min is less than the first set threshold M1, trigger the acousto-optic early warning, that is

[0085] d min <M1

[0086] Otherwise, continue the real-time monitoring of the safety distance for live working in proximity.

[0087] S7: Judge in real time whether the closest distance between the wire and the motion monitoring target reaches an over-limit locking threshold, and control the over-limit locking controller to act in time to automatically lock the transmission mechanism of the mechanical equipment. Further, it may include:

[0088] S7.1: Based on the calculation result of S5, judge in real time whether the closest distance d between the wire and the motion monitoring target min is less than a second set threshold M2 (M2<M1); M2 is set according to actual demands, which is not limited in the present invention;

[0089] S7.2: If the closest distance d between the wire and the motion monitoring target min is less than the second set threshold M2, trigger the early warning device to control the over-limit locking controller to act, so that the transmission mechanism of the mechanical equipment is automatically locked, that is

[0090] d min <M2

[0091] Otherwise, continue the real-time monitoring of the safety distance for live working in proximity.

[0092] In this embodiment, it is judged whether the real-time distance between the wire and the motion monitoring target reaches the over-limit locking threshold, and the over-limit locking controller is controlled to act in time to automatically lock the transmission mechanism of the mechanical equipment.

[0093] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0094] This invention also provides a real-time automatic ranging and early warning system for near-electric work based on laser point cloud and image fusion, such as... Figure 2 As shown, it includes:

[0095] Data fusion module: used to fuse laser point cloud and image data collected at the near-electricity work site to obtain the mapping relationship between pixels in the image and point cloud data;

[0096] Point cloud processing module: used to extract motion monitoring target point cloud data corresponding to motion monitoring targets in image data and guide wire point cloud data corresponding to static targets in image data from point cloud data according to the mapping relationship;

[0097] Distance measuring module: used to calculate the closest distance between the traverse point cloud data and the motion monitoring target point cloud data;

[0098] Early warning module: used to issue early warnings for near-electric work based on the nearest distance.

[0099] In some embodiments, a joint calibration module is also included, which is used to jointly calibrate the lidar and the monocular camera to obtain an extrinsic parameter calibration matrix; the extrinsic parameter calibration matrix is ​​used to fuse the lidar point cloud and image data to obtain the mapping relationship between pixels in the image and point cloud data.

[0100] In some embodiments, the method for extracting guide wire point cloud data in the point cloud processing module includes:

[0101] The target recognition network is used to obtain the detection boxes of all wires in the image, and the wire point cloud within each detection box is obtained according to the mapping relationship.

[0102] For the conductor point cloud data within each detection box, a density-based clustering algorithm is used to divide the conductor point cloud data into different clusters, and the different clusters constitute the clustering result;

[0103] Construct a sprawling point cloud set and a traverse point cloud set; select a point p from the clustering results. i As the current point of spread of the conductor, it is added to the set of spread points;

[0104] Conductor spread: Calculate the current conductor spread point p i Direction vector α i ; Get at point p i Direction vector α i tangent plane Si ; Obtain the tangent plane S i The set of point clouds whose planar distance is less than a first set distance Obtain point cloud set The radius r of the envelope circle i If r i relative to r i-1 If a mutation occurs: remove point p from the spread point cloud set. i And no longer spreading to point p i Otherwise, point p will be used. i Add to the traverse point cloud set, which is used to store the point cloud data of the traverse; add the current r i The value is updated to r i-1 From the remaining unprocessed point cloud in the clustering results, select the next traverse spread point as the current traverse spread point P. i And add it to the spread point cloud set;

[0105] Repeat the above traverse propagation steps until no new point clouds are added to the propagation point cloud set; the traverse point cloud set is the traverse point cloud data representing all traverses.

[0106] In some embodiments, the method for extracting motion monitoring target point cloud data in the point cloud processing module includes:

[0107] Constructing a point cloud map P map =[P D ,P S ] and its corresponding rasterized map P grid , where P D P is the set of dynamic point clouds from the most recent n-1 frames. S It is the set of static point clouds from the most recent n-1 frames;

[0108] The point cloud data acquired by the LiDAR is segmented according to the moving target point cloud segmentation algorithm to obtain the dynamic point cloud of the most recent frame. and static point cloud Save to P respectively D and P S middle;

[0109] Remove point cloud map P map The point cloud of the in-th frame, to ensure P D and P S It contains only the dynamic and static point clouds of the most recent n-1 frames;

[0110] Based on dynamic point cloud and static point cloud In the current raster map P gridThe number of points in the corresponding grid position is used to update the attribute of the corresponding grid as either a static grid or a dynamic grid.

[0111] A density-based propagation algorithm is employed, using the dynamic point cloud of the current frame. For the initial point cloud to the point cloud map P map The point cloud within is spread out so that the spread point cloud only contains dynamic point clouds, and the spread point cloud constitutes a spread point cloud set.

[0112] Repeat the above process until no new point cloud is spread. The final set of spread point clouds is the motion monitoring target point cloud.

[0113] In some embodiments, the method of updating the attribute of the corresponding raster to a static raster or a dynamic raster includes:

[0114] If the current raster map P grid If the number of dynamic point clouds in any grid cell of the current frame is greater than the number of static point clouds, then the grid cell is a dynamic grid cell; otherwise, it is a static grid cell.

[0115] In some embodiments, the method for making the point cloud obtained by the spread consist only of dynamic point clouds includes:

[0116] If, during the spread process, the point cloud obtained from the spread of a certain moving target contains point clouds within a static grid, then the portion of the point cloud belonging to the static grid is removed from the point cloud obtained from the spread of the certain moving target.

[0117] If the point cloud obtained by the spread of a certain moving target point is entirely within a static grid, then the spread of that moving target point shall be stopped.

[0118] In some embodiments, the method for calculating the nearest distance includes:

[0119] Construct a kd-tree for the point cloud of motion monitoring targets;

[0120] For each point in the motion monitoring target point cloud data, find the point that is spatially closest to it from the guide wire point cloud data, ultimately forming a point cloud pair set P. pair ,in This represents the pair of points closest to the i-th point in the motion monitoring target point cloud data, and its calculation formula is:

[0121]

[0122] In the formula, For the i-th point in the motion monitoring target point cloud data, Distance in point cloud data for guide lines The nearest point;

[0123] The point cloud pair set P is calculated using the Euclidean distance formula. pairs The minimum distance between any two points is the closest distance between the guide wire and the motion monitoring target.

[0124] In some embodiments, the method for providing early warning for near-power work includes:

[0125] When the nearest distance is less than a first preset threshold, an early warning is triggered to remind that there may be operational risks.

[0126] When the nearest distance is less than the second set threshold, the warning device is triggered to control the over-limit lock controller to automatically lock the mechanical equipment transmission mechanism; the second set threshold is less than the first set threshold.

[0127] This invention also provides a non-transitory computer-readable storage medium storing a computer program. The computer program includes program instructions that, when executed by a processor, implement the various steps of the method described in this invention, which will not be elaborated further here.

[0128] The computer-readable storage medium can be the data transmission apparatus or the internal storage unit of a computer device provided in any of the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium can also be the external storage device of the computer device, such as the plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the computer device.

[0129] Furthermore, the computer-readable storage medium may include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that is to be output or has already been output.

[0130] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0132] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0133] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0134] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the real-time automatic ranging and early warning method for near-electric work based on laser point cloud and image fusion.

[0135] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A real-time automatic ranging and early warning system for near-electric work based on laser point cloud and image fusion, characterized in that, include: Data fusion module: used to fuse laser point cloud data and image data collected at the near-electricity work site to obtain the mapping relationship between pixels in the image and point cloud data; Point cloud processing module: used to extract motion monitoring target point cloud data corresponding to the motion monitoring target in the image data and guide wire point cloud data corresponding to the static target in the image data from the laser point cloud data according to the mapping relationship; Distance measuring module: used to calculate the closest distance between the traverse point cloud data and the motion monitoring target point cloud data; Early warning module: used to issue early warnings for near-electric work based on the nearest distance; The method for extracting conductor point cloud data in the point cloud processing module includes: The target recognition network is used to obtain the detection boxes of all wires in the image data, and the point cloud data of the wires in each detection box is obtained according to the mapping relationship. For the conductor point cloud data within each detection box, a density-based clustering algorithm is used to divide the conductor point cloud data into different clusters, and the different clusters constitute the clustering result; Construct a sprawling point cloud set and a traverse point cloud set; select a point p from the clustering results. i As the current point of spread of the conductor, it is added to the set of spread points; Conductor spread: Calculate the current conductor spread point p i Direction vector α i ; Get at point p i Direction vector α i tangent plane S i ; Obtain the tangent plane S i The set of point clouds whose planar distance is less than a first set distance Obtain point cloud set The radius r of the envelope circle i If r i relative to r i-1 If a mutation occurs: remove point p from the spread point cloud set. i And no longer spreading to point p i Otherwise, the point will be... Add to the conductor point cloud set; set the current envelope circle radius value r. i Updated to r i−1 The value; select the next traverse spread point as the current traverse spread point P from the remaining unprocessed point cloud in the clustering results. i And add it to the spread point cloud set; Repeat the above traverse propagation steps until no new point clouds are added to the propagation point cloud set; the traverse point cloud set is the traverse point cloud data representing all traverses.

2. The real-time automatic ranging and early warning system for near-electric work based on laser point cloud and image fusion as described in claim 1, characterized in that, The method for extracting motion monitoring target point cloud data in the point cloud processing module includes: Constructing a point cloud map P map =[P D ,P S ] and its corresponding rasterized map P grid , where P D Let P be the set of dynamic point clouds from the most recent n-1 frames. S It is the set of static point clouds from the most recent n-1 frames; The point cloud data acquired by the LiDAR is segmented according to the moving target point cloud segmentation algorithm to obtain the dynamic point cloud of the most recent frame. and static point cloud Save to P respectively D and P S middle; Remove point cloud map P map The point cloud of the in-th frame; Based on dynamic point cloud and static point cloud In the current raster map P grid The number of points in the corresponding grid position is used to update the attribute of the corresponding grid as either a static grid or a dynamic grid. A density-based propagation algorithm is employed, using the dynamic point cloud of the current frame. For the initial point cloud to the point cloud map P map The point cloud within is spread out so that the spread point cloud only contains dynamic point clouds, and the spread point cloud constitutes a spread point cloud set. Repeat the above process until no new point cloud is spread. The final set of spread point clouds is the motion monitoring target point cloud.

3. The real-time automatic ranging and early warning system for near-electric work based on laser point cloud and image fusion as described in claim 1, characterized in that, The method for updating the attribute of the corresponding raster to a static raster or a dynamic raster includes: If the current raster map P grid If the number of dynamic point clouds in any grid cell of the current frame is greater than the number of static point clouds, then the grid cell is a dynamic grid cell; otherwise, it is a static grid cell.

4. The real-time automatic ranging and early warning system for near-electric work based on laser point cloud and image fusion as described in claim 2, characterized in that, The method for ensuring that the point cloud obtained from the spread contains only dynamic point clouds includes: If, during the spread process, the point cloud obtained from the spread of a certain moving target contains point clouds within a static grid, then the portion of the point cloud belonging to the static grid is removed from the point cloud obtained from the spread of the certain moving target. If the point cloud obtained by the spread of a certain moving target point is entirely within a static grid, then the spread of that moving target point shall be stopped.

5. The real-time automatic ranging and early warning system for near-electric work based on laser point cloud and image fusion as described in any one of claims 1, 3, and 4, characterized in that, Methods for calculating the nearest distance include: Construct a kd-tree for motion monitoring target point cloud data; For each point in the motion monitoring target point cloud data, find the point that is spatially closest to it from the guide wire point cloud data, ultimately forming a point cloud pair set P. pair ,in This represents the pair of points closest to the i-th point in the motion monitoring target point cloud data, and its calculation formula includes: ； In the formula, For the i-th point in the motion monitoring target point cloud data, Distance in point cloud data for guide lines The nearest point; The point cloud pair set P is calculated using the Euclidean distance formula. pairs The minimum distance between any two points is the closest distance between the guide wire and the motion monitoring target.

6. The real-time automatic ranging and early warning system for near-electric work based on laser point cloud and image fusion as described in any one of claims 1, 3, and 4, characterized in that, Methods for issuing early warnings for near-powered work include: When the nearest distance is less than a first preset threshold, an early warning is triggered to remind that there may be operational risks. When the nearest distance is less than the second set threshold, the warning device is triggered to control the over-limit lock controller to automatically lock the mechanical equipment transmission mechanism; the second set threshold is less than the first set threshold.

7. A real-time automatic ranging and early warning method for near-electric work based on laser point cloud and image fusion applied to the system of claim 1, characterized in that, include: The mapping relationship between pixels in the image and point cloud data is obtained by fusing laser point cloud data and image data collected at the near-electricity operation site; Based on the mapping relationship, the motion monitoring target point cloud data corresponding to the motion monitoring target in the image data and the guide wire point cloud data corresponding to the static target in the image data are extracted from the laser point cloud data; Calculate the closest distance between the conductor point cloud data and the motion monitoring target point cloud data; Early warning is issued for near-electric work based on the aforementioned closest distance.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the real-time automatic ranging and early warning method for near-electric work based on laser point cloud and image fusion as described in claim 7.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the real-time automatic ranging and early warning method for near-electric work based on laser point cloud and image fusion as described in claim 7.

Images