Near-electricity work real-time automatic distance measurement early warning system and method based on laser point cloud and image fusion

Abstract

The invention discloses a near-electricity work real-time automatic distance measurement early warning system and method based on laser point cloud and image fusion. The method comprises the following steps: fusing laser point cloud data and image data collected in a near-electricity working site to obtain a mapping relation between pixels in an image and the laser point cloud data; motion monitoring target point cloud data corresponding to a motion monitoring target in the image data and lead point cloud data corresponding to a static target in the image data are extracted from the point cloud data according to the mapping relation; calculating the nearest distance between the conductor point cloud data and the motion monitoring target point cloud data; and carrying out early warning on the near-electricity operation according to the nearest distance. According to the invention, through fusion of the laser point cloud and the image data and automatic spreading of the wire and the moving target point cloud, real-time distance measurement is carried out on the wire and the moving monitoring target, the automation degree of point cloud extraction of the wire and the moving monitoring target is improved, and real-time automatic distance measurement early warning of near-electricity operation is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of near-electricity operation monitoring and early warning, and particularly relates to a real-time automatic ranging and early warning system and method for near-electricity operation based on the fusion of laser point cloud and image. Background Art

[0002] With the continuous advancement of power grid construction, the use of large construction machinery such as cranes for near-electricity operation in transmission lines has become increasingly common. Due to the limited space and complex environment of the operation scene, it is easy to cause operation accidents due to the collision of construction machinery with conductors. At present, in order to avoid the collision accident between construction machinery and conductors, manual safety distance monitoring is mainly relied on, which is not only time-consuming and laborious, but also the monitoring is not timely and comprehensive enough to effectively guarantee the safety of near-electricity operation. While using machines to replace manual labor for real-time monitoring and early warning of near-electricity operation can greatly reduce the occurrence of operation accidents.

[0003] In the current technical solutions, the real-time monitoring and early warning of near-electricity operation are mainly achieved by methods based on laser point cloud and image. Among them, the method based on laser point cloud can obtain accurate three-dimensional spatial information of the operation scene. However, due to the lack of information such as color and texture, and the sparsity of the point cloud, it is difficult to identify targets directly and difficult to apply for visualization directly; the method based on image can obtain rich visual information of the scene and achieve fast and accurate detection of targets. However, the image lacks depth information and it is difficult to accurately locate three-dimensional space targets. The method based on the fusion of laser point cloud and image combines the advantages of point cloud and image data, resulting in high accuracy of three-dimensional space distance monitoring and good visualization effect. However, the existing solutions have limited application scenarios, weak interactivity and single monitoring function, and it is difficult to prevent and control the risks of near-electricity operation more effectively, intuitively and accurately. Summary of the Invention

[0004] The present invention provides a real-time automatic ranging and early warning system and method for near-electricity operation based on the fusion of laser point cloud and image. By fusing laser point cloud and image data and using the automatic spread of the point cloud of conductors and moving monitoring targets, real-time ranging and early warning for near-electricity operation are realized, with a wider monitoring range, higher safety, and results more in line with the requirements of actual engineering applications.

[0005] A real-time automatic ranging and early warning system for near-electricity operation based on the fusion of laser point cloud and image to achieve one of the purposes of the present invention includes:

[0006] A data fusion module: used to fuse the laser point cloud data and image data collected at the near-electricity operation site to obtain the mapping relationship between the pixels in the image and the point cloud data;

[0007] Point cloud processing module: used to extract the point cloud data of the motion monitoring target corresponding to the image data and the point cloud data of the wire corresponding to the static target from the laser point cloud data according to the mapping relationship; the motion monitoring target refers to mechanical equipment during near-electrical operation; the static target refers to static power facilities such as transmission lines;

[0008] Range measurement module: used to calculate the shortest distance between the point cloud data of the wire and the point cloud data of the motion monitoring target;

[0009] Early warning module: used to give an early warning for near-electrical operation according to the shortest distance.

[0010] Further, in the point cloud processing module, the method for extracting the point cloud data of the wire includes:

[0011] Through the target recognition network, obtain the detection frames of all wires in the image data, and according to the mapping relationship, obtain the point cloud data of the wire within each detection frame;

[0012] Point cloud clustering: For the point cloud data of the wire within each detection frame, use the density-based clustering algorithm to divide the point cloud data of the wire into different clusters, and the different clusters constitute the clustering result;

[0013] Construct a spreading point cloud set and a wire point cloud set; select a point p i from the clustering result as the current wire spreading point and add it to the spreading point cloud set;

[0014] Wire spreading: Calculate the direction vector α i of the current wire spreading point p i ; obtain the tangent plane S i of the direction vector α i at the point p i ; obtain the point cloud set whose plane distance from the tangent plane S i is less than the first set distance Obtain the point cloud set and get the radius r i of the circumscribing circle of the point cloud set; if r i mutates relative to r i-1 : then remove the point p i from the spreading point cloud set and stop spreading the point p i ; otherwise add the point p i to the wire point cloud set, and the wire point cloud set is used to store the point cloud data of the wire; update the value of the current circumscribing circle radius r i to the value of r i-1 ; select the next wire spreading point from the remaining unprocessed point clouds in the clustering result as the current wire spreading point P iAnd add it to the spreading point cloud set;

[0015] Repeat the above wire spreading steps until no new point cloud is added to the spreading point cloud set; the wire point cloud set is the wire point cloud data representing all wires.

[0016] The technical effects of the above method for extracting wire point cloud data include: obtaining the wire detection box through the target recognition network, then obtaining the wire point cloud based on the mapping relationship, and combining operations such as point cloud clustering and wire spreading, which can accurately extract the wire point cloud data representing all wires from complex point cloud data, effectively excluding other interference factors, and improving the accuracy and integrity of wire point cloud extraction; during the wire spreading process, by calculating the direction vector and tangent plane of the point, and determining whether to continue spreading according to the mutation of the envelope circle radius, the system can adaptively process wire point clouds with different shapes and distributions, ensuring that the extraction result is more in line with the actual wire shape.

[0017] Further, in the point cloud processing module, the method for extracting moving monitoring target point cloud data includes:

[0018] Construct a point cloud map P map =[P D , P S and its corresponding rasterized map P grid , where P D is the dynamic point cloud set of the nearest n - 1 frames, and P S is the static point cloud set of the nearest n - 1 frames;

[0019] Segment the point cloud data collected by the lidar according to the moving target point cloud segmentation algorithm to obtain the dynamic point cloud P i D and the static point cloud P i S , and save them to P D and P S respectively;

[0020] Remove the point cloud of the (i - n)-th frame in the point cloud map P map to ensure that only the dynamic point cloud and static point cloud of the nearest n - 1 frames are in P D and P S ;

[0021] Update the attributes of the corresponding grid to static grid or dynamic grid according to the number of point clouds at the corresponding grid positions of the dynamic point cloud P i D and the static point cloud P i S in the current rasterized map P grid ;

[0022] Adopt a density-based spreading algorithm, and use the dynamic point cloud P of the current frame i D as the initial point cloud to spread the point cloud in the point cloud map P map so that the spread point cloud only contains dynamic point clouds, and the spread point cloud forms a spread point cloud set;

[0023] Repeat the above process until no new point cloud is spread, and the finally obtained spread point cloud set is the motion monitoring target point cloud.

[0024] The technical effects of the above method for extracting motion monitoring target point cloud data include: constructing a point cloud map and performing rasterization processing, combining with a motion target point cloud segmentation algorithm, can effectively distinguish dynamic and static point clouds, achieve accurate extraction of the motion monitoring target point cloud, and provide accurate data for subsequent calculation of the distance between the motion monitoring target and the wire; by saving the point cloud data of the last n - 1 frames and continuously updating and removing the old frame point clouds, the system can make full use of the information of multiple frames of data, improve the tracking and monitoring ability of moving targets, and adapt to the complex motion states of moving targets in the operation scenario.

[0025] Furthermore, the method for updating the attribute of the corresponding grid to a static grid or a dynamic grid includes:

[0026] If the number of dynamic point clouds in the current frame in any grid in the grid map P at the current moment grid is greater than the number of static point clouds, then the grid is a dynamic grid, otherwise it is a static grid.

[0027] The technical effects of updating the grid attributes include: determining the grid attributes as static or dynamic according to the number of dynamic point clouds and static point clouds in the grid in the current frame, providing a clear classification basis for subsequent grid-based point cloud processing and analysis, and helping to more accurately identify and process dynamic point clouds; this method is simple in calculation and easy to implement, while ensuring accuracy, it improves the operation efficiency of the system, can quickly update the grid attributes, and meets the real-time requirements.

[0028] Further technical solutions include: the method for making the spread point cloud only contain dynamic point clouds includes:

[0029] During the spreading process, if the point cloud obtained by spreading a certain moving target spread point contains point clouds in a static grid, then the part of the point cloud belonging to the static grid is removed from the point cloud obtained by spreading the certain moving target spread point;

[0030] If the point cloud obtained by spreading a certain moving target spread point is all point clouds in a static grid, then stop spreading the certain moving target point.

[0031] The technical effects of the above technical solution include: removing the point cloud within the static grid during the spreading process, and stopping the spreading of the point cloud entirely within the static grid, ensuring that the finally obtained spreading point cloud only contains dynamic point clouds, improving the purity of the point cloud for motion monitoring targets, and reducing the interference of static point clouds on the monitoring of moving targets; effectively avoiding misjudging static point clouds as moving target point clouds, making the positioning and tracking of the system for motion monitoring targets more accurate, and thus improving the monitoring accuracy of the entire system for moving targets in near-electrical operations.

[0032] Further, the method for calculating the nearest distance includes:

[0033] Construct a kd-tree for the point cloud of the motion monitoring target; a kd-tree (K-Dimensional Tree) is a data structure for organizing data points in a k-dimensional space, which can perform nearest neighbor search more efficiently; the kd-tree organizes the point cloud data according to certain rules, enabling most impossible points to be quickly excluded when searching for the nearest point, thus greatly improving the search efficiency;

[0034] For each point in the point cloud data of the motion monitoring target, find the point in the wire point cloud data that is closest to it in spatial distance, and finally form a set of point pairs where represents the point pair closest to the i-th point in the point cloud data of the motion monitoring target, and its calculation formula is:

[0035]

[0036] In the formula, is the i-th point in the point cloud data of the motion monitoring target, is the point in the wire point cloud data that is closest to ;

[0037] Use the Euclidean distance formula to calculate the distance between every two points in the set of point pairs P pairs The minimum value of the distances is the nearest distance between the wire and the motion monitoring target.

[0038] The beneficial effects of the above technical solution for calculating the nearest distance include: the kd-tree data structure can quickly find the point closest in spatial distance to each point in the point cloud data of the motion monitoring target from the wire point cloud data, greatly improving the efficiency of nearest point search, and having obvious advantages when dealing with large-scale point cloud data; using the Euclidean distance formula to calculate the distance between every two points in the set of point pairs can accurately obtain the nearest distance between the wire and the motion monitoring target, providing accurate distance data support for subsequent early warnings and ensuring the accuracy and reliability of the early warnings.

[0039] Further, the method for implementing early warning for near-electricity operation includes:

[0040] When the nearest distance is less than the first set threshold, trigger an early warning reminder that there may be operation risks;

[0041] When the nearest distance is less than the second set threshold, trigger the early warning device to control the over-limit locking controller to act, so that the transmission mechanism of the mechanical equipment is automatically locked; the second set threshold is less than the first set threshold.

[0042] The technical effects of the above early warning include: the mechanism of setting different thresholds for hierarchical early warning can take different countermeasures according to the degree of danger, providing multi-level safety guarantees; by triggering the early warning device to control the over-limit locking controller to act, the active protection against the risks of near-electricity operation is realized, and measures can be taken in time before the danger occurs, effectively reducing the incidence of near-electricity operation accidents and protecting the lives of operators and the normal operation of equipment.

[0043] Further, it further includes a joint calibration module for jointly calibrating the lidar and the monocular camera to obtain an external parameter calibration matrix; the external parameter calibration matrix is used to fuse the lidar point cloud and the image data to obtain the mapping relationship between the pixels in the image and the point cloud data.

[0044] Furthermore, the calibration method includes: using an edge computing processor equipped with a lidar and a camera to collect target feature points; the target feature points are usually easily recognizable markers; jointly calibrating the lidar and the camera by extracting the image feature points and point cloud feature points of the target to obtain an external parameter calibration matrix for converting the lidar coordinate system and the camera coordinate system and realizing the fusion of the point cloud and the image.

[0045] A method for real-time automatic ranging and early warning of near-electricity operation based on the fusion of lidar point cloud and image to achieve the second object of the present invention includes:

[0046] Fuse the lidar point cloud data and the image data collected at the near-electricity operation site to obtain the mapping relationship between the pixels in the image and the point cloud data;

[0047] Extract the moving monitoring target point cloud data corresponding to the moving monitoring target in the image data and the wire point cloud data corresponding to the static target in the image data from the lidar point cloud data according to the mapping relationship;

[0048] Calculate the nearest distance between the wire point cloud data and the moving monitoring target point cloud data;

[0049] Implement early warning for near-electricity operation according to the nearest distance.

[0050] A non-transitory computer-readable storage medium for achieving the third object of the present invention, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the near-electrical operation real-time automatic ranging and warning method based on laser point cloud and image fusion are implemented.

[0051] A computer program product for achieving the fourth object of the present invention, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the near-electrical operation real-time automatic ranging and warning method based on laser point cloud and image fusion are implemented.

[0052] The beneficial effects of the present invention include:

[0053] The technical solution of the present invention covers multiple modules such as fusion, extraction, calculation, and warning, forming a complete near-electrical operation monitoring and warning system, realizing the full-process function from data collection to final warning, and providing comprehensive guarantee for the safety of near-electrical operations; the data fusion module fuses laser point cloud and image data, can fully combine the spatial information of the laser point cloud and the visual information such as texture and color of the image, improve the perception accuracy of the operation scene, and lay a foundation for subsequent accurate extraction of target point cloud data and calculation of distance; by automatically spreading the wire and the moving target point cloud, real-time ranging is performed on the wire and the moving monitoring target, improving the automation degree of wire and moving monitoring target point cloud extraction, and realizing real-time automatic ranging and warning for near-electrical operations. Description of the Drawings

[0054] Figure 1 It is a schematic flow chart of the method of the present invention;

[0055] Figure 2 The system block diagram of the present invention. Detailed Embodiments

[0056] The following detailed embodiments are used to explain the technical solution of the present invention so that those skilled in the art can understand the present invention. The protection scope of the present invention is not limited to the following specific implementation structures. Those made by those skilled in the art that include the technical solution of the present invention and are different from the following specific embodiments are also within the protection scope of the present invention.

[0057] An embodiment of the present invention provides a near-electrical operation real-time automatic ranging and warning method based on laser point cloud and image fusion, as Figure 1 shown, including:

[0058] S1: Collect the target feature points using an edge computing processor equipped with a lidar and a camera; the target feature points are usually easily recognizable markers; perform joint calibration of the lidar and the camera by extracting the image feature points and point cloud feature points of the target, such as using the EPnP algorithm to solve for the 2D feature points and 3D feature points, to obtain an external parameter calibration matrix for converting the lidar coordinate system and the camera coordinate system and realizing the fusion of the point cloud and the image;

[0059] S2: Obtain the real-time lidar point cloud and image data of the lidar and the monocular camera, and fuse the lidar point cloud and the image data according to the external parameter calibration matrix to obtain the mapping relationship between the pixels in the image and the point cloud data, so as to realize the real-time automatic association and mapping of the pixels and the point cloud;

[0060] S3: Use the automatic spreading method of wire point cloud to automatically spread and extract the point cloud data P of the wire (such as power line, cable, etc.) body Furthermore, the method for extracting the point cloud data of all wires includes:

[0061] S3.1 Through the target recognition network, obtain the detection frames of all wires in the image, and then according to the mapping relationship, obtain the wire point cloud within each detection frame; the point cloud data corresponding to each detection frame can be stored in different temporary sets for subsequent separate processing;

[0062] S3.2 For the wire point cloud within each detection frame, use a density-based clustering algorithm (such as DBSCAN) for automatic spreading. This algorithm will divide the point cloud into different clusters according to the density relationship between points, and select a point as the first wire spreading point p1 in the clustering result; the selection rule can be random selection or select a point near the center of the detection frame; p1 is the initial spreading point cloud, and as the subsequent steps progress, this spreading point cloud set will continue to expand;

[0063] S3.3 For the current wire spreading point p i , calculate the direction vector α of point p i , and the method for calculating the direction vector can be to calculate the main direction of this point and the points in its neighborhood, such as using the principal component analysis (PCA) method, so as to obtain the tangent plane S of α at point p i ; the equation of the tangent plane can be determined by the point-normal form equation; i ; at α i of p i ;

[0064] S3.4 Obtain the point cloud whose plane distance from the tangent plane S i is less than M S ; the point cloud can be obtained by traversing all the point clouds, calculating the distance from each point to the tangent plane, and screening out the point clouds with a distance less than Ms ​ And use the minimum enclosing circle algorithm (such as the Welzl algorithm) to obtain the point cloud set The radius r of the enclosing circle i , judge whether |r i - r i-1 | > M r holds; for the case of i = 1, since there is no r0, an initial r0 value can be set (such as 0 or a suitable value set according to experience); if r i mutates, that is, |r i - r i-1 | > M r , remove the point p from the spreading point cloud set i , and no longer spread the point p i ; if the condition is not met, add the point p i to the wire point cloud set;

[0065] S3.5 Update the current r i value to r i-1 , to prepare for the next iteration; from the remaining unprocessed point clouds, select the next wire spreading point P i+1 according to a certain rule (such as the point closest to the current spreading point and not marked as not spreading), and add it to the spreading point cloud set;

[0066] S3.6 Repeat S3.2 - S3.5 until no new point clouds are spread;

[0067] S3.7 After processing the wire point clouds corresponding to each detection box, the wire point cloud set is the wire point cloud data P of all wires in the fused image body .

[0068] S4: Use the moving target automatic spreading method to automatically spread the monitored moving target (such as large machinery during operation) to obtain the moving monitoring target point cloud P target , further, it may include:

[0069] S4.1 Construct a point cloud map P map = [P D , P S and its corresponding rasterized map P grid , where P D is the dynamic point cloud set of the nearest n - 1 frames, P S is the static point cloud set of the nearest n - 1 frames, and the attribute of each grid in the rasterized map P grid is either static or dynamic;

[0070] S4.2 Segment the point cloud data of each frame collected by the lidar according to the moving target point cloud segmentation algorithm to obtain the dynamic point cloud and static point cloud of each frame; the dynamic point cloud here refers to the point cloud data generated by the moving target, and the static point cloud is the point cloud data generated by the stationary object. For example, in the scenario of working near electricity, the point cloud generated by the large machinery operated by the operator belongs to the dynamic point cloud, while the point cloud generated by the surrounding fixed power facilities, buildings, etc. belongs to the static point cloud. If the point cloud data of the current i-th frame is segmented to obtain the dynamic point cloud P i D and the static point cloud P i S , and save them to P D and P S respectively. At the same time, remove the point cloud of the (i - n)-th frame in P map to ensure that there are only the dynamic point cloud and static point cloud of the nearest n - 1 frames in P D and P S . That is:

[0071]

[0072] n is a fixed parameter value, that is, the number of frames of the retained points; for example, when n = 5 is set, it means that there are the dynamic point cloud and static point cloud of the nearest 4 frames saved in P D and P S . i represents the current frame number, which increases with time and i > n;

[0073] S4.3 Update the attributes of the grid according to the positions of the dynamic point cloud P i D and the static point cloud P i S in the rasterized map P grid . If the number of dynamic point clouds of the current i-th frame in any grid of the raster map P grid at the current moment is greater than the number of static point clouds, then the grid is a dynamic grid, otherwise it is a static grid; in this way, as the point cloud data of each frame is updated, the attributes of each grid in the rasterized map P grid will be dynamically updated accordingly, so as to accurately reflect the distribution of dynamic and static objects in the current scene;

[0074] S4.4 Adopt the density-based spreading algorithm, and use the dynamic point cloud P i D of the current frame as the initial point cloud to spread the point cloud in P map . The density-based spreading algorithm (such as the extended application of the DBSCAN algorithm) will expand from the initial point cloud to the surrounding according to the density relationship between points, and gather the point clouds with connected densities together. For example, for the dynamic point cloud P iD For each point in it, the algorithm searches for points within its neighborhood whose density meets certain conditions and adds these points to the growing point cloud set; during the growth process, if a growing point p of a moving object j in the growing point cloud contains points within a static grid, then from the growing point p of the moving object j remove from the growing point cloud the part of the point cloud that belongs to the static grid, so as to ensure that the growing point cloud contains as much as possible only the dynamic point cloud related to the moving object, so as to more accurately perform subsequent operations such as monitoring and analyzing the moving object; if point p j the growing point cloud is all points within the static grid, indicating that the growing point has spread to the stationary area, then stop the spread of point p j to avoid wrongly including the point cloud of the stationary object in the moving object point cloud;

[0075] S4.5 Repeat the above spreading and processing process until no new points can be spread, and the finally obtained point cloud set is the moving object monitoring target point cloud P target .

[0076] S5: Calculate the closest distance between the wire and the moving object monitoring target. Further, it may include:

[0077] S5.1: Construct the kd-tree (K-Dimensional Tree, a data structure for organizing data points in k-dimensional space, which can perform nearest neighbor search more efficiently. The kd-tree organizes the point cloud data according to certain rules, enabling most impossible points to be quickly excluded during the search for the nearest point, thus greatly improving the search efficiency) of the moving object monitoring target point cloud P target . For each point in P target , find the point in the wire point cloud P body that is closest to it in terms of spatial distance, and finally form a set of point pairs where represents the point pair closest to the i-th point in the wire point cloud P body , and its calculation formula is:

[0078]

[0079] In the formula, is the i-th point in the moving object monitoring target point cloud P target , is the point in the wire point cloud P body closest to ;

[0080] S5.2: Use the Euclidean distance formula to calculate the set of point pairs P pairsThe distance between every two points is calculated, and the minimum distance is found. The minimum distance is denoted as d. min , which is the closest distance between the wire and the moving monitoring target.

[0081] S6: Real-time judgment of the closest distance d min whether it reaches the set warning threshold, and timely trigger the anti-collision sound and light alarm. Further, it may include:

[0082] S6.1: Real-time judgment of the closest distance d between the wire and the moving monitoring target min whether it is less than the first set threshold M1; M1 is set according to actual needs and is not limited in the present invention;

[0083] S6.2: If the closest distance d between the wire and the moving monitoring target min is less than the first set threshold M1, then trigger the sound and light warning, that is

[0084] d min < M1

[0085] Otherwise, continue the real-time monitoring of the safe distance for live working near electricity.

[0086] S7: Real-time judgment of whether the closest distance between the wire and the moving monitoring target reaches the overlimit locking threshold, and timely control the overlimit locking controller to act to automatically lock the transmission mechanism of the mechanical equipment. Further, it may include:

[0087] S7.1: According to the calculation result of S5, real-time judgment of whether the closest distance d between the wire and the moving monitoring target min is less than the second set threshold M2 (M2 < M1); M2 is set according to actual needs and is not limited in the present invention;

[0088] S7.2: If the closest distance d between the wire and the moving monitoring target min is less than the second set threshold M2, then trigger the warning device to control the overlimit locking controller to act, so that the transmission mechanism of the mechanical equipment is automatically locked, that is

[0089] d min < M2

[0090] Otherwise, continue the real-time monitoring of the safe distance for live working near electricity.

[0091] In this embodiment, it is judged whether the real-time distance between the wire and the moving monitoring target reaches the overlimit locking threshold, and the overlimit locking controller is timely controlled to act to automatically lock the transmission mechanism of the mechanical equipment.

[0092] It should be understood that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0093] An embodiment of the present invention also provides a real-time automatic ranging and warning system for near-electric operation based on the fusion of laser point cloud and image, as Figure 2 shown, including:

[0094] A data fusion module: used to fuse the laser point cloud and image data collected at the near-electric operation site to obtain the mapping relationship between the pixels in the image and the point cloud data;

[0095] A point cloud processing module: used to extract the moving monitoring target point cloud data corresponding to the moving monitoring target in the image data and the wire point cloud data corresponding to the static target in the image data from the point cloud data according to the mapping relationship;

[0096] A ranging module: used to calculate the closest distance between the wire point cloud data and the moving monitoring target point cloud data;

[0097] A warning module: used to implement a warning for near-electric operation according to the closest distance.

[0098] In some embodiments, it further includes a joint calibration module for jointly calibrating the lidar and the monocular camera to obtain an external parameter calibration matrix; the external parameter calibration matrix is used to fuse the laser point cloud and image data to obtain the mapping relationship between the pixels in the image and the point cloud data.

[0099] In some embodiments, in the point cloud processing module, the method for extracting wire point cloud data includes:

[0100] Through the target recognition network, obtain the detection frames of all wires in the image, and according to the mapping relationship, obtain the wire point cloud within each detection frame;

[0101] For the wire point cloud data within each detection frame, use the density-based clustering algorithm to divide the wire point cloud data into different clusters, and the different clusters constitute the clustering result;

[0102] Construct a spreading point cloud set and a wire point cloud set; select a point p i from the clustering result as the current wire spreading point and add it to the spreading point cloud set;

[0103] Wire spreading: Calculate the direction vector α i of the current wire spreading point p i ; obtain the tangent plane S i of the direction vector α i at the point p i; Obtain the point cloud set whose plane distance from the tangent plane S i is less than the first set distance Obtain the point cloud set of the circumradius r of the envelope circle i ; If r i mutates relative to r i-1 : Then remove the point p from the spreading point cloud set i , and no longer spread the point p i ; Otherwise, add the point p i to the wire point cloud set, where the wire point cloud set is used to store the point cloud data of the wire; update the current r i value to r i-1 ; Select the next wire spreading point from the remaining unprocessed point clouds in the clustering result as the current wire spreading point P i and add it to the spreading point cloud set;

[0104] Repeat the above wire spreading steps until no new point cloud is added to the spreading point cloud set; the wire point cloud set is the wire point cloud data representing all wires.

[0105] In some embodiments, in the point cloud processing module, the method for extracting the point cloud data of the motion monitoring target includes:

[0106] Construct a point cloud map P map =[P D , P S and its corresponding rasterized map P grid , where P D is the dynamic point cloud set of the nearest n - 1 frames, and P S is the static point cloud set of the nearest n - 1 frames;

[0107] Segment the point cloud data collected by the lidar according to the motion target point cloud segmentation algorithm to obtain the dynamic point cloud P i D and the static point cloud P i S , and save them to P D and P S respectively;

[0108] Remove the point cloud of the (i - n)-th frame in the point cloud map P map to ensure that there are only the dynamic point clouds and static point clouds of the nearest n - 1 frames in P D and P S ;

[0109] According to the dynamic point cloud P i D and the static point cloud P i SIn the current rasterized map P grid Update the attribute of the corresponding grid to a static grid or a dynamic grid according to the number of point clouds at the corresponding grid position in it;

[0110] Adopt a density-based spreading algorithm, and use the dynamic point cloud P of the current frame i D As the initial point cloud, spread the point cloud in the point cloud map P map So that the spread point cloud only contains dynamic point clouds, and the spread point cloud forms a set of spread point clouds;

[0111] Repeat the above process until no new point cloud is spread, and the finally obtained set of spread point clouds is the motion monitoring target point cloud.

[0112] In some embodiments, the method for updating the attribute of the corresponding grid to a static grid or a dynamic grid includes:

[0113] If the number of dynamic point clouds in the current frame in any grid in the grid map P at the current moment grid Is greater than the number of static point clouds, then the grid is a dynamic grid, otherwise it is a static grid.

[0114] In some embodiments, the method for making the spread point cloud only contain dynamic point clouds includes:

[0115] During the spreading process, if the point cloud obtained by spreading a certain moving target spread point contains the point cloud in the static grid, then the part of the point cloud belonging to the static grid is removed from the point cloud obtained by spreading the certain moving target spread point;

[0116] If the point cloud obtained by spreading a certain moving target spread point is all the point cloud in the static grid, then stop spreading the certain moving target point.

[0117] In some embodiments, the method for calculating the nearest distance includes:

[0118] Construct a kd-tree of the motion monitoring target point cloud;

[0119] For each point in the motion monitoring target point cloud data, find the point in the wire point cloud data that is closest to it in spatial distance, and finally form a set of point cloud pairs Wherein Represents the point pair closest to the i-th point in the motion monitoring target point cloud data, and its calculation formula is:

[0120]

[0121] In the formula, Is the i-th point in the motion monitoring target point cloud data, The point closest to distance p in the wire point cloud data i target ;

[0122] The Euclidean distance formula is used to calculate the distance between every two points in the set P of point clouds pairs The minimum value of the distances is the closest distance between the wire and the moving monitoring target.

[0123] In some embodiments, the method for implementing early warning for near-electrical operation includes:

[0124] When the closest distance is less than the first set threshold, an early warning reminder is triggered to indicate that there may be operation risks;

[0125] When the closest distance is less than the second set threshold, the early warning device is triggered to control the over-limit locking controller to act, so that the transmission mechanism of the mechanical equipment is automatically locked; the second set threshold is less than the first set threshold.

[0126] An embodiment of the present invention also provides a non-transitory computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program includes program instructions. When the program instructions are executed by a processor, each step of the method described in the present invention is implemented, which will not be elaborated here.

[0127] The computer-readable storage medium may be the internal storage unit of the data transmission device or computer device provided in any of the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device.

[0128] Furthermore, the computer-readable storage medium may also include both the internal storage unit and the external storage device of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store the data to be output or already output.

[0129] Those skilled in the art should understand that the embodiments of the present invention may be provided as a method, a system, or a computer program product. Therefore, the present invention may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0131] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0133] Embodiments of the present invention also provide a computer program product, including a computer program / instructions, which when executed by a processor, implement the steps of the method for real-time automatic ranging warning for near-electrical operation based on laser point cloud and image fusion.

[0134] Content not described in detail in this specification belongs to the prior art well-known to those skilled in the art.

Claims

1. A real-time automatic distance measurement and warning system for near-electrical operations based on laser point cloud and image fusion, characterized in that: include: Data fusion module: used to fuse the laser point cloud data and image data collected at the near-electric operation site to obtain the mapping relationship between the pixels in the image and the point cloud data; Point cloud processing module: used for extracting the motion monitoring target point cloud data corresponding to the motion monitoring target in the image data and the wire point cloud data corresponding to the static target in the image data from the laser point cloud data according to the mapping relationship; Distance measurement module: used to calculate the shortest distance between the wire point cloud data and the motion monitoring target point cloud data; Early warning module: used for issuing early warning for near-electrical operations according to the shortest distance.

2. The real-time automatic distance measurement and warning system for near-electrical operation based on laser point cloud and image fusion according to claim 1 is characterized in that: In the point cloud processing module, the method for extracting wire point cloud data includes: Obtain detection frames of all wires in the image data through a target recognition network, and obtain wire point cloud data in each detection frame according to the mapping relationship; For the wire point cloud data in each detection frame, a density-based clustering algorithm is used to divide the wire point cloud data into different clusters, and the different clusters constitute the clustering results; Constructing a spreading point cloud set and a wire point cloud set; selecting a point pi from the clustering result as the current wire spreading point, and adding it to the spreading point cloud set; Wire spread: Calculate the direction vector α of the current wire spread point pi i ; Get the direction vector α at point pi i The tangent plane S i ; Get the tangent plane S i The point cloud set P whose plane distance is less than the first set distance Si ; Get the point cloud set P Si The radius of the enveloping circle r i If r i Relative r i -1 mutation occurs: point pi is removed from the spread point cloud set and point pi is no longer spread; otherwise, point pi is added to the wire point cloud set; the current envelope radius value r is i Update to r i -1; select the next wire spreading point from the remaining unprocessed point cloud in the clustering result as the current wire spreading point P i and adding it to the spreading point cloud set; Repeat the above wire propagation steps until no new point cloud is added to the propagation point cloud set; the wire point cloud set is the wire point cloud data representing all wires.

3. The real-time automatic distance measurement and warning system for near-electrical operation based on laser point cloud and image fusion according to claim 1 or 2, characterized in that: In the point cloud processing module, the method for extracting motion monitoring target point cloud data includes: Constructing point cloud map P map =[P D ,P S ] and its corresponding rasterized map P grid , where P D is the dynamic point cloud set of the latest n-1 frames, P S is the static point cloud set of the most recent n-1 frames; According to the moving target point cloud segmentation algorithm, the point cloud data collected by the lidar is segmented to obtain the dynamic point cloud P of the most recent frame. i D and the static point cloud P i S , respectively saved to P D and P S middle; Remove point cloud map P map Point cloud of frame in; According to the dynamic point cloud P i D and the static point cloud P i S In the current rasterized map P grid The number of point clouds at the corresponding grid position in the grid is used to update the attribute of the corresponding grid to be a static grid or a dynamic grid; Using the density-based spreading algorithm, the dynamic point cloud P of the current frame i D is the initial point cloud to point cloud map P map Propagate the point cloud within the point cloud so that the point cloud obtained by the propagation only contains the dynamic point cloud, and the point cloud obtained by the propagation constitutes a propagated point cloud set; The above process is repeated until no new point cloud is propagated, and the final propagation point cloud set is the motion monitoring target point cloud.

4. The real-time automatic distance measurement and warning system for near-electrical operation based on laser point cloud and image fusion as claimed in claim 3 is characterized in that: The method for updating the attribute of the corresponding grid to be a static grid or a dynamic grid includes: If the current grid map P grid If the number of dynamic point clouds in the current frame in any grid is greater than the number of static point clouds, the grid is a dynamic grid, otherwise it is a static grid.

5. The real-time automatic distance measurement and warning system for near-electrical operation based on laser point cloud and image fusion as claimed in claim 3 is characterized in that: The method for making the point cloud obtained by spreading contain only dynamic point clouds includes: During the spreading process, if the point cloud obtained by spreading a certain moving target spreading point contains a point cloud in the static grid, then the point cloud in the static grid is removed from the point cloud obtained by spreading a certain moving target spreading point; If the point clouds obtained by propagating a certain moving target propagation point are all point clouds within the static grid, the propagation of the certain moving target point is stopped.

6. The real-time automatic distance measurement and warning system for near-electrical operation based on laser point cloud and image fusion according to any one of claims 1, 2, 4, and 5, characterized in that: The method for calculating the closest distance includes: Construct a kd-tree of motion monitoring target point cloud data; For each point in the motion monitoring target point cloud data, find the point closest to it in spatial distance from the wire point cloud data, and finally form a point cloud pair set in It represents the point pair closest to the i-th point in the motion monitoring target point cloud data, and its calculation formula includes: In the formula, is the i-th point in the motion monitoring target point cloud data, The distance in the wire point cloud data nearest point; The Euclidean distance formula is used to calculate the point cloud pair set P pairs The distance between every two points in the graph is the minimum distance between the wire and the motion monitoring target.

7. The real-time automatic distance measurement and warning system for near-electrical operation based on laser point cloud and image fusion according to any one of claims 1, 2, 4, and 5, characterized in that: Methods for early warning of near-electrical operations include: When the closest distance is less than a first set threshold, an early warning is triggered to remind that there may be an operation risk; When the closest distance is less than a second set threshold, the warning device is triggered to control the over-limit locking controller to operate, so that the transmission mechanism of the mechanical equipment is automatically locked; the second set threshold is less than the first set threshold.

8. A real-time automatic distance measurement and warning method for near-electrical operations based on laser point cloud and image fusion, characterized in that: include: The laser point cloud data collected at the near-electrical operation site is fused with the image data to obtain the mapping relationship between the pixels in the image and the point cloud data; Extracting the moving monitoring target point cloud data corresponding to the moving monitoring target in the image data and the wire point cloud data corresponding to the static target in the image data from the laser point cloud data according to the mapping relationship; Calculating the shortest distance between the wire point cloud data and the motion monitoring target point cloud data; An early warning is implemented for near-electrical operations based on the closest distance.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of the real-time automatic ranging and warning method for near-electrical operations based on laser point cloud and image fusion as described in claim 8 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by the processor, the steps of the real-time automatic ranging and warning method for near-electrical operations based on laser point cloud and image fusion as described in claim 8 are implemented.

Images