Transformer substation external damage prevention monitoring method based on multi-modal fusion

Abstract

The invention relates to the technical field of power equipment monitoring, and provides a substation external damage prevention monitoring method based on multi-modal fusion, which fully utilizes the space (depth) information of laser point cloud and the color texture information of images to carry out construction machinery detection, and adopts the D-S evidence theory to carry out confidence fusion on the two detection results, so as to obtain the substation external damage prevention monitoring method. The problem of low confidence caused by insufficient light, too small target and the like when the construction machinery is detected by independently using image data is solved, and the detection precision of the construction machinery is effectively improved; by matching the laser point cloud three-dimensional coordinate information with the two-dimensional construction machinery detection result, high-precision construction machinery positioning is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of power equipment monitoring, and in particular to a substation anti-external force damage monitoring method based on multi-modal fusion. Background Art

[0002] At present, the construction of substation infrastructure is accelerating, and the number of substation renovation, expansion and construction projects is increasing. It is difficult for mechanical drivers to accurately measure the distance between substation live equipment and construction machinery with the naked eye, and construction workers lack protection awareness. As a result, external force damage accidents such as equipment discharge, tripping short circuit and collision damage occur from time to time, which not only affects the normal order of electricity use and causes huge economic losses, but also may cause casualties. Therefore, it is very important to monitor the substation against external force damage.

[0003] With the rapid development of computer vision technology and its application in substation video image monitoring devices, automatic detection of construction machinery such as tower cranes, cranes, cement tankers, bulldozers, excavators, etc. in substation monitoring scenes has been realized. Therefore, based on intelligent recognition technology, hidden dangers of external force damage in substations can be automatically identified and timely alarmed, greatly improving the efficiency and effect of line monitoring.

[0004] However, the existing monitoring methods based on image data are limited by data characteristics. Objects in the image are naturally larger when they are near and smaller when they are far away, lens distortion, etc. It is difficult to directly determine the actual distance between various construction hidden danger targets and live equipment from the image, resulting in frequent alarms of hidden danger images. Secondly, the detection accuracy of the construction machinery detection method based on image data is greatly affected by the image illumination intensity and target size, which reduces the overall stability and reliability of the monitoring method.

[0005] In order to solve the above-mentioned deficiencies in the prior art, the present invention combines the advantages of image data and laser point cloud data and proposes a substation anti-external force damage monitoring method based on multi-modal fusion. Summary of the invention

[0006] The purpose of the present invention is to solve the deficiencies of the prior art and provide a substation anti-external force damage monitoring method based on multi-modal fusion to improve the accuracy of substation anti-external force damage monitoring and reduce invalid alarms.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A substation anti-external force damage monitoring method based on multi-modal fusion includes the following steps:

[0009] Step 1: Jointly calibrate and time-synchronize the laser radar and camera at the substation monitoring site, collect the 3D point cloud data and RGB image data at the site, and record the 3D information of the live equipment within the monitoring range in the radar coordinate system;

[0010] Step 2: According to the projection matrix from the radar coordinate system to the image coordinate system, the correspondence between the pixel points in the image and the laser point cloud is obtained to generate a sparse depth map, and the sparse depth map is converted into a dense depth map through a depth completion method based on a bilateral filter;

[0011] Step 3: Design a multimodal fusion construction machinery detection framework, use the image and dense depth map as the output of the YOLOv5 model respectively, fuse the model output results based on DS evidence theory, and output the construction machinery detection box results;

[0012] Step 4: Cluster the original laser point cloud based on the DBSCAN algorithm, and perform bounding box fitting on each group of point clouds;

[0013] Step 5: Based on the construction machinery detection frame obtained in step 3 and the clustered point cloud boundary frame obtained in step 4, the construction machinery is positioned, and anti-external force damage monitoring is achieved by calculating the actual distance to the live equipment.

[0014] Preferably, in step 1, a checkerboard rectangular calibration plate is used, and the position of the calibration plate is repeatedly moved within the common field of view of the camera and the lidar to collect multiple sets of images and point cloud data, and the Autoware calibration tool is used for joint calibration to obtain the rotation matrix R from the radar coordinate system to the camera coordinate system. l2c , the translation vector T from the radar coordinate system to the camera coordinate system l2c And the camera intrinsic parameter K c2i ; Finally, the projection matrix T from the radar coordinate system to the image coordinate system is calculated by the following formula: l2i :

[0015]

[0016] In step 1, the output period is calculated according to the sampling frequency of the laser radar and the camera, and the time synchronization of the data is performed. The output period T of the two types of data after synchronization is p It can be calculated by the following formula:

[0017]

[0018] Among them, F lidar and F camera Respectively represent the number of frames collected per second by the laser radar and camera; T p The unit is seconds, that is, every T pThe data collected every second is the data at the same moment, realizing the time synchronization of the two sensors.

[0019] Preferably, in step 2, according to the projection matrix T from the radar coordinate system to the image coordinate system obtained in step 1 l2i , establish the mapping relationship between the three-dimensional point cloud coordinates and the image plane coordinates, the mapping formula is as follows:

[0020]

[0021] where the vector [x l y l z l 1] T is the spatial coordinate of the three-dimensional point cloud in the radar coordinate system, x l Indicates the coordinate value of the point cloud on the X axis, y l Indicates the coordinate value of the point cloud on the Y axis, z l Represents the coordinate value of the point cloud on the Z axis; vector [uv1] T It is the plane coordinate in the image coordinate system after the three-dimensional point cloud is mapped, u represents the horizontal coordinate value, and v represents the vertical coordinate value;

[0022] After obtaining the one-to-one correspondence between each point cloud coordinate and the plane coordinate, a sparse depth map consistent with the length and width of the image captured by the camera is generated; specifically: first, according to the length H and width W of the image, the coordinate points whose u value is not in the interval [0, W] or the v value is not in the interval [0, H] are removed; secondly, a two-dimensional matrix with all zeros in length and width consistent with the image is created, and all the remaining plane coordinate points are traversed, and the values ​​of the coordinates (u, v) in the two-dimensional matrix are assigned to z l , as the depth of the position; after the final traversal is completed, a sparse depth map with a size of H×W×1 is obtained.

[0023] Preferably, in step 2, a bilateral filter is applied to complete the depth of the sparse depth map to generate a dense panoramic image, specifically: traverse each pixel point p in the image, and the depth value I at the pixel point p after filtering is p ′ The following formula can be used for calculation:

[0024]

[0025] Among them, S represents the neighborhood window centered on p; p and q represent the physical coordinates of the pixel points; I p ,I q W represents the pixel (depth) value corresponding to pixel points p and q on the sparse depth map; p is the normalization coefficient; G c and G sRepresent the distance weight and similarity weight respectively, which are calculated by Gaussian function. The calculation formula is as follows:

[0026]

[0027] Where exp(·) represents an exponential function with the natural constant e as the base; (u p ,v p ) is the coordinate of pixel point p; (u q ,v q ) is the coordinate of pixel point q; σ c and σ s are the two standard deviations corresponding to the Gaussian function;

[0028] After traversing the entire sparse panorama through a bilateral filter, depth completion is achieved and it is replicated twice at the channel level to obtain a dense panorama of size H×W×3.

[0029] Preferably, in step 3, the DS evidence theory is applied to the confidence fusion of the detection box results of the two data, and the specific steps are:

[0030] a) Image-based detection box B c and the detection box B based on the dense depth map d Perform pairwise intersection over union (IoU) calculations and select B with IoU greater than 0.75. c and B d They are grouped together for the next step of confidence fusion, and the other detection boxes are directly output;

[0031] b) Based on DS evidence theory, define the recognition framework θ = {correct detection, wrong detection}, B c and B d are two pieces of evidence under θ, representing the detection box based on the image and the detection box based on the dense depth map respectively. The basic probability comes from the confidence of the detection box before fusion; the basic probability is assigned as follows:

[0032] For evidence B c , the probability of correct detection is assigned to The probability of detecting error is assigned as

[0033] For evidence B d , the probability of correct detection is assigned to The probability of detecting error is assigned as

[0034] in, Represents the detection box B c The confidence score of Represents the detection box B d Confidence score of

[0035] c) Based on the probability synthesis rule of DS evidence theory, the detection box B c and B d The confidence after fusion is calculated by the following formula:

[0036]

[0037] in, represents the confidence after the fusion of the two detection frames, and K represents the conflict factor, which is calculated by the following formula:

[0038]

[0039] d) For the detection box B c and B d The position information of the fused detection frame is fused and the position information of the fused detection frame is output. The position information is represented by the upper left corner coordinates and the lower right corner coordinates The composition is obtained by the following formula:

[0040]

[0041] in and They are detection box B c The upper left corner coordinates and lower right corner coordinates of; and They are detection box B d The upper left corner coordinates and lower right corner coordinates of;

[0042] e) Repeat steps b-d to achieve all detection boxes B with IoU greater than 0.75 c and B d Fusion, output the location information and confidence of the fused detection box.

[0043] Preferably, the step 4 clusters the original laser point cloud obtained in step 1 and generates a bounding box, and the specific steps are as follows:

[0044] a) In order to reduce the subsequent clustering algorithm overhead, the SACSegmentation function in the PCL toolkit is used to perform plane segmentation on the original laser point cloud to remove the ground point cloud;

[0045] b) Applying the DBSCAN algorithm to the non-ground point cloud data after the ground points are removed, dividing it into different clusters, and for each cluster, extracting the corresponding point cloud data to form a point cloud group for each cluster;

[0046] c) For each point cloud group, extract the minimum value (x min ,ymin ,z min ) and the maximum value (x max ,y max ,z max ), these 6 values ​​are used to fit the minimum bounding box of each group of cluster point clouds.

[0047] Preferably, in step 5, when there is a construction machinery target in the multimodal fusion detection result, the point cloud boundary box is matched with the construction machinery detection box to realize the positioning of the construction machinery; finally, in the same radar coordinate system, the Euclidean distance between the three-dimensional coordinates of the construction machinery and the three-dimensional coordinates of the recorded live equipment is calculated. If the measured distance is less than the preset safety threshold, an alarm is given through voice broadcast in the monitoring scenario, and the screenshot is pushed to the back-end monitoring console to remind relevant staff.

[0048] Preferably, the construction machinery positioning comprises the following steps:

[0049] a) According to the projection matrix T obtained in step 1 l2i , project each cluster point cloud bounding box to obtain its two-dimensional bounding box on the image. The projection formula is as follows:

[0050]

[0051] in, Represents the coordinates of the upper left corner of the two-dimensional bounding box in the image after projection; Represents the coordinates of the upper right corner of the two-dimensional bounding box in the image after projection;

[0052] b) Traverse the construction machinery target detection box results obtained in step 3, and for each detection box, perform IoU calculations with all two-dimensional bounding boxes to obtain an IoU value list;

[0053] c) For each IoU value list corresponding to the detection box, the clustered point cloud bounding box corresponding to the two-dimensional bounding box with the largest IoU value is regarded as the actual bounding box of the construction machinery to realize the positioning of the construction machinery.

[0054] The present invention discloses a substation protection against external force damage monitoring method based on multi-modal fusion, which has the following beneficial effects.

[0055] The present invention makes full use of the spatial (depth) information of the laser point cloud and the color and texture information of the image to detect construction machinery, and adopts the DS evidence theory to fuse the confidence of the two detection results, thereby overcoming the problem of low confidence caused by insufficient light, too small target, etc. when using image data alone for construction machinery detection, and effectively improves the detection accuracy of construction machinery; by matching the three-dimensional coordinate information of the laser point cloud and the two-dimensional construction machinery detection results, high-precision construction machinery positioning is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a construction machinery detection framework diagram of multi-modal fusion of the present invention;

[0057] Figure 2 This is a flow chart of the multi-modal fusion substation anti-external force damage monitoring method of the present invention; DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0059] In the description of the present invention, it is necessary to understand that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0060] Example 1

[0061] like Figure 2 As shown, a substation anti-external force damage monitoring method based on multi-modal fusion includes the following steps:

[0062] A substation anti-external force damage monitoring method based on multi-modal fusion includes the following steps:

[0063] Step 1: Jointly calibrate and time-synchronize the laser radar and camera at the substation monitoring site, collect the 3D point cloud data and RGB image data at the site, and record the 3D information of the live equipment within the monitoring range in the radar coordinate system;

[0064] As a preferred embodiment, in step 1, it is worth noting that the camera and the laser radar are arranged at the substation monitoring site. Since the laser radar and the camera are installed at different positions, the projections of the same target in the radar coordinate system and the camera coordinate system will not overlap. It is necessary to calibrate the sensor's internal and external parameters to obtain a projection matrix that aligns the laser radar information with the visual information in space. Therefore, a checkerboard rectangular calibration plate is used. By repeatedly moving the calibration plate in the common field of view of the camera and the laser radar, multiple sets of images and point cloud data are collected, and the Autoware calibration tool is used for joint calibration to obtain the rotation matrix R from the radar coordinate system to the camera coordinate system. l2c , the translation vector T from the radar coordinate system to the camera coordinate system l2c And the camera intrinsic parameter K c2i; Finally, the projection matrix T from the radar coordinate system to the image coordinate system is calculated by the following formula: l2i :

[0065]

[0066] It is worth noting that the sampling frequencies of the laser radar and the camera are different, and there is a time difference between the collected point cloud data and the image data. Therefore, it is necessary to synchronize the frequencies of the collected data information of the two sensors. Therefore, in step 1, the output period is calculated according to the sampling frequencies of the laser radar and the camera, and the time synchronization of the data is performed. After synchronization, the output period of the two data is T p It can be calculated by the following formula:

[0067]

[0068] Among them, F lidar and F camera Respectively represent the number of frames collected per second by the laser radar and camera; T p The unit is seconds, that is, every T p The data collected every second is the data at the same moment, realizing the time synchronization of the two sensors.

[0069] Step 2: According to the projection matrix from the radar coordinate system to the image coordinate system, the correspondence between the pixel points in the image and the laser point cloud is obtained to generate a sparse depth map, and the sparse depth map is converted into a dense depth map through a depth completion method based on a bilateral filter;

[0070] Preferably, in step 2, according to the projection matrix T from the radar coordinate system to the image coordinate system obtained in step 1 l2i , establish the mapping relationship between the three-dimensional point cloud coordinates and the image plane coordinates, the mapping formula is as follows:

[0071]

[0072] where the vector [x l y l z l 1] T is the spatial coordinate of the three-dimensional point cloud in the radar coordinate system, x l Indicates the coordinate value of the point cloud on the X axis, y l Indicates the coordinate value of the point cloud on the Y axis, z l Represents the coordinate value of the point cloud on the Z axis; vector [uv1] T It is the plane coordinate in the image coordinate system after the three-dimensional point cloud is mapped, u represents the horizontal coordinate value, and v represents the vertical coordinate value;

[0073] After obtaining the one-to-one correspondence between each point cloud coordinate and the plane coordinate, a sparse depth map consistent with the length and width of the image captured by the camera is generated; specifically: first, according to the length H and width W of the image, the coordinate points whose u value is not in the interval [0, W] or the v value is not in the interval [0, H] are removed; secondly, a two-dimensional matrix with all zeros in length and width consistent with the image is created, and all the remaining plane coordinate points are traversed, and the values ​​of the coordinates (u, v) in the two-dimensional matrix are assigned to z l , as the depth of the position; after the final traversal is completed, a sparse depth map with a size of H×W×1 is obtained.

[0074] As a preferred embodiment, in step 2, it is worth noting that compared with densely arranged image pixels, the lidar point cloud is sparse, so the resolution of the sparse depth map generated by the projection matrix is ​​much lower than the image resolution. Bilateral filtering is a nonlinear filtering method that relies on a smoothing operator to transfer local information to the entire image, thereby filling in unknown areas. Compared with other filters such as median filtering, mean filtering, and Gaussian filtering, the weight of bilateral filtering not only considers the Euclidean distance of pixels, but also considers the similarity between pixel values, so that the depth image retains edge information while denoising. Therefore, a bilateral filter is applied to the sparse depth map for depth completion to generate a dense panoramic image, specifically: traverse each pixel point p in the image, and the depth value I at the pixel point p after filtering p ′ The following formula can be used for calculation:

[0075]

[0076] Among them, S represents the neighborhood window centered on p; p and q represent the physical coordinates of the pixel points; I p ,I q W represents the pixel (depth) value corresponding to pixel points p and q on the sparse depth map; p is the normalization coefficient; G c and G s Represent the distance weight and similarity weight respectively, which are calculated by Gaussian function. The calculation formula is as follows:

[0077]

[0078] Where exp(·) represents an exponential function with the natural constant e as the base; (u p ,v p ) is the coordinate of pixel point p; (u q ,v q ) is the coordinate of pixel point q; σ c and σ s are the two standard deviations corresponding to the Gaussian function;

[0079] After traversing the entire sparse panorama through a bilateral filter, depth completion is achieved and it is replicated twice at the channel level to obtain a dense panorama of size H×W×3.

[0080] like Figure 1 As shown in the figure, step 3, design a multimodal fusion construction machinery detection framework, take the image and dense depth map as the output of the YOLOv5 model respectively, fuse the model output results based on the DS evidence theory, and output the construction machinery detection frame result; it is worth noting that when using image data alone for construction machinery detection, there are problems such as insufficient light and too small targets, which lead to low confidence and reduce the model detection accuracy. In order to improve the target confidence in this case, a multimodal fusion construction machinery detection framework is proposed, which introduces the detection confidence of the dense depth map with rich spatial information, and uses the DS evidence theory to fuse the two. It is worth noting that It is worth noting that the YOLOv5 models corresponding to the two types of data have the same network structure but have different network weight parameters after separate training. They output the detection box results based on the image and the detection box results based on the dense depth map respectively. Each detection box includes the location information and detection confidence of the construction machinery, where the location information is composed of the upper left corner coordinate point and the lower right corner coordinate point of the detection box of the construction machinery target in the image; secondly, based on the DS evidence theory, the confidence fusion of the detection box results of the two types of data is performed, and the fused detection box results are output; finally, the fused detection box results are screened, and the detection boxes with detection confidence lower than 0.5 in the results are deleted, and the final results are output.

[0081] Since the DS evidence theory can be used to deal with uncertain problems, it can fuse different prediction results according to its own rules and finally generate new prediction results. It is often used in reliability assessment, data fusion, fault diagnosis and other fields. As a preferred method, the DS evidence theory is applied to the confidence fusion of the detection box results of the two data in step 3. The specific steps are as follows:

[0082] a) Image-based detection box B c and the detection box B based on the dense depth map d Perform pairwise intersection over union (IoU) calculations and select B with IoU greater than 0.75 c and B d They are grouped together for the next step of confidence fusion, and the other detection boxes are directly output;

[0083] b) Based on DS evidence theory, define the recognition framework θ = {correct detection, wrong detection}, B c and B dare two pieces of evidence under θ, representing the detection box based on the image and the detection box based on the dense depth map respectively. The basic probability comes from the confidence of the detection box before fusion; the basic probability is assigned as follows:

[0084] For evidence B c , the probability of correct detection is assigned to The probability of detecting error is assigned as

[0085] For evidence B d , the probability of correct detection is assigned to The probability of detecting error is assigned as

[0086] in, Represents the detection box B c The confidence score of Represents the detection box B d The confidence score

[0087] c) Based on the probability synthesis rule of DS evidence theory, the detection box B c and B d The confidence after fusion is calculated by the following formula:

[0088]

[0089] in, represents the confidence after the fusion of the two detection frames, and K represents the conflict factor, which is calculated by the following formula:

[0090]

[0091] d) For the detection box B c and B d The position information of the fused detection frame is fused and the position information of the fused detection frame is output. The position information is represented by the upper left corner coordinates and the lower right corner coordinates The composition is obtained by the following formula:

[0092]

[0093] in and They are detection box B c The upper left corner coordinates and lower right corner coordinates of; and They are detection box B d The upper left corner coordinates and lower right corner coordinates of;

[0094] e) Repeat steps b-d to achieve all detection boxes B with IoU greater than 0.75 c and B dFusion, output the location information and confidence of the fused detection box.

[0095] Confidence fusion through DS evidence theory is beneficial to adjust the confidence of the detection box in situations such as insufficient light and too small a target. The low confidence of the misdetected target can be maintained, while the confidence of the correctly detected target but with low confidence can be improved through fusion, so that it will not be deleted during screening, effectively improving the detection accuracy of the model.

[0096] Step 4: cluster the original laser point cloud based on the DBSCAN algorithm, and perform bounding box fitting on each group of point clouds; preferably, step 4 clusters the original laser point cloud obtained in step 1 and generates a bounding box, and the specific steps are as follows:

[0097] a) In order to reduce the subsequent clustering algorithm overhead, the SACSegmentation function in the PCL toolkit is used to perform plane segmentation on the original laser point cloud to remove the ground point cloud;

[0098] b) Applying the DBSCAN algorithm to the non-ground point cloud data after the ground points are removed, dividing it into different clusters, and for each cluster, extracting the corresponding point cloud data to form a point cloud group for each cluster;

[0099] c) For each point cloud group, extract the minimum value (x min ,y min ,z min ) and the maximum value (x max ,y max ,z max ), these 6 values ​​are used to fit the minimum bounding box of each group of cluster point clouds.

[0100] Step 5: Based on the construction machinery detection frame obtained in step 3 and the clustered point cloud boundary frame obtained in step 4, the construction machinery is positioned, and anti-external force damage monitoring is achieved by calculating the actual distance to the live equipment.

[0101] As a preference, Figure 2 As shown, in step 5, when there is a construction machinery target in the multimodal fusion detection result, the point cloud boundary box is matched with the construction machinery detection box to realize the positioning of the construction machinery; finally, in the same radar coordinate system, the Euclidean distance between the three-dimensional coordinates of the construction machinery and the three-dimensional coordinates of the recorded live equipment is calculated. If the measured distance is less than the preset safety threshold, an alarm is given through voice broadcast in the monitoring scenario, and the screenshot is pushed to the back-end monitoring console to remind relevant staff.

[0102] It is worth noting that the construction machinery target detection frame obtained in step 3 has semantic information and two-dimensional image position but does not have actual positioning effect, while the point cloud bounding box obtained in step 4 has actual position information but each box lacks semantics. Therefore, it is necessary to match the point cloud bounding box with the construction machinery target detection frame to realize the construction machinery positioning. As a preferred embodiment, the construction machinery positioning includes the following steps:

[0103] a) According to the projection matrix T obtained in step 1 l2i , project each cluster point cloud bounding box to obtain its two-dimensional bounding box on the image. The projection formula is as follows:

[0104]

[0105] in, Represents the coordinates of the upper left corner of the two-dimensional bounding box in the image after projection; Represents the coordinates of the upper right corner of the two-dimensional bounding box in the image after projection;

[0106] b) Traverse the construction machinery target detection box results obtained in step 3, and for each detection box, perform IoU calculations with all two-dimensional bounding boxes to obtain an IoU value list;

[0107] c) For each IoU value list corresponding to the detection box, the clustered point cloud bounding box corresponding to the two-dimensional bounding box with the largest IoU value is regarded as the actual bounding box of the construction machinery to realize the positioning of the construction machinery.

[0108] The above are only preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The substitution may be a substitution of a part of the structure, device, method step, or a complete technical solution. The equivalent replacement or change according to the technical solution and the inventive concept of the present invention shall be covered within the protection scope of the present invention.

Claims

1. A substation anti-external force damage monitoring method based on multi-modal fusion, characterized in that: The steps include: Step 1: Jointly calibrate and time-synchronize the laser radar and camera at the substation monitoring site, collect the 3D point cloud data and RGB image data at the site, and record the 3D information of the live equipment within the monitoring range in the radar coordinate system; Step 2: According to the projection matrix from the radar coordinate system to the image coordinate system, the correspondence between the pixel points in the image and the laser point cloud is obtained to generate a sparse depth map, and the sparse depth map is converted into a dense depth map through a depth completion method based on a bilateral filter; Step 3: Design a multimodal fusion construction machinery detection framework, use the image and dense depth map as the output of the YOLOv5 model respectively, fuse the model output results based on DS evidence theory, and output the construction machinery detection box results; Step 4: Cluster the original laser point cloud based on the DBSCAN algorithm, and perform bounding box fitting on each group of point clouds; Step 5: Based on the construction machinery detection frame obtained in step 3 and the clustered point cloud boundary frame obtained in step 4, the construction machinery is positioned, and anti-external force damage monitoring is achieved by calculating the actual distance to the live equipment.

2. The substation anti-external force damage monitoring method based on multi-modal fusion according to claim 1 is characterized in that: In step 1, a checkerboard rectangular calibration plate is used to repeatedly move the calibration plate within the common field of view of the camera and the lidar, collect multiple sets of images and point cloud data, and use the Autoware calibration tool to perform joint calibration to obtain the rotation matrix R from the radar coordinate system to the camera coordinate system. l2c , the translation vector T from the radar coordinate system to the camera coordinate system l2c And the camera intrinsic parameter K c2i ; Finally, the projection matrix T from the radar coordinate system to the image coordinate system is calculated by the following formula: l2i : In step 1, the output period is calculated according to the sampling frequency of the laser radar and the camera, and the time synchronization of the data is performed. The output period T of the two types of data after synchronization is p It can be calculated by the following formula: Among them, F lidar and F camera Respectively represent the number of frames collected per second by the laser radar and camera; T p The unit is seconds, that is, every T p The data collected every second is the data at the same moment, realizing the time synchronization of the two sensors.

3. The substation anti-external force damage monitoring method based on multi-modal fusion according to claim 1 or 2, characterized in that: In step 2, according to the projection matrix T from the radar coordinate system to the image coordinate system obtained in step 1 l2i , establish the mapping relationship between the three-dimensional point cloud coordinates and the image plane coordinates, the mapping formula is as follows: where the vector [x l y l z l 1] T is the spatial coordinate of the three-dimensional point cloud in the radar coordinate system, x l Indicates the coordinate value of the point cloud on the X axis, y l Indicates the coordinate value of the point cloud on the Y axis, z l Indicates the coordinate value of the point cloud on the Z axis; vector [uv1] T It is the plane coordinate in the image coordinate system after the three-dimensional point cloud is mapped, u represents the horizontal coordinate value, and v represents the vertical coordinate value; After obtaining the one-to-one correspondence between each point cloud coordinate and the plane coordinate, a sparse depth map consistent with the length and width of the image captured by the camera is generated; specifically, first, according to the length H and width W of the image, the coordinate points whose u values ​​are not in the interval [0, W] or the v values ​​are not in the interval [0, H] are removed; Next, create a two-dimensional matrix with all zeros in length and width that matches the image, traverse all remaining plane coordinate points, and assign z to the value of the coordinate (u, v) in the two-dimensional matrix. l , as the depth of the position; after the final traversal is completed, a sparse depth map with a size of H×W×1 is obtained.

4. The substation anti-external force damage monitoring method based on multi-modal fusion according to claim 1 is characterized in that: In step 2, a bilateral filter is applied to complete the depth of the sparse depth map to generate a dense panoramic image. Specifically, each pixel point p in the image is traversed, and the depth value I at the pixel point p after filtering is p ′ The following formula can be used for calculation: Among them, S represents the neighborhood window centered on p; p and q represent the physical coordinates of the pixel points; I p ,I q W represents the pixel (depth) value corresponding to pixel points p and q on the sparse depth map; p is the normalization coefficient; G c and G s Represent the distance weight and similarity weight respectively, which are calculated by Gaussian function. The calculation formula is as follows: Where exp(·) represents an exponential function with the natural constant e as the base; (u p ,v p ) is the coordinate of pixel point p; (u q ,v q ) is the coordinate of pixel point q; σ c and σ s are the two standard deviations corresponding to the Gaussian function; After traversing the entire sparse panorama through a bilateral filter, depth completion is achieved and it is replicated twice at the channel level to obtain a dense panorama of size H×W×3.

5. The substation anti-external force damage monitoring method based on multi-modal fusion according to claim 1 is characterized in that: In step 3, the DS evidence theory is applied to the confidence fusion of the detection box results of the two data, and the specific steps are: a) Image-based detection box B c and the detection box B based on the dense depth map d Perform pairwise intersection over union (IoU) calculations and select B with IoU greater than 0.

75. c and B d They are grouped together for the next step of confidence fusion, and the other detection boxes are directly output; b) Based on DS evidence theory, define the identification framework B c and B d for The two pieces of evidence below represent the detection box based on the image and the detection box based on the dense depth map. The basic probability comes from the confidence of the detection box before fusion. The basic probability assignment is as follows: For evidence B c , the probability of correct detection is assigned to The probability of detecting an error is assigned as For evidence B d , the probability of correct detection is assigned to The probability of detecting error is assigned as in, Represents the detection box B c The confidence score of Represents the detection box B d Confidence score of c) Based on the probability synthesis rule of DS evidence theory, the detection box B c and B d The confidence after fusion is calculated by the following formula: in, It represents the confidence after the fusion of the two detection frames, and K represents the conflict factor, which is calculated by the following formula: d) For the detection box B c and B d The position information of the fused detection frame is fused and the position information of the fused detection frame is output. The position information is represented by the upper left corner coordinates and the lower right corner coordinates The composition is obtained by the following formula: in and They are detection box B c The upper left corner coordinates and lower right corner coordinates of; and They are detection box B d The upper left corner coordinates and lower right corner coordinates of; e) Repeat steps b-d to achieve all detection boxes B with IoU greater than 0.75 c and B d Fusion, output the location information and confidence of the fused detection box.

6. The substation anti-external force damage monitoring method based on multi-modal fusion according to claim 1 is characterized in that: The step 4 clusters the original laser point cloud obtained in step 1 and generates a bounding box. The specific steps are as follows: a) In order to reduce the subsequent clustering algorithm overhead, the SACSegmentation function in the PCL toolkit is used to perform plane segmentation on the original laser point cloud to remove the ground point cloud; b) Applying the DBSCAN algorithm to the non-ground point cloud data after the ground points are removed, dividing it into different clusters, and for each cluster, extracting the corresponding point cloud data to form a point cloud group for each cluster; c) For each point cloud group, extract the minimum value of the point cloud group in the X-axis, Y-axis and Z-axis respectively. (x min ,y min ,z min ) and the maximum value (x max ,y max ,z max ), these 6 values ​​are used to fit the minimum bounding box of each group of cluster point clouds.

7. The substation anti-external force damage monitoring method based on multi-modal fusion according to claim 1 is characterized in that: In step 5, when there is a construction machinery target in the multimodal fusion detection result, the point cloud boundary box is matched with the construction machinery detection box to realize the positioning of the construction machinery; finally, in the same radar coordinate system, the Euclidean distance between the three-dimensional coordinates of the construction machinery and the three-dimensional coordinates of the recorded live equipment is calculated. If the measured distance is less than the preset safety threshold, an alarm is given through voice broadcast in the monitoring scenario, and the screenshot is pushed to the back-end monitoring console to remind relevant staff.

8. The substation protection against external force damage monitoring method based on multi-modal fusion according to claim 7 is characterized in that: The construction machinery positioning comprises the following steps: a) According to the projection matrix T obtained in step 1 l2i , project each cluster point cloud bounding box to obtain its two-dimensional bounding box on the image. The projection formula is as follows: in, Represents the coordinates of the upper left corner of the two-dimensional bounding box in the image after projection; Represents the coordinates of the upper right corner of the two-dimensional bounding box in the image after projection; b) Traverse the construction machinery target detection box results obtained in step 3, and for each detection box, perform IoU calculations with all two-dimensional bounding boxes to obtain an IoU value list; c) For each IoU value list corresponding to the detection box, the clustered point cloud bounding box corresponding to the two-dimensional bounding box with the largest IoU value is regarded as the actual bounding box of the construction machinery to realize the positioning of the construction machinery.

Images