A tree obstacle intelligent detection method and system based on multimodal perception

Abstract

The present invention relates to an intelligent tree barrier detection method and system based on multimodal perception. The method uses sensors carried by drones to collect data, including RGB images, LIDAR point clouds, multispectral data, etc. First, the data is preprocessed to form a standardized input. Then, a Hough transform is performed on the LIDAR point cloud to extract the position of the power line and generate a three-dimensional line model. Then, a deep learning algorithm is used to locate the tree bounding box from the RGB image, and the high-risk point cloud area is screened in combination with the line model. Then, a three-dimensional tree model is constructed, and the minimum safe clearance distance between the power line and the tree is calculated in real time through a collision detection algorithm to obtain a three-dimensional tree model with a safety distance annotation. Finally, a spatial risk assessment is performed based on this model. The present invention can significantly improve the reliability and efficiency of power line inspections.

Description

Technical Field

[0001] The present invention relates to the field of intelligent recognition, and in particular to a tree obstacle intelligent detection method and system based on multimodal perception. Background Art

[0002] With the rapid expansion of power transmission line networks, the safe operation of transmission lines faces increasing challenges. Tree obstructions (tree barriers) are a major factor in line failures. Statistics show that a high proportion of accidents occur due to trees growing too fast, collapsing, or encroaching into the safe clearance of power lines during severe weather (such as strong winds and heavy rain), leading to short circuits, power outages, and even equipment damage. Traditional tree barrier detection methods rely primarily on manual inspections and ground-based mapping, but this approach is not only costly and inefficient, but also complex environments such as rain and strong winds can increase detection difficulty and risks. Furthermore, traditional methods have significant limitations in scenarios involving large areas, complex terrain (such as mountainous areas and high altitudes), or long line spans. Summary of the Invention

[0003] To achieve the above object, the present invention adopts the following technical solutions:

[0004] A tree obstacle intelligent detection method based on multimodal perception includes the following steps:

[0005] S1: Collect data through sensors onboard drones, obtain sensor data, and preprocess the sensor data to form standardized input; the sensor data includes RGB images, LIDAR point clouds, multispectral data, and drone data;

[0006] S2: Perform Hough transform on the LIDAR point cloud in the standardized input to extract the primary location of the power line and generate a 3D line model of the power line;

[0007] S3: Use a deep learning target detection algorithm to locate the tree target bounding box from the standardized input RGB image, and combine it with the 3D line model of the power line to screen high-risk point cloud areas in the space near the power line;

[0008] S4: Based on the high-risk point cloud areas in the space near the power lines, a 3D tree model is constructed. Using a collision detection algorithm, the minimum safe clearance distance between the power lines and the trees is calculated in real time from the 3D tree model to obtain a 3D tree model with safety distance annotations.

[0009] S5: Perform spatial risk assessment based on a 3D tree model with safety distance annotations.

[0010] Furthermore, the pre-processing of the sensor data to form a standardized input in step S1 is specifically as follows:

[0011] Spatiotemporal synchronization processing: sensor data with different acquisition frequencies are normalized based on timestamps to ensure spatiotemporal correlation;

[0012] Spatial registration and coordinate system 1, using a checkerboard calibration plate, solve the rotation matrix R and translation vector t through joint calibration, and the conditions are expressed as:

[0013] ;

[0014] in, represents a point in the camera coordinate system, Represents a point in the radar coordinate system;

[0015] In this process, the Levenberg-Marquardt algorithm is used to minimize the reprojection error, which is expressed as:

[0016] ;

[0017] in, is the RGB camera projection model, is the first The coordinates of the corner points, that is, the actual observation points in the camera coordinate system; is the number of point pairs matching the camera coordinate system and the radar coordinate system, represents the minimum evaluation function; Indicates the first 3D points;

[0018] The Brown-Conrady model is used to process RGB images to correct lens distortion, which is expressed as:

[0019] ;

[0020] in, Indicates a point Radial distance to the optical axis, is the radial distortion coefficient, is the tangential distortion coefficient; is the actual pixel position of the RGB image obtained after correction; is the pixel position of the RGB image before correction;

[0021] Perform voxel downsampling on the LIDAR point cloud and divide the LIDAR point cloud into sides of length A voxel grid with each voxel retaining the centroid is expressed as:

[0022] ;

[0023] in, Voxel The center of mass; is the total number of points in voxel v, Voxel The coordinates of each point in

[0024] Perform feature matching on multispectral data based on SIFT algorithm to achieve multispectral band alignment and obtain aligned multispectral data;

[0025] Using extended Kalman filtering, the UAV data and Perform fusion to obtain the latitude, longitude, elevation and attitude data of the UAV; Represents the real-time differential positioning data of the UAV data; IMU represents the inertial measurement data of the UAV data;

[0026] Based on this, a standardized output is formed, formalized as a time-synchronized tuple , expressed as:

[0027] ;

[0028] in, is the rectified RGB image; is the downsampled LIDAR point cloud; is the aligned multispectral data; is the latitude, longitude and elevation, The posture data.

[0029] Furthermore, the extended Kalman filter is used to transform the UAV data into and Perform fusion to obtain the latitude, longitude, elevation and attitude data of the UAV; specifically:

[0030] Obtain the drone's state vector based on its latitude, longitude, elevation and attitude data ,in, is the position of the drone; is the speed of the drone; is the attitude quaternion of the drone, which is used to represent spatial rotation; represents transpose;

[0031] pass The driving prediction step is expressed as:

[0032] ;

[0033] ;

[0034] ;

[0035] in, For drones at all times location information; For drones at all times speed; For drones at all times The attitude quaternion of is the time increment; Represents the rotation matrix from the IMU coordinate system to the global coordinate system, which is calculated by the drone at time The attitude quaternion is calculated; For drones at all times acceleration; is the acceleration due to gravity; is quaternion multiplication; For drones at all times angular velocity;

[0036] The update step based on the GPS+RTK observation of the UAV data is expressed as:

[0037] ;

[0038] in, GPS+RTK measures the drone's position in real time to correct the IMU's accumulated error. is the observation matrix; For drones at all times The state vector of .

[0039] Furthermore, the three-dimensional line model of the wire generated in step S2 is specifically:

[0040] Representing the wires of a power line as parametric equations of lines in 3D space , expressed as:

[0041] ;

[0042] in, is the parametric equation of a straight line Any point on is the parametric equation of a straight line The reference point on is the parametric equation of a straight line The direction vector, is the parametric equation of a straight line The straight line parameters of

[0043] The Hough parameter space is defined as: ;

[0044] Requirements:

[0045]

[0046] ;

[0047] in, is the distance from the straight line to the origin; is the azimuth of the line in the XY plane; is the pitch angle between the straight line and the XY plane; represents the normal vector of the line; represents any point on the line;

[0048] For each point , calculate the voting value in the Hough parameter space, expressed as:

[0049] ;

[0050] For the rough line selected by Hough detection, the residual error of the distance from the point to the line is minimized by nonlinear optimization, which is expressed as:

[0051] ;

[0052] in, is the number of LIDAR point clouds of wires;

[0053] The wire exhibits a parabolic sag under its own weight, with the height h distributed along the arc length s, expressed as:

[0054] ;

[0055] in, represents the height at arc length s, is the starting height of the wire; w is the weight per unit length of the wire; is the horizontal tension of the wire; is the span, i.e. the horizontal distance between the supporting points of the wires;

[0056] Fitting the sag point cloud data, by weighted least squares fitting w and , expressed as:

[0057] ;

[0058] in, represents the predicted height of the power line, for point The projected arc length along the line direction of the wire; M represents the total number of LIDAR point clouds of the wire; represents the minimum evaluation function;

[0059] Finally, the 3D line model of the wires It is expressed as follows:

[0060] .

[0061] Furthermore, S3 is specifically:

[0062] Use deep learning target detection algorithms to locate tree targets in the standardized input RGB image and obtain the two-dimensional position of each tree in the image by detecting the target bounding box;

[0063] Combine the 3D line model and LIDAR point cloud to map the tree targets in the RGB image to the corresponding LIDAR point cloud;

[0064] 3D line model based on wires Construct a space search area; specifically, establish a cylindrical safety area with the wire as the center, and the cylindrical safety area is parameterized as radius r and length Cylinder, length is the corresponding gear distance;

[0065] The radius r is defined as follows:

[0066] ;

[0067] The following judgment is made for each point of the LIDAR point cloud:

[0068] Through KD-Tree query, each point set in the LIDAR point cloud located near the power lines is retained, and the points within the cylindrical safety area are further filtered by combining the tree point cloud clusters mapped from the RGB image corresponding to each bounding box;

[0069] Calculate the minimum distance from the tree point cloud cluster to the power line using the shortest distance formula from a straight line to a point , which is the minimum clearance distance; expressed as:

[0070] ;

[0071] in, Represents a tree point cloud cluster;

[0072] Conduct risk assessment, specifically:

[0073] Setting safety thresholds ,like , marked as high-risk tree point cloud clusters, that is, high-risk point cloud areas in the space near power lines.

[0074] Furthermore, S4 is specifically:

[0075] S41: Cluster the LIDAR point cloud in the high-risk point cloud area using the DBSCAN algorithm to segment the point cloud clusters of individual trees;

[0076] S42: Extract the height-direction trunk from the point cloud cluster of a single tree and use the RANSAC algorithm to fit the trunk into a cylindrical model;

[0077] S43: Separate the trunk and branches based on the local geometric properties of the LIDAR point cloud, and use spatial continuity to grow from the trunk and classify adjacent points as part of the crown;

[0078] S44: Generate a gradient field based on the normal vector of the LIDAR point cloud, solve the Poisson equation based on the gradient field to construct an implicit function, perform Poisson surface reconstruction, and fit the tree surface model;

[0079] S45: fusing the corrected RGB image into the aligned multispectral data, specifically calculating a vegetation index using the aligned multispectral data, and assigning a texture map to a mesh surface of a tree surface model using the corrected RGB image to obtain a three-dimensional tree model;

[0080] S46: Using the constructed 3D power line model and 3D tree model, a safe clearance distance is calculated through a collision detection algorithm to obtain a 3D tree model with a safe distance annotation.

[0081] Furthermore, S5 is specifically:

[0082] Extract tree features from tree models with safety distance annotations, including tree height and tree density ; and combined with wind speed , calculate the dynamic minimum clearance distance;

[0083] Based on this, the feature vector is constructed , expressed as:

[0084] ;

[0085] in, is the average annual growth rate of trees; is the minimum clearance distance; The health score is obtained by multispectral calculation of NDVI value; is the tension factor;

[0086] A risk assessment model is constructed to output the risk level R for each tree based on the LIDAR point cloud and 3D tree model, expressed as:

[0087] ;

[0088] in, A risk score is obtained based on the risk assessment model;

[0089] Maintenance recommendations are provided based on the risk level of each tree, specifically pruning or clearing dangerous areas; regular monitoring of warning areas; and no treatment required in safe areas.

[0090] Furthermore, the risk assessment model is expressed as:

[0091] ;

[0092] ;

[0093] ;

[0094] ;

[0095] ;

[0096] in, Score points for clearance distance; is the risk-free reference distance; scoring health risks; score for density risk; are the minimum and maximum values ​​of density, respectively; Score for windage risk; is the wind speed, These are all preset weight coefficients.

[0097] A tree obstacle intelligent detection system based on multimodal perception includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps of the above-mentioned tree obstacle intelligent detection method based on multimodal perception.

[0098] The present invention has the following beneficial effects:

[0099] 1. This invention integrates RGB images, LIDAR point clouds, multispectral data, and drone data, and achieves real-time detection and safety assessment of tree obstacles through efficient edge computing and lightweight AI models. Furthermore, through intelligent sensor collaboration, precise alignment of data preprocessing, a multi-stage fusion detection network, and a cloud-edge collaborative update mechanism, this solution not only significantly improves detection accuracy and real-time response capabilities, but also enables long-term and stable inspection tasks, ensuring safe power grid operation.

[0100] 2. This invention uses a tree model with safety distance markings, combined with multi-dimensional features such as geometry, health, and density, to quantify the risk of trees to power lines, and provides scientific support for power inspections through dynamic threshold adjustment and visualization. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0102] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0103] refer to Figure 1 , a tree obstacle intelligent detection method based on multimodal perception, comprising the following steps:

[0104] S1: Collect data through sensors onboard drones, obtain sensor data, and preprocess the sensor data to form standardized input; the sensor data includes RGB images, LIDAR point clouds, multispectral data, and drone data;

[0105] S2: Perform Hough transform on the LIDAR point cloud in the standardized input to extract the primary location of the power line and generate a 3D line model of the power line;

[0106] S3: Use a deep learning target detection algorithm to locate the tree target bounding box from the standardized input RGB image, and combine it with the 3D line model of the power line to screen high-risk point cloud areas in the space near the power line;

[0107] S4: Based on the high-risk point cloud areas in the space near the power lines, a 3D tree model is constructed. Using a collision detection algorithm, the minimum safe clearance distance between the power lines and the trees is calculated in real time from the 3D tree model to obtain a 3D tree model with safety distance annotations.

[0108] S5: Perform spatial risk assessment based on a 3D tree model with safety distance annotations.

[0109] Furthermore, the pre-processing of the sensor data to form a standardized input in step S1 is specifically as follows:

[0110] Spatiotemporal synchronization processing: sensor data with different acquisition frequencies are normalized based on timestamps to ensure spatiotemporal correlation;

[0111] Spatial registration and coordinate system 1, using a checkerboard calibration plate, solve the rotation matrix R and translation vector t through joint calibration, and the conditions are expressed as:

[0112] ;

[0113] in, represents a point in the camera coordinate system, Represents a point in the radar coordinate system;

[0114] In this process, the Levenberg-Marquardt algorithm is used to minimize the reprojection error, which is expressed as:

[0115] ;

[0116] in, is the RGB camera projection model, is the first The coordinates of the corner points, that is, the actual observation points in the camera coordinate system; is the number of point pairs matching the camera coordinate system and the radar coordinate system, represents the minimum evaluation function; Indicates the first 3D points;

[0117] The Brown-Conrady model is used to process RGB images to correct lens distortion, which is expressed as:

[0118] ;

[0119] in, Indicates a point Radial distance to the optical axis, is the radial distortion coefficient, is the tangential distortion coefficient; is the actual pixel position of the RGB image obtained after correction; is the pixel position of the RGB image before correction;

[0120] Perform voxel downsampling on the LIDAR point cloud and divide the LIDAR point cloud into sides of length A voxel grid with each voxel retaining the centroid is expressed as:

[0121] ;

[0122] in, Voxel The center of mass; is the total number of points in voxel v, Voxel The coordinates of each point in

[0123] Perform feature matching on multispectral data based on SIFT algorithm to achieve multispectral band alignment and obtain aligned multispectral data;

[0124] Using extended Kalman filtering, the UAV data and Perform fusion to obtain the latitude, longitude, elevation and attitude data of the UAV; Represents the real-time differential positioning data of the UAV data; IMU represents the inertial measurement data of the UAV data;

[0125] Based on this, a standardized output is formed, formalized as a time-synchronized tuple , expressed as:

[0126] ;

[0127] in, is the rectified RGB image; is the downsampled LIDAR point cloud; is the aligned multispectral data; is the latitude, longitude and elevation, The posture data.

[0128] Furthermore, the extended Kalman filter is used to transform the UAV data into and Perform fusion to obtain the latitude, longitude, elevation and attitude data of the UAV; specifically:

[0129] Obtain the drone's state vector based on its latitude, longitude, elevation and attitude data ,in, is the position of the drone; is the speed of the drone; is the attitude quaternion of the drone, which is used to represent spatial rotation; represents transpose;

[0130] pass The driving prediction step is expressed as:

[0131] ;

[0132] ;

[0133] ;

[0134] in, For drones at all times location information; For drones at all times speed; For drones at all times The attitude quaternion of is the time increment; Represents the rotation matrix from the IMU coordinate system to the global coordinate system, which is calculated by the drone at time The attitude quaternion is calculated; For drones at all times acceleration; is the acceleration due to gravity; is quaternion multiplication; For drones at all times angular velocity;

[0135] The update step based on the GPS+RTK observation of the UAV data is expressed as:

[0136] ;

[0137] in, GPS+RTK measures the drone's position in real time to correct the IMU's accumulated error. is the observation matrix; For drones at all times The state vector of .

[0138] Furthermore, the three-dimensional line model of the wire generated in step S2 is specifically:

[0139] Representing the wires of a power line as parametric equations of lines in 3D space , expressed as:

[0140] ;

[0141] in, is the parametric equation of a straight line Any point on is the parametric equation of a straight line The reference point on is the parametric equation of a straight line The direction vector, is the parametric equation of a straight line The straight line parameters of

[0142] The Hough parameter space is defined as: ;

[0143] Requirements:

[0144]

[0145] ;

[0146] in, is the distance from the straight line to the origin; is the azimuth of the line in the XY plane; is the pitch angle between the straight line and the XY plane; represents the normal vector of the line; represents any point on the line;

[0147] For each point , calculate the voting value in the Hough parameter space, expressed as:

[0148] ;

[0149] For the rough line selected by Hough detection, the residual error of the distance from the point to the line is minimized by nonlinear optimization, which is expressed as:

[0150] ;

[0151] in, is the number of LIDAR point clouds of wires;

[0152] The wire exhibits a parabolic sag under its own weight, with the height h distributed along the arc length s, expressed as:

[0153] ;

[0154] in, represents the height at arc length s, is the starting height of the wire; w is the weight per unit length of the wire; is the horizontal tension of the wire; is the span, i.e. the horizontal distance between the supporting points of the wires;

[0155] Fitting the sag point cloud data, by weighted least squares fitting w and , expressed as:

[0156] ;

[0157] in, represents the predicted height of the power line, for point The projected arc length along the line direction of the wire; M represents the total number of LIDAR point clouds of the wire; represents the minimum evaluation function;

[0158] Finally, the 3D line model of the wires It is expressed as follows:

[0159] .

[0160] Furthermore, S3 is specifically:

[0161] Use deep learning target detection algorithms to locate tree targets in the standardized input RGB image and obtain the two-dimensional position of each tree in the image by detecting the target bounding box;

[0162] Combine the 3D line model and LIDAR point cloud to map the tree targets in the RGB image to the corresponding LIDAR point cloud;

[0163] 3D line model based on wires Construct a space search area; specifically, establish a cylindrical safety area with the wire as the center, and the cylindrical safety area is parameterized as radius r and length Cylinder, length is the corresponding gear distance;

[0164] The radius r is defined as follows:

[0165] ;

[0166] The following judgment is made for each point of the LIDAR point cloud:

[0167] Through KD-Tree query, each point set in the LIDAR point cloud located near the power lines is retained, and the points within the cylindrical safety area are further filtered by combining the tree point cloud clusters mapped from the RGB image corresponding to each bounding box;

[0168] Calculate the minimum distance from the tree point cloud cluster to the power line using the shortest distance formula from a straight line to a point , which is the minimum clearance distance; expressed as:

[0169] ;

[0170] in, Represents a tree point cloud cluster;

[0171] Conduct risk assessment, specifically:

[0172] Setting safety thresholds ,like , marked as high-risk tree point cloud clusters, that is, high-risk point cloud areas in the space near power lines.

[0173] Furthermore, S4 is specifically:

[0174] S41: Cluster the LIDAR point cloud in the high-risk point cloud area using the DBSCAN algorithm to segment the point cloud clusters of individual trees;

[0175] S42: Extract the height-direction trunk from the point cloud cluster of a single tree and use the RANSAC algorithm to fit the trunk into a cylindrical model;

[0176] S43: Separate the trunk and branches based on the local geometric properties of the LIDAR point cloud, and use spatial continuity to grow from the trunk and classify adjacent points as part of the crown;

[0177] S44: Generate a gradient field based on the normal vector of the LIDAR point cloud, solve the Poisson equation based on the gradient field to construct an implicit function, perform Poisson surface reconstruction, and fit the tree surface model;

[0178] S45: fusing the corrected RGB image into the aligned multispectral data, specifically calculating a vegetation index using the aligned multispectral data, and assigning a texture map to a mesh surface of a tree surface model using the corrected RGB image to obtain a three-dimensional tree model;

[0179] S46: Using the constructed 3D power line model and 3D tree model, a safe clearance distance is calculated through a collision detection algorithm to obtain a 3D tree model with a safe distance annotation.

[0180] Furthermore, S5 is specifically:

[0181] Extract tree features from tree models with safety distance annotations, including tree height and tree density ; and combined with wind speed , calculate the dynamic minimum clearance distance;

[0182] Based on this, the feature vector is constructed , expressed as:

[0183] ;

[0184] in, is the average annual growth rate of trees; is the minimum clearance distance; The health score is obtained by multispectral calculation of NDVI value; is the tension factor;

[0185] A risk assessment model is constructed to output the risk level R for each tree based on the LIDAR point cloud and 3D tree model, expressed as:

[0186] ;

[0187] in, A risk score is obtained based on the risk assessment model;

[0188] Maintenance recommendations are provided based on the risk level of each tree, specifically pruning or clearing dangerous areas; regular monitoring of warning areas; and no treatment required in safe areas.

[0189] Furthermore, the risk assessment model is expressed as:

[0190] ;

[0191] ;

[0192] ;

[0193] ;

[0194] ;

[0195] in, Score points for clearance distance; is the risk-free reference distance; scoring health risks; score for density risk; are the minimum and maximum values ​​of density, respectively; Score for windage risk; is the wind speed, These are all preset weight coefficients.

[0196] A tree obstacle intelligent detection system based on multimodal perception includes a processor, a memory, and a computer program stored in the memory. When the processor executes the computer program, it specifically performs the steps of the above-mentioned tree obstacle intelligent detection method based on multimodal perception.

[0197] In one example, the sensors include an RGB camera, LIDAR, a multispectral camera, a GPS+RTK module, and an IMU module, specifically as follows: the RGB camera acquires high-definition visual information, namely RGB images; the LIDAR collects three-dimensional point cloud data of tree barriers and power grids, namely LIDAR point clouds, which are used to model the spatial distance relationship between trees and wires; the multispectral camera detects the health of leaves and identifies diseased trees that may fall or die; the GPS+RTK module achieves centimeter-level real-time position calibration, clearly recording the position and spatial distribution of each tree; the IMU module stabilizes flight in real time and compensates for sensor errors caused by drone movement and vibration.

[0198] Specifically, a deep learning target detection algorithm (such as YOLOv8 or improved Faster R-CNN) is used to locate tree targets in the RGB image, and the two-dimensional position of each tree is obtained by detecting the bounding box.

[0199] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A tree obstacle intelligent detection method based on multimodal perception, characterized in that: The following steps are involved: S1: Collect data through sensors onboard drones, obtain sensor data, and preprocess the sensor data to form standardized input; the sensor data includes RGB images, LIDAR point clouds, multispectral data, and drone data; S2: Perform Hough transform on the LIDAR point cloud in the standardized input to extract the primary location of the power line and generate a 3D line model of the power line; S3: Use a deep learning target detection algorithm to locate the tree target bounding box from the standardized input RGB image, and combine it with the 3D line model of the power line to screen high-risk point cloud areas in the space near the power line; S4: Based on the high-risk point cloud areas in the space near the power lines, a 3D tree model is constructed. Using a collision detection algorithm, the minimum safe clearance distance between the power lines and the trees is calculated in real time from the 3D tree model to obtain a 3D tree model with safety distance annotations. S5: Perform spatial risk assessment based on a 3D tree model with safety distance annotations; The three-dimensional line model of the wire generated in step S2 is specifically as follows: Representing the wires of a power line as parametric equations of lines in 3D space , expressed as: ； in, is the parametric equation of a straight line Any point on is the parametric equation of a straight line The reference point on is the parametric equation of a straight line The direction vector, is the parametric equation of a straight line The straight line parameters of The Hough parameter space is defined as: ; Requirements: ； ； in, is the distance from the straight line to the origin; is the azimuth of the line in the XY plane; is the pitch angle between the straight line and the XY plane; represents the normal vector of the line; represents any point on the line; For each point , calculate the voting value in the Hough parameter space, expressed as: ； For the rough line selected by Hough detection, the residual error of the distance from the point to the line is minimized by nonlinear optimization, which is expressed as: ； in, is the number of LIDAR point clouds of wires; The wire exhibits a parabolic sag under its own weight, with the height h distributed along the arc length s, expressed as: ； in, represents the height at arc length s, is the starting height of the wire; w is the weight per unit length of the wire; is the horizontal tension of the wire; is the span, i.e. the horizontal distance between the supporting points of the wires; Fitting the sag point cloud data, by weighted least squares fitting w and , expressed as: ； in, represents the predicted height of the power line, for point The projected arc length along the line direction of the wire; M represents the total number of LIDAR point clouds of the wire; represents the minimum evaluation function; Finally, the 3D line model of the wires It is expressed as follows: ； S3 specifically: Use deep learning target detection algorithms to locate tree targets in the standardized input RGB image and obtain the two-dimensional position of each tree in the image by detecting the target bounding box; Combine the 3D line model and LIDAR point cloud to map the tree targets in the RGB image to the corresponding LIDAR point cloud; 3D line model based on wires Construct a space search area; specifically, establish a cylindrical safety area with the wire as the center, and the cylindrical safety area is parameterized as radius r and length Cylinder, length is the corresponding gear distance; The radius r is defined as follows: ； The following judgment is made for each point of the LIDAR point cloud: Through KD-Tree query, each point set in the LIDAR point cloud located near the power lines is retained, and the points within the cylindrical safety area are further filtered by combining the tree point cloud clusters mapped from the RGB image corresponding to each bounding box; Calculate the minimum distance from the tree point cloud cluster to the power line using the shortest distance formula from a straight line to a point , which is the minimum clearance distance; expressed as: ； in, Represents a tree point cloud cluster; Conduct risk assessment, specifically: Setting safety thresholds ,like , marked as high-risk tree point cloud clusters, that is, high-risk point cloud areas in the space near power lines.

2. The method for intelligent tree obstacle detection based on multimodal perception according to claim 1, characterized in that: The preprocessing of sensor data to form standardized input in step S1 is specifically as follows: Spatiotemporal synchronization processing: sensor data with different acquisition frequencies are normalized based on timestamps to ensure spatiotemporal correlation; Spatial registration and coordinate system 1, using a checkerboard calibration plate, solve the rotation matrix R and translation vector t through joint calibration, and the conditions are expressed as: ； in, represents a point in the camera coordinate system, Represents a point in the radar coordinate system; In this process, the Levenberg-Marquardt algorithm is used to minimize the reprojection error, which is expressed as: ; in, is the RGB camera projection model, is the first The coordinates of the corner points, that is, the actual observation points in the camera coordinate system; is the number of point pairs matching the camera coordinate system and the radar coordinate system, represents the minimum evaluation function; Indicates the first 3D points; The Brown-Conrady model is used to process RGB images to correct lens distortion, which is expressed as: ； in, Indicates a point Radial distance to the optical axis, is the radial distortion coefficient, is the tangential distortion coefficient; is the actual pixel position of the RGB image obtained after correction; is the pixel position of the RGB image before correction; Perform voxel downsampling on the LIDAR point cloud and divide the LIDAR point cloud into sides of length A voxel grid with each voxel retaining the centroid is expressed as: ; in, Voxel The center of mass; is the total number of points in voxel v, Voxel The coordinates of each point in Perform feature matching on multispectral data based on SIFT algorithm to achieve multispectral band alignment and obtain aligned multispectral data; Using extended Kalman filtering, the UAV data and Perform fusion to obtain the latitude, longitude, elevation and attitude data of the UAV; Represents the real-time differential positioning data of the UAV data; IMU represents the inertial measurement data of the UAV data; Based on this, a standardized output is formed, formalized as a time-synchronized tuple , expressed as: ； in, is the rectified RGB image; is the downsampled LIDAR point cloud; is the aligned multispectral data; is the latitude, longitude and elevation, The posture data.

3. The method for intelligent tree obstacle detection based on multimodal perception according to claim 2, characterized in that: Using extended Kalman filtering, the UAV data and Perform fusion to obtain the latitude, longitude, elevation and attitude data of the UAV; specifically: Obtain the drone's state vector based on its latitude, longitude, elevation and attitude data ,in, is the position of the drone; is the speed of the drone; is the attitude quaternion of the drone, which is used to represent spatial rotation; represents transpose; pass The driving prediction step is expressed as: ； ； ； in, For drones at all times location information; For drones at all times speed; For drones at all times The attitude quaternion of is the time increment; Represents the rotation matrix from the IMU coordinate system to the global coordinate system, which is calculated by the drone at time The attitude quaternion is calculated; For drones at all times acceleration; is the acceleration due to gravity; is quaternion multiplication; For drones at all times angular velocity; The update step based on the GPS+RTK observation of the UAV data is expressed as: ； in, GPS+RTK measures the drone's position in real time to correct the IMU's accumulated error. is the observation matrix; For drones at all times The state vector of .

4. The method for intelligent tree obstacle detection based on multimodal perception according to claim 1, characterized in that: S4 is specifically: S41: Cluster the LIDAR point cloud in the high-risk point cloud area using the DBSCAN algorithm to segment the point cloud clusters of individual trees; S42: Extract the height-direction trunk from the point cloud cluster of a single tree and use the RANSAC algorithm to fit the trunk into a cylindrical model; S43: Separate the trunk and branches based on the local geometric properties of the LIDAR point cloud, and use spatial continuity to grow from the trunk and classify adjacent points as part of the crown; S44: Generate a gradient field based on the normal vector of the LIDAR point cloud, solve the Poisson equation based on the gradient field to construct an implicit function, perform Poisson surface reconstruction, and fit the tree surface model; S45: fusing the corrected RGB image into the aligned multispectral data, specifically calculating a vegetation index using the aligned multispectral data, and assigning a texture map to a mesh surface of a tree surface model using the corrected RGB image to obtain a three-dimensional tree model; S46: Using the constructed 3D power line model and 3D tree model, a safe clearance distance is calculated through a collision detection algorithm to obtain a 3D tree model with a safe distance annotation.

5. The method for intelligent tree obstacle detection based on multimodal perception according to claim 4, characterized in that: S5 is specifically: Extract tree features from tree models with safety distance annotations, including tree height and tree density ; and combined with wind speed , calculate the dynamic minimum clearance distance; Based on this, the feature vector is constructed , expressed as: ； in, is the average annual growth rate of trees; is the minimum clearance distance; The health score is obtained by multispectral calculation of NDVI value; is the tension factor; A risk assessment model is constructed to output the risk level R for each tree based on the LIDAR point cloud and 3D tree model, expressed as: ； in, A risk score is obtained based on the risk assessment model; Maintenance recommendations are provided based on the risk level of each tree, specifically pruning or clearing dangerous areas; regular monitoring of warning areas; and no treatment required in safe areas.

6. The method for intelligent tree obstacle detection based on multimodal perception according to claim 5, characterized in that: The risk assessment model is expressed as: ； ； ； ； ； in, Score points for clearance distance; is the risk-free reference distance; scoring health risks; score for density risk; are the minimum and maximum values ​​of density, respectively; Score for windage risk; is the wind speed, These are all preset weight coefficients.

7. A tree obstacle intelligent detection system based on multimodal perception, characterized in that: The system comprises a processor, a memory and a computer program stored in the memory. When the processor executes the computer program, the system specifically performs the steps of the tree obstacle intelligent detection method based on multimodal perception according to any one of claims 1 to 6.

Images