Power transmission line tree obstacle identification and positioning method and device based on RF-CNN, and medium

Abstract

The invention discloses an RF-CNN-based power transmission line tree obstacle recognition and positioning method and device and a medium, and belongs to the technical field of image processing, and the method comprises the steps: collecting data through a laser radar system and an RGB image collection device, carrying out the preprocessing of the collected data, carrying out the feature fusion through an RF-CNN model, and carrying out the recognition and positioning of a tree obstacle. And training a multi-task learning model to perform three-dimensional positioning of tree types and tree positions, performing classification through fused feature data in tree identification, identifying different types of trees, positioning the position of each tree in a three-dimensional space, calculating horizontal and vertical distances between the tree and the power transmission line, and evaluating whether the tree forms a tree obstacle risk or not. According to the method, the defects of the prior art in the aspects of tree identification, positioning precision, tree obstacle risk assessment and the like are effectively overcome, the efficiency, precision and reliability of tree obstacle monitoring are improved, and a powerful technical support is provided for intelligent inspection and safety management of a power transmission line.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a transmission line tree barrier identification and positioning method, device and medium based on RF-CNN. BACKGROUND

[0002] In the daily maintenance and monitoring of transmission lines, tree barrier problems have always been an important factor affecting power safety. Trees or other obstacles contacting or approaching transmission lines not only may cause equipment damage, but also may cause more serious power accidents. Therefore, accurately identifying and positioning the tree barriers around the transmission lines has become an important task to ensure the safety of power transmission. However, the existing tree barrier detection technology has many challenges in practical application, mainly in terms of recognition accuracy, real-time performance, environmental adaptability, and data processing complexity.

[0003] Traditional tree barrier identification methods mostly rely on image processing or LiDAR point cloud data. In the field of image processing, many methods are based on RGB (color system) images or infrared images, and use edge detection, image segmentation, feature matching and other algorithms to identify obstacles. However, these methods are easily affected by light, weather and changes in viewing angle, resulting in low recognition accuracy, especially in complex weather conditions (such as fog, overcast or strong light), which may not be able to accurately identify tree barriers. Moreover, image processing methods usually cannot directly provide three-dimensional spatial information of the target, which is crucial for accurately positioning the spatial position of the tree barrier.

[0004] LiDAR technology, as an important means of obtaining three-dimensional spatial data, has been widely used in tree barrier detection in recent years. LiDAR generates high-precision three-dimensional point cloud data by emitting laser beams and receiving reflected signals, which can provide detailed information about the shape of the terrain and objects. LiDAR data itself also has many problems. Due to the sparsity of point cloud data and noise interference, relying solely on LiDAR point cloud for tree barrier identification presents certain challenges. Noise points in the point cloud need to be eliminated through complex processing such as denoising and registration, which not only takes time but also may affect real-time performance. In addition, point cloud data lacks direct semantic information, so feature extraction and classification methods are needed to identify trees or other obstacles. Existing point cloud processing methods usually rely on traditional geometric feature extraction algorithms, which are difficult to provide accurate and efficient obstacle identification when dealing with complex scenes.

[0005] In recent years, deep learning techniques have made significant progress in image and point cloud recognition. Especially the deep learning method based on convolutional neural network (CNN) can automatically learn and extract complex features in images and point clouds, greatly improving the accuracy and robustness of recognition. The application of deep learning methods in tree barrier recognition still faces some challenges. Although CNN can automatically learn features in data, its processing capacity often depends on a large amount of labeled data, and when dealing with large-scale three-dimensional point clouds, the computational complexity is high, making it difficult to meet the real-time requirements. Moreover, traditional deep learning models usually focus on the processing of single modal data, such as images or point clouds, but in complex environments, single modal often cannot fully express all the information of the target. Therefore, the fusion of multi-modal data, especially the fusion of LiDAR point cloud data and other sensor data (such as RGB images or infrared images), has become the key to improving the accuracy of tree barrier recognition.

[0006] In view of the deficiencies of the prior art, the present patent technology proposes a power line tree barrier recognition and positioning method based on RF-CNN. RF-CNN (Recurrent Feature Convolutional Neural Networks) is a deep learning model that combines the characteristics of convolutional neural network (CNN) and recurrent neural network (RNN). It strengthens the spatio-temporal correlation of features through recurrent structure, and is particularly suitable for processing data with temporal and spatial structure characteristics. SUMMARY

[0007] In view of the above existing problems, the present application effectively solves the deficiencies of the prior art in tree recognition, positioning accuracy, tree barrier risk assessment, etc., improves the efficiency, accuracy and reliability of tree barrier monitoring, and provides strong technical support for intelligent inspection and safety management of power lines.

[0008] To solve the above technical problems, a power line tree barrier recognition and positioning method based on RF-CNN is proposed, which includes, Using a LiDAR system and RGB image acquisition equipment, 3D point cloud data and image data of the power transmission line area were collected, covering different environmental and climatic conditions. Individual data was collected for each type of tree. The collected point cloud data underwent noise removal, sparse point cloud filling, and high-density region reduction. The image data underwent image enhancement preprocessing, and a convolutional neural network was used to extract the visual features of the trees. After preprocessing, the point cloud data and image data were fused using an RF-CNN model to train a multi-task learning model to identify tree species and simultaneously perform 3D localization of tree locations. The model was validated by dividing the dataset and optimizing the model parameters. Tree identification was performed by classifying the fused feature data to identify different types of trees. Trees that did not meet the criteria were filtered out by the confidence value output by the RF-CNN model. The position of each tree in three-dimensional space was located using three-dimensional point cloud data and the visual features of the trees, and the horizontal and vertical distances to the power transmission lines were calculated to assess whether the trees posed a tree obstacle risk. The identification accuracy was evaluated by the accuracy, precision, recall and F1-score indicators. Potential tree obstacles were identified by calculating the spatial relationship between the trees and the power transmission lines, and risk levels were assigned to different trees.

[0009] As a preferred embodiment of the RF-CNN-based tree obstacle identification and localization method for power transmission lines described in this invention, the method of collecting three-dimensional point cloud data and image data of the power transmission line area includes using a LiDAR system to collect information on tree species for n types of trees specified in the power transmission line channel. The LiDAR sensor is configured with 16-line laser scanning, a scanning range of 100 meters, each scanning angle set to 0.5°, and a sampling accuracy of 1cm. Multiple data acquisitions were conducted under different environmental conditions. The LiDAR scanner was activated, and multiple scanning positions were set up within the measurement area for 360° omnidirectional scanning. When scanning at any position, the LiDAR system emitted a laser beam. By measuring the laser reflection time, the distance between each laser beam and the object surface was calculated to generate point cloud data. The point cloud data obtained from all scanning positions were synthesized. Through point cloud registration, the point cloud data from all scanning positions were aligned and fused to generate three-dimensional point cloud data. At the same time, a point cloud file containing spatial coordinate information and additional point intensity values ​​was generated for all scanning positions and timestamps. High-resolution RGB cameras are used for image acquisition. While LiDAR scanning is being performed, the RGB cameras are moved in multiple directions along the power transmission line to capture images of the power transmission line channel. The image resolution is set to 4000*3000, and each image contains tree samples in the scanned area. Each captured image is stored synchronously with the corresponding LiDAR point cloud data, and is accompanied by timestamps and annotation information. All collected point cloud data and image data are manually labeled. The manual labeling includes labeling of point cloud data and image data, generating labeling files, and storing point cloud data, image data and labeling files in different formats. Meanwhile, during the data acquisition process, multiple data acquisitions are performed within the same area, and the acquired data is calibrated and the image is aligned with the point cloud for quality control.

[0010] As a preferred embodiment of the RF-CNN-based power line tree obstacle identification and localization method of the present invention, the preprocessing includes: Preprocessing of point cloud data includes, Noise is removed by eliminating outliers and ground points; Sparsity processing is performed using voxel grid filtering; Image data preprocessing includes, Image denoising, contrast adjustment, color enhancement, and image resizing; Gaussian blur algorithm is used to denoise and smooth the image; Histogram equalization is applied to the image to adjust its sharpness; By adjusting the grayscale distribution of an image, pixel values ​​are evenly distributed across the entire grayscale range, thus enhancing the image's contrast. The image is cropped to select the area containing trees, and the image is uniformly scaled to a size of 224*224. At the same time, the pixel values ​​of the image are normalized and scaled to the range of [0,1].

[0011] As a preferred embodiment of the RF-CNN-based tree obstacle identification and localization method for power transmission lines described in this invention, the extraction of visual features of trees includes: For point cloud feature extraction, specifically, in the processed point cloud data, geometric features of each tree are extracted using PCA, including the tree's height, width, and crown shape. The PCA algorithm is then used to reduce the dimensionality of the point cloud for each tree, calculate the principal orientation of the point cloud data, and extract the main features of the tree. Where X is the point cloud dataset, μ is the mean of the data, cov(X) is the covariance matrix of the main features, M is the total number of pixels in the image, i is the variable index, and T is... The transpose of the matrix, Point cloud data in a point cloud dataset; Perform curvature analysis on point cloud data, calculate the local curvature of each point, and determine the surface curvature of trees; Image feature extraction, specifically, The SIFT algorithm is used to extract local features from RGB images; Use convolutional neural networks to extract global features from images; Images are processed using a pre-trained network to extract visual features at different levels.

[0012] As a preferred embodiment of the RF-CNN-based method for identifying and locating tree obstacles on power transmission lines according to the present invention, the multi-task learning model includes an RF-CNN model combined with a random forest algorithm and a convolutional neural network. Input LiDAR point cloud data and RGB image data. Each sample includes the point cloud data and image data of one tree, and each tree is accompanied by a tree category label. Use the point cloud data to train a random forest model to build a decision tree. Each feature vector corresponding to each tree is used as input, and the output is geometric features. Set the extracted features to any geometric feature of the tree, and perform random forest training: in, is the output of the i-th decision tree, and N is the number of decision trees; An RGB image is input into a convolutional neural network (CNN) to train and output the visual features of trees. The feature vectors output by the random access network (RF) and the CNN are concatenated to generate a joint feature vector, which is then input into a fully connected layer to output the tree classification result. ,in, These are the geometric features output by the RF model. These are the visual features output by the CNN model, concatenated into a vector. For final input; After fusing features, the concatenated feature vectors are input into a fully connected layer for final classification. During model training, cross-validation is used to evaluate the model's performance. Through training and evaluation, a trained RF-CNN model is obtained and used to identify different types of trees.

[0013] As a preferred embodiment of the RF-CNN-based tree obstacle identification and localization method for power transmission lines described in this invention, the identification of different types of trees includes tree identification and classification and confidence assessment. The tree identification and classification includes inputting the joint feature vector obtained by training the RF-CNN model into the classification network, classifying the trees according to the point cloud geometric features and image visual features extracted by the RF-CNN model, using a multilayer perceptron as the classifier, inputting the joint feature vector into the network, and performing nonlinear transformation through multiple fully connected layers, and finally outputting the tree category and identification result. The confidence assessment includes assigning a confidence value to each classification result, representing the RF-CNN model's confidence in the current classification result. When the confidence of the RF-CNN model is below 80%, it is considered a suspected tree barrier and requires further manual verification.

[0014] As a preferred embodiment of the RF-CNN-based method for identifying and locating tree obstacles on power transmission lines according to the present invention, the method for assessing whether a tree constitutes a tree obstacle risk includes: locating the spatial position of the tree using the three-dimensional coordinates of each point in the point cloud data; performing density clustering analysis on the point cloud data to generate a spatial distribution model of the trees; using the DBSCAN algorithm to cluster the point cloud data; calculating the density between points; and automatically identifying point cloud sets belonging to the same tree. in, For point and points The distance between them and and , and and For point and points The three-dimensional coordinates are used to cluster points belonging to the same tree together. The clustered point cloud is used to generate a three-dimensional model of the tree. The precise location of the tree in space is further determined by calculating the distance between the tree and the power transmission line and the vertical distance between the tree height and the power transmission line. The distance between the trees and the power transmission line is calculated. The shortest distance from the tree's centroid to the power transmission line is used to determine whether the trees interfere with the power transmission line. The coordinates of the power transmission line are set as follows: The centroid coordinates of the tree are The shortest distance from the tree to the power transmission line is calculated as follows: in, This indicates the horizontal distance between trees and power transmission lines. When this distance is less than a set threshold of 10 meters, the trees are considered to be tree obstacles. The vertical distance between the tree height and the power transmission line is defined as follows: if the tree height is greater than the safe height of the power transmission line and the vertical distance between the tree and the power transmission line is less than 5 meters, it is considered a high-risk tree obstacle and the tree is treated as such; if the tree height is greater than the safe height of the power transmission line, but the vertical distance between the tree and the power transmission line is greater than 5 meters, it is considered a medium-risk tree obstacle and maintenance personnel are notified to handle it; if the tree height is less than the safe height of the power transmission line, it is considered a low-risk obstacle.

[0015] The present invention provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the RF-CNN-based method for identifying and locating tree obstacles on power transmission lines.

[0016] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the RF-CNN-based method for identifying and locating tree obstacles on power transmission lines.

[0017] The beneficial effects of this invention are as follows: By combining LiDAR data and RGB image data, this invention fully utilizes the advantages of different data sources. LiDAR data provides accurate three-dimensional spatial information of trees, enabling the identification of tree morphology and structure; while RGB images provide color and texture information of trees, enhancing the model's ability to identify tree species. The RF-CNN model can integrate the geometric features of point clouds and the visual features of images to accurately identify the categories of different tree species.

[0018] By training and fusing RF-CNN models, trees can be accurately located in three-dimensional space. The model's positioning error is controlled within 0.5 meters, meeting the accuracy requirements for locating tree obstacles along power transmission lines. This accuracy is crucial for analyzing the spatial relationship between trees and power transmission lines, effectively determining which trees are within danger distance. This method not only identifies tree species and locations but also performs tree obstacle risk assessment based on information such as the distance between trees and power transmission lines and tree height. The relative positions of trees and power transmission lines are monitored in real time, enabling accurate identification of potential tree obstacle hazards and early intervention, thereby reducing the probability of power transmission line accidents.

[0019] Traditional tree obstruction detection requires extensive manual inspections, which is not only costly but also inefficient. By introducing RF-CNN and laser 3D point cloud data fusion technology, the system can automatically identify and locate trees, reducing the workload of manual inspections and improving efficiency. This patented technology can monitor tree growth in real time, promptly detect potential tree obstructions, and avoid oversights by manual inspections. This method enables 24 / 7 automated monitoring, significantly reducing the frequency and expenses of manual inspections. Especially in remote or inaccessible power transmission line areas, the automated detection system can replace manual labor, significantly saving manpower and operating costs.

[0020] By accurately assessing tree-related risks, this method can predict the risk of trees contacting power lines in advance, providing real-time warnings to relevant departments and facilitating timely action, such as clearing trees, pruning, or re-laying lines. This early warning system can prevent large-scale power outages and ensure the safety of power transmission. Early detection of potential tree-related hazards reduces power interruptions and equipment damage caused by fallen trees or branches contacting power lines, thereby ensuring the stability and continuity of power supply and minimizing safety accidents and property losses caused by tree-related obstacles.

[0021] This method is adaptable to different terrains, climates, and tree species, maintaining high recognition accuracy even in diverse environments. By using multimodal data (point clouds and images), the model's adaptability to different tree species is enhanced, enabling it to handle complex tree morphologies and environmental backgrounds. The RF-CNN model exhibits strong robustness, effectively handling complex tree morphologies, tree areas with varying densities, and point cloud data under different climatic conditions, avoiding performance degradation that may occur in traditional methods due to weather changes, lighting differences, or other environmental factors.

[0022] This patented technology framework is highly scalable and can be extended to more tree species or more monitoring areas according to actual needs. With appropriate adjustments and optimizations, the system can be applied to other types of environmental monitoring. This method can adapt to power transmission line monitoring tasks of different scales, from small local lines to large-scale national power grids, and can be flexibly deployed. Furthermore, by combining cloud computing and edge computing technologies, the system can achieve remote real-time monitoring, improving the speed and efficiency of data processing. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 The above is a flowchart of the overall process for identifying and locating tree obstacles on power transmission lines based on RF-CNN, according to an embodiment of the present invention.

[0025] Figure 2 The diagram shows the RF-CNN model architecture of a power transmission line tree obstacle identification and localization method based on RF-CNN, as provided in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of the three-dimensional coordinate positioning error of a tree obstacle identification and positioning method for power transmission lines based on RF-CNN, provided in an embodiment of the present invention. Detailed Implementation

[0027] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0028] Example 1, referring to Figures 1-2 This is the first embodiment of the present invention, which provides a method for identifying and locating tree obstacles on power transmission lines based on RF-CNN, including: S1: Using a lidar system and RGB image acquisition equipment, three-dimensional point cloud data and image data of the power transmission line area are collected respectively. The collected data covers different environmental and climatic conditions, and each type of tree is collected separately.

[0029] When using the RF-CNN-based tree obstacle identification and localization method for power transmission lines to identify and locate tree obstacles in power transmission line corridors, nine common tree species are set for power transmission line corridors (e.g., eucalyptus, camphor, paper mulberry, oak, tung oil tree, cypress, pine, fir, and bamboo). Specifically, LiDAR (Light Detection and Ranging) technology is widely used in geospatial data acquisition, enabling rapid and accurate acquisition of 3D point cloud data. For the identification and location of tree obstructions along power transmission line corridors, LiDAR systems (such as Velodyne VLP-16 or Riegl VZ-4000) were used to scan the corridor area, collecting information on different tree species. The selected locations were within the power transmission line corridor, encompassing areas containing nine common tree species, ensuring that the collected point cloud data covered representative samples of these trees. A LiDAR system was used to collect information on tree species for n types of trees specified in the transmission line corridor. The LiDAR sensor was configured with 16-line laser scanning, a scanning range of 100 meters, a scanning angle of 0.5°, and a sampling accuracy of 1 cm. Multiple data acquisitions are conducted under different environmental conditions. The LiDAR scanner is activated, and multiple scanning positions are set within the measurement area for 360° omnidirectional scanning. When scanning at any position, the LiDAR system emits a laser beam. By measuring the laser reflection time, the distance between each laser beam and the object surface is calculated, generating point cloud data. The point cloud data obtained from all scanning positions are synthesized. Through point cloud registration (such as using the ICP algorithm, Iterative Closest Point), the point cloud data from all scanning positions are aligned and fused to generate three-dimensional point cloud data. At the same time, a point cloud file (usually in .pcd or .las format) is generated for all scanning positions and timestamps, containing spatial coordinate information and additional point intensity values. The coordinates of each point represent the position of a certain point within the scanning area, and the point intensity value reflects the reflection intensity of the laser beam, which helps to identify the reflection characteristics of the object surface.

[0030] It should be noted that a high-resolution RGB camera was used for image acquisition. While the LiDAR was scanning, the RGB camera was moved in multiple directions along the power transmission line to capture images of the power transmission line channel. The image resolution was set to 4000*3000, and each image contained tree samples in the scanned area. Each captured image was stored synchronously with the corresponding LiDAR point cloud data, and included a timestamp and annotation information. It should be noted that all collected point cloud data and image data are manually labeled. The manual labeling includes labeling of point cloud data and image data, generating labeling files, and storing point cloud data, image data and labeling files in different formats. Among them, point cloud data annotation: For the point cloud data of each scan location, the spatial location of the trees is manually labeled and each tree is assigned a unique ID; It should be noted that the image data annotation is as follows: For each RGB image, the bounding boxes of the trees are marked, and a corresponding classification label is assigned to each tree. The classification labels include nine common tree types to train the model for multi-class recognition.

[0031] Meanwhile, during the data acquisition process, multiple data acquisitions are performed within the same area, and the acquired data is calibrated and the image is aligned with the point cloud for quality control.

[0032] In this process, multiple data collections were conducted under different environments (sunny days, cloudy days, smog, rainy days, etc.) to ensure the diversity of the dataset and improve the generalization ability of the model; the LiDAR scanner was periodically calibrated using known standard objects to ensure that the acquired point cloud data had sufficient accuracy.

[0033] S2: The collected point cloud data is subjected to noise removal, sparse point cloud filling and high-density region simplification. The image data is preprocessed for image enhancement. The visual features of trees are extracted using a convolutional neural network.

[0034] Point cloud data and RGB image data acquired from LiDAR are cleaned, denoised, and standardized, and useful features are extracted. These features will help the model understand the spatial distribution, texture features, and local structure of trees in the image, thereby improving the accuracy of recognition and localization.

[0035] Furthermore, ① preprocessing of the point cloud data is necessary. Point cloud data often contains noise and irrelevant information, which can affect subsequent feature extraction and model training. To improve the quality of point cloud data, noise removal and sparsification are performed, including outlier removal and ground point removal. Noise removal is achieved by removing outliers and ground points. Specifically, in point cloud data, outliers are typically caused by measurement errors, poorly reflecting objects, or sensor malfunctions. A statistical outlier removal algorithm is used. For each point, the average distance to its 10 nearest neighbors is calculated; if a point's distance is significantly greater than the average distance of its neighbors, it is considered an outlier. These outliers are then removed by setting a threshold. in, It is a point To its nearest neighbor distance, is the number of neighboring points, and j is the variable index; Ground near power transmission lines often interferes with tree detection, necessitating the removal of ground points from point cloud data. Ground point extraction algorithms (such as RANSAC) are used to identify and remove ground points. RANSAC (Random Consensus Sample Consensus) continuously selects a set of points for line or plane fitting to find the best-fitting planar model (i.e., the ground) and removes the ground points.

[0036] Voxel grid filtering is used for sparsification. Specifically, for regions with high point cloud density, voxel grid filtering is used for sparsification. By setting the size of the voxel grid, the point cloud data within each grid is represented by the center point of that grid, thereby reducing the number of points and improving computational efficiency. The point cloud data is divided into voxels of fixed size, and the point cloud within each voxel represents the region by calculating its center point. The sparsified point cloud still retains the main geometric structure of the trees. Where V is the point set within a voxel. As a representative point of voxels, The average coordinates of all points within the voxel are denoised and sparsified. After the above denoising and sparsification processes, the point cloud data will be cleaner and the spatial information of each point will be more representative, which will help with the subsequent extraction and classification of tree morphological features.

[0037] ② Preprocess the image data, including image denoising, contrast adjustment, color enhancement, and image size unification; It should be noted that Gaussian blur is used for image denoising and smoothing. Specifically, Gaussian blur can effectively remove noise from an image while preserving its overall structure.

[0038] in, Let g and h be the Gaussian function, where g and h are the horizontal and vertical offsets of any point in the Gaussian kernel relative to the center point. The standard deviation is used. Convolution operations are employed to smooth the image and remove noise. It should be noted that histogram equalization of the image adjusts the sharpness, effectively improving the saliency of trees in the image, making it easier for the model to identify trees. Specifically, by adjusting the grayscale distribution of the image, the pixel values ​​are evenly distributed across the entire grayscale range, enhancing the image contrast. : in, Here, M represents the grayscale value of the image, and M represents the total number of pixels in the image. It should be noted that the image was cropped to select the area containing trees, and the image was uniformly scaled to a size of 224*224. At the same time, the pixel values ​​of the image were standardized and scaled to the range of [0, 1].

[0039] Furthermore, for point cloud feature extraction, specifically, in the processed point cloud data, geometric features of each tree are extracted using PCA (Principal Component Analysis), including the tree's height, width, and crown shape. The PCA algorithm is then used to reduce the dimensionality of the point cloud for each tree, calculate the principal orientation of the point cloud data, and extract the main features of the tree. Where X is the point cloud dataset, μ is the mean of the data, cov(X) is the covariance matrix of the main features, M is the total number of pixels in the image, i is the variable index, and T is... The transpose of the matrix, Point cloud data in a point cloud dataset; It should be noted that performing curvature analysis on point cloud data and calculating the local curvature of each point helps to identify the shape of tree branches and trunks. Specifically, by calculating the normal direction and curvature value of the neighborhood around each point, the surface curvature of the tree can be determined; the trunk of a tree usually has a lower curvature value, while the branches and leaf areas have a higher curvature value. It should be noted that image feature extraction specifically involves, ①SIFT Feature Extraction: The SIFT (Scale Invariant Feature Transform) algorithm is used to extract local features from RGB images. SIFT can stably extract feature points at different scales and rotation angles, which is crucial for the recognition of tree texture and local morphology. The SIFT algorithm finds key points with high contrast in the image and extracts descriptors through gradient information in local regions. These descriptors can effectively represent the texture features of trees.

[0040] ② Convolutional Feature Extraction: Using traditional convolutional neural networks (CNNs) to extract global features of the image, such as the color and shape of trees. Using pre-trained networks (such as VGG16 or ResNet50) to process the image and extract higher-level visual features.

[0041] S3: After preprocessing the point cloud data and image data, feature fusion is performed using the RF-CNN model to train a multi-task learning model to identify tree species and simultaneously perform 3D localization of tree locations. During the training process, the dataset is divided for model validation and model parameters are optimized.

[0042] Specifically, the RF-CNN model combines the Random Forest algorithm and a convolutional neural network. The RF module uses the Random Forest algorithm to extract geometric features from point cloud data, including tree height, width, curvature, and density distribution. The Random Forest reduces the risk of overfitting and improves classification accuracy by building multiple decision trees.

[0043] CNN module: Uses convolutional neural networks (such as ResNet or VGG16) to process RGB images, extracting texture features, color features, and local structural features of trees. Stacking convolutional and pooling layers helps capture the complex morphology of trees.

[0044] Data fusion layer: This layer fuses the features extracted by RF and CNN to improve the accuracy of tree recognition and localization. A simple feature-level fusion method is used, which involves concatenating the feature vectors output by RF and CNN, and then performing joint classification through a fully connected layer.

[0045] Furthermore, LiDAR point cloud data and RGB image data are input, with each sample including point cloud data and image data of a single tree, and each tree accompanied by a tree category label. A random forest model is trained using the point cloud data to build a decision tree. Each feature vector corresponding to each tree is used as input, and the output is geometric features (such as tree height, branch thickness, curvature, etc.). The extracted features are set to any geometric feature of the tree, and random forest training is performed. in, This is the output of the i-th decision tree, where N is the number of decision trees. These are the geometric features output by the RF model; RGB images are input into a convolutional neural network, trained and output by the CNN to extract visual features of trees (such as tree texture, color, shape, etc.). A pre-trained convolutional neural network (such as ResNet50 or VGG16) is used to extract visual features from RGB images. Through multi-layer convolution and pooling operations, the CNN can extract important texture, color and shape information from the image. Each RGB image is input into a convolutional neural network, and transfer learning is used to extract the visual features of trees by fine-tuning a pre-trained network (such as an ImageNet pre-trained model). The features extracted by the convolutional layers are then passed through fully connected layers to output a high-dimensional feature vector, which represents the morphological features of the tree.

[0046] Next, the feature vectors output by the RF and CNN are concatenated to generate a joint feature vector (assuming the feature dimension of the RF output is 100, the feature dimension of the CNN output is 512, and the dimension of the concatenated feature vector is 612). This joint feature vector is then input into a fully connected layer to output the tree classification result. ,in, These are the geometric features output by the RF model. These are the visual features output by the CNN model, concatenated into a vector. For final input; After fusing the features, the concatenated feature vector is input into a fully connected layer for final classification.

[0047] It should also be noted that during model training, cross-validation is used to evaluate the model's performance. Through training and evaluation, a trained RF-CNN model is finally obtained, which is then used to identify different types of trees. The specific steps are as follows: The collected data on nine tree species were divided into training and testing sets. An 80% training set and a 20% testing set ratio was used to ensure sufficient training data.

[0048] Training was performed using the standard Adam optimizer, with the cross-entropy loss function employed. It can effectively handle multi-class classification problems: in, For the number of categories, For actual labels, This represents the probability predicted by the model.

[0049] The performance of the model is evaluated using metrics such as accuracy, precision, recall, and F1 score. : To accurately predict the number, The total number of predictions is used to train and evaluate the model, resulting in a trained RF-CNN model that can efficiently identify and locate different types of trees.

[0050] In summary, the RF-CNN (Recurrent Fusion Convolutional Neural Network) model is used to deeply fuse laser 3D point cloud data with RGB image data. This fusion method does not rely on a single data source but improves the accuracy of tree identification and localization by leveraging the complementary advantages of multimodal data.

[0051] Laser point cloud data provides precise three-dimensional spatial information of trees, effectively describing their geometric morphology, such as height, width, and crown characteristics. RGB image data provides visual features of trees, including appearance, color, and texture, enhancing the accuracy of tree species classification.

[0052] Through this multimodal data fusion, the model can not only identify tree morphology from point clouds but also identify tree species by combining images, thereby improving recognition accuracy, especially when tree species are complex and similar. This data fusion approach solves the problem of traditional methods' excessive reliance on a single data source, improving the model's accuracy and robustness.

[0053] Traditional convolutional neural networks (CNNs) are typically limited to processing two-dimensional image data, while RF-CNN, by introducing a recursive fusion mechanism, can better process and fuse features from different data sources (such as point clouds and images). The RF-CNN model can progressively optimize the fusion of various features through recursive modules, thereby improving the accuracy of classification and regression tasks.

[0054] RF-CNN not only integrates data from different modalities when processing tree recognition tasks, but also plays a role in tree localization, tree obstacle detection, and risk assessment, making the model more efficient and accurate in multi-task learning. This innovative deep learning model greatly improves the multi-task processing capability of tree recognition and localization tasks, especially in situations with diverse trees and complex environments, significantly improving recognition accuracy and localization accuracy.

[0055] S4: Tree identification classifies different types of trees by fusing feature data. The confidence value output by the RF-CNN model filters out trees that do not meet the conditions. Using 3D point cloud data and the visual features of trees, the position of each tree in 3D space is located, and the horizontal and vertical distances to the power transmission lines are calculated to assess whether the trees pose a tree obstacle risk.

[0056] Furthermore, the identification of different types of trees includes tree identification, classification, and confidence assessment; The tree identification and classification includes inputting the joint feature vector obtained by training the RF-CNN model into the classification network, classifying the trees according to the point cloud geometric features and image visual features extracted by the RF-CNN model, using a multilayer perceptron as the classifier, inputting the joint feature vector into the network, and performing nonlinear transformation through multiple fully connected layers, and finally outputting the tree category and identification result. The confidence assessment includes assigning a confidence value to each classification result, representing the RF-CNN model's confidence in the current classification result. When the confidence of the RF-CNN model is below 80%, it is considered a suspected tree barrier and requires further manual verification.

[0057] S5: The accuracy of identification is evaluated by the accuracy, precision, recall and F1-score indicators. Potential tree obstacles are identified by calculating the spatial relationship between trees and power transmission lines, and risk levels are assigned to different trees.

[0058] Furthermore, the spatial location of trees is located using the three-dimensional coordinates of each point in the point cloud data. Density clustering analysis is performed on the point cloud data to generate a spatial distribution model of the trees. The DBSCAN algorithm is used to cluster the point cloud data, calculate the density between points, and automatically identify point cloud sets belonging to the same tree. in, For point and points The distance between them and and , and and For point and points The three-dimensional coordinates are used to cluster points belonging to the same tree together. The clustered point cloud is used to generate a three-dimensional model of the tree. The precise location of the tree in space is further determined by calculating the distance between the tree and the power transmission line and the vertical distance between the tree height and the power transmission line. The distance between the trees and the power transmission line is calculated. The shortest distance from the tree's centroid to the power transmission line is used to determine whether the trees interfere with the power transmission line. The coordinates of the power transmission line are set as follows: The centroid coordinates of the tree are The shortest distance from the tree to the power transmission line is calculated as follows: in, This indicates the horizontal distance between trees and power transmission lines. When this distance is less than a set threshold of 10 meters, the trees are considered to be tree obstacles. The vertical distance between the tree height and the power transmission line is defined as follows: if the tree height is greater than the safe height of the power transmission line and the vertical distance between the tree and the power transmission line is less than 5 meters, it is considered a high-risk tree obstacle and the tree is treated as such; if the tree height is greater than the safe height of the power transmission line, but the vertical distance between the tree and the power transmission line is greater than 5 meters, it is considered a medium-risk tree obstacle and maintenance personnel are notified to handle it; if the tree height is less than the safe height of the power transmission line, it is considered a low-risk tree obstacle. In summary, this method enables precise calculation of the three-dimensional spatial relationship between trees and power transmission lines, particularly in the identification and localization of tree obstacles. By utilizing the features of 3D point cloud data and image fusion, this method can not only accurately locate trees but also calculate the relative positions of trees and power transmission lines in real time.

[0059] By precisely calculating the horizontal distance and vertical height difference between trees and power transmission lines, this method can assess whether trees pose a tree barrier risk. This precise risk assessment mechanism ensures that potential hazards between trees and power transmission lines can be identified and warned in advance, preventing power accidents caused by fallen trees or branches contacting power lines. This innovation goes beyond tree identification in tree barrier monitoring, further improving the accuracy and real-time nature of tree risk assessment, thereby effectively preventing potential safety hazards to power transmission lines.

[0060] Compared to existing technologies, the innovation of this patented technology lies in its breakthrough of the limitations of traditional tree obstacle identification methods. By fusing RF-CNN models with laser 3D point cloud data, it can more effectively cope with complex environmental changes, improving identification accuracy and real-time performance. Leveraging the advantages of RF-CNN models in multimodal data fusion, this patented technology can fully utilize the complementarity of LiDAR point cloud data and other sensor data, thereby achieving more efficient and accurate tree obstacle identification and positioning, ensuring the safe operation of power transmission lines. It not only possesses strong innovation but also effectively solves various problems in existing technologies, demonstrating broad application prospects.

[0061] Example 2, as Figure 3 As shown, this is the second embodiment of the present invention, which provides a method for identifying and locating tree obstacles on power transmission lines based on RF-CNN. To verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0062] During model training, the performance of tree recognition and localization was evaluated using metrics such as accuracy, precision, recall, and F1-score, yielding the following results: (1) Accuracy of tree species identification The performance of the RF-CNN model in identifying different tree species is shown in Table 1 after experiments. Table 1. Performance of the RF-CNN model in identifying different tree species.

[0063] Among all nine tree species, the recognition accuracy of oak and eucalyptus was the highest, reaching over 95%, indicating that the geometric and visual features of these trees are relatively obvious and easy for the model to recognize.

[0064] The identification accuracy of cedar trees is slightly lower, at about 89.9%. This may be related to the fact that cedar trees have similar appearances and high growth density, making it more difficult to distinguish them from other tree species.

[0065] The precision and recall rates for tung trees and Chinese arborvitae were relatively balanced, indicating that the recognition performance of these trees was relatively stable.

[0066] (2) Error in tree location and three-dimensional coordinate positioning like Figure 3 Regarding tree positioning, the main evaluation focused on the errors in three-dimensional coordinate positioning, including horizontal positioning error (XY plane) and vertical positioning error (Z axis). Table 2 shows the positioning errors for different trees: Table 2. Positioning errors of different trees

[0067] The localization error for most trees is between 0.3 meters and 0.5 meters, indicating that the RF-CNN model has high accuracy in the three-dimensional spatial localization of trees.

[0068] The localization errors for cedar and paper mulberry trees are relatively large, especially in the vertical direction (Z-axis). This may be related to the shape of the tree crowns and the density of the point cloud data, leading to some deviation in the model's localization in the vertical direction.

[0069] (3) Accuracy of tree barrier risk assessment In terms of tree barrier risk assessment, the distance between trees and power transmission lines, tree height, and relative position of trees and power transmission lines are used to assess whether a tree poses a tree barrier risk. Table 3 shows the tree barrier risk assessment results for nine types of trees (assessment thresholds are 10 meters horizontal distance and 5 meters vertical distance): Table 3. Results of Tree Barrier Risk Assessment

[0070] The tree barrier risk identification for oak and eucalyptus trees is highly accurate, accurately identifying which trees constitute tree barriers and effectively filtering out trees that pose no risk.

[0071] The tree barrier risk identification rate for cedar and paper mulberry is slightly lower, indicating that these trees have more complex crown morphologies and are easily misjudged as low-risk trees.

[0072] The experimental conclusions are as follows: High accuracy in tree species identification: The RF-CNN model performs well in tree species identification, with an accuracy of over 90% for most tree species, especially for eucalyptus and oak, where the identification accuracy is relatively high.

[0073] Low positioning error: The three-dimensional positioning error of trees is small, and the horizontal and vertical positioning errors of most trees are controlled within 0.5 meters, which meets the accuracy requirements for tree obstacle identification of power transmission lines.

[0074] Effectiveness of tree obstacle risk assessment: Spatial relationship analysis between trees and power transmission lines effectively identified potential tree obstacles, especially with high accuracy in identifying high-risk trees.

[0075] The influence of tree species and environmental factors: The accuracy of tree species identification is significantly affected by tree morphological characteristics, point cloud data quality, and environmental factors. For some dense and complex tree species (such as fir and paper mulberry), the model faces certain challenges in terms of identification and localization accuracy.

[0076] In summary, the tree obstacle identification and localization method for power transmission lines based on the fusion of RF-CNN and laser 3D point cloud data can effectively identify tree species, locate tree positions, and accurately assess the risk of tree obstacles, providing strong technical support for the safety inspection and tree obstacle prevention of power transmission lines.

[0077] Example 3, the third embodiment of the present invention, differs from the previous two embodiments in that: If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0079] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0080] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any of the following techniques known in the art and combinations thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

Claims

1. A method for transmission line tree barrier identification and positioning based on RF-CNN, characterized in that: The application relates to a tree obstacle risk identification method based on point cloud and image fusion. Three-dimensional point cloud data and image data of a power transmission line area are collected by using a laser radar system and an RGB image acquisition device, the collected data cover different environmental and climate conditions, and each kind of tree is collected separately; Noise removal, sparse point cloud filling and high-density area simplification are performed on the collected point cloud data, image enhancement preprocessing is performed on the image data, and a convolutional neural network is used to extract visual features of the trees; After preprocessing of the point cloud data and the image data, feature fusion is performed through an RF-CNN model, a multi-task learning model is trained to identify the tree species, and three-dimensional positioning of the tree position is simultaneously performed, data sets are divided during the training process to verify the model, and model parameters are optimized; The tree identification is classified through the fused feature data, different types of trees are identified, a confidence value is output by the RF-CNN model to filter the trees that do not meet the conditions, the three-dimensional point cloud data and the visual features of the trees are used to locate the position of each tree in the three-dimensional space, and the horizontal and vertical distances between the trees and the power transmission line are calculated to evaluate whether the trees constitute a tree obstacle risk; The identification accuracy is evaluated through accuracy, precision, recall and F1-score indexes, potential tree obstacles are calculated through the spatial relationship between the trees and the power transmission line, and risk levels are assigned to different trees.

2. The RF-CNN-based transmission line tree obstacle identification and positioning method of claim 1, wherein: The three-dimensional point cloud data and the image data of the power transmission line area are collected respectively, the LiDAR system is used to collect information containing tree species of n types of trees in the power transmission line channel, the LiDAR sensor is configured as a 16-line laser scanner, the scanning range is 100 meters, each scanning angle is set as 0.5 degrees, and the sampling accuracy is 1 cm; Multiple data collection is performed under different environmental conditions, the LiDAR scanner is started, multiple scanning positions are set in the measurement area for 360-degree omnidirectional scanning, when any position is scanned, the LiDAR system emits a laser beam, the distance between each laser beam and the surface of an object is calculated by measuring the reflection time of the laser, point cloud data is generated, the point cloud data obtained from all scanning positions is synthesized, the point cloud data of all scanning positions is aligned and fused through point cloud registration, three-dimensional point cloud data is generated, and a point cloud file is generated for all scanning positions and time stamps, containing spatial coordinate information and additional point intensity values; High-resolution RGB cameras are used for image acquisition, the RGB cameras are moved in multiple directions along the power transmission line to shoot images of the power transmission line channel while the LiDAR is scanning, the image resolution is set as 4000*3000, and each image contains tree samples in the scanning area, each shot image is stored synchronously with corresponding LiDAR point cloud data, and each image is attached with a time stamp and annotation information; All collected point cloud data and image data are manually annotated, the manual annotation content includes point cloud data annotation and image data annotation, an annotation file is generated, and the point cloud data, the image data and the annotation file are stored in different formats respectively; Meanwhile, multiple acquisitions in the same area, calibration of the acquired data, and quality control of image and point cloud alignment are performed during the data acquisition process. 3.The method of claim 2, wherein: The preprocessing includes, The preprocessing of the point cloud data includes, Noise removal by removing outliers and ground points; Sparse processing using voxel grid filtering; The preprocessing of the image data includes, Denoising, contrast adjustment, color enhancement, and image size unification of the image; Gaussian blur algorithm is used to denoise and smooth the image; Histogram equalization is performed to adjust the clarity of the image; The gray scale distribution of the image is adjusted so that the pixel values are uniformly distributed within the entire gray scale range, enhancing the contrast of the image; The image is cropped to select the area containing the trees, and the image is uniformly scaled to 224*224 size. The pixel values are standardized to the range [0, 1].

4. The RF-CNN-based transmission line tree obstacle identification and positioning method of claim 3, wherein: The visual feature extraction of the trees includes, Point cloud feature extraction, specifically, in the processed point cloud data, PCA is used to extract the geometric features of each tree, including the height, width, and shape of the tree crown. PCA algorithm is used to reduce the dimensionality of each tree's point cloud, calculate the main direction of the point cloud data, and extract the main features of the tree: where X is the point cloud dataset, μ is the mean of the data, cov(X) is the covariance matrix of the principal components, M is the total number of pixels of the image, i is the index of the variable, T is the transpose matrix, is the point cloud data in the point cloud dataset;​ Curvature analysis of the point cloud data is performed to calculate the local curvature of each point and judge the surface curvature of the tree. Image feature extraction, specifically, SIFT algorithm is used to extract local features from RGB images; Convolutional neural network is used to extract global features of the image; A pre-trained network is used to process the image and extract visual features at different levels.

5. The RF-CNN-based transmission line tree obstacle identification and positioning method of claim 4, wherein: The multi-task learning model includes an RF-CNN model combining random forest algorithm and convolutional neural network, LiDAR point cloud data and RGB image data are input, each sample includes point cloud data and image data of a tree, and each tree is accompanied by a tree class label. Point cloud data is used to train a random forest model to build a decision tree. Each feature vector corresponding to each tree is input, and the output is geometric features. Set the extracted features as any geometric features of the tree, and perform random forest training: wherein, is the output of the i-th decision tree, and N is the number of decision trees. The RGB image is input into a convolutional neural network, trained and output CNN extracts visual features of the tree, the feature vectors output by RF and CNN are spliced to generate a joint feature vector, and the joint feature vector is input into a fully connected layer to output the classification result of the tree wherein, is a geometric feature output by the RF model, is a visual feature output by the CNN model, and the spliced vector is the final input After fusing the features, the concatenated feature vector is input into a fully connected layer for final classification. During model training, cross-validation is used to evaluate the performance of the model. Through training and evaluation, a trained RF-CNN model is obtained, and different types of trees are identified.

6. The RF-CNN-based transmission line tree obstacle identification and positioning method of claim 5, wherein: The identification of different types of trees includes tree identification classification and confidence evaluation; The tree identification classification includes inputting the joint feature vector obtained by training the RF-CNN model into the classification network, classifying the trees according to the point cloud geometric features and image visual features extracted by the RF-CNN model, using a multi-layer perceptron as a classifier, passing the joint feature vector into the network, and performing nonlinear transformation through multiple fully connected layers to finally output the tree class and identification result. The confidence evaluation includes that each classification result is accompanied by a confidence value representing the confidence of the RF-CNN model for the current classification result, and when the confidence of the RF-CNN model is lower than 80%, it is regarded as a suspected tree barrier and requires manual further verification.

7. The RF-CNN-based transmission line tree obstacle identification and positioning method of claim 6, wherein: The method for evaluating whether the tree constitutes a tree barrier risk comprises locating the spatial position of the tree by using the three-dimensional coordinates of each point in the point cloud data, performing density clustering analysis on the point cloud data to generate a spatial distribution model of the tree, using a DBSCAN algorithm to cluster the point cloud data, calculating the density between points, and automatically identifying the point cloud set belonging to the same tree: wherein, is the distance between points and points , and and , and and is the three-dimensional coordinate of point and point , by calculating the distance between all points, clustering the points belonging to the same tree together, generating a three-dimensional model of the tree from the clustered point cloud, and further determining the accurate position of the tree in space by calculating the distance between the tree and the power transmission line and the vertical distance between the tree height and the power transmission line.

8. The RF-CNN-based transmission line tree obstacle identification and positioning method of claim 7, wherein: The distance between the tree and the power transmission line is calculated, the shortest distance between the gravity center of the tree and the power transmission line is calculated to determine whether the tree interferes with the power transmission line, and the coordinates of the power transmission line are set as , the gravity center coordinates of the tree are , and the shortest distance between the tree and the power transmission line is calculated as wherein, represents the horizontal distance of the tree from the power transmission line, and when this distance is less than a set threshold of 10 meters, the tree is considered to be a tree barrier; The vertical distance between the height of the tree and the power transmission line is that when the height of the tree is greater than the safe height of the power transmission line and the vertical distance between the tree and the power transmission line is less than 5 meters, the tree is determined as a high-risk tree barrier and is regarded as a tree barrier; when the height of the tree is greater than the safe height of the power transmission line, but the vertical distance between the tree and the power transmission line is greater than 5 meters, the tree is determined as a medium-risk tree barrier and a maintenance personnel is notified to handle; when the height of the tree is lower than the safe height of the power transmission line, the tree is determined as a low-risk. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-7. The processor executes the computer program to realize the steps of the RF-CNN-based power transmission line tree barrier identification and positioning method in any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the RF-CNN-based power transmission line tree barrier identification and positioning method in any one of claims 1 to 7.

Images