Power transmission line tree obstacle distance measurement method based on binocular vision and deep learning fusion

Abstract

The invention discloses a power transmission line tree obstacle distance measurement method based on binocular vision and deep learning fusion, and relates to the technical field of electric power, and the method comprises the steps: collecting a binocular image, carrying out the calibration and stereo correction, and carrying out the single-scale Retinex enhancement processing of the corrected image; and performing power transmission line and tree obstacle region segmentation on the enhanced image based on a DeepLabv3 + model in combination with a coordinate attention mechanism, and performing semantic-guided parallax estimation and optimization on the enhanced image based on a segmentation result to obtain an optimized parallax map. According to the method, a multi-level feature map is extracted through Res2Net-50, target area positioning is enhanced in combination with a coordinate attention mechanism, an ASPP module is used for fusing multi-scale context information, the segmentation precision of a power transmission line and a tree barrier is improved, a Kalman filter for adaptively adjusting observation noise variance is used for processing multi-frame distance measurement data, and the accuracy of distance measurement is improved. And the stability and the reliability of distance measurement are improved.

Description

Technical Field

[0001] This invention relates to the field of power technology, and in particular to a method for measuring tree obstacles on power transmission lines based on the fusion of binocular vision and deep learning. Background Technology

[0002] Monitoring the distance between power transmission lines and nearby trees is a crucial measure to prevent power outages caused by wind-induced discharge, tree encroachment, or tree fall. If trees are blown off course by the wind, grow naturally, or fall close to the conductors, they can trigger discharges or even power outages. Therefore, accurately measuring the distance between power transmission lines and trees is essential. Traditional monitoring techniques rely on manual inspections or lidar ranging, but the former is inefficient and risky, while the latter is expensive and difficult to deploy on a large scale using drones. In recent years, with the widespread adoption of drone inspections, visual ranging methods based on binocular cameras have gained attention due to their low cost and high versatility. This method reconstructs the three-dimensional structure of the scene by analyzing the parallax between the left and right camera images, thereby calculating the spatial distance between the power transmission lines and tree obstacles.

[0003] Although existing technologies have made some progress in ranging distances from tree obstacles along power transmission lines, significant shortcomings remain. First, current technologies directly perform stereo matching on the original images, failing to adequately consider the complex lighting conditions during power transmission line inspections (such as backlighting and low illumination at dawn and dusk). This results in inconsistent brightness and contrast between the left and right images, undermining the fundamental premise of stereo matching. Second, even when image segmentation is used to identify power transmission lines and tree-obstructed areas, blurry segmentation boundaries or missed detections cannot effectively improve ranging reliability. Therefore, existing binocular vision tree obstacle ranging technologies have significant deficiencies in image quality improvement, accurate target identification, and robust ranging calculation. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a method for measuring tree obstacles on power transmission lines based on the fusion of binocular vision and deep learning, which solves the problems of image quality improvement, accurate target recognition and robust distance measurement calculation.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for measuring tree obstacles on power transmission lines based on the fusion of binocular vision and deep learning, comprising, Binocular images were acquired and calibrated and stereo corrected, and the corrected images were then subjected to single-scale Retinex enhancement processing. Based on the DeepLabv3+ model combined with the coordinate attention mechanism, the enhanced image is segmented into power line and tree obstacle regions. Based on the segmentation results, the enhanced image is semantically guided to estimate and optimize disparity, and an optimized disparity map is obtained. Based on the optimized disparity map, 3D point cloud reconstruction and target point set extraction are performed. The target point set is fitted to the guide space curve and the minimum distance from the tree obstacle point to the curve is calculated. Kalman filtering is used to obtain the ranging result.

[0007] As a preferred embodiment of the transmission line tree obstacle ranging method based on binocular vision and deep learning fusion described in this invention, the single-scale Retinex enhancement processing of the correction image refers to using an image adaptive enhancement method based on the single-scale Retinex algorithm to enhance two images, obtaining left and right enhanced images respectively. and .

[0008] As a preferred embodiment of the transmission line tree obstacle ranging method based on binocular vision and deep learning fusion described in this invention, wherein: the segmentation of the transmission line and tree obstacle regions in the enhanced image based on the DeepLabv3+ model combined with the coordinate attention mechanism refers to the left enhanced image... The input is fed into the Res2Net-50 network structure, where feature maps from three layers are extracted sequentially, and a coordinate attention mechanism is used to move along the deep semantic feature maps. Global average pooling is performed in both the horizontal and vertical directions to obtain two receptive vectors. These two receptive vectors are then input into a one-dimensional convolution operation to obtain attention weights. Finally, these attention weights are combined with the deep semantic feature map. The enhanced feature map is obtained by fusion and then input into the ASPP module. The feature map corresponding to each branch in the ASPP module is obtained and then concatenated to obtain a multi-scale context information aggregation feature map. Aggregate multi-scale contextual information into feature maps The upsampled feature map is obtained by bilinear interpolation upsampling. With shallow feature map Obtain by channel splicing The data is then fed into a 3×3 convolutional layer, where features from different sources are integrated through feature learning to output the final semantic segmentation feature map. For pixel-level classification of corresponding category labels, a multi-task loss function is used to optimize the network during the semantic segmentation model training phase. After the model training is completed, the final semantic segmentation feature map is analyzed. Softmax activation is performed, and the image is upsampled to its original resolution using bilinear interpolation to obtain the class probability distribution for each pixel location. The class with the highest probability value at each pixel coordinate is selected as the final class for that pixel. When the class label output M(p,q) equals 1, the pixel belongs to the transmission line area, and the mask is... Otherwise, it is 0, indicating that the pixel belongs to a tree barrier region, and the mask is 0. .

[0009] As a preferred embodiment of the transmission line tree obstacle ranging method based on binocular vision and deep learning fusion described in this invention, wherein: the disparity estimation and optimization of the enhanced image based on the segmentation results to obtain an optimized disparity map refers to the left and right enhanced images and Using the input image, a semi-global matching algorithm is used to calculate disparity and obtain an initial disparity map. and the transmission line area mask and tree barrier area mask As input for the region constraint, the optimized disparity map is obtained through the optimization process of the region constraint.

[0010] As a preferred embodiment of the transmission line tree obstacle ranging method based on binocular vision and deep learning fusion described in this invention, wherein: the 3D point cloud reconstruction and target point set extraction based on the optimized disparity map refers to the process based on the optimized disparity map. By combining the camera's internal and external geometric parameters, triangulation is performed on corresponding pixels in the left and right views. Based on the triangulation, a complete 3D point cloud dataset of the scene is obtained pixel by pixel. Voxel filtering is then used to smooth the 3D point cloud dataset, resulting in smooth and structurally consistent point cloud data. And according to the transmission line area mask and tree barrier area mask Point cloud data The point sets of power lines and tree obstacles are obtained by classifying and extracting them according to the semantic tags of the corresponding pixels.

[0011] As a preferred embodiment of the transmission line tree obstacle ranging method based on binocular vision and deep learning fusion described in this invention, wherein: the step of fitting the conductor space curve through the target point set and calculating the minimum distance from the tree obstacle point to the curve refers to the point set of the transmission line. The points are sorted according to the main direction of the transmission line in space. The sorted point set is then uniformly sampled at fixed intervals to obtain a sampled node set. A cubic spline interpolation method is then used to fit the sampled node set to a curve, resulting in a continuous spatial curve for the transmission line. This process is then applied to the tree obstacle point set. The curve of each tree obstacle point in the calculation is then used to determine the transmission line curve. The shortest Euclidean distance is calculated based on the shortest Euclidean distance. The set of shortest distances is obtained, and the minimum value is selected from the set of shortest distances of all tree obstacle points as the minimum spatial distance between the tree obstacle and the transmission line in the current frame.

[0012] As a preferred embodiment of the transmission line tree obstacle ranging method based on binocular vision and deep learning fusion described in this invention, wherein: obtaining the ranging result using Kalman filtering refers to obtaining the minimum spatial distance between the tree obstacle and the transmission line. Then, a one-dimensional Kalman filter is used to perform temporal smoothing on the multi-frame ranging results. During the operation, the ranging state variable is set as follows: , representing the true distance estimate at time t, where the distance observation value is... The state equation and observation equation of the Kalman filter are defined separately. Before each frame update, the stable distance estimate output after the Kalman filter update at the previous time step is obtained. and the corresponding estimation error covariance Construct a sliding window containing ranging observations from the current frame and the previous n frames, and calculate the standard deviation based on the sliding window. When the standard deviation... satisfy At that time, among them, If it is an empirical coefficient, then it is determined that a sudden change in the real state has occurred, and the model parameters are reset. When the standard deviation... satisfy If the state is considered continuous and stationary, standard prediction is performed using default parameters. The Kalman filter prediction step is executed, and the observation residual of the detection frame at time t is calculated, based on the prediction error covariance. Calculate the covariance corresponding to the observation residuals; Set confidence threshold When satisfied At that time, among them, It is the observation residual. If the variance is the observation residual covariance, it is determined that the current observation is disturbed. An adaptive update is performed on the observation noise variance. Based on the judgment of the predicted value and the observation reliability, the fusion weighting coefficient is calculated, and then the fusion weighting coefficient is used... The predicted values ​​are fused with the original observations to generate a one-step prediction fusion result, keeping the original observation noise variance unchanged. Based on the current ranging observation value k(t), the Kalman gain is calculated, and the fused prediction value is used as the basis for the result. The state estimate and covariance are updated with Kalman gain K(t).

[0013] As a preferred embodiment of the transmission line tree obstacle ranging method based on binocular vision and deep learning fusion described in this invention, the following steps are included: acquiring binocular images and performing calibration and stereo correction: Before transmission line inspection, the binocular cameras are calibrated using the Zhang Zhengyou calibration method to obtain the intrinsic parameter matrix and distortion parameters of each camera. Through the calibration process, the extrinsic parameter matrix between the left and right cameras is obtained. During transmission line inspection, a synchronous binocular visible light camera system simultaneously acquires visible light images from the left and right perspectives. The remapping matrix required for stereo correction is calculated using the extrinsic parameter matrix. Stereo correction is then performed on the left and right images to obtain the corrected left and right images. and .

[0014] In a second aspect, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the transmission line tree obstacle ranging method based on binocular vision and deep learning fusion as described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the transmission line tree obstacle ranging method based on binocular vision and deep learning fusion as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: This invention uses a Res2Net-50 / DeepLabv3+ segmentation network with a coordinate attention mechanism to accurately extract slender wires and irregular tree obstacle regions, generate a high-confidence semantic mask, and implement semantically guided disparity optimization based on the mask—enhancing matching constraints in low-texture areas of tree obstacles and filtering along the skeleton direction in transmission line areas, significantly improving the geometric consistency of the disparity map. It also combines semantic labels to extract target point clouds and achieves precise "point-to-curve" ranging by fitting the spatial curve of the wires with cubic splines. Through Kalman filtering, it effectively suppresses single-frame jitter and outputs a stable and reliable temporal distance, improving the accuracy and robustness of tree obstacle ranging in complex scenes. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the transmission line tree obstacle ranging method based on the fusion of binocular vision and deep learning in Example 1.

[0019] Figure 2 This is a flowchart of the 3D point cloud reconstruction and ranging process in Example 1. Detailed Implementation

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0023] Example 1, referring to Figure 1 and Figure 2 This is the first embodiment of the present invention, which provides a method for measuring tree obstacles on power transmission lines based on the fusion of binocular vision and deep learning, including the following steps: S1. Acquire binocular images and perform calibration and stereo correction, and perform single-scale Retinex enhancement processing on the corrected images; Specifically, acquiring binocular images and performing calibration and stereo correction refers to calibrating the binocular cameras using the Zhang Zhengyou calibration method before power transmission line inspection, obtaining the intrinsic parameter matrix and distortion parameters of each camera. Through the calibration process, the extrinsic parameter matrix between the left and right cameras (including the rotation matrix and translation vector, defining the spatial geometric relationship between the coordinate systems of the two cameras) is obtained. During the power transmission line inspection, a synchronous binocular visible light camera system mounted on a UAV platform is used to simultaneously acquire visible light images from the left and right perspectives, denoted as the left image. With right image To ensure the accuracy of subsequent 3D reconstruction, the remapping matrix required for stereo correction is calculated using the extrinsic parameter matrix. This remapping matrix is ​​then used to remap the left image. With right image Stereo correction is performed to perfectly align the left and right images along the pixel row direction, eliminating geometric distortion errors during parallax calculation, resulting in corrected left and right images. and .

[0024] By accurately obtaining the intrinsic and extrinsic parameters of the binocular camera through Zhang Zhengyou's calibration method and combining it with remapping to achieve epipolar correction, the left and right images are strictly aligned, effectively eliminating geometric errors caused by lens distortion and installation deviation, significantly improving the efficiency and accuracy of stereo matching, and providing a geometrically consistent and structurally accurate image foundation for subsequent high-reliability 3D reconstruction and power transmission line-tree obstacle ranging.

[0025] Furthermore, single-scale Retinex enhancement processing of the corrected image refers to improving the visibility of the image under complex weather conditions such as low light, backlight, and fog, while maintaining the consistency of brightness and contrast between the left and right images. An adaptive image enhancement method based on the single-scale Retinex algorithm is used to process the two images. The calculation formula for Retinex enhancement processing is as follows: in, It is the grayscale or brightness value at pixel coordinates (p, q) after enhancement. It is an image enhancement, where I(p,q) is the grayscale or brightness value of the input image at pixel coordinates (p,q). It is a two-dimensional convolution operation. It is a Gaussian smoothing kernel function used to estimate illumination components. It is a numerical stability constant to prevent zero values ​​from occurring in logarithmic operations, and log(·) is the natural logarithm operation; The left and right enhanced images were obtained based on the enhancement processing. and .

[0026] By employing an adaptive image enhancement step based on the single-scale Retinex algorithm, effective estimation and separation of illumination components in left and right binocular images under complex weather conditions such as low illumination, backlight, or fog are achieved. Gaussian kernels are used to smooth the input image to approximate the ambient illumination, and logarithmic domain reflectance component extraction is used to enhance the local contrast and detail visibility of the image, avoiding the destruction of the grayscale consistency assumption on which binocular matching depends due to enhancement asymmetry.

[0027] S2. Based on the DeepLabv3+ model combined with the coordinate attention mechanism, the enhanced image is segmented into power line and tree obstacle regions. Based on the segmentation results, the enhanced image is semantically guided to estimate and optimize disparity, and an optimized disparity map is obtained. Specifically, the segmentation of transmission line and tree barrier regions in the enhanced image based on the DeepLabv3+ model combined with the coordinate attention mechanism refers to effectively extracting semantic features of the transmission line and tree barrier regions from the enhanced image. The input is fed into the Res2Net-50 network structure, and feature maps from three layers are extracted sequentially: in, It is a shallow feature map, used to preserve detailed information such as image edges and textures. It is a mid-level feature map that incorporates contextual semantic information. It is a deep semantic feature map, which contains higher-level semantic abstraction information; To further enhance the spatial localization capability of transmission lines and tree-obstacle target areas in deep semantic feature maps, and to use a coordinate attention mechanism along the deep semantic feature map... Global average pooling is performed in both the horizontal and vertical directions to obtain two sensing vectors: in, It is the perception vector of channel u in the vertical direction. It is the perception vector of channel u in the horizontal direction. , These are the height and width of the input feature map; By inputting the sensing vectors from both directions into a one-dimensional convolution operation, position-sensitive weights are obtained: in, , These are the attention weights in the i-th row and j-th column of channel u, respectively. It is the Sigmoid activation function; Integrate the attention weights in both directions with the deep semantic feature map The feature maps are then fused together to obtain the enhanced feature maps: in, It is the enhanced feature map; Enhanced feature map The input is fed into the ASPP module (a standard DeepLabv3+ module, consisting of a 1×1 convolution branch, a 3×3 dilated convolution branch (dilation rates r=6, r=12, and r=18), and a global pooling branch), obtaining the feature map corresponding to each branch in the ASPP module. The feature maps obtained from each branch are then concatenated to obtain a multi-scale context information aggregation feature map. in, It is a feature map aggregated from multi-scale contextual information. Upsample is the upsampling operation, GAP is the global average pooling operation, and r is the inflation rate. Aggregate multi-scale contextual information into feature maps By upsampling by a factor of 4 using bilinear interpolation, the spatial resolution is restored to that of the original image. Size, to obtain the upsampled feature map This facilitates fusion with shallow features, and the upsampled feature map With shallow feature map Obtain by channel splicing This allows for the combination of boundary information from shallow features and contextual information from deep features, further enhancing the ability to finely segment target regions (such as power transmission lines and tree barriers). The data is then input into a 3×3 convolutional layer, where feature learning integrates features from different sources to output the final semantic segmentation feature map. The pixel-level classification of corresponding category labels (background, power lines, tree obstacles) is performed. During the training phase of the semantic segmentation model (using the DeepLabv3+ network model as the semantic segmentation algorithm framework and the Res2Net-50 network as the backbone network), a multi-task loss function is used to optimize the network, resulting in the final semantic segmentation feature map. The main output loss function is calculated by comparing the data with the true labels. in, It is the main output loss function. It's a real label. Yes An upsampling operation is performed to restore the image to its original resolution. CE is the cross-entropy loss function, which calculates the error between the prediction and the true label. By using the middle layer feature map The number of channels is converted to match the number of categories in the final output, and then upsampled to the original resolution size of the input image to obtain an auxiliary output. Calculate the cross-entropy loss as an auxiliary output loss function: in, It is an auxiliary output loss function. It is Upsampled 8 times to the true label Same size; The main output loss function With auxiliary output loss function The total loss function is obtained by performing a weighted summation. The Adam optimizer is used for iterative training until convergence. After the model training is completed, the final semantic segmentation feature map is... Softmax activation is performed, and the image is upsampled to its original resolution using bilinear interpolation to obtain the class probability distribution for each pixel location: in, It is the pixel-level category probability distribution after upsampling; For each pixel coordinate, the category with the highest probability value is selected as the final category for that pixel: Where M(p,q) is the category label output at pixel coordinates (p,q), c is the category identifier, c=0 represents the background area, c=1 represents the power transmission line area, and c=2 represents the tree barrier area. It is the probability value of category c at pixel coordinates (p, q) output by the network; When the category label output M(p,q) equals 1, the pixel belongs to the power transmission line area, and the mask is... Otherwise, it is 0, indicating that the pixel belongs to a tree barrier region, and the mask is 0. Both types of masks are binary images, and the pixel values ​​are defined as follows: in, It is the transmission line region mask at pixel coordinates (p, q). It is the tree barrier region mask at pixel coordinates (p,q). A pixel value of 1 indicates the corresponding target region (wire or tree barrier), and a pixel value of 0 indicates a non-target background region.

[0028] A Res2Net-50 network structure is employed to extract multi-level feature maps, ensuring effective capture of image details and contextual information, and improving the discriminability of target regions. A coordinate attention mechanism is used to enhance the spatial localization capability of deep semantic features, significantly improving the detection accuracy of key targets and avoiding segmentation fragmentation or offset. The ASPP module fuses multi-scale contextual information, enhancing the model's adaptability to targets of different sizes, particularly suitable for identifying tree obstacles of varying distances and shapes. Combining shallow boundary information with deep semantic features restores and refines segmentation boundaries, improving pixel-level classification accuracy. Finally, the training process is optimized using a main-supplement dual-output loss function, enhancing the model's convergence stability and generalization ability. This not only improves the segmentation quality of power transmission lines and tree obstacle regions but also provides a reliable guarantee for subsequent disparity optimization and 3D ranging.

[0029] Furthermore, based on the segmentation results, semantically guided disparity estimation and optimization are performed on the enhanced image to obtain an optimized disparity map that guides the left and right enhanced images. and For the input image, a semi-global matching algorithm is used to calculate disparity. To ensure global consistency and edge continuity in disparity estimation, a total energy function is defined: Where E(D) is the total energy function, representing the disparity smoothing cost of the entire image. It is the disparity value of pixel p. It is the disparity value of pixel q. It is the matching cost of pixel p. It is a smoothing penalty term when the disparity between adjacent pixels changes by 1 pixel. It is a smoothing penalty term when the disparity between adjacent pixels exceeds 1 pixel, and l(·) is an indicator function; The semi-global matching algorithm obtains the optimal solution for the disparity distribution across the entire image by minimizing the total energy function E(D), which is the initial disparity map. To further improve parallax matching accuracy in areas with power lines and tree obstructions, the transmission line area mask will be applied. and tree barrier area mask As a region constraint input, when the semantic mask is a tree barrier region mask When the pixel is located in a tree barrier region, traditional matching is prone to errors due to the weak surface texture of tree canopies and other areas. Therefore, the number of aggregated paths in the semi-global matching algorithm is increased to enhance multi-directional constraints and improve matching stability in low-texture regions. This is especially relevant when the semantic mask is a transmission line region mask. If the pixel in question belongs to the transmission line area, then to suppress noise in the conductor parallax and maintain the continuity of the conductor structure, the initial parallax map is modified accordingly. Median filtering is performed along the transmission line skeleton direction within the transmission line area mask to smooth isolated error points. Through optimization based on area constraints, an optimized disparity map is obtained. in, This is the optimized disparity map. This is the optimized disparity result at pixel coordinates (p, q). It is the disparity result at pixel coordinates (p, q). It is an optimization function based on semantic regions.

[0030] By combining a semi-global matching algorithm with semantic region constraints, high-precision disparity calculation for transmission lines and tree-block areas was achieved. The initial disparity map was optimized for different features (low texture and slender structure), including increasing the number of aggregated paths and median filtering along the skeleton direction, which significantly improved the matching stability of tree-block areas and the depth continuity of conductor areas. The final optimized disparity map not only reduced holes and mismatches in low-texture areas, but also performed well in maintaining the geometric continuity of conductors, providing high-quality depth information for subsequent 3D reconstruction and greatly improving the accuracy and reliability of tree-block ranging.

[0031] S3. Based on the optimized disparity map, 3D point cloud reconstruction and target point set extraction are performed. The target point set is fitted to the guide space curve and the minimum distance from the tree obstacle point to the curve is calculated. Kalman filtering is used to obtain the ranging result.

[0032] Specifically, 3D point cloud reconstruction and target point set extraction based on optimized disparity maps refer to the process of reconstructing 3D point clouds and extracting target points based on optimized disparity maps. By combining the camera's internal and external geometric parameters, triangulation is performed on corresponding pixels in the left and right views: Where Z(p,q) is the depth value corresponding to pixel coordinate (p,q), and X(p,q) and Y(p,q) are the horizontal and vertical coordinates of pixel coordinate (p,q) in the three-dimensional coordinate system, respectively. , These are the camera's focal lengths in the x and y directions, respectively. , B is the position of the principal point (optical center) in the image coordinate system, and B is the baseline length of the stereo camera. A complete 3D point cloud dataset of the scene is obtained by pixel-by-pixel calculation based on triangulation. To eliminate isolated noise points caused by matching errors or uneven illumination, voxel filtering is used on the 3D point cloud dataset. Smoothing is performed to effectively reduce uneven point density and high-frequency noise, resulting in smooth and structurally consistent point cloud data. And according to the transmission line area mask and tree barrier area mask Point cloud data Based on the semantic labels of the corresponding pixels, the following classifications were performed to obtain: in, It is a set of transmission line points. It is a tree obstacle point set, only retaining The number of pixels is determined to ensure that the point cloud contains only valid spatial points corresponding to the true parallax.

[0033] By optimizing the disparity map and combining it with camera parameters for triangulation, a precise conversion from 2D images to 3D point cloud data was achieved, providing a physical-scale geometric basis for the analysis of transmission lines and tree obstacles. Voxel filtering technology was used to smooth the generated 3D point cloud, effectively eliminating noise points caused by matching errors and uneven illumination, thus improving the quality and consistency of the point cloud data. Finally, based on semantic label classification, clean transmission line and tree obstacle point sets were extracted, eliminating background noise interference and ensuring the accuracy and stability of subsequent spatial distance calculations.

[0034] Furthermore, fitting the conductor space curve to the target point set and calculating the minimum distance from the tree obstacle point to the curve refers to the point set of the transmission line. The points are sorted according to their main spatial direction (i.e., along the direction of their extension). To avoid curve fitting errors caused by uneven point cloud density, the sorted point set is uniformly sampled at fixed intervals to obtain a sampled node set. Then, a cubic spline interpolation method is used to fit the sampled node set to obtain a continuous spatial curve of the transmission line. in, It is a continuous curve of the transmission line in three-dimensional space, s is the curve parameter, and x(s), y(s), z(s) are the three-dimensional coordinate components corresponding to any point on the curve. Tree obstacle point set The curve of each tree obstacle point in the calculation is then used to determine the transmission line curve. Shortest Euclidean distance: in, It is the shortest distance from the j-th tree obstacle point to the transmission line curve. It is the Euclidean norm. It is the j-th tree obstacle in the tree obstacle set; The shortest distance set is obtained based on the shortest Euclidean distance calculation. Among the shortest distances of all tree obstacle points, the minimum value is selected as the minimum spatial distance between the tree obstacle and the transmission line in the current frame. That is, the shortest spatial distance between the tree barrier and the power transmission line in the detection frame corresponding to time t.

[0035] The point cloud of the transmission line is sorted along the main direction, uniformly sampled, and fitted with cubic spline interpolation to transform the discrete point set into a smooth and continuous parametric curve. This overcomes the fitting bias caused by the sparseness and uneven distribution of the point cloud, and provides a geometric benchmark that conforms to the actual physical shape for distance calculation. Based on the "point-curve" model, the shortest Euclidean distance from each tree barrier point to the conductor is calculated, and the minimum value is taken as the single-frame ranging result. Compared with the traditional "point-point" method, this method more realistically reflects the spatial relationship between the overhead conductor and the tree barrier, and more realistically reflects the closest spatial relationship between the tree barrier and the flexible overhead conductor.

[0036] Furthermore, using Kalman filtering to obtain ranging results refers to obtaining the minimum spatial distance between tree barriers and transmission lines. Subsequently, to suppress single-frame disparity matching errors, point cloud reconstruction noise, and ranging fluctuations caused by short-term occlusion, a one-dimensional Kalman filter is used to perform temporal smoothing on the multi-frame ranging results. During the operation, the ranging state variable is set as follows: , representing the true distance estimate at time t, where the distance observation value is... The state equation and observation equation for Kalman filtering are defined as follows: Where w(t-1) is the process noise, which follows a Gaussian distribution with a mean of 0 and a process noise variance of Q, and v(t) is the observation noise, which follows a Gaussian distribution with a mean of 0 and an observation noise variance of T. To improve the filter's response to abrupt changes, before each frame update, the stable distance estimate output after Kalman filtering from the previous time step is obtained. and the corresponding estimation error covariance Construct a sliding window containing ranging observations from the current frame and the previous n frames, represented as: in, It is the ranging observation value in the detection frame corresponding to time tm; Calculating standard deviation using a sliding window: in, It is the standard deviation. It is the mean of the distance measurement observations within the sliding window; When standard deviation satisfy At that time, among them, It is an empirical coefficient, set through experiments. If a sudden change in the real state occurs (such as a drone rapidly approaching or a tree branch swaying significantly), the model parameters are reset. The reset steps include: Set the initial value for the process noise variance: in, It is the initial value of the process noise variance, and k(t-1) is the ranging observation value of the detection frame corresponding to the previous time step; Set the initial value for the observation noise variance: in, This is the initial value of the observation noise variance. It is an empirical proportionality coefficient, set through experiments; Set prior prediction values: in, This is the predicted distance value; Set the prediction error covariance equal to the initial value of process noise variance When the standard deviation satisfy If the state is considered to be continuous and stationary, standard prediction is performed using default parameters, and the Kalman filter prediction steps are executed: in, It is the predicted distance value. It is the prediction error covariance. W(t-1) is the filtering result of the detection frame corresponding to the previous time step, and W(t-1) is the estimation error covariance of the detection frame corresponding to the previous time step. To improve the filter's ability to identify anomalous observations and adaptively adjust the observation noise variance when unreliable observations occur, the observation residual of the detection frame at time t is calculated, expressed as: in, It is the observation residual of the detection frame corresponding to time t; Based on prediction error covariance The covariance corresponding to the observation residuals is calculated to assess the current observation uncertainty. The expression is as follows: Where S(t) is the observation residual covariance; Set confidence thresholds using cross-validation. When the observed residuals satisfy If the current observation is disturbed, the observation noise variance is adaptively updated using the following formula: in, This is the adjusted observation noise variance; To further improve the stability and robustness of the state estimation, after assessing the reliability of the predicted and observed values, a fusion weighting coefficient is calculated to balance the contributions of the predicted and observed values ​​in the estimation. The expression is as follows: in, It is the fusion weighting coefficient; Using fusion weighting coefficients The predicted values ​​are fused with the original observations to generate a one-step prediction fusion result: in, It is a fused prediction value. This is the predicted distance value; Keeping the original observation noise variance constant, calculate the Kalman gain based on the current ranging observation value k(t): Where K(t) is the Kalman gain, used to balance the weights of the prediction model and the observed data; Based on fusion prediction values Update state estimates and covariance with Kalman gain K(t): in, W(t) is the filtered distance estimate, i.e., the stable tree barrier-transmission line distance value, and W(t) is the estimation error covariance of the detection frame corresponding to time t.

[0037] The sliding window mechanism is used to evaluate the statistical characteristics of the ranging sequence in real time. When a sudden change in the real state is detected (such as a drone rapidly approaching or a tree branch swaying significantly), the process and observation noise variance are automatically reset, and the initial prediction value is adjusted. This overcomes the problem of hysteresis in the response of traditional filters. The credibility of the current observation is judged by the observation residual and confidence threshold. When there is occlusion or noise interference, the observation noise variance is adaptively increased and the weight of outliers is reduced. The weighted coefficient is fused to perform weighted fusion of the predicted value and the original observation to generate a more reliable prior estimate. Then, the state is updated by combining adaptive Kalman gain. This not only effectively suppresses the ranging jitter caused by single-frame disparity error, point cloud noise and short-term occlusion, but also quickly tracks changes when the real distance changes abruptly. It achieves high-precision, high-robustness, and strong dynamic response tree obstacle safe distance output.

[0038] This embodiment also provides a computer device applicable to the transmission line tree obstacle ranging method based on the fusion of binocular vision and deep learning, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the transmission line tree obstacle ranging method based on the fusion of binocular vision and deep learning proposed in the above embodiment.

[0039] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0040] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the transmission line tree obstacle ranging method based on the fusion of binocular vision and deep learning as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

Claims

1. A method for measuring tree obstacles on power transmission lines based on the fusion of binocular vision and deep learning, characterized in that: include, Binocular images were acquired and calibrated and stereo corrected, and the corrected images were then subjected to single-scale Retinex enhancement processing. Based on the DeepLabv3+ model combined with the coordinate attention mechanism, the enhanced image is segmented into power line and tree obstacle regions. Based on the segmentation results, the enhanced image is semantically guided to estimate and optimize disparity, and an optimized disparity map is obtained. Based on the optimized disparity map, 3D point cloud reconstruction and target point set extraction are performed. The target point set is fitted to the guide space curve and the minimum distance from the tree obstacle point to the curve is calculated. Kalman filtering is used to obtain the ranging result.

2. The method for measuring tree obstacles on power transmission lines based on the fusion of binocular vision and deep learning as described in claim 1, characterized in that: The single-scale Retinex enhancement processing of the corrected image refers to using an image adaptive enhancement method based on the single-scale Retinex algorithm to enhance the two images, obtaining left and right enhanced images respectively. and .

3. The method for measuring tree obstacles on power transmission lines based on the fusion of binocular vision and deep learning as described in claim 2, characterized in that: The method of segmenting power line and tree obstacle regions in the enhanced image based on the DeepLabv3+ model and coordinate attention mechanism refers to the left enhanced image. The input is fed into the Res2Net-50 network structure, where feature maps from three layers are extracted sequentially, and a coordinate attention mechanism is used to move along the deep semantic feature maps. Global average pooling is performed in both the horizontal and vertical directions to obtain two receptive vectors. These two receptive vectors are then input into a one-dimensional convolution operation to obtain attention weights. Finally, these attention weights are combined with the deep semantic feature map. The enhanced feature map is obtained by fusion and then input into the ASPP module. The feature map corresponding to each branch in the ASPP module is obtained and then concatenated to obtain a multi-scale context information aggregation feature map. Aggregate multi-scale contextual information into feature maps The upsampled feature map is obtained by bilinear interpolation upsampling. With shallow feature map Obtain by channel splicing The data is then fed into a 3×3 convolutional layer, where features from different sources are integrated through feature learning to output the final semantic segmentation feature map. For pixel-level classification of corresponding category labels, a multi-task loss function is used to optimize the network during the semantic segmentation model training phase. After the model training is completed, the final semantic segmentation feature map is analyzed. Softmax activation is performed, and the image is upsampled to its original resolution using bilinear interpolation to obtain the class probability distribution for each pixel location. The class with the highest probability value at each pixel coordinate is selected as the final class for that pixel. When the class label output M(p,q) equals 1, the pixel belongs to the transmission line area, and the mask is... Otherwise, it is 0, indicating that the pixel belongs to a tree barrier region, and the mask is 0. .

4. The method for measuring tree obstacles on power transmission lines based on the fusion of binocular vision and deep learning as described in claim 3, characterized in that: The disparity estimation and optimization based on the segmentation results for semantically guided enhancement of the image yields an optimized disparity map that directs the left and right enhancement images. and Using the input image, a semi-global matching algorithm is used to calculate disparity and obtain an initial disparity map. and the transmission line area mask and tree barrier area mask As input for the region constraint, the optimized disparity map is obtained through the optimization process of the region constraint.

5. The method for measuring tree obstacles on power transmission lines based on the fusion of binocular vision and deep learning as described in claim 4, characterized in that: The 3D point cloud reconstruction and target point set extraction based on the optimized disparity map refers to the process based on the optimized disparity map. By combining the camera's internal and external geometric parameters, triangulation is performed on corresponding pixels in the left and right views. Based on the triangulation, a complete 3D point cloud dataset of the scene is obtained pixel by pixel. Voxel filtering is then used to smooth the 3D point cloud dataset, resulting in smooth and structurally consistent point cloud data. And according to the transmission line area mask and tree barrier area mask Point cloud data The point sets of power lines and tree obstacles are obtained by classifying and extracting them according to the semantic tags of the corresponding pixels.

6. The method for measuring tree obstacles on power transmission lines based on the fusion of binocular vision and deep learning as described in claim 5, characterized in that: The process of fitting the conductor space curve to the target point set and calculating the minimum distance from the tree barrier point to the curve refers to the process for the transmission line point set. The points are sorted according to the main direction of the transmission line in space. The sorted point set is then uniformly sampled at fixed intervals to obtain a sampled node set. A cubic spline interpolation method is then used to fit the sampled node set to a curve, resulting in a continuous spatial curve for the transmission line. This process is then applied to the tree obstacle point set. Each tree obstacle point in the calculation is converted to the transmission line curve. The shortest Euclidean distance is calculated based on the shortest Euclidean distance. The set of shortest distances is obtained, and the minimum value is selected from the set of shortest distances of all tree obstacle points as the minimum spatial distance between the tree obstacle and the transmission line in the current frame.

7. The method for measuring tree obstacles on power transmission lines based on the fusion of binocular vision and deep learning as described in claim 6, characterized in that: The use of Kalman filtering to obtain ranging results refers to obtaining the minimum spatial distance between the tree barrier and the transmission line. Then, a one-dimensional Kalman filter is used to perform temporal smoothing on the multi-frame ranging results. During the operation, the ranging state variable is set as follows: , representing the true distance estimate at time t, where the distance observation value is... The state equation and observation equation of the Kalman filter are defined separately. Before each frame update, the stable distance estimate output after the Kalman filter update at the previous time step is obtained. and the corresponding estimation error covariance Construct a sliding window containing ranging observations from the current frame and the previous n frames, and calculate the standard deviation based on the sliding window. When the standard deviation... satisfy At that time, among them, If it is an empirical coefficient, then it is determined that a sudden change in the real state has occurred, and the model parameters are reset. When the standard deviation... satisfy If the state is considered continuous and stationary, standard prediction is performed using default parameters. The Kalman filter prediction step is executed, and the observation residual of the detection frame at time t is calculated, based on the prediction error covariance. Calculate the covariance corresponding to the observation residuals; Set confidence threshold When satisfied At that time, among them, It is the observation residual. If the variance is the observation residual covariance, it is determined that the current observation is disturbed. An adaptive update is performed on the observation noise variance. Based on the judgment of the predicted value and the observation reliability, the fusion weighting coefficient is calculated, and then the fusion weighting coefficient is used... The predicted values ​​are fused with the original observations to generate a one-step prediction fusion result, keeping the original observation noise variance unchanged. Based on the current ranging observation value k(t), the Kalman gain is calculated, and the fused prediction value is used as the basis for the result. The state estimate and covariance are updated with Kalman gain K(t).

8. The method for measuring tree obstacles on power transmission lines based on the fusion of binocular vision and deep learning as described in claim 7, characterized in that: The acquisition, calibration, and stereo correction of binocular images refer to the following steps: Before the transmission line inspection, the binocular cameras are calibrated using the Zhang Zhengyou calibration method to obtain the intrinsic parameter matrix and distortion parameters of each camera. Through the calibration process, the extrinsic parameter matrix between the left and right cameras is obtained. During the transmission line inspection, a synchronous binocular visible light camera system simultaneously acquires visible light images from the left and right perspectives. The remapping matrix required for stereo correction is calculated using the extrinsic parameter matrix. Stereo correction is then performed on the left and right images to obtain the corrected left and right images. and .

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the transmission line tree obstacle ranging method based on the fusion of binocular vision and deep learning as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the transmission line tree obstacle ranging method based on the fusion of binocular vision and deep learning as described in any one of claims 1 to 8.

Images