Power transmission channel management and control device and method based on binocular vision and laser point cloud prior

Abstract

The application relates to the technical field of power transmission, and discloses a power transmission channel management and control device and method based on binocular vision and laser point cloud priori, which is arranged on a high-voltage power transmission frame and comprises a left-eye camera, a right-eye camera, a first fixing frame, a second fixing frame, a third fixing frame, a horizontal rotating shaft, a vertical rotating shaft, a rotating controllable holder and a microprocessor. The left-eye camera and the right-eye camera are symmetrically arranged and are respectively installed on the second fixing frame through the first fixing frame. The two ends of the second fixing frame are respectively connected with one end of the third fixing frame, and the other end of the third fixing frame is fixed on the rotating controllable holder. The horizontal rotating shaft is arranged on the second fixing frame, and the second fixing frame is connected with the two third fixing frames through the horizontal rotating shaft. The microprocessor is arranged on the rotating controllable holder, and the vertical rotating shaft is arranged on the lower part of the rotating controllable holder. The application has the beneficial effect that visible light information and radar point cloud information can be fused to realize three-dimensional management and control of the power transmission channel.

Description

Technical Field

[0001] This invention relates to the field of power transmission technology, and to a power transmission channel management device, system, and method based on zoomable binocular vision and laser point cloud prior. Background Technology

[0002] With the continuous progress of the country and the booming development of the Internet era, electricity demand has increased significantly across the country in recent years. The power grid is a power hub integrating substations, distribution, and transmission. The transmission channels, composed of power poles and transmission lines, are an indispensable link in the power grid's energy transmission, and their stable operation is a crucial guarantee for safe electricity use. Transmission lines are generally erected in outdoor environments, characterized by complex environments and diverse terrains, and may contain potential hazards such as cranes and trees that could affect the safe operation of the transmission lines. When the distance between a potential hazard and the transmission line is less than the prescribed safe distance, the high-voltage transmission line can discharge onto the hazard, causing significant casualties and property damage. To prevent these problems, it is necessary to monitor potential hazards in the transmission channels in real time and measure the clearance distance between the transmission line and the potential hazard.

[0003] The earliest detection method was for electricians to observe the distance between the two objects, which was inefficient and prone to misjudgment. With the development of binocular vision technology, methods for detecting potential hazards in power transmission channels and measuring clearance distances using binocular image information have been developed. However, these methods use binocular cameras with fixed focal lengths, which have problems such as low accuracy in identifying potential hazards and low accuracy in measuring at long distances, and cannot adapt to the changing environment of power transmission channels.

[0004] Meanwhile, provincial power grid companies are gradually advancing radar point cloud modeling of transmission channels. The main method involves using drones equipped with lidar to fly along the transmission channels, scanning for power poles, transmission lines, and external obstacles to create a model. However, external obstacles such as trees change significantly over time, and this method is costly to use per application, has long intervals between applications, and has limited effectiveness in preventing safety incidents. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a three-dimensional control device and method for power transmission channels based on zoom-controllable binocular vision and laser point cloud priors. It can integrate visible light information and radar point cloud information to achieve three-dimensional control of power transmission channels, greatly improving the accuracy of hazard target identification and positioning, as well as the three-dimensional control capability of power transmission channels.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention proposes a power transmission channel control device based on binocular vision and laser point cloud priors. The control device is mounted on a high-voltage power transmission frame and includes a left-eye camera, a right-eye camera, a first fixed frame, a second fixed frame, a third fixed frame, a horizontal rotation axis, a vertical rotation axis, a rotatable gimbal, and a microprocessor. The left-eye camera and the right-eye camera are symmetrically arranged and are respectively mounted on the second fixed frame via the first fixed frame. Both ends of the second fixed frame are connected to one end of a third fixed frame, and the other end of the third fixed frame is fixed to the rotatable gimbal. The horizontal rotation axis is mounted on the second fixed frame, and the second fixed frame is connected to the two third fixed frames via the horizontal rotation axis. The microprocessor is mounted on the rotatable gimbal, and the vertical rotation axis is located at the lower part of the rotatable gimbal.

[0008] In conjunction with the first aspect, both the left and right cameras are controllable zoom cameras.

[0009] Secondly, this invention proposes a power transmission channel management method based on binocular vision and laser point cloud priors, comprising the following steps:

[0010] Step 1: Obtain the initial parameters of the binocular zoom camera when the transmission lines and power poles are within the field of view of the binocular zoom camera;

[0011] Step 2: Complete the camera calibration under the initial conditions, and build an autonomous update model for the calibration parameters of the binocular zoom camera to obtain the calibration results;

[0012] Step 3: Perform distortion correction and binocular stereo correction on the image based on the calibration results;

[0013] Step 4: Construct an instance segmentation model for power poles, transmission lines, and hidden danger targets based on background suppression and key area enhancement to achieve the detection and segmentation of key hidden danger targets;

[0014] Step 5: Adjust the camera focus and rotate the camera gimbal to ensure the target is centered and clearly visible in the binocular image;

[0015] Step 6: Calculate the clearance distance between the potential hazard and the transmission line;

[0016] Step 7: Based on the results of hazard target detection and segmentation and clearance distance measurement, realize three-dimensional intelligent management and control of the power transmission channel.

[0017] In conjunction with the second aspect, the specific method of step 1 is as follows: based on the prior information of the lidar point cloud of the power transmission channel, the initial depth between the hardware platform and the power transmission line and power pole is obtained; the gimbal angle and zoom factor of the binocular zoom camera are adjusted by the microprocessor so that the power transmission line and power pole are within the field of view of the binocular zoom camera; the shooting angle and focal length of the binocular zoom camera at this moment are obtained, and the two are used as the initial parameters of the binocular zoom camera.

[0018] In conjunction with the second aspect, the specific method of step 2 is as follows: construct an imaging model of a binocular zoom camera, realize coordinate transformation between the world coordinate system, camera coordinate system, image coordinate system and pixel coordinate system, complete camera calibration under initial conditions, obtain the extrinsic parameters between the lidar and the left eye camera through the calibration of the lidar and the left eye camera, obtain the intrinsic parameters of the left eye camera and the right eye camera respectively and the extrinsic parameters between the left eye camera and the right eye camera through the calibration of the left eye camera and the right eye camera, and construct an autonomous update model for the calibration parameters of the binocular zoom camera.

[0019] In conjunction with the second aspect, the specific method of step 3 is as follows: based on the calibration results in step 2, the binocular images are subjected to distortion correction and binocular stereo correction, and then the prior information of the lidar point cloud is used to achieve high-precision stereo matching of the binocular images with spatial consistency constraints, and the positional correspondence between the binocular images is obtained. The binocular images include the left eye camera image and the right eye camera image.

[0020] In conjunction with the second aspect, further, the specific method of step 4 is as follows: based on the binocular images of the power transmission channel captured by the binocular zoom camera under the initial parameters, an instance segmentation model of power poles, transmission lines and hidden danger targets based on background suppression and key area enhancement is constructed to realize the detection and segmentation of key hidden danger targets.

[0021] In conjunction with the second aspect, the specific method of step 5 is as follows: construct a model of the relationship between the camera focal length and the number of pixels occupied by the hidden danger target, construct a model of the correspondence between the average distance from the hidden danger target to the binocular zoom camera and the camera focal length, adjust the camera focal length, analyze the influence of the rotation angle of the rotating camera gimbal on the position of the hidden danger target in the binocular image, and rotate the camera gimbal so that the hidden danger target is in the center of the binocular image and is clearly visible.

[0022] In conjunction with the second aspect, the specific method of step 6 is as follows: after adjusting the camera focal length and rotating the camera gimbal so that the potential hazard target and the nearby power transmission line are in the central area of ​​the image, construct the line segment feature to be tested based on the minimum number of pixels of the potential hazard target and the instance segmentation result, derive the correspondence between the number of pixels and the actual distance, and calculate the clearance distance between the potential hazard target and the power transmission line.

[0023] In conjunction with the second aspect, the specific method of step 7 is as follows: based on the results of the detection and segmentation of hidden danger targets and the results of the clearance distance measurement, an artificial intelligence model is constructed, and based on three-dimensional quantitative analysis, it is determined whether there are hidden danger targets threatening the normal operation of the transmission line during the operation of the transmission channel, so as to realize the three-dimensional intelligent control of the transmission channel.

[0024] Compared with existing technologies, this invention provides a three-dimensional control device and method for power transmission channels based on zoom-controllable binocular vision and laser point cloud priors, which has the following beneficial effects:

[0025] (1) The power transmission channel control device of the present invention can integrate visible light information and radar point cloud information to realize three-dimensional control of the power transmission channel, which greatly improves the identification and positioning accuracy of hidden danger targets and the three-dimensional control capability of the power transmission channel.

[0026] (2) Based on the calibration, correction and stereo matching of the zoom binocular camera, the three-dimensional control device of the present invention realizes the detection and segmentation of power poles, transmission lines and hidden danger targets according to artificial intelligence algorithms, constructs the camera gimbal rotation angle and hidden danger target position model, derives the relationship between the number of pixels occupied by the hidden danger target and its actual size, obtains the accurate distance between the hidden danger target and the transmission line, and realizes the accurate positioning of the hidden danger target.

[0027] (3) The three-dimensional control method of the present invention mainly obtains binocular image information and lidar point cloud information of hidden danger targets and transmission lines in the power transmission channel. Based on the zoom factor, binocular camera rotation angle and position relationship of hidden danger targets at different distances, as well as the number of pixels occupied by hidden danger targets and the size model of hidden danger targets, a three-dimensional control model of power transmission channel targets is constructed. Thus, the clearance distance between hidden danger targets and transmission lines is obtained based on the number of pixels between hidden danger targets and transmission lines in the power transmission channel, thereby realizing real-time three-dimensional monitoring of the power transmission channel. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the structure of the three-dimensional control device for power transmission channels of the present invention;

[0029] Figure 2 This is a schematic diagram of the three-dimensional control system for power transmission channels of the present invention;

[0030] Figure 3 This is a schematic diagram illustrating the implementation process of the three-dimensional control method for power transmission channels of the present invention;

[0031] Figure 4 This is a diagram showing the calibration position relationship of the zoom camera in this invention;

[0032] Figure 5 This is a flowchart illustrating the segmentation of power poles, transmission lines, and potential hazards in this invention.

[0033] Figure 6 This is a schematic diagram of the camera gimbal rotation in this invention;

[0034] Figure 7 This is a diagram showing the relationship between the potential hazard target and the clearance distance of the transmission line in this invention.

[0035] Meaning of the reference numerals in the diagram:

[0036] 1-Left eye camera; 2-Right eye camera; 3-First mounting bracket; 4-Second mounting bracket; 5-Horizontal rotation axis; 6-Third mounting bracket; 7-Rotating controllable gimbal; 8-Vertical rotation axis; 9-Microprocessor. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] like Figure 1 As shown, this invention proposes a three-dimensional control device for power transmission channels based on zoomable binocular vision and laser point cloud priors. The device is fixedly mounted on a high-voltage power transmission frame and includes: a left-eye camera 1, a right-eye camera 2, a first fixed frame 3, a second fixed frame 4, a third fixed frame 6, a horizontal rotation axis 5, a vertical rotation axis 8, a controllable gimbal 7, and a microprocessor 9. The left-eye camera 1 and right-eye camera 2 are symmetrically arranged and mounted on the second fixed frame 4 via the first fixed frame 3. Both ends of the second fixed frame 4 are connected to one end of a third fixed frame 6, and the other end of the third fixed frame 6 is fixed to the controllable gimbal 7. The second fixed frame 4, through the third fixed frame 6, positions the left-eye camera 1 and right-eye camera 2 on the controllable gimbal 7. The second fixed frame 4 is equipped with a horizontal rotation axis 5, which connects to the two third fixed frames 6, enabling the left-eye camera 1 and right-eye camera 2 to rotate horizontally. The microprocessor 9 is mounted on the rotatable gimbal 7, and the lower part of the rotatable gimbal 7 is provided with a vertical rotation axis 8. The entire rotatable gimbal 7 can rotate around the vertical direction through the vertical rotation axis 8.

[0039] In one specific implementation of this embodiment, the control device of the present invention is mounted on a high-voltage transmission line.

[0040] In one specific implementation of this embodiment, both the left eye camera 1 and the right eye camera 2 are controllable zoom cameras, and the two cameras are collectively referred to as a binocular zoom camera.

[0041] like Figure 2As shown, the present invention proposes a three-dimensional control system for power transmission channels based on zoomable binocular vision and laser point cloud priors, which is integrated into a microprocessor 9. It includes an initial camera parameter acquisition module, a power transmission channel target detection and segmentation module, a zoomable binocular camera calibration module, a binocular correction and stereo matching module, a camera gimbal rotation angle calculation module, a hazard target clearance distance measurement module, and a power transmission channel artificial intelligence monitoring module.

[0042] The initial camera parameter acquisition module is used to obtain the distribution of the power transmission channel based on the prior data of the lidar point cloud of the power transmission channel, and to determine the initial shooting angle and camera focal length of the binocular zoom camera.

[0043] The target detection and segmentation module for power transmission channels is used to detect power transmission lines, power poles and towers and potential hazards, and to segment these three targets to obtain the target detection and segmentation results of power equipment on power transmission lines.

[0044] The zoom binocular camera calibration module and the binocular correction and stereo matching module work together to complete the process of binocular zoom camera calibration, distortion correction and stereo matching.

[0045] The camera gimbal rotation angle calculation module is used to solve the problem of potential targets exceeding the image boundary during the focal length change of a binocular zoom camera, ensuring that potential targets are centered in the image.

[0046] The target clearance distance measurement module utilizes the monocular and binocular vision collaborative measurement method disclosed in the paper "A measurement system based on internal cooperation of cameras in binocular vision" to solve the distance from the target to the transmission line by deriving the relationship between the number of pixels and their corresponding actual distance.

[0047] The AI-powered monitoring module for power transmission channels displays target detection and segmentation results, along with distance calculation results, within the images monitored by inspectors. Based on three-dimensional quantitative analysis, it determines whether any potential hazards could affect the operation of the transmission lines. This algorithm module is integrated into a microprocessor, enabling real-time three-dimensional control of the power transmission channels through a hardware platform.

[0048] like Figure 3 As shown, the three-dimensional control method for power transmission channels based on zoom-controllable binocular vision and laser point cloud priors of the present invention is as follows:

[0049] Step 1: Based on the prior information of the lidar point cloud of the power transmission channel, obtain the initial depth between the hardware platform and the power transmission line and power pole. Adjust the gimbal angle and zoom of the binocular zoom camera through the microprocessor 9 so that the power transmission line and power pole are within the camera's field of view. Obtain the shooting angle and focal length of the binocular zoom camera at this moment, and use the two as the initial parameters of the binocular zoom camera.

[0050] Step 2: Construct an imaging model for the binocular zoom camera, achieving coordinate transformations between the world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system. Complete camera calibration under initial conditions. Obtain the extrinsic parameters between the LiDAR and the left eye camera through calibration. Obtain the intrinsic parameters of the left and right eye cameras, as well as the extrinsic parameters between them, through calibration. Construct an autonomous update model for the binocular zoom camera calibration parameters. The specific process is as follows:

[0051] Step 2-1: As Figure 4 As shown, corresponding feature points in the radar point cloud and the left eye camera are selected. The PNP algorithm is used to calibrate the left eye camera and the lidar, obtaining the extrinsic parameter matrices of the zoom camera and lidar, including the rotation matrix R and the translation matrix T. Then, the rotation matrix R and the translation matrix T are used to process the coordinates of the target in the world coordinate system (the radar point cloud contains the world coordinate information of each point), obtaining the corresponding coordinates in the camera coordinate system. The three-dimensional coordinates in the world coordinate system are denoted as [X...]. w ,Y w Z w The obtained camera coordinate system coordinates are denoted as [X]. c ,Y c Z c The conversion formula is shown below, where R represents the rotation matrix, T represents the translation matrix, and r 11 ~r 33 t1 to t3 are elements in the rotation matrix R, and t1 to t3 are elements in the translation matrix T.

[0052]

[0053] Step 2-2: Under the ideal pinhole imaging model, the coordinates in the image coordinate system can be calculated based on the coordinates in the camera coordinate system in Step 2-1 and the principle of triangle similarity, as shown in the following formula, where f0 represents the focal length of the binocular camera, and the coordinates in the image coordinate system are marked as (x,y).

[0054]

[0055] Step 2-3: In Step 2-2, the image coordinates (x, y) represent the position information of the three-dimensional object on the two-dimensional image. Without considering the influence of lens distortion, the image coordinates (x, y) are discretized according to the pixel size of the camera to obtain the coordinates in the pixel coordinate system. The coordinates in the pixel coordinate system are denoted as (u, v), where dx and dy represent the physical size of the pixel in the x-axis and y-axis directions, respectively, i.e., pixel coordinates u, v. In the formula, u0 and v0 represent the principal point coordinates in the x-axis and y-axis directions, respectively.

[0056]

[0057] Steps 2-4: As Figure 4 As shown, the scaling factor of the binocular zoom camera is calculated based on the correspondence between spatial matching point pairs before and after zooming, and the homography matrix of the zooming process is calculated using geometric constraints to calibrate the intrinsic parameters of the binocular zoom camera. The specific process is as follows:

[0058] Step 2-4-1: The corresponding points of spatial point P on the imaging plane of the left eye camera before and after zooming are respectively and This indicates the center point of the left eye camera's imaging plane before zooming. The center point of the left eye camera's image plane after zooming. This indicates the center point of the right eye camera's imaging plane before zooming. O represents the center point of the image plane of the right eye camera after zooming. L O represents the location of the optical center of the left eye camera. R B represents the optical center of the right eye camera, and B represents the distance between the left and right eye cameras. This represents the homography matrix of the intrinsic parameters of the left and right cameras before zooming. This represents the homography matrix between the intrinsic parameters of the left and right cameras after zooming; the intrinsic parameters of the right camera before zooming are the new intrinsic parameters after zooming, and the transformation relationship between the two is expressed as... Homography matrix representation, i.e. Z represents the homography matrix representing the intrinsic parameters of the right eye camera before and after zooming; L and Z R Let r be the Z-axis of the world coordinate system corresponding to the left and right cameras. Assume there are M matching point pairs before and after zooming, i = 1, 2, 3...M. Calculate the scaling factor of the binocular zoom camera based on the changes in the x, y coordinates in the left eye image, where r is the scaling factor of the binocular zoom camera. u and r v These are the zoom scaling factors for the x-axis and y-axis, respectively, r f This is the focal length scaling factor; and These are the focal lengths of the left eye camera before and after zooming;

[0059]

[0060] Wherein, the coordinates of the spatial point P before and after zoom on the left eye image are (x' Li ,y' Li )and The principal point coordinates of the left eye image before zooming are (u' L0 ,v' L0 After zooming, the principal point coordinates of the left eye image are:

[0061] Step 2-4-2: Based on the plane parallelism constraint r before and after zooming u =r v =r f Calculate the homography matrix of the left camera during zooming using the principal point invariance principle. Estimate the intrinsic parameters of the binocular zoom camera, where the intrinsic parameters of the left eye camera before and after zoom are K and K, respectively. L1 and K L2 This enabled the calibration of the intrinsic parameters of a binocular zoom camera.

[0062]

[0063]

[0064]

[0065] Steps 2-5: Based on the transformation relationship of the stereo camera's rotation and translation matrices before and after zooming, construct an autonomous parameter update model for the stereo zoom camera. The rotation matrix R remains unchanged after zooming, R1 represents the rotation matrix before zooming, and R2 represents the rotation matrix after zooming. The translation matrix element t' after zooming can be calculated using the translation matrix element t before zooming and the intrinsic parameters of the stereo zoom camera. and These are the distances between the image planes of the left and right cameras before and after zooming, respectively. This can be expressed by the formula... and formula Calculate r f This refers to the focal length scaling factor mentioned in step 2-4-1. and These are the focal lengths of the left and right cameras before zooming, respectively.

[0066]

[0067] Step 3: Based on the camera calibration results under the initial conditions in Step 2, perform distortion correction and stereo correction on the binocular images. Then, use the prior information of the LiDAR point cloud to achieve high-precision stereo matching of the binocular images with spatial consistency constraints, and obtain the positional correspondence between the binocular images. The binocular images include the left camera image and the right camera image. The specific process is as follows:

[0068] Step 3-1: Correct the binocular images captured by the binocular zoom camera on the drone, including image distortion correction and binocular stereo correction. The specific steps are as follows:

[0069] Step 3-1-1: Based on the coordinate system transformation method in Step 2-1, obtain the rotation matrix R1 and translation matrix T1 of the right eye camera before zooming the left eye camera at the current moment. Divide the rotation matrix R1 before zooming into two parts, denoted as the first rotation matrix R3 and the second rotation matrix R4 respectively. Rotate the left eye image and the right eye image according to the first rotation matrix R3 and the second rotation matrix R4 respectively, so that the imaging planes of the left eye image and the right eye image are coplanar.

[0070] Step 3-1-2: Construct a rotation vector from the baseline to the epipolar line using three orthogonal vectors e1, e2, and e3. Rotate the x-axis of the imaging plane along the optical axis that has been adjusted to be parallel to the x-axis component of the translation matrix T1 to achieve alignment of the rows of the binocular image, thereby achieving stereo correction of the binocular image. The corrected binocular image is denoted as I1.

[0071] Step 3-2: Based on the traditional SURF algorithm, a stereo matching model is constructed by multi-step fusion of LiDAR point cloud and corrected binocular image I1 to achieve high-precision stereo matching of binocular images with spatial consistency constraints. The specific steps are as follows:

[0072] Step 3-2-1: Calculate the average distance between each point of the LiDAR and its k nearest neighbors, and then calculate the distance threshold. Points exceeding the distance threshold are classified as isolated points. Filter all abnormal points in the LiDAR point cloud using a statistical outlier elimination algorithm.

[0073] Step 3-2-2: As shown in the following formula, based on the projection points of the LiDAR point cloud onto the left eye image, a weighted Gaussian distribution model W(m,n) is created based on the center pixel space and intensity function. This searches for pixels within the spatial neighborhood of the projection points whose intensity or color value changes by less than 10%. Where (m,n) and (x... L ,y L I(m,n) and I(x) represent the current pixel coordinates and the center pixel coordinates, respectively. L ,y L ) represent the intensity functions of the current pixel and the center pixel, respectively, σ xy and σl These are the standard deviations of the current pixel and the center pixel, respectively.

[0074]

[0075] Step 3-2-3: Construct a stereo matching function for binocular images based on the SURF algorithm to achieve coarse matching of binocular images;

[0076] Step 3-2-4: Construct feature descriptions for each pixel in the left and right eye images based on the intensity function of the pixels. Calculate the matching cost by measuring the feature description distance of coarse matching points. Here, C((x,y),d) represents the cost function of the pixel with disparity d at coordinates (x,y), I(x,y) and I(x+d,y) are the feature descriptions of the left and right eye images, respectively, and the Dist function is a similarity measure of the feature descriptions.

[0077] C((x,y),d)=Dist(I(x,y),I(x+d,y))

[0078] Step 3-2-5: By considering the spatial consistency of uniform pixels, a matching enhancement function G is proposed. The Euclidean distance between the projection points of the LiDAR point cloud onto the binocular image and their corresponding binocular image pixel neighborhoods is calculated. Within a certain disparity range, the cost is adjusted using confidence. Outside this range, the cost is continuously enhanced through the matching enhancement function to optimize the binocular stereo matching result and generate a high-precision disparity map. As shown in the following equation, W(i,j) is the weighted Gaussian distribution model obtained in step 3-2-2, d is the binocular disparity, and di... m denoted as parallax of the LiDAR reprojection point, F as initial cost, k as enhancement coefficient, c as standard deviation, and w as an empirical value set between pixel coordinates and LiDAR reprojection point coordinates.

[0079]

[0080] Step 4: As Figure 3 and Figure 5 Based on the binocular images of the power transmission channel captured by the binocular zoom camera under initial parameters, an instance segmentation model for power poles, transmission lines, and potential hazards is constructed, based on background suppression and key area enhancement, to achieve the detection and segmentation of key potential hazards. The specific process is as follows:

[0081] Step 4-1: First, a feature extraction layer is constructed using a convolutional network to obtain the edge features of power poles, transmission lines, and potential hazards. Then, optimized feature maps of power poles, transmission lines, and potential hazards are obtained through hybrid pooling, upsampling, and weighted fusion, denoted as C. Z ;

[0082] Step 4-2: Construct a background filter to process the feature map C after hybrid pooling, upsampling, and weighted fusion. Z For feature map C Z Background suppression is performed, and then local filtering is applied to the key regions (ROIs) of the binocular images of the power transmission channel captured by the binocular zoom camera under initial parameters to output candidate regions C for the key targets to be segmented. r ;

[0083] Step 4-3: Combine the optimized feature map C of power poles, transmission lines, and potential hazards. Z Candidate regions C of the key target to be segmented r To achieve the aggregation of characteristic regions of key hidden danger targets, a key target feature map C of fixed size is obtained. rz ;

[0084] Step 4-4: Transfer the key target feature map C rz As input, an RCNN segmentation prediction model is constructed based on the RCNN detector to predict the feature maps C of each key target. rz Based on the accuracy of the data, a corresponding confidence score is obtained;

[0085] Steps 4-5: Based on the confidence score and key target feature map C rz The system generates segmentation results for power poles, transmission lines, and potential hazards. It then predicts the intersection-over-union (IoU) ratio between the segmented results and the ground truth. A multi-task loss function is used to optimize the IoU, achieving instance segmentation of power poles, transmission lines, and potential hazards within transmission channels. The multi-task loss function L(p...) e ,t e The formula is shown below, where e represents the e-th target to be identified, e = 1, 2, 3, ..., N, and N represents the number of segments the hazard target is divided into. cls p represents the number of classification points. e The output score is the classification result. For the classification truth value, L cls It is the classification loss function for foreground and background, N bbox t is the number of candidate regions generated in step 6-3. e This represents the output score of the generated candidate regions. L represents the truth value of the candidate region. bbox It is the logarithmic regression loss of the candidate region.

[0086]

[0087] Step 5: Construct a model showing the relationship between camera focal length and the number of pixels occupied by the potential hazard, and a model showing the correspondence between the average distance from the potential hazard to the binocular zoom camera and the camera focal length. Adjust the camera focal length and analyze the impact of the rotation angle of the camera pan-tilt head on the position of the potential hazard in the binocular image. Rotate the camera pan-tilt head to ensure the potential hazard is centered and clearly visible in the binocular image. The specific process is as follows:

[0088] Step 5-1: First, adjust the camera focal length according to the distance between different potential hazards and the binocular zoom camera, and construct a model showing the relationship between the average distance between the potential hazard and the binocular zoom camera and the camera focal length. The specific steps are as follows:

[0089] Step 5-1-1: Determine the position of the hazard target in the stereo image based on the instance segmentation results, and obtain the pixel coordinates (u) of a single hazard target. k ,v k ), where k represents a single potential hazard target;

[0090] Step 5-1-2: Calculate the average distance Z between the single hazard target and the binocular zoom camera using the principle of binocular vision size measurement. k The calculation formula is as follows, where f0 is the initial camera focal length, b is the camera baseline distance, and d is the left and right eye parallax of the binocular zoom camera, or simply binocular parallax. This represents the x-coordinate of the single hazard target in the left-eye camera. The x-coordinate of the single hazard target in the right eye camera;

[0091]

[0092] Step 5-1-3: Calculate the average distance from different potential hazards to the binocular zoom camera for the initial camera focal length f0. Construct a linear model graphically, considering the distance from the potential hazard to the binocular zoom camera, the camera focal length, and the number of pixels occupied by the potential hazard. Adjust the camera focal length according to this linear model so that the number of pixels occupied is approximately 1 / 4 of the total number of pixels in the binocular image. The linear model is shown in the following equation, where S is the number of pixels, K is the camera intrinsic parameter composed of camera focal length, principal point coordinates, etc., γ is the bias coefficient, α and β are the corresponding distance coefficient and intrinsic parameter coefficient, respectively, and u... l and v l These are the coordinates of the principal points on the horizontal and vertical axes of the image, respectively.

[0093] maxS=αZ k +βK(f,u l ,v l )+γ

[0094] Step 5-1-4: Construct the average distance Z from the hazard target to the binocular zoom camera using the nonlinear least squares method. kThe nonlinear model of the relationship between the focal length f and the corresponding camera focal length is shown in the following equation. Through this model, the focal length corresponding to the binocular zoom camera can be obtained directly using distance information. F(f) is a function of the camera focal length f.

[0095]

[0096] Step 5-2: As Figure 6 As shown, during the zooming process of potential hazards, the hazard may extend beyond the image boundary, leading to a decrease in the subsequent three-dimensional control capability of the power transmission channel. To address this, a camera gimbal rotation method is proposed (the camera gimbal rotation includes the rotation of the binocular zoom camera around the horizontal rotation axis and the rotation of the controllable gimbal around the vertical rotation axis). By rotating the binocular zoom camera, the potential hazard is centered in the image. The specific steps are as follows:

[0097] Step 5-2-1: When the camera focal length changes from O1O c0 Change to O1O c1 At that time, the projection point of point P in the world coordinate system onto the pixel coordinate system is from P c0 Change to P c1 ;

[0098] Step 5-2-2: When the new projection point P c1 When the image point P is outside the image boundary or located in the image edge region, the camera gimbal needs to be rotated to make point P appear smaller. c1 Move the point P within the image boundary, assuming we need to make it... c1 Move to point P c0 The position is first determined by obtaining vector P through geometric relationships. c1 P c0 ;

[0099] Step 5-2-3: Based on the relationship between a point and a plane, place point P... c1 Move to point P c0 The problem is transformed into the problem of moving the origin O1 of the existing image coordinate system to the position of point O2.

[0100] Step 5-2-4: Connect the origin O of the camera coordinate system after zooming. c1 And the new image coordinate system origin O2, because O1O c1 ⊥O1xy, calculate the rotation angle α of the camera gimbal using the law of cosines;

[0101]

[0102] Step 5-2-5: Based on geometric relationships, we can obtain O c1 O1 2 =O c1 O2 2 +O2O1 2Further simplification yields the rotation angle of the camera gimbal as follows:

[0103] Step 5-3: Decompose the rotation angle of the camera gimbal in the horizontal and vertical directions. Rotate the horizontal rotation axis and the vertical rotation gimbal by servo control to determine whether the potential target is within the image range. If not, continue to step 5-2-2. If so, complete the rotation of the controllable zoom camera so that the potential target in the power transmission channel and the power transmission lines around it are in the central area of ​​the image and the number of pixels they occupy is about 1 / 4 of the number of pixels in the binocular image.

[0104] exist Figure 6 In the middle, X w O w Y w Z w The representation of the world coordinate system, X c0 O c0 Y c0 Z c0 X represents the world coordinate system before zooming. c1 O c1 Y c1 Z c0 Let O0 be the world coordinate system after zooming, and O1 be the origin of the image coordinate system before and after zooming, respectively. u and v are the horizontal and vertical axes of the image coordinate system, respectively. x and y are the extensions of the origin on the horizontal and vertical axes before zooming, respectively.

[0105] Step 6: As Figure 7 As shown, after adjusting the camera focal length and rotating the camera pan-tilt head to position the potential hazard and the nearby power transmission line in the central region of the image, the features of the line segment to be tested are constructed based on the minimum number of pixels in the potential hazard and instance segmentation results. The correspondence between the number of pixels and the actual distance is derived, and the clearance distance between the potential hazard and the power transmission line is accurately calculated. The specific process is as follows:

[0106] Step 6-1: Extract the edge features of the hidden danger target and the transmission line based on the instance segmentation results, search for the shortest line segment among the lines connecting all edge points of the two, and construct the corresponding line segment feature L1;

[0107] Step 6-2: As Figure 7 As shown, assuming the angle between line segment feature L1 and the imaging plane of the binocular zoom camera is θ, calculate the proportional relationship of each line segment in the figure based on the principle of triangle similarity.

[0108]

[0109] Where f is the camera focal length, m1(x1,y1) and m2(x2,y2) represent the projected coordinates of the line segment feature on the imaging plane, respectively, and O c1 For the camera optical center, and Points A and B are the world coordinates of the starting and ending points of the line segment, respectively. Point A is the intersection of the extension of the camera's optical center and the extension of point M1 in the camera plane. Point B is the intersection of the perpendicular line segment passing through point M2 and line segment AM1. ΔZ is the length of this perpendicular line segment. Point M'2 is the intersection of the line connecting point M2 and the camera's optical center with line segment AB.

[0110] Step 6-3: Calculate the projection length ΔZ of the target under test in the direction perpendicular to the camera optical axis based on the geometric relationship, where b is the camera baseline distance, d is the binocular parallax, and u0 is the x-coordinate of the camera principal point;

[0111]

[0112] Step 6-4: As Figure 7 As shown, assuming that m1 and m2 on the imaging plane are two adjacent pixels, the actual distance represented by these two adjacent pixels is calculated based on the proportional relationship and ΔZ obtained in steps 6-1 and 6-2.

[0113]

[0114] in, x1 represents the distance from point A to the camera's optical center, and x2 represents the x-axis coordinate of pixel m2.

[0115] Step 6-5: Determine the number of pixels occupied by line segment feature L1 and the angle θ between the potential hazard target and the binocular zoom camera. Calculate the actual distance corresponding to line segment feature L1 according to step 6-4 to realize the measurement of the clearance distance between the potential hazard target and the power line.

[0116] Step 7: Based on the hazard detection and segmentation results and the clearance distance measurement results, the status of hazard targets and clearance distance measurement results in the transmission channel are displayed in real time in the images monitored by the inspectors. Based on three-dimensional quantitative analysis (which compares the real-time display of hazard targets and clearance distance measurement results in the images monitored by the inspectors with the national standards for power safety distances in the power industry to obtain the comparison results), it is determined whether any hazard targets threaten the normal operation of the transmission line during the operation of the transmission channel, thus achieving three-dimensional intelligent control of the transmission channel.

[0117] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0118] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A power transmission channel management method based on binocular vision and laser point cloud priors, characterized in that, Includes the following steps: Step 1: Obtain the initial parameters of the binocular zoom camera when the transmission lines and power poles are within the field of view of the binocular zoom camera; Step 2: Complete the camera calibration under the initial conditions, and build an autonomous update model for the calibration parameters of the binocular zoom camera to obtain the calibration results; Step 3: Perform distortion correction and binocular stereo correction on the image based on the calibration results; Step 4: Construct an instance segmentation model for power poles, transmission lines, and hidden danger targets based on background suppression and key area enhancement to achieve the detection and segmentation of key hidden danger targets; Step 5: Adjust the camera focus and rotate the camera gimbal to ensure the target is centered and clearly visible in the binocular image; Step 6: Calculate the clearance distance between the potential hazard and the transmission line; Step 7: Based on the results of hazard target detection and segmentation and clearance distance measurement, realize three-dimensional intelligent management and control of the power transmission channel; The specific method of step 1 is as follows: Based on the prior information of the lidar point cloud of the power transmission channel, the initial depth between the hardware platform and the power transmission line and power tower is obtained. The gimbal angle and zoom of the binocular zoom camera are adjusted by the microprocessor so that the power transmission line and power tower are within the field of view of the binocular zoom camera. The shooting angle and focal length of the binocular zoom camera at this moment are obtained, and the two are used as the initial parameters of the binocular zoom camera. The specific method of step 5 is as follows: construct a model of the relationship between the camera focal length and the number of pixels occupied by the hidden danger target, construct a model of the correspondence between the average distance from the hidden danger target to the binocular zoom camera and the camera focal length, adjust the camera focal length, analyze the influence of the rotation angle of the rotating camera gimbal on the position of the hidden danger target in the binocular image, and rotate the camera gimbal to make the hidden danger target in the center of the binocular image and clearly visible. The specific method of step 6 is as follows: after adjusting the camera focal length and rotating the camera gimbal so that the hidden danger target and the nearby transmission line are in the central area of ​​the image, construct the line segment feature to be tested based on the minimum number of pixels in the segmentation results of the hidden danger target and the transmission line instance, derive the correspondence between the number of pixels and the actual distance, and calculate the clearance distance between the hidden danger target and the transmission line. The specific method of step 7 is as follows: construct an artificial intelligence model based on the detection and segmentation results of the hidden danger targets and the measurement results of the clearance distance, and judge whether there are hidden danger targets threatening the normal operation of the transmission line during the operation of the transmission channel based on three-dimensional quantitative analysis, so as to realize three-dimensional intelligent control of the transmission channel.

2. The power transmission channel control method based on binocular vision and laser point cloud prior as described in claim 1, characterized in that, The specific method of step 2 is as follows: construct an imaging model of a binocular zoom camera, realize coordinate transformation between the world coordinate system, camera coordinate system, image coordinate system and pixel coordinate system, complete camera calibration under initial conditions, obtain the extrinsic parameters between the lidar and the left eye camera through the calibration of the lidar and the left eye camera, obtain the intrinsic parameters of the left eye camera and the right eye camera and the extrinsic parameters between the left eye camera and the right eye camera through the calibration of the left eye camera and the right eye camera, and construct an autonomous update model for the calibration parameters of the binocular zoom camera.

3. The power transmission channel control method based on binocular vision and laser point cloud prior as described in claim 1, characterized in that, The specific method of step 3 is as follows: based on the calibration results in step 2, the binocular images are subjected to distortion correction and binocular stereo correction. Then, the prior information of the lidar point cloud is used to achieve high-precision stereo matching of the binocular images with spatial consistency constraints, and the positional correspondence between the binocular images is obtained. The binocular images include the left eye camera image and the right eye camera image.

4. The power transmission channel control method based on binocular vision and laser point cloud prior as described in claim 1, characterized in that, The specific method of step 4 is as follows: Based on the binocular images of the power transmission channel captured by the binocular zoom camera under the initial parameters, an instance segmentation model of power poles, transmission lines and hidden danger targets based on background suppression and key area enhancement is constructed to realize the detection and segmentation of key hidden danger targets.

5. A power transmission channel control device based on binocular vision and laser point cloud prior, used to implement the power transmission channel control method based on binocular vision and laser point cloud prior as described in any one of claims 1-4, characterized in that: The control device is mounted on a high-voltage transmission line and includes a left-eye camera, a right-eye camera, a first fixed frame, a second fixed frame, a third fixed frame, a horizontal rotation axis, a vertical rotation axis, a rotatable pan-tilt unit, and a microprocessor. The left-eye camera and the right-eye camera are symmetrically arranged and are respectively mounted on the second fixed frame via the first fixed frame. Both ends of the second fixed frame are connected to one end of a third fixed frame, and the other end of the third fixed frame is fixed to the rotatable pan-tilt unit. The horizontal rotation axis is mounted on the second fixed frame, and the second fixed frame is connected to the two third fixed frames via the horizontal rotation axis. The microprocessor is mounted on the rotatable pan-tilt unit, and the vertical rotation axis is located at the lower part of the rotatable pan-tilt unit.

6. The power transmission channel control device based on binocular vision and laser point cloud prior as described in claim 5, characterized in that: Both the left and right cameras are controllable zoom cameras.

Images