A visual sag detection method, system and apparatus

Abstract

The application discloses a kind of visual sag detection method, system and device, method includes: S1, camera and laser radar external parameter are pre-calibrated and the internal parameter and relative external parameter of both are obtained;S2, the image data and three-dimensional point cloud data of the transmission conductor to be measured are respectively synchronously acquired;S3, the partial point cloud of conductor is extracted by fitting using conductor local three-dimensional point cloud, and the vertical plane where the conductor is located is determined in combination with gravity direction information and external parameter;S4, the image pixel of conductor is segmented in the image collected by camera, and the pixel coordinates of each conductor in camera coordinate system are determined;S5, the segmented conductor image pixel is projected to the vertical plane, the conductor image pixel is extended to three-dimensional space point cloud under camera coordinate system, and the catenary of transmission conductor line is fitted based on three-dimensional space point cloud, and the sag value of conductor is calculated.The application can improve the perception accuracy of conductor state in sag detection technology, and can continuously carry out high-precision sag detection after single system installation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of power automation and computer vision, in particular to the field of transmission line state and safety monitoring, and more particularly to a visual sag detection method, system and device. BACKGROUND

[0002] Transmission line sag refers to the vertical distance between the lowest point of the sag curve formed by the self-weight and external load between two adjacent towers and the connecting line of the two suspension points. The size of the sag directly affects the safe operation and transmission capacity of the transmission line. Too small sag will cause excessive tension of the conductor, which may cause accidents such as conductor breakage, tower collapse or insulator string falling, especially under extreme weather conditions (such as icing and strong wind). Too large sag may cause conductor wind swing, dancing, and even contact discharge with ground vegetation and buildings, leading to inter-phase short circuit or tripping, threatening the stable operation of the power grid.

[0003] With the development of power systems towards high voltage and large capacity, and the application of dynamic capacity increasing technology, the operating conditions of conductors (such as temperature and mechanical load) change more frequently, and the demand for dynamic monitoring of sag is increasingly prominent. In addition, China's transmission network covers complex terrain, and traditional manual inspection cannot meet the efficient and accurate monitoring requirements. Therefore, the development of high-precision, all-weather and intelligent sag detection technology is of great significance to ensure the safety of the power grid, improve the operation efficiency and reduce the risk of accidents.

[0004] Currently, sag detection mainly adopts laser radar scheme or image detection scheme, but both have different degrees of technical bottlenecks:

[0005] Laser radar scheme: Laser radar can provide high-precision three-dimensional point cloud with millimeter-level precision, but due to its limited sensing distance (usually within 200m) and sparse point cloud, its effective sensing distance of the transmission conductor is very limited, usually within 30m-50m, and direct three-dimensional fitting of the conductor based on these local point clouds cannot achieve the required precision. The local data obtained cannot describe the overall characteristics of the transmission line. The main method to overcome this limitation in existing schemes is to use unmanned aerial vehicles to carry laser radars to scan the transmission line and fuse the point cloud to obtain the complete three-dimensional point cloud of the transmission line, but this method is high in operation cost and cannot monitor the line in real time.

[0006] As disclosed in Chinese patent application CN119642719A, a kind of unmanned aerial vehicle wire sag measurement device and method, the patent proposes a measurement device carried on unmanned aerial vehicle, by on-board collimator and laser radar range finder to the whole transmission line is scanned, to obtain the overall state of wire, calculate sag;Chinese patent application CN119090939A proposes a kind of flat area transmission line sag measurement method based on airborne laser radar, by pre-training a point cloud segmentation network, the point cloud data obtained by unmanned aerial vehicle laser radar scanning is segmented, reconstructed and straight line fitting, calculate the numerical value of sag.

[0007] Image sensor scheme: traditional vision scheme is mostly binocular camera fixed on transmission tower, combined with depth estimation and semantic segmentation network to detect transmission line form. The perception range of this kind of scheme is farther, can observe the overall state of wire, but it depends on the disparity analysis of left and right images to estimate depth, for small, complex background wire, its stereo matching is difficult, the precision of estimated three-dimensional coordinates is insufficient, it is difficult to meet the requirements of sag measurement.

[0008] Among them, as disclosed in Chinese patent application CN118071678A, a kind of transmission line sag real-time measurement method based on binocular vision, proposes a kind of real-time measurement method of sag based on binocular stereo vision, through unmanned aerial vehicle to collect pole and tower interval photo, by epipolar correction and minimum spanning tree (MST) stereo matching algorithm to generate disparity map, then through triangulation to convert to three-dimensional point cloud, combined with catenary model fitting to calculate sag. Chinese patent application CN118505537A discloses a kind of real-time monitoring system and method for transmission line sag based on multi-angle image fusion, the patent designs a set of multi-angle image fusion real-time monitoring system, obtains wire image through multi-view image acquisition module, after pretreatment such as denoising, edge detection, fuses multi-angle image information to reconstruct complete three-dimensional form of wire, combined with catenary model to calculate sag. Chinese patent application CN202010494477 discloses an image recognition algorithm for transmission line sag measurement, the patent proposes a kind of catenary fitting algorithm based on unmanned aerial vehicle aerial image, extracts transmission line edge pixel coordinates through image recognition, constructs catenary equation and calculates maximum sag value by derivation.

[0009] In summary, the current sag detection technology has significant shortcomings: the effective detection distance of the laser radar is limited, and it cannot accurately fit the long-span transmission lines (such as 500m or more). The unmanned aerial vehicle carries out segmented scanning to obtain the overall shape of the transmission line, and the system has high operating cost and insufficient real-time performance; the visual scheme based on binocular cameras or multi-view images can directly obtain the overall shape of the transmission line, but is limited by algorithm accuracy and complex background interference, and the detection accuracy is insufficient. At present, there is still a lack of technical solutions that combine the sensing advantages of the two sensors. Therefore, it is urgent to develop a sag detection technology that organically combines the advantages of each sensor, has higher accuracy, and can be operated normally, to support the operation and maintenance needs of the smart grid. SUMMARY

[0010] In view of the problems in the background art, the present application provides a visual sag detection method, system and device, which can improve the perception accuracy of the conductor state in the sag detection technology and continuously perform high-precision sag detection after single system installation. The present application is realized by the following technical solutions.

[0011] S1, pre-calibrate the camera and laser radar external parameters to obtain the internal parameters of the camera and laser radar and the relative external parameters between them;

[0012] S2, use the camera and laser radar to synchronously acquire image data and three-dimensional point cloud data of the transmission line to be measured, respectively, the image data acquired by the camera contains the complete conductor state, and the laser radar acquires local three-dimensional point cloud data of the transmission line at a short distance;

[0013] S3, use the local three-dimensional point cloud of the conductor collected by the laser radar to perform fitting, extract the partial point cloud of the conductor, combine the gravity direction information provided by the gravity acceleration sensor, and determine the vertical plane in which the conductor is located by constructing a joint optimization function for solving the vertical plane in the camera coordinate system based on the relative external parameters of the laser radar and the camera, to obtain the equation of the vertical plane in which the conductor is located in the camera coordinate system;

[0014] S4, use an image semantic segmentation method to segment the image pixels of the conductor in the image collected by the camera, and use a region growing or grouping algorithm to group the image pixels belonging to each conductor, to determine the pixel coordinates of each conductor in the camera coordinate system;

[0015] S5, use the pixel coordinates of the conductor in the camera coordinate system obtained in step S4 and the equation of the vertical plane in which the conductor is located in the camera coordinate system obtained in S3, project the segmented conductor image pixels onto the vertical plane, extend the conductor image pixels projected onto the vertical plane to three-dimensional space point cloud in the camera coordinate system, and fit the catenary of the transmission line based on the three-dimensional space point cloud, and calculate the sag value of the transmission line.

[0016] Further, the camera and the laser radar are mounted on a rigid structure, and the camera is a monocular camera or a binocular camera.

[0017] Further, when the camera is a binocular camera, the step S3 uses the binocular camera to generate dense point cloud information when determining the vertical plane of the conductor, and a dense point cloud information space coordinate constraint term of the binocular camera is included in a joint optimization function for solving the vertical plane.

[0018] Further, the internal parameters obtained in the step S1 include a camera focal length, a resolution and a radar field of view angle; and the relative external parameters include a relative displacement and a rotation angle.

[0019] Further, after the step S1 is performed and before the step S2 is performed, the following operations are further performed:

[0020] In the camera coordinate system, a point cloud subset corresponding to a static background is identified, and the point cloud subset is registered with a static background feature in image data to calibrate or verify a preset rigid transformation relationship between the laser radar and the camera in real time.

[0021] Further, the method for determining the vertical plane of the conductor includes: fitting a power line point cloud subset to screen a high-confidence reference point set, calculating an internal vector of the plane by using the coordinates in the point set, performing a cross multiplication operation on the internal vector of the plane and a gravity acceleration vector to obtain a normal vector of the spatial plane, and then establishing a plane equation.

[0022] Further, the calculation method of the sag value is: fitting a catenary model based on the three-dimensional coordinate point set, determining a lowest point and two suspension points of a curve of the catenary model, and calculating a vertical distance between the lowest point and the two suspension points.

[0023] The application further discloses a visual sag detection system, which comprises:

[0024] a camera configured to collect image data of the conductor;

[0025] a laser radar configured to collect three-dimensional point cloud data;

[0026] a gravity acceleration sensor configured to obtain a gravity direction and calculate a vertical plane of the power line;

[0027] a parameter calibration module configured to pre-calibrate external parameters of the camera and the laser radar to obtain relative external parameters therebetween;

[0028] a data collection module configured to obtain image data and three-dimensional point cloud data of a power conductor to be measured;

[0029] A plane construction module is configured to fit a local three-dimensional point cloud of a conductor collected by a laser radar to determine a vertical plane where the complete conductor is located, and then to obtain a pose of the vertical plane of the conductor in a camera coordinate system by using relative external parameters of the camera and the laser radar.

[0030] A two-dimensional pixel segmentation module is configured to segment image pixels of the conductor in an image collected by the camera, project the segmented image pixels of the conductor to the vertical plane by using the pose relationship of the vertical plane of the conductor in the camera coordinate system obtained by the spatial plane construction module.

[0031] An arc sag calculation module is configured to fit a catenary of the transmission conductor line based on the image pixels of the conductor and calculate an arc sag value of the conductor.

[0032] Further, the camera and the laser radar are mounted on a rigid structure, and the camera is a monocular camera or a binocular camera.

[0033] The application further discloses a visual arc sag detection device, characterized in that comprising:

[0034] a camera, a laser radar, a gravity acceleration sensor and a processing unit.

[0035] The camera and the laser radar are configured to obtain image data and three-dimensional point cloud data of the transmission conductor, the gravity acceleration sensor is configured to obtain a gravity direction in a camera coordinate system, and the processing unit is configured to calculate an arc sag value of the conductor based on visual arc sag detection.

[0036] The application has the following beneficial effects by adopting the above technical scheme:

[0037] The application organically combines the advantages of high precision of the laser radar and low cost of the image sensor, accurately obtains the camera external parameters, the vertical plane where the transmission line is located and other calculation references by scanning the high-precision local point cloud obtained by the laser radar, and then performs projection calculation on the position of the whole transmission line by using the image data obtained by the camera, so that the problems of high cost of the laser radar scheme and insufficient precision of the binocular camera scheme are effectively overcome, and the conductor arc sag is accurately detected in real time. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 FIG. 1 is a flowchart of a visual arc sag detection method according to an embodiment of the application.

[0039] Figure 2 FIG. 2 is a hardware and software module structure diagram of a visual arc sag detection system according to an embodiment of the application.

[0040] Figure 3 FIG. 3 is a flowchart of determining conductor pixel coordinates by a semantic segmentation network according to an embodiment of the application.

[0041] Figure 4The flowchart for calculating the three-dimensional coordinates of the conductor by combining the laser radar data and the pixel coordinates according to the embodiment of the present application.

[0042] Figure 5 The schematic diagram of the core principle of three-dimensional reconstruction in the camera coordinate system by using laser radar assistance according to the embodiment of the present application. DETAILED DESCRIPTION

[0043] The present application will be further described in detail below in combination with the drawings and specific embodiments. EMBODIMENT

[0044] The embodiment of the present application provides a visual sag detection method, which is used to solve the problems of high cost of the laser radar scheme and insufficient precision of the binocular camera scheme, and realizes real-time and accurate detection of conductor sag at a lower cost. Figure 1 The method of the present application comprises:

[0045] Before real-time detection, a deep learning model for accurately segmenting the power line needs to be pre-trained; in the present embodiment, U-Net is selected as the semantic segmentation network architecture.

[0046] Dataset construction: a large number of conductor images containing various backgrounds, lighting and weather conditions are collected, and the conductor images are manually labeled at the pixel level to generate binary label images corresponding to the original images. In the binary label image, the power line pixel region is assigned a value of 1, and the background region is assigned a value of 0.

[0047] Model training: the constructed dataset is divided into a training set, a validation set and a test set; the U-Net network is trained end-to-end using the training set; during the training process, a cross-entropy loss function is used to measure the difference between the model output and the true label, and an Adam optimizer is used to iteratively update the network weights until the model performance converges on the validation set.

[0048] Model deployment: the model weights of the trained and performance-optimized model are saved and deployed in the software processing unit of the system for subsequent real-time image segmentation tasks.

[0049] S1, pre-calibrate the camera and laser radar to obtain the internal parameters of the camera and laser radar and the relative external parameters between them and optimize the parameters, and the step S1 specifically comprises:

[0050] S101, rigidly fix the camera and laser radar together;

[0051] S102, use a rectangular checkerboard calibration board or other reference object, which is fixed and placed at a proper position in front of the camera and laser radar, to ensure that the camera and laser radar have a complete calibration board in their field of view.

[0052] S103, respectively scan the calibration board using the camera and the laser radar, and obtain the calibration board image and the overall point cloud of the space where the calibration board is located;

[0053] S104, in the point cloud, extract the coordinates of at least three calibration board vertexes in the laser radar coordinate system through clustering analysis;

[0054] S105, in the camera, determine the pixel coordinates of the calibration board vertexes in the image through the Canny edge extraction algorithm and the artificial screening method;

[0055] S106, calculate the conversion matrix of the laser radar coordinate system and the camera pixel coordinate system, i.e., the relative external parameters between the two, and the calculation formula is as follows:

[0056]

[0057] S107, external parameter optimization. When the system is installed, the relative position of the camera and the laser radar is fixed, and the parameter state of the camera and the laser radar can be determined according to steps S101-S106, but in the actual operation process of the system, the external parameters of the device are easy to deviate from the initial state due to external factors, resulting in unstable measurement results. Therefore, the external parameters of the camera in the world coordinate system need to be optimized regularly. Considering that the external parameters between the camera and the laser radar are fixed, as long as the external parameters of the laser radar to the world coordinate system are optimized in real time, based on the formula:

[0058]

[0059] the external parameter updating of the camera can be realized, and the optimization process includes:

[0060] Referring to Figure 5 , in the embodiment, the positive direction of the z-axis is defined as the direction from the power transmission tower where the device is installed to the power transmission tower suspended at the other end of the conductor, the positive direction of the y-axis is vertical upward, and the positive direction of the x-axis is parallel to the ground, which conforms to the right-handed coordinate system and is perpendicular to the y and z axes.

[0061] Suppose that the three direction axes and the origin of the world coordinate system are respectively represented as , which are represented as in the laser radar coordinate system. In fact, the are respectively defined as ,

[0062] , and . Based on the above formula, there is a transformation relationship between the two sets of parameters as follows:

[0063]

[0064] The specific process of the external parameter optimization algorithm is as follows:

[0065] The 3D straight line fitting is performed on the wire point cloud using the least square method, and the straight line direction is . The plane fitting is performed on the wire point cloud using RANSAC, and the normal vector of the fitted plane is taken as , and is taken as .

[0066] The centers of the left and right point clouds are calculated, and then the midpoint of the two centers is calculated. The midpoint is translated along the axis to the position where the z value is 0, and this position is taken as .

[0067] The rotation matrix and the translation matrix between the laser radar coordinate system and the world coordinate system are calculated according to the above formula.

[0068] Considering random interference, the external parameter calculated from the point cloud of the laser radar may have errors, and the external parameter needs to be further optimized. In this embodiment, the external parameter expressed by Euler angles in a specific range is traversed, and the external parameter is used to project the subsequent wire each time. The error between the laser radar point cloud and the projected point cloud is calculated, and the external parameter expressed by Euler angles with the smallest error is found. The traversal search adopts the principle of from-coarse-to-fine, and the initial value is the Euler angle corresponding to the external parameter calculated by the above process. Then the traversal search precision is set to 1°, 0.1° and 0.01° respectively, and the specific value can be adjusted appropriately.

[0069] The step S107 can be periodically performed after the system is installed to ensure the accuracy of the camera and the external parameter of the laser radar.

[0070] S2, data acquisition, coordinate system replacement; the image data and three-dimensional point cloud data of the transmission line to be measured are synchronously acquired by the camera and the laser radar respectively, the image data acquired by the camera contains the complete state of the transmission line, and the laser radar acquires the local three-dimensional point cloud data of the transmission line in the near distance. The step S2 specifically comprises:

[0071] S201, the camera and the laser radar work synchronously, and the camera acquires an RGB image containing the wire . The laser radar device (Lidar) acquires the three-dimensional point cloud of the surrounding environment The point coordinates in the three-dimensional point cloud are defined in the Lidar coordinate system). However, since the laser radar is fixedly installed in the central part of the transmission tower, the scanning range is limited, and only the local three-dimensional point cloud data of the transmission line to be measured in the near distance (about 50m) can be acquired; ​

[0072] S202, transform all Lidar point clouds into camera internal coordinate system, get point clouds ; the transformation is completed by a rigid transformation matrix previously calibrated: The specific transformation process described by the matrix is as follows:

[0073] pixel coordinates using the camera intrinsic matrix , get normalized plane coordinates

[0074] , the laser radar coordinates are first translated t and then rotated R to get the camera coordinates, that is: ,

[0075] Further transformation, get:

[0076]

[0077] Let , we get

[0078]

[0079] This is the conversion equation of the laser radar point cloud coordinates to the camera coordinate system, where the relative position relationship between the laser radar and the camera is determined when the system is installed; all subsequent three-dimensional calculations are performed in the camera coordinate system, unless otherwise specified.

[0080] S3, construct the vertical plane where the power transmission conductor is located; use the local three-dimensional point cloud of the conductor collected by the laser radar to fit and extract the conductor part point cloud, and combine the gravity direction information provided by the gravity acceleration sensor and the extrinsic parameters of the laser radar and the camera in the camera coordinate system to determine the vertical plane where the conductor is located. The step S3 constructs the spatial vertical plane where the to-be-measured power transmission line is located in the ground coordinate system. Referring to Figure 5 , the hardware layer of the embodiment of the application is installed in the central part of the power transmission tower, and this deployment method makes the collected three-dimensional point cloud data usually contain information of multiple power transmission lines suspended on both sides of the power transmission tower. The step S3 specifically includes:

[0081] S301, according to the camera coordinate system point cloud obtained in step S202, the symmetry center is divided into single-sided conductor coordinates

[0082] , respectively performing the following calculation steps.

[0083] S302, in the single-sided camera coordinate system point cloud In the embodiment, the sparse point cloud subset belonging to the power transmission line itself is identified and extracted by clustering and filtering based on the catenary equation :

[0084] The piecewise quadratic curve and the piecewise linear curve of x and y with respect to z are respectively established

[0085]

[0086] S303, the curve constraint and parallel constraint of the power transmission line are established; for the polynomial function f(z), its curvature is as follows, wherein is the quadratic term coefficient corresponding to the above formula. The maximum value of the curvature k of f(z) appears at the pole, so the curvature k at the pole is constrained, and the problem becomes a constrained optimization problem, which is expressed as follows, and R represents the maximum curvature radius set in advance.

[0087]

[0088] The curvature constraint is performed on each curve, which in the embodiment is embodied as ensuring that each fitting curve f(z) has the same quadratic term and linear term coefficient, and each fitting curve only has the difference in the translation amount. That is, each curve on the single side has the same , and parameters, and only and are different.

[0089] After the above curve fitting step, the preliminary power transmission line point cloud distribution can be obtained. To further improve the positioning stability, the embodiment also adopts a post-processing method:

[0090] S304, the RANSAC fitting method is used to remove outliers with large deviations, and high-credibility power transmission line points are reserved to form the true value of part of the power transmission line position.

[0091] S305, two points are taken in , wherein are the coordinates of each point, and a vector of the plane where the power transmission line is located is calculated.

[0092] S306, since the local power transmission line angle is not obvious, using only the local point cloud of the power transmission line to calculate the plane where the power transmission line is located will cause a large error, and the gravity acceleration sensor is used to obtain the gravity vector g in the camera coordinate system as the calculation reference; by vector cross multiplication operation, the normal vector of the vertical plane is obtained.

[0093] S307, in order to improve the reliability of plane calculation, other two points on the power transmission line are selected to repeat steps S305 and S306, the normal vectors of the planes where the two points are located are calculated, and the average of the normal vectors after filtering outliers is taken as the normal vector of the vertical plane where the conductor is located.

[0094] Thus, the equation of the vertical plane where the power transmission line is located is determined as: wherein is any point on the plane, is the point used in the calculation process.

[0095] The conventional laser radar can provide high-precision three-dimensional point cloud with millimeter-level precision, but due to its limited sensing distance (usually within 200m) and sparse point cloud, its effective sensing distance of the power transmission line is very limited, usually within 30m-50m, so only partial three-dimensional point cloud of the power transmission line can be obtained. Direct three-dimensional fitting of the power transmission line based on the partial point cloud of the power transmission line cannot achieve the required precision for application. To overcome the above problems, the point cloud collected by the laser radar is only used for fitting calculation of the vertical plane where the power transmission line is located, that is, the equation of the plane where the power transmission line is located is determined in the camera coordinate system with the help of the gravity acceleration sensor, and the laser radar point cloud is not directly used for catenary fitting of the power transmission line in three-dimensional space.

[0096] S4, two-dimensional pixel segmentation of the power transmission line; the image pixel of the power transmission line is segmented in the image collected by the camera by using an image semantic segmentation method, and a region growing grouping algorithm is used to group the image pixels belonging to each power transmission line to determine the pixel coordinates of each power transmission line in the camera coordinate system. See Figure 3 , Figure 3 The flowchart for determining the pixel coordinates of the power transmission line by the semantic segmentation network according to an embodiment of the present application is shown in FIG. 4. The step S4 specifically includes:

[0097] S401, the U-Net model trained in the preparation stage is used to process the RGB image collected in the step S2, and a binary segmentation mask image is outputted; the segmentation mask image has the same size as the original image, and the set of pixels with a value of 1 accurately identifies the position of the power transmission line on the two-dimensional image.

[0098] ​​​​S402, since a plurality of wires exist in a segmentation mask, the pixels of the wires belonging to the same group in the segmentation mask also need to be divided, and the region growing algorithm is selected to group the segmentation results, starting from a seed point, and continuously adding adjacent pixels that meet certain conditions to the same region until no new pixels can be added; according to the observation of the segmentation results, the pixels on the same wire are close, and the pixels between different wires are far apart, so the growing condition predefined in this embodiment is determined as the Manhattan distance between two points, and if it is less than the threshold, it is determined to have similar properties and is divided into the same group of wires.

[0099] S403, set a threshold to filter the noise with the characteristics of fragmentation, small number of points, and far distance between adjacent pixels, as shown in Figure 3 , traverse all segmentation groups, judge the number of pixels, and if the condition is met, determine it as an effective wire group, and finally obtain the pixel coordinates of each group of wires.

[0100] S5, three-dimensional coordinate reconstruction, calculate the sag; in the camera coordinate system, project the pixel coordinates of each group of wires to the vertical plane where the wire is located and expand it to a three-dimensional point cloud, fit the catenary of the power transmission line based on the three-dimensional point cloud, and calculate the sag value of the wire; as shown in Figure 4 , Figure 4 is the flow chart of the present application for calculating the three-dimensional coordinates of the wire combined with the laser radar data and the pixel coordinates, and the step S5 specifically includes:

[0101] S501, refer to Figure 4 , first construct the camera optical center ray equation, the pixel coordinates of a wire in the image are , and the translated image plane coordinates can be obtained according to the camera intrinsic matrix.

[0102]

[0103] The corresponding three-dimensional space coordinates in the camera coordinate system can be expressed as , wherein is the focal length of the camera.

[0104] S502, combine the image plane coordinates in the camera coordinate system with the optical center position, i.e. the device position

[0105] to obtain the direction vector of the optical center ray equation:

[0106]

[0107] The optical center ray equation can be written as , and t represents the distance measured from the starting point along the ray.

[0108] S503, finally, the equation of the vertical plane of the traverse is combined with the equation of the camera optical center ray constructed by all the split points of the traverse:

[0109]

[0110] Solve the three-dimensional coordinates of the traverse in the camera coordinate system .

[0111] S504, convert the coordinates to the vertical plane of the traverse, and according to the catenary formula

[0112]

[0113] Least square fitting is performed on each point on the traverse to obtain the complete three-dimensional space coordinates of the traverse.

[0114] S505, from the fitted catenary curve, the two end points in the vertical plane and the lowest point of the curve are analyzed, and the distance from the lowest point to the line connecting the two end points is calculated, which is the required traverse sag value.

[0115] Example 2

[0116] This embodiment is improved on the basis of example 1, and the camera is replaced by a binocular camera, specifically a binocular stereo camera. The binocular camera can directly calculate the dense depth information of the scene through stereo matching, providing more abundant geometric constraints for determining the vertical plane of the traverse, thereby improving the accuracy of traverse fitting and sag detection. Compared with steps S1-S5 of example 1, the main changes of using a binocular stereo camera are as follows:

[0117] T1, binocular camera calibration. In addition to calibrating the internal parameters of each camera (left camera and right camera) and the relative external parameters between the camera and the laser radar, it is also necessary to accurately calibrate the relative position relationship between the two cameras, i.e. the external parameters of the binocular camera: rotation matrix R and translation vector T.

[0118] The specific steps of the step T1 include:

[0119] T101, image acquisition; similar to example 1, a high-precision checkerboard calibration plate is used, and multiple groups (usually 15-20 groups) of checkerboard calibration plate images are captured from different angles and distances using left and right cameras to ensure that the checkerboard occupies more than two-thirds of the image area.

[0120] T102, Corner detection; For the collected left and right image pairs, use a corner detection algorithm (such as Harris corner detection) to extract the pixel coordinates of the checkerboard corner points; To improve the accuracy of calibration, use optimization methods such as quadratic interpolation to optimize the corner coordinates to the sub-pixel level.

[0121] T103, Single target calibration; First, apply mature camera calibration algorithms such as Zhang Zhengyou calibration to the left and right cameras respectively, and calculate the internal parameters of each camera according to the detected corner coordinates.

[0122] T104, Stereo calibration; Using the corresponding corner pairs in the left and right camera images and their respective internal parameters, the rotation matrix R and translation vector T of the right camera relative to the left camera are calculated by solving the epipolar geometry constraint; R and T describe the rigid transformation relationship between the two camera coordinate systems, which is the basis for stereo matching and three-dimensional reconstruction.

[0123] T2, Generate dense point cloud of scene using binocular vision; After data acquisition (step S2), the calibrated binocular camera parameters can be used to process the left and right images collected synchronously to generate a dense depth point cloud of the scene. The detailed process of this step is:

[0124] T201, Stereo rectification. Use the binocular camera internal and external parameters obtained in T1 to perform epipolar rectification on the left and right images. In the rectified images, the original epipolar line will become a horizontal line, making the search for corresponding points one-dimensional instead of two-dimensional, reducing the difficulty of stereo matching.

[0125] T202, Stereo matching. In the rectified left and right images, use a stereo matching algorithm such as SGBM - Semi-Global Block Matching to find the best matching point for each pixel in the left image on the corresponding epipolar line in the right image.

[0126] T203, Calculate disparity map. The coordinate difference between two corresponding pixel points in the horizontal direction is the disparity. After matching all pixels in the image, a disparity map can be generated, where the gray value of each pixel represents the disparity of that point.

[0127] T204, Three-dimensional reconstruction. According to the principle of triangulation, using the disparity value, camera focal length, and binocular baseline length (the distance between the optical centers of the left and right cameras, which can be obtained from the translation vector T), the disparity map can be converted into a depth map. Combined with pixel coordinates and camera internal parameters, the three-dimensional spatial coordinates (X, Y, Z) of each pixel in the image can be calculated in the camera coordinate system, forming a dense point cloud information of the scene .

[0128] T3, the point cloud obtained by binocular stereo estimation and the radar point cloud data are jointly optimized to determine the traverse vertical plane. In step S3 of embodiment 1, the determination of the traverse vertical plane mainly depends on the local sparse point cloud collected by the lidar and the gravity vector. In this embodiment, the dense point cloud information generated by the binocular camera is fused into the local traverse position to form a joint optimization problem, so that a more dense and accurate local traverse position is obtained. The main processing method of this step is as follows:

[0129] Before step S301, the three-dimensional point cloud generated by binocular vision in step T2 .

[0130] and the point cloud collected by the lidar and converted into the camera coordinate system are aligned. Since the two are unified in the same camera coordinate system through calibration, they can be directly fused.

[0131] The fused point cloud is denser than the point cloud obtained by the lidar, and can be used as the point cloud in step S3 to calculate a more dense traverse point cloud distribution and improve the accuracy of the plane calculation in S3.

[0132] The subsequent processing steps of this embodiment are the same as those of embodiment 1. However, since the dense point cloud generated by binocular vision is introduced as an additional spatial constraint, this embodiment effectively compensates for the shortcomings of the sparse and limited range of the lidar point cloud in embodiment 1, improves the accuracy and stability of the vertical plane fitting, and ultimately makes the calculated sag value more reliable. Embodiment

[0133] The embodiment of the present application discloses a visual sag detection device, mainly including a hardware layer and a software processing unit. The hardware layer includes a camera module and a lidar (Lidar) module, which are rigidly fixed together, and the relative position relationship (i.e. the rigid transformation matrix from the Lidar coordinate system to the camera coordinate system ) is obtained through installation and subsequent periodic calibration to ensure the conversion accuracy between the two sensor coordinate systems; the software processing unit is responsible for executing all algorithm processes of the present application.

[0134] The above only describes the preferred embodiments of the present application, but the specific embodiments described herein are only used to explain the present application and do not limit the present application. Any skilled person in the art can make any simple modification, equivalent change and modification to the above embodiments without departing from the principles and spirits of the present application according to the technical essence of the present application, which shall be included in the protection scope of the present application.

Claims

1. A method of visual sag detection, characterized by, The method comprises the following steps: S1, pre-calibrate the camera and the laser radar external parameters, obtain the internal parameters of the camera, the laser radar and the relative external parameters between them; S2, use the camera and the laser radar to synchronously acquire image data and three-dimensional point cloud data of the transmission line to be measured respectively, the image data acquired by the camera contains the complete state of the transmission line, and the laser radar acquires local three-dimensional point cloud data of the transmission line at a short distance; S3, fitting is performed on the local three-dimensional point cloud of the transmission line collected by the laser radar, part of the point cloud of the transmission line is extracted, the gravity direction information provided by the gravity acceleration sensor, and the relative external parameters between the laser radar and the camera are combined to determine the vertical plane where the transmission line is located in the camera coordinate system by constructing a joint optimization function for solving the vertical plane, that is, the equation of the vertical plane where the transmission line is located in the camera coordinate system is obtained; the determination mode of the vertical plane where the transmission line is located comprises: fitting the point cloud subset of the transmission line to screen a high-confidence reference point set, calculating the internal vector of the plane by using the coordinates in the point set, performing cross multiplication operation on the internal vector of the plane and the gravity direction vector to obtain the normal vector of the vertical plane, and then establishing the plane equation; S4, the image pixel of the transmission line is segmented in the image collected by the camera by using an image semantic segmentation method, and the image pixels belonging to each transmission line are grouped by using a region growing segmentation algorithm to determine the pixel coordinates of each transmission line in the camera coordinate system; S5, the pixel coordinates of the transmission line in the camera coordinate system obtained in step S4 and the equation of the vertical plane where the transmission line is located in the camera coordinate system obtained in step S3 are used to project the segmented transmission line image pixels to the vertical plane, the transmission line image pixels projected to the vertical plane are expanded to three-dimensional space point cloud in the camera coordinate system, the catenary of the transmission line is fitted based on the three-dimensional space point cloud, and the sag value of the transmission line is calculated; the calculation mode of the sag value is that the catenary model is fitted based on the local three-dimensional point cloud of the transmission line, the lowest point and two hanging points of the catenary model curve are determined, and the vertical distance between the lowest point and the two hanging points is calculated.

2. The visual sag detection method of claim 1, wherein The camera and the laser radar are installed on a rigid structure, and the camera is a monocular camera or a binocular camera.

3. The visual sag detection method of claim 2, wherein, When the camera is a binocular camera, the dense point cloud information of the binocular camera is generated when the vertical plane where the transmission line is located is determined in step S3, and the dense point cloud information space coordinate constraint term of the binocular camera is included in the joint optimization function for solving the vertical plane.

4. The visual sag detection method of claim 1, wherein The internal parameters obtained in step S1 include the focal length, resolution and radar field of view angle of the camera; and the relative external parameters include the relative displacement and rotation angle.

5. The visual sag detection method of claim 1, wherein, After step S1 is performed and before step S2 is performed, the following operation is further performed: In the camera coordinate system, a point cloud subset corresponding to the static background is identified, and the point cloud subset is matched with the static background features in the image data to calibrate or verify the preset rigid transformation relationship between the laser radar and the camera in real time.

6. A visual sag detection system as claimed in any one of claims 1 to 5, wherein The method comprises the following steps: The camera is used for collecting image data of the transmission line; The laser radar is used for collecting three-dimensional point cloud data; The gravity acceleration sensor is used for acquiring the gravity direction and calculating the vertical plane where the transmission line is located; a parameter calibration module, configured to pre-calibrate camera and laser radar external parameters to obtain relative external parameters therebetween; a data acquisition module, configured to acquire image data and three-dimensional point cloud data of a power transmission conductor to be measured; a plane construction module, configured to fit local three-dimensional point cloud of the conductor collected by the laser radar to determine a vertical plane where the complete conductor is located, and then obtain a pose of the conductor vertical plane in a camera coordinate system by using relative external parameters of the camera and the laser radar; a two-dimensional pixel segmentation module, configured to segment image pixels of the conductor in an image collected by the camera, project the segmented image pixels of the conductor to the vertical plane by using the pose of the conductor vertical plane in the camera coordinate system obtained by the plane construction module; an sag calculation module, configured to fit a catenary of the conductor based on the image pixels of the conductor and calculate a sag value of the conductor.

7. The system of claim 6, wherein, The camera and the laser radar are mounted on a rigid structure, and the camera is a monocular camera or a binocular camera.

8. A visual sag detection device, characterized by comprise: a camera, a laser radar, a gravity acceleration sensor and a processing unit; the camera and the laser radar acquire image data and three-dimensional point cloud data of a power transmission conductor, the gravity acceleration sensor acquires a gravity direction in a camera coordinate system, and the processing unit calculates a sag value of the conductor based on the visual sag detection in any one of claims 1-7.

Images