A power line real-time posture estimation method for a distribution network live working robot

Abstract

The embodiment of the present application provides a kind of power line real-time posture estimation method of network matching live working robot, belong to automation and robot technical field.The method is by obtaining the RGB image and depth map about power line under the same frame, and according to the RGB image and depth map, the point cloud diagram about the power line is obtained, after the RGB image is input to Mask R-CNN model, the mask about power line can be obtained, then the mask and point cloud diagram are matched to obtain the point cloud about power line, and the point cloud about power line is fitted, to realize the estimation of the posture of power line.

Description

Technical Field

[0001] This invention relates to the fields of automation and robotics, and more specifically to a method for real-time attitude estimation of power lines for live-line working robots in power distribution networks. Background Technology

[0002] In existing robotic live-line work on power distribution networks, identifying and locating power lines under strong outdoor sunlight has always been a problem. Because the intensity of sunlight varies depending on the weather, images captured by RGB cameras tend to be relatively dark, making accurate identification and segmentation of target power lines impossible. There are many common power line segmentation methods, which can be broadly divided into two categories: one uses thresholding based on the difference between the power line's color and the background color, or color-based region growing; the other uses deep learning methods. While these two methods are relatively simple, they are significantly affected by the background and overall image color, resulting in less accurate identification and location of power lines. Summary of the Invention

[0003] The purpose of this invention is to provide a real-time attitude estimation method for power lines of a live-line working robot in a power distribution network. This method can realize real-time attitude estimation for multiple power lines in a power distribution network scenario.

[0004] To achieve the above objectives, embodiments of the present invention provide a method for real-time attitude estimation of power lines for live-line working robots in power distribution networks, the method comprising:

[0005] Acquire RGB images and depth maps of power lines;

[0006] A point cloud map of the electric field lines is obtained based on the RGB image and the depth map.

[0007] The RGB image of the electric field line is input into the trained Mask R-CNN model to obtain a mask of the electric field line.

[0008] Obtain the first pixel region of the electric field mask in the RGB image;

[0009] Obtain the second pixel region of each point in the point cloud image;

[0010] The first pixel region and the second pixel region are matched to obtain a point cloud corresponding to the mask, which is used as the first point cloud.

[0011] Determine whether the electric field line is a straight line based on the RGB image;

[0012] If the power line is determined to be a straight line, the first point cloud is fitted using the RANSAC algorithm to fit it into a straight line, and the unit space vector of the straight line is obtained.

[0013] Obtain the second centroid of the mask;

[0014] The position of the pixel at the second centroid of the mask in the first point cloud is taken as the position of the straight line, and the position of the straight line is the same as the position of the electric field line.

[0015] Optionally, if it is determined that the electric field line is not a straight line, the first point cloud is fitted with a B-spline curve to obtain the orientation of the electric field line.

[0016] Optionally, if it is determined that the electric field line is not a straight line, fitting the first point cloud with a B-spline curve to obtain the orientation of the electric field line includes:

[0017] Obtain a first point cloud of the power lines;

[0018] Divide the first point cloud with respect to the power line into regions;

[0019] Calculate the first centroid of all the first point clouds within each region;

[0020] The first centroid represents all the first point clouds within the region;

[0021] Obtain the first centroid of each region and sort the first centroids according to the X-axis direction;

[0022] Substitute the coordinates of the sorted first centroid into formula (1) of the cubic B-spline curve to determine the three-dimensional coordinates of the control points:

[0023] , formula (1)

[0024] in, Represents the control point The three-dimensional coordinates Let be a single parameter of the B-spline curve, and Variation within a preset threshold, , , Let these be the three-dimensional coordinates of the first centroid. Let be the basis functions of the B-spline curve. Let the index of the basis function be denoted as . Let be the degree of the basis function. , ;

[0025] The attitude of the electric field line is obtained based on the three-dimensional coordinates of the control points and formula (2):

[0026] , formula (2)

[0027] in, The orientation of the electric field line.

[0028] Optionally, obtaining the second centroid of the mask includes:

[0029] Obtain the grayscale value of the mask;

[0030] The zeroth and first moments of the mask image are obtained according to formula (3);

[0031]

[0032] , formula (3)

[0033]

[0034] in, The image of the mask is shown in The grayscale value of the point The zeroth moment of the image of the mask is represented. , The first moment of the image of the mask is represented by the image of the mask.

[0035] The second centroid of the mask is obtained according to formula (4):

[0036] , , formula (4)

[0037] in, This is the second centroid of the mask.

[0038] Optionally, the method includes:

[0039] Acquire images of the power lines under different conditions;

[0040] Construct the initial Mask R-CNN model;

[0041] The image of the power line is input into the initial Mask R-CNN model to train the initial Mask R-CNN model.

[0042] Optionally, acquiring images of the power lines under different conditions includes: acquiring images of the power lines under different lighting conditions during sunny days and images of the power lines under low light conditions on cloudy days.

[0043] Optionally, the method includes:

[0044] Obtain a first point cloud of the power lines;

[0045] Obtain the fitted centerline of the straight line obtained by the RANSAC algorithm;

[0046] The average distance from all the first point clouds to the fitted center line is obtained according to formula (5):

[0047] , formula (5)

[0048] in, The fitted centerline was obtained using the RANSAC algorithm. It is a point on the fitted center line. The average distance from all the first point clouds to the fitted center line. It is the unit space vector of the fitted center line. It is a point The distance to the fitted center line;

[0049] The diameter of the electric field line is obtained based on the average distance from the first point cloud to the fitted center line and formula (6):

[0050] , formula (6)

[0051] in, The diameter of the electric field line is given.

[0052] Optionally, the RGB image and depth map of the power line are acquired using a binocular camera.

[0053] Through the above technical solution, the present invention provides a real-time attitude estimation method for power lines of live-line working robots in power distribution networks. This method acquires RGB images and depth maps of power lines in the same frame, and obtains point cloud maps of power lines based on the RGB images and depth maps. After inputting the RGB images into the Mask R-CNN model, a mask of the power lines can be obtained. Then, the mask and the point cloud map are matched to obtain point clouds of the power lines. Finally, the point clouds of the power lines are fitted to achieve the attitude estimation of the power lines.

[0054] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0055] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0056] Figure 1 This is a flowchart of a real-time attitude estimation method for a live-line working robot in a power distribution network according to an embodiment of the present invention;

[0057] Figure 2 This is a flowchart illustrating the acquisition of the power line attitude when the power line is not straight, according to a real-time attitude estimation method for a live-line working robot in a power distribution network based on an embodiment of the present invention.

[0058] Figure 3 This is a flowchart of a method for obtaining the second centroid in a real-time attitude estimation of power lines for a live-line working robot in a distribution network, according to an embodiment of the present invention.

[0059] Figure 4 This is a flowchart of a method for obtaining a trained Mask R-CNN model for real-time attitude estimation of power lines for a live-line working robot according to an embodiment of the present invention.

[0060] Figure 5 This is a flowchart illustrating the method for obtaining the diameter of a power line in a real-time attitude estimation method for a live-line working robot according to an embodiment of the present invention.

[0061] Figure 6 This is a method for real-time attitude estimation of power lines for live-line working robots in distribution networks according to an embodiment of the present invention, for straight power lines in a distribution network scenario;

[0062] Figure 7 This invention relates to a method for real-time attitude estimation of power lines for live-line working robots in distribution networks, based on an embodiment of the present invention, for curved power lines in a distribution network scenario. Detailed Implementation

[0063] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0064] Figure 1 This is a flowchart illustrating a real-time attitude estimation method for power lines using a live-line working robot in a power distribution network, according to an embodiment of the present invention. In this invention, the method for obtaining the attitude of the power line may include:

[0065] In step S1, the RGB image and depth map of the power line are acquired.

[0066] In step S2, a point cloud map of the power lines is obtained based on the RGB image and the depth map.

[0067] In step S3, the RGB image of the power line is input into the trained Mask R-CNN model to obtain a mask of the power line.

[0068] In step S4, the first pixel region of the electric field mask in the RGB image is obtained.

[0069] In step S5, the second pixel region of each point cloud in the point cloud image is obtained.

[0070] In step S6, the first pixel region and the second pixel region are matched to obtain the point cloud corresponding to the mask as the first point cloud.

[0071] In step S7, it is determined whether the electric field lines are straight based on the RGB image.

[0072] In step S8, if the power line is determined to be a straight line, the first point cloud is fitted using the RANSAC algorithm to fit it into a straight line, and the unit space vector of the straight line is obtained.

[0073] In step S9, the second centroid of the mask is obtained.

[0074] In step S10, the position of the pixel of the second centroid of the mask is taken as the position of the straight line at the position of the first point cloud, and the position of the straight line is the same as the position of the electric field line.

[0075] In this invention, after acquiring the RGB image and corresponding depth map of a power line in the same frame, a point cloud map of the power line is obtained based on the acquired RGB image and depth map. The number of power lines can be one or more. The RGB image is input into a trained Mask R-CNN model to obtain the mask of the power line, which is the pixel region occupied by the power line in the RGB image. Because the point cloud map is obtained from the RGB image and depth map, all points in the point cloud map have corresponding pixel regions in the RGB image. Therefore, the mask can be matched with the point clouds in the point cloud map. After obtaining the first pixel region corresponding to the mask and the second pixel region of each point cloud, the first pixel region and the second pixel region are matched to obtain the point cloud corresponding to the mask. This point cloud can be the first point cloud, which is the point cloud related to the power line. After obtaining the first point cloud, it can be determined whether the power line is a straight line based on the acquired RGB image. Figure 6As shown, if the power line is the main line on an overhead power line, and the value being pulled is relatively large, then the power line can be treated as a straight line to solve its attitude. The RANSAC algorithm can be used to fit the first point cloud to form a straight line. After fitting with the RANSAC algorithm, the unit space vector of the straight line can be obtained, which can be the unit space vector of the power line. After obtaining the unit space vector of the power line, the second centroid of the mask about the power line can be obtained. Because the position of the pixel of the second centroid corresponds to the position of the pixel region in the first point cloud, the position of the pixel of the second centroid of the mask in the first point cloud can be used as the position of the straight line. Since the straight line is fitted with the first point cloud about the power line, the position of the straight line can be the same as the position of the power line. Therefore, the position of the pixel of the centroid of the mask in the first point cloud can be used as the position of the power line.

[0076] In one embodiment of the present invention, such as Figure 1 As shown, the orientation of a power line can also be obtained when it is determined that the power line is not a straight line.

[0077] In step S11, the orientation of the electric field line is obtained by fitting the first point cloud of the B-spline curve.

[0078] like Figure 7 As shown, when the electric field line is disconnected and suspended, it is naturally curved and cannot be fitted to the first point cloud using the RANSAC algorithm, nor can the algorithm be used to solve the attitude of the electric field line. In this case, a B-spline curve can be used to fit the first point cloud and then obtain the curve equation of the electric field line. The curve equation of the electric field line can then represent the attitude of the electric field line.

[0079] In one embodiment of the present invention, such as Figure 2 As shown, the first point cloud is fitted with a B-spline curve to obtain the attitude of the electric field lines, including:

[0080] In step S12, the first point cloud about the power lines is obtained.

[0081] In step S13, the first point cloud about the power line is divided into regions.

[0082] In step S14, the first centroid of all first point clouds within each region is calculated.

[0083] In step S15, the first centroid represents all the first point clouds within the region.

[0084] In step S16, the first centroid of each region is obtained, and the first centroids are sorted according to the X-axis direction.

[0085] In step S17, the sorted first centroid coordinates are substituted into formula (1) of the 3rd degree B-spline curve segment to determine the three-dimensional coordinates of the control points:

[0086] , formula (1)

[0087] in, Representative control point The three-dimensional coordinates Let be a single parameter of the B-spline curve, and Variation within a preset threshold, , , The three-dimensional coordinates of the first centroid Let be the basis functions of the B-spline curve. Let the index of the basis function be denoted as . Let be the degree of the basis function. , .

[0088] In step S18, the attitude of the electric field line is obtained based on the three-dimensional coordinates of the control points and formula (2):

[0089] , formula (2)

[0090] in, The orientation of the power line.

[0091] In this invention, when the power line is in a naturally curved state, the RANSAC algorithm cannot be directly used to fit the first point cloud. Therefore, the first point cloud can be fitted using a B-spline curve. The first point cloud of the power line can be obtained, and the first point cloud can be divided into regions, each with the same range. Therefore, the first centroid of all point clouds within a region can be calculated, and this first centroid can represent all the first point clouds within that region. Thus, after obtaining the first centroid of each region and sorting it, a discrete point curve representing the changing trend of the point cloud centerline can be obtained. This discrete point curve representing the changing trend of the point cloud centerline can represent the power line or the B-spline curve. Therefore, the coordinates of these first centroids can be substituted into formula (1) of a cubic B-spline curve to determine the three-dimensional coordinates of the control points. This B-spline curve can represent the attitude of the power line. The three-dimensional coordinates of the control points of the B-spline curve can be obtained by formula (1). Then, the three-dimensional coordinates of the control points can be substituted into formula (2) to obtain the parametric equation of the B-spline curve. The equation of the B-spline curve can represent the attitude of the electric field line.

[0092] In one embodiment of the present invention, such as Figure 3 As shown, obtaining the second centroid of the mask may include:

[0093] In step S19, the grayscale value of the mask is obtained.

[0094] In step S20, the zeroth and first moments of the mask image are obtained according to formula (3):

[0095]

[0096] , formula (3)

[0097]

[0098] in, The image representing the mask is in The grayscale value of the point The zeroth moment of the image representing the mask. , The first moment of the mask image is represented.

[0099] In step S21, the second centroid of the mask is obtained according to formula (4):

[0100] , , formula (4)

[0101] in, It is the second centroid of the mask.

[0102] In this invention, the images of each mask obtained through the Mask R-CNN model are already single-channel binary images. Therefore, the centroid of the mask can be obtained from the zeroth and first moments of the mask image. Since the first two dimensions of the point cloud image and the first two dimensions of the RGB image are the same, after obtaining the second centroid, a point cloud that matches the second centroid can be found in the point cloud image. Then, the position of this point cloud can represent the position of the line containing the point cloud.

[0103] In one embodiment of the present invention, such as Figure 4 As shown, training this Mask R-CNN model can include:

[0104] In step S22, images of power lines under different conditions are acquired.

[0105] In step S23, the initial Mask R-CNN model is constructed.

[0106] In step S24, the image of the power line is input into the initial Mask R-CNN model to train the initial Mask R-CNN model.

[0107] The quantity and richness of the dataset greatly affect the accuracy of deep learning training results. In this invention, the initial Mask R-CNN model, before training, may output masks that do not meet the standards when segmenting images. Therefore, it is necessary to input images of power lines under different environments and conditions into the initial Mask R-CNN model to train it. This ensures that when images of power lines are input into the trained Mask R-CNN model, the model can output masks of power lines that meet the standards.

[0108] In one embodiment of the invention, when acquiring an image of a power line, one can ascend to the underside of the power line by riding in an insulated bucket of a boom truck and handheld with a camera to capture the image. The number of power lines in the image is between 1 and 6, and the shooting distance is between 0.5 meters and 3 meters. The captured images can include conditions under various lighting conditions at different times of day, as well as conditions under low light conditions on cloudy days. The background of the image can mostly be the sky, but can also include photos with tree branches, leaves, and buildings as backgrounds.

[0109] In one embodiment of the present invention, when it is determined that the electric field line is a straight line, determining the diameter of the electric field line may include:

[0110] In step S25, the first point cloud about the power lines is obtained.

[0111] In step S26, the fitted center line of the straight line obtained by the RANSAC algorithm is obtained.

[0112] In step S27, the average distance from all first point clouds to the fitted center line is obtained according to formula (5):

[0113] , formula (5)

[0114] in, The fitted centerline was obtained using the RANSAC algorithm. It is a point on the fitted center line. The average distance from all first point clouds to the fitted center line. It is the unit space vector of the fitted centerline. It is a point Distance to the fitted center line.

[0115] In step S28, the diameter of the electric field line is obtained based on the average distance from the first point cloud to the fitted center line and formula (6):

[0116] , formula (6)

[0117] in, The diameter of the power line.

[0118] Since the electric field line is a narrow, strip-shaped, black, low-texture object, the point cloud obtained from the RGB image and depth map is not a semi-cylinder, but rather a strip-shaped planar point cloud. After obtaining the fitting center line of the straight line using the RANSAC algorithm, the average distance from the first point cloud of the electric field line to the fitting center line can be calculated using formula (5). Because the first point cloud can be symmetrical with respect to the fitting center line, when the electric field line is close to a straight line, the diameter of the electric field line can be calculated using the average distance from the first point cloud to the fitting center line and formula (6).

[0119] In one embodiment of the present invention, workers can acquire RGB images and depth maps of power lines using a binocular camera. When acquiring RGB images and depth maps of power lines using the binocular camera, the two cameras can be placed completely parallel with the same focal length to obtain the RGB image and depth map of the power lines in the same frame.

[0120] Through the above technical solution, the present invention provides a real-time attitude estimation method for power lines of live-line working robots in power distribution networks. This method acquires RGB images and depth maps of power lines in the same frame, and obtains point cloud maps of power lines based on the RGB images and depth maps. After inputting the RGB images into the Mask R-CNN model, a mask of the power lines can be obtained. Then, the mask and the point cloud map are matched to obtain point clouds of the power lines. Finally, the point clouds of the power lines are fitted to achieve the attitude estimation of the power lines.

[0121] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0122] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for real-time attitude estimation of power lines for live-line working robots in power distribution networks, characterized in that, The method includes: Acquire RGB images and depth maps of power lines; A point cloud map of the electric field lines is obtained based on the RGB image and the depth map. The RGB image of the electric field line is input into the trained Mask R-CNN model to obtain a mask of the electric field line. Obtain the first pixel region of the electric field mask in the RGB image; Obtain the second pixel region of each point in the point cloud image; The first pixel region and the second pixel region are matched to obtain a point cloud corresponding to the mask, which is used as the first point cloud. Determine whether the electric field line is a straight line based on the RGB image; If the power line is determined to be a straight line, the first point cloud is fitted using the RANSAC algorithm to fit it into a straight line, and the unit space vector of the straight line is obtained. Obtain the second centroid of the mask; The position of the pixel at the second centroid of the mask in the first point cloud is taken as the position of the straight line, and the position of the straight line is the same as the position of the electric field line; If it is determined that the electric field line is not a straight line, the first point cloud is fitted with a B-spline curve to obtain the attitude of the electric field line, including: Obtain a first point cloud of the power lines; Divide the first point cloud with respect to the power line into regions; Calculate the first centroid of all the first point clouds within each region; The first centroid represents all the first point clouds within the region; Obtain the first centroid of each region and sort the first centroids according to the X-axis direction; Substitute the coordinates of the sorted first centroid into the formula representing the B-spline curve segment to determine the three-dimensional coordinates of the control points: The three-dimensional coordinates of the control points are substituted into the formula representing the attitude of the electric field line to obtain the attitude of the electric field line.

2. The method according to claim 1, characterized in that, A B-spline curve segment is a cubic segment, and the formula for representing a B-spline curve segment is: , Official (1) in, Represents the control point The three-dimensional coordinates Let be a single parameter of the B-spline curve, and Variation within a preset threshold, , , Let the three-dimensional coordinates of the first centroid be... Let be the basis functions of the B-spline curve. Let the index of the basis function be denoted as . Let be the degree of the basis function. , ; The formula for expressing the attitude of an electric field line is: , Official (2) in, The orientation of the electric field line.

3. The method according to claim 1, characterized in that, Obtaining the second centroid of the mask includes: Obtain the grayscale value of the mask; The zeroth and first moments of the mask image are obtained according to formula (3); , Official (3) in, The image of the mask is shown in The grayscale value of the point The zeroth moment of the image of the mask is represented. , The first moment of the image of the mask is represented by the image of the mask. The second centroid of the mask is obtained according to formula (4): , , Official (4) in, This is the second centroid of the mask.

4. The method according to claim 1, characterized in that, The method includes: Acquire images of the power lines under different conditions; Construct the initial Mask R-CNN model; The image of the power line is input into the initial Mask R-CNN model to train the initial Mask R-CNN model.

5. The method according to claim 4, characterized in that, Acquiring images of the power lines under different conditions includes: acquiring images of the power lines under various lighting conditions during sunny days and images of the power lines under low light conditions on cloudy days.

6. The method according to claim 1, characterized in that, The method includes: Obtain a first point cloud of the power lines; Obtain the fitted centerline of the straight line obtained by the RANSAC algorithm; The average distance from all the first point clouds to the fitted center line is obtained according to formula (5): , Official (5) in, The fitted centerline was obtained using the RANSAC algorithm. It is a point on the fitted center line. The average distance from all the first point clouds to the fitted center line. It is the unit space vector of the fitted center line. It is a point The distance to the fitted center line; The diameter of the electric field line is obtained based on the average distance from the first point cloud to the fitted center line and formula (6): , Official (6) in, The diameter of the electric field line is given.

7. The method according to claim 1, characterized in that, The RGB image and depth map of the power line were acquired using a binocular camera.

Images