A method for identifying, positioning and grasping a drainage wire

Abstract

The present invention proposes a method for identifying, positioning and grasping a drainage line. During the automatic operation of live connection and ignition operation, the camera at the robotic arm is used to accurately identify the drainage line, so that the gripper is positioned directly below the drainage line and the X-axis or Y-axis of the gripper with the camera is parallel to the exposed wire part of the drainage line to complete the grasping task. When positioning the gripper, the method identifies the exposed wire part based on the instance segmentation of the drainage line and fits the positions of its endpoints. Specifically, by controlling the robotic arm to translate a preset distance in the four directions of right, down, left and up respectively to draw a spatial virtual rectangular frame, a user coordinate system is established with the plane of this rectangular frame as the coordinate plane. At this time, the pixel coordinates of the drainage line endpoints in the four images are P1, P2, P3, and P4, and the PnP method is used to solve the motion problem of the gripper point in the 3D-2D space coordinate system. The present invention is applicable to the 10kV live connection and ignition operation robot, meeting the requirements for accurate identification and grasping of the drainage line during the automatic operation of live connection and ignition operation.

Description

Technical Field

[0001] The present invention relates to the technical field of power equipment, and in particular to a method for identifying, positioning and grasping a bypass conductor. Background Art

[0002] With the continuous improvement of the social demand for the safety and stability of power supply, the requirements for the operation quality of operators are getting higher and higher. It is very difficult for operators to improve the operation quality and efficiency again on the premise of ensuring operation safety. At present, manual live working operations are all carried out by using manual tools to cut and strip the cable outer sheath and connect the bypass conductor. However, the distribution network lines are usually very compact, and the phase distance between lines is small. When operators connect the bypass conductor live, it is easy to cause a short circuit and trigger accidents such as personal injury and death.

[0003] At present, in China, replacing manual labor with a robotic arm to complete the live connection of bypass conductors in the distribution network has extremely broad application prospects. However, during the operation of the robotic arm, a large number of application scenarios for visual auxiliary positioning are required, such as: identification of conductors, identification of bypass conductors, identification of connection positions, etc. Among them, due to the non-fixed position and posture of the bypass conductor itself, it is difficult to visually identify and accurately grasp the bypass conductor. Therefore, a reliable algorithm for identifying, positioning and grasping the bypass conductor needs to be designed, which is applicable to the 10kV live connection operation robot and meets the requirements for accurately identifying and grasping the bypass conductor during the automatic operation of the live connection operation. Summary of the Invention

[0004] The present invention provides a method for identifying, positioning and grasping a bypass conductor, which is applicable to the 10kV live connection operation robot and meets the requirements for accurately identifying and grasping the bypass conductor during the automatic operation of the live connection operation.

[0005] The present invention adopts the following technical solutions.

[0006] A method for identifying, positioning and grasping a bypass conductor is used to accurately identify the bypass conductor through a camera at the robotic arm during the automatic operation of the live connection operation, and then grasp the exposed conductor part with the gripper of the robotic arm. The method positions the gripper directly below the bypass conductor and makes the X-axis or Y-axis of the gripper with the camera parallel to the exposed conductor part of the bypass conductor to complete the grasping task. When positioning the gripper, the method identifies the exposed conductor part based on the instance segmentation of the bypass conductor and fits the position of its endpoints. Specifically, by controlling the robotic arm to translate a preset distance in the four directions of right, down, left, and up respectively to draw a spatial virtual rectangular frame, a user coordinate system is established with the plane of this rectangular frame as the coordinate plane. At this time, the pixel coordinates of the bypass conductor endpoints in the four images are P 1 、P 2 、P 3 、P 4 , and the PnP method is used to solve the motion problem of the gripper point in the 3D-2D space coordinate system.

[0007] The method uses the SOLOV2 image segmentation technology to train a multi-classification network for instance segmentation of the drainage line to identify the bare conductor at the stripping position.

[0008] The PnP method solves the motion problem of the drainage line endpoints. Specifically, it uses EPnP to solve the pose change required for the camera to reach the drainage line position according to the coordinates of the fiducial points P 1 、P 2 、P 3 、P 4 in the camera coordinate system and the coordinates of the fiducial points in the virtual coordinate plane. The process is as follows:

[0009] Calculate the coordinates of the fiducial points in the camera reference frame:

[0010]

[0011] Calculate the coordinates of the 3D reference points in the camera reference frame:

[0012]

[0013] Calculate the centroid and matrix A:

[0014]

[0015]

[0016] Calculate the centroid p i} i＝1，...n of {p O and matrix B

[0017]

[0018]

[0019] Calculate the height H where the camera is located

[0020] H = B T A

[0021] Calculate the SVD decomposition of H:

[0022] H = U∑V T

[0023] Calculate the rotation R required for the camera pose change:

[0024] R = UV T

[0025] Calculate the translation t of the camera pose:

[0026]

[0027] The exposed wire part of the drainage wire is located at the wire that has been stripped in the hanging section of the drainage wire, and its length is a known value. Let the end point of the drainage wire be A and the junction of the exposed wire be B. Taking to represent the spatial vector from B to A, the camera and the position of the robot gripper are installed horizontally. Then, when the robot gripper moves and the camera reaches the horizontal position with the exposed wire position after the attitude transformation about the Y-axis, and the camera moves through the translation transformation solved by the EPnP algorithm, it can reach the grasping position of the drainage wire. The specific calculation methods for the attitude transformation amount and the movement amount of the camera are as follows:

[0028] Step A1: Calculate the angle θ between and the Y-axis of the camera in the camera coordinate system. The formula is

[0029]

[0030] Step A2: Calculate the rotation axis of and the Y-axis which is given by the cross product of and

[0031] Let a = (a 1 , a 2 , a 3 ) and b = (b 1 , b 2 , b 3 ) be the unit vectors of and the Y-axis respectively. Then the cross product is expressed as:

[0032]

[0033] Step A3: Calculate the rotation vector Step A4: Take the middle position of the exposed drainage wire as the grasping point. The transformation relationship between the camera and the grasping point is:

[0034]

[0035] The subsequent wire grasping operation includes the following steps:

[0036] Step A5: Identify the drainage wire target in the image through the deep learning object detection algorithm;

[0037] Step A6: Calculate the pose of the drainage wire relative to the position and orientation of the camera according to the p3p algorithm;

[0038] Step A7: Calculate the spatial coordinates of the drainage wire in the robot coordinate system through the pose;

[0039] Step A8: Control the gripper of the robotic arm to move to the coordinate position of the drainage line for wire grasping operation;

[0040] During the wire grasping operation, the gripper should avoid contacting the exposed wire part. The recognition of the exposed wire part adopts a combined algorithm that uses the HIS space model to segment the background image, applies morphological methods to denoise the environment, segments the exposed wire part, uses the region growing method to further segment the exposed wire part, and uses the morphological erosion method to remove the fragments generated during the segmentation for the segmented exposed wire;

[0041] The combined algorithm includes the following steps:

[0042] Step B1: Select the initial growing pixel point A on the foreground image of the drainage line captured by the camera, marked as f(i, j). In the neighborhood of A, the growing criterion is that the gray value of the point to be detected differs from the gray value of the growing point by 1 or 0;

[0043] After the first region growing, f(i - 1, j), f(i, j - 1), f(i, j + 1) all differ from the gray value of the selected center point by 1, so they are merged. After the second region growing, f(i + 1, j) is merged;

[0044] After the third region growing, f(i + 1, j - 1), f(i + 2, j) are merged;

[0045] When there are no pixel points that meet the growing criterion, the region on the target image stops growing;

[0046] The purpose of using mathematical morphological erosion is to remove some of the fragments generated during image segmentation, further optimize the image details, and improve the image quality;

[0047] The erosion method is:

[0048] For the target image X, use the set B to erode the set A, defined as: Translate the structure element B by a to get Ba. If Ba is contained in X, record this a point. The set composed of all a points that meet the above conditions is called the result of X eroded by B.

[0049] The recognition formula expression for the exposed wire is:

[0050] The robotic arm translates 5 centimeters to the right, down, left, and up respectively to draw a spatial virtual rectangular frame.

[0051] The drainage line is an arc-shaped wire hanging on both sides of the tower pole during the live connection operation, in a suspended state and with a section of the stripped wire exposed.

[0052] The robotic arm is the robotic arm of a live wire connection and ignition operation robot. During the automatic live wire connection and ignition operation, first manually guide the movement of the robotic arm to capture an image of the bypass wire by the camera on the robotic arm. Then, perform instance segmentation manually, that is, click on a point on the image to determine the stripping section of the bypass wire and a grasping point on the insulated section behind the stripping section that the robotic arm needs to grasp with the gripper, so that the robot can accurately identify the entire bypass wire on the image and filter the background image. Then, the robot controls the movement position and rotation direction of the robotic arm, calculates the spatial coordinates of the specified grasping point on the bypass wire according to the relationship between the new position of the wire and the movement of the robotic arm, and controls the robotic arm to accurately grasp the bypass wire.

[0053] The solution of the present invention is based on a bypass wire recognition, positioning and grasping algorithm, which is applicable to a 10kV live wire connection and ignition operation robot, meets the requirements of accurate recognition and grasping of the bypass wire during the automatic live wire connection and ignition operation process, can meet the operation and maintenance requirements of live working, save labor costs, improve safety and reliability, and reduce the impact on other areas of the circuit. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present invention will be further described in detail below with reference to the drawings and specific embodiments:

[0055] Att Figure 1 is a schematic diagram of the bypass wire;

[0056] Att Figure 2 is a schematic diagram of the robotic arm gripper grasping the bypass wire;

[0057] In the figure: 1 - bypass wire; 2 - bare wire part; 3 - bypass wire end point; 4, robotic arm; 5 - camera. SPECIFIC EMBODIMENTS

[0058] As shown in the figure, a method for recognizing, positioning and grasping a bypass wire is used to accurately recognize the bypass wire 1 by the camera 5 at the robotic arm 4 during the automatic live wire connection and ignition operation process, and then grasp its bare wire part 2 with the gripper of the robotic arm. The method positions the gripper directly below the bypass wire and makes the X-axis or Y-axis of the gripper with the camera parallel to the bare wire part of the bypass wire to complete the grasping task; when positioning the gripper, the method recognizes the bare wire part based on bypass wire instance segmentation and fits the position of its end points. Specifically, by controlling the robotic arm to translate a preset distance in the four directions of right, down, left, and up respectively to draw a spatial virtual rectangular frame, and establishing a user coordinate system with the plane of this rectangular frame as the coordinate plane. At this time, the pixel coordinates of the bypass wire end points in the four images are P 1 、P 2 、P 3 、P 4, the PnP method is used to solve the problem of the motion of the gripper point in the 3D-2D space coordinate system.

[0059] The method uses the SOLOV2 image segmentation technology to train a multi-classification network for instance segmentation of the drainage line to identify the bare conductor at the stripping position.

[0060] The PnP method solves the motion problem of the end point 3 of the drainage line. Specifically, EPnP is used to solve the pose change required for the camera to reach the position of the drainage line according to the coordinates of the fiducial points P 1 、P 2 、P 3 、P 4 in the camera coordinate system and the coordinates of the fiducial points on the virtual coordinate plane. The process is as follows:

[0061] Calculate the coordinates of the fiducial points in the camera reference frame:

[0062]

[0063] Calculate the coordinates of the 3D reference points in the camera reference frame:

[0064]

[0065] Calculate the centroid and matrix A:

[0066]

[0067]

[0068] Calculate {p i} i＝1，...n the centroid p O and matrix B

[0069]

[0070]

[0071] Calculate the height H where the camera is located

[0072] H = B T A

[0073] Calculate the SVD decomposition of H:

[0074] H = U∑V T

[0075] Calculate the rotation R required for the camera pose change:

[0076] R = UV T

[0077] Calculating the translational displacement \(t\) of the camera pose:

[0078]

[0079] The exposed wire part of the drainage wire is located at the wire that has been stripped in the hanging section of the drainage wire, and its length is a known value. Let the end point of the drainage wire be \(A\), and the junction of the exposed wire be \(B\). Taking to represent the spatial vector from \(B\) to \(A\), the camera is installed horizontally with the position of the robot gripper. Then, when the robot gripper moves, and the camera reaches the horizontal position of the exposed wire after the attitude transformation around the \(Y\)-axis, and the translational transformation solved by the EPnP algorithm for the camera movement is performed, the camera can reach the grasping position of the drainage wire. The specific calculation methods for the attitude transformation amount and the movement amount of the camera are as follows:

[0080] Step A1: Calculate the angle \(\theta\) between the

[0081]

[0082] and the \(Y\)-axis of the camera in the camera coordinate system. The formula is Rotation axis of the around the and The cross product gives. Let \(a=(a 1 ,a 2 ,a 3 ), b=(b 1 ,b 2 ,b 3 ) be the unit vectors of and the \(Y\)-axis respectively. Then the cross product is expressed as:

[0083]

[0084] Step A3: Calculate the rotation vector

[0085] Step A4: Take the middle position of the exposed drainage wire as the grasping point. The transformation relationship between the camera and the grasping point is:

[0086]

[0087] The subsequent wire grasping operation includes the following steps:

[0088] Step A5: Identify the drainage wire target in the image through the deep learning object detection algorithm;

[0089] Step A6: Calculate the pose of the drainage wire relative to the position and orientation of the camera according to the P3P algorithm;

[0090] Step A7: Calculate the spatial coordinates of the drainage wire in the robot coordinate system through the pose;

[0091] Step A8: Control the gripper of the robotic arm to move to the coordinate position of the drainage line for wire grasping operation;

[0092] During the wire grasping operation, the gripper should avoid contacting the exposed wire part. The combination algorithm of using the HIS spatial model to segment the background image is adopted to identify the exposed wire part. The morphological method is used to denoise the environment, segment the exposed wire part, and the region growing method is used to further segment the exposed wire part. The morphological erosion method is used to remove the fragments generated during the segmentation for the segmented exposed wire;

[0093] The combination algorithm includes the following steps:

[0094] Step B1: Select the initial growth pixel point A on the foreground image of the drainage line captured by the camera, marked as f(i, j). In the neighborhood of A, the growth criterion is that the gray value of the point to be detected differs from the gray value of the growth point by 1 or 0;

[0095] After the first region growth, f(i - 1, j), f(i, j - 1), f(i, j + 1) all differ from the gray value of the selected center point by 1, so they are merged. After the second region growth, f(i + 1, j) is merged;

[0096] After the third region growth, f(i + 1, j - 1), f(i + 2, j) are merged;

[0097] Step B4: When there are no pixel points that meet the growth criterion, the region on the target image stops growing;

[0098] The purpose of using mathematical morphological erosion is to remove some fragments generated during the image segmentation, further optimize the image details, and improve the image quality;

[0099] The erosion method is as follows:

[0100] For the target image X, use the set B to erode the set A, defined as: Translate the structure element B by a to get Ba. If Ba is contained in X, record this a point. The set composed of all a points that meet the above conditions is called the result of X eroded by B.

[0101] The recognition formula expression of the exposed wire is:

[0102] The robotic arm translates 5 centimeters to the right, down, left, and up directions respectively to draw a spatial virtual rectangular frame.

[0103] The drainage wire is an arc-shaped wire hung on both sides of the tower pole during live connection and ignition operation, in a suspended state with a section of stripped wire exposed.

[0104] The robotic arm is the robotic arm of the live connection and ignition operation robot. During the automatic operation of the live connection and ignition operation, first, the robotic arm is manually guided to move so that the camera on the robotic arm captures an image of the drainage wire. Then, manual instance segmentation is performed, that is, a point on the stripped section of the drainage wire is determined by clicking on the image, and a grasping point on the insulated section behind the stripped section that the robotic arm needs to grasp with the gripper is determined, enabling the robot to accurately identify the entire drainage wire on the image and filter the background image. Then, the robot controls the movement position and rotation direction of the robotic arm, calculates the spatial coordinates of the specified grasping point on the drainage wire based on the relationship between the new position of the wire obtained and the movement of the robotic arm, and controls the robotic arm to accurately grasp the drainage wire.

Claims

1. A method for identifying, positioning and grasping a drainage line, which is used in the automatic operation process of live connection and ignition operation. The drainage line is accurately identified by a camera at the robotic arm, and then the exposed wire part thereof is grasped by the gripper of the robotic arm. It is characterized in that: The method positions the gripper directly below the drainage line, and the X-axis or Y-axis of the gripper with a camera is parallel to the exposed wire part of the drainage line to complete the grasping task. When positioning the gripper, the method identifies the exposed wire part based on the instance segmentation of the drainage line and fits the positions of its endpoints. Specifically, by controlling the robotic arm to translate a preset distance in the four directions of right, down, left, and up respectively to draw a spatial virtual rectangular frame, a user coordinate system is established with the plane of this rectangular frame as the coordinate plane. At this time, the pixel coordinates of the endpoints of the drainage line in the four images are P 1 , P 2 , P 3 , P 4 . The PnP method is used to solve the motion problem of the gripper points in the 3D-2D space coordinate system; The bare wire part of the drainage wire is located at the wire that has been stripped and processed in the hanging section of the drainage wire, and its length is a known value; let the end point of the drainage wire be A and the junction of the bare wire be B. Taking to represent the spatial vector from B to A, the camera is installed horizontally with the position of the manipulator gripper. Then, when the manipulator gripper moves to make the camera reach the horizontal position of the bare wire after the attitude transformation of the Y axis, and the camera moves through the translation transformation solved by the EPnP algorithm, it can reach the grasping position of the drainage wire. The specific calculation methods for the attitude transformation amount and the movement amount of the camera are as follows: Step A1, calculate the angle θ between the camera Y-axis and the camera coordinate system, and the formula is Step A2, calculate the axis of rotation of the shaft given by the and cross product Let \(a=(a 1 ,a 2 ,a 3 )\) and \(b=(b 1 ,b 2 ,b 3 )\) be the unit vectors of and the Y-axis respectively, then the cross product is expressed as: Step A3, calculate the rotation vector Step A4: Take the middle position of the exposed drainage line as the grasping point. The transformation relationship between the camera and the grasping point is: The subsequent wire grasping operation includes the following steps: Step A5: Identify the drainage line target in the image through the deep learning object detection algorithm. Step A6: Calculate the pose of the drainage line relative to the position and orientation of the camera according to the P3P algorithm. Step A7: Calculate the spatial coordinates of the drainage line in the robot coordinate system through the pose. Step A8: Control the gripper of the robot arm to move to the coordinate position of the drainage line to perform the wire grasping action. During the wire grasping operation, the gripper needs to avoid contacting the exposed wire part. The identification of the exposed wire part adopts a combined algorithm for segmenting the background image using the HIS space model, uses the morphological method to denoise the environment, segments the exposed wire part, uses the region growing method to continue segmenting the exposed wire part, and uses the morphological erosion method to remove the fragments generated during the segmentation for the segmented exposed wire. The combined algorithm includes the following steps: Step B1: Select the initial growing pixel point as A on the foreground image of the drainage line captured by the camera, marked as f(i, j). In the neighborhood of A, the growing criterion is that the gray value of the point to be detected differs from the gray value of the growing point by 1 or 0. Step B2: After the first region growing, f(i - 1, j), f(i, j - 1), f(i, j + 1) all differ from the gray value of the selected center point by 1, so they are merged. After the second region growing, f(i + 1, j) is merged. Step B3: After the third region growing, f(i + 1, j - 1), f(i + 2, j) are merged. Step B4: When there are no pixel points that meet the growing criterion, the region on the target image stops growing. Step B5: The purpose of using mathematical morphological erosion is to remove some fragments generated during image segmentation, further optimize the image details, and improve the image quality. The erosion method is: For the target image X, the erosion of set A by set B is defined as: After translating the structuring element B by a to obtain Ba, if Ba is included in X, The set composed is called the result of X being eroded by B. The recognition formula for bare conductors is expressed as:

2. A method for identifying, positioning and grasping a drainage line according to claim 1, It is characterized in that: The method uses the SOLOV2 image segmentation technology to train a multi-classification network for performing instance segmentation of the drainage line to identify the exposed wire at the stripping position.

3. A method for identifying, positioning and grasping a drainage line according to claim 1, It is characterized in that: The PnP method is used to solve the problem of the movement of the end points of the drainage line. Specifically, EPnP is used to solve the pose change required for the camera to reach the position of the drainage line based on the coordinates of the fiducial points P 1 , P 2 , P 3 , P 4 in the camera coordinates and the coordinates of the fiducial points on the virtual coordinate plane. The process is as follows: Calculate the coordinates of the fiducial point in the camera reference frame: Calculate the coordinates of the 3D reference point in the camera reference frame: Calculate the centroid and matrix A: Calculate the centroid p of i { i＝1，...n} O and matrix B Calculate the height H where the camera is located H = B T A Calculate the SVD decomposition of H: H = U∑V T Calculate the rotation R required for the camera pose change: R = UV T Calculate the translation t of the camera pose:

4. A method for identifying, positioning and grasping a drainage line according to claim 1, It is characterized in that: The robotic arm translates 5 centimeters in the right, down, left, and up directions respectively to draw a spatial virtual rectangular frame.

5. A method for identifying, positioning and grasping a drainage line according to claim 1, It is characterized in that: The drainage line is an arc-shaped wire hung on both sides of the tower pole during live connection and energization operations. It is in a suspended state and has a section of stripped wire exposed.

6. A method for identifying, positioning, and grasping a drainage line according to claim 1, characterized in that: the robotic arm is the robotic arm of a live connection and energization operation robot. During the automatic live connection and energization operation, first, manually guide the movement of the robotic arm to allow the camera on the robotic arm to capture an image of the drainage line. Then, manually perform instance segmentation, that is, click on a point on the image to determine a point on the stripped section of the drainage line and a grasping point on the insulated section behind the stripped section that the robotic arm needs to grasp with the gripper, so that the robot can accurately identify the entire drainage line on the image and filter the background image. Then, the robot controls the movement position and rotation direction of the robotic arm, calculates the spatial coordinates of the specified grasping point on the drainage line according to the relationship between the new position of the wire obtained and the movement of the robotic arm, and controls the robotic arm to accurately grasp the drainage line.

Images