Mechanical arm control method and system for executing wire clamp disassembly and assembly operation

Abstract

The invention discloses a mechanical arm control method and system for executing wire clamp dismounting operation, and the method comprises the steps: obtaining a first color image and first point cloud data of a collected target operation position, carrying out the image segmentation and contour extraction of the first color image, and obtaining a first contour corresponding to each wire clamp; calculating a direction vector corresponding to the second point cloud data so as to output first pose information of the operation lead; calculating geometric information of a bounding box corresponding to each first contour to obtain a second color image of each wire clamp; performing attitude estimation on the second color image to obtain second pose information of each wire clamp; and according to the first pose information and the second pose information, an action plan of the mechanical arm is generated, so that the mechanical arm executes the grabbing operation of the wire and the disassembly and assembly operation of the wire clamp according to the action plan, and automation of the disassembly and assembly operation of the wire clamp is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of mechanical equipment, and in particular to a method and system for controlling a mechanical arm for performing wire clamp disassembly and assembly operations. Background Art

[0002] With the application of technology in the process of power production, in order to ensure the safety of power workers, a large number of live operations have gradually been transformed from manual to live operation robots. During the process of disassembling and assembling wire clamps by live operation robots in distribution networks, the robot operation process mainly includes grabbing wires and disassembling and assembling wire clamps.

[0003] Currently, robot disassembly and assembly of wire clamps mainly relies on manual observation, constantly adjusting the position between the operating tool at the end of the robotic arm and the target wire clamp, and completing the task of disassembly and assembly of the wire clamp by manually controlling the robotic arm.

[0004] The method of manually adjusting the robotic arm relies on the staff's judgment of the position of the wire clamp. In the actual adjustment process, the staff's perception accuracy of the wire clamp cannot reach the centimeter level, which cannot meet the accuracy requirements of the State Grid when operating the cable. Furthermore, when performing the task of disassembling and assembling the wire clamp at high altitudes, the safety of the staff must be guaranteed, which cannot meet the safety requirements of live working. Summary of the invention

[0005] In order to solve the above technical problems, the present invention discloses a robot arm control method and system for performing wire clamp disassembly and assembly operations, which are used to automatically perform wire clamp disassembly and assembly operations and improve the efficiency and accuracy of disassembly and assembly.

[0006] In order to achieve the above object, the present invention discloses a robot arm control method for performing a wire clamp disassembly and assembly operation, comprising:

[0007] Acquire a first color image and first point cloud data of the collected target working position, and perform image segmentation on the first color image to obtain a mask image;

[0008] Extracting second point cloud data corresponding to the operating conductor from the first point cloud data, and performing contour extraction on the mask image to obtain first contours corresponding to each wire clamp in the target operating position; wherein the operating conductor is determined by comparing the first color image and the first point cloud data;

[0009] Calculating a direction vector corresponding to the second point cloud data and geometric information of a bounding box corresponding to each of the first contours;

[0010] Performing posture estimation on the second point cloud data according to the direction vector to obtain the first posture information corresponding to the working guide wire, and intercepting the first color image according to the geometric information to obtain the second color image corresponding to each wire clip;

[0011] Performing posture estimation on the second color image to obtain second posture information corresponding to each wire clip;

[0012] An action plan of the robot arm is generated according to the first posture information and the second posture information, so as to control the robot arm to perform a grasping operation of the wire and an assembly and disassembly operation of the wire clamp according to the action plan.

[0013] The present invention discloses a method for controlling a manipulator for performing a wire clamp disassembly and assembly operation. First, a first color image and a first point cloud data corresponding to the position where the operation is performed are obtained, and the operation wire and the operation wire clamp are determined according to the image and the point cloud data, so as to realize automatic perception of the wire clamp and the wire. Then, the second point cloud data corresponding to the operation wire is intercepted from the first point cloud data, so as to calculate the direction vector in the point cloud data, and then the wire posture fitting is performed in combination with the direction vector and the point cloud data, so as to control the manipulator according to the posture of the wire to realize the grasping operation of the wire. Furthermore, while the posture perception of the wire is being performed, the first color image is segmented to extract a mask image composed of multiple wire clamps contained in the current image, and then the contour of the mask image is extracted to extract the area of ​​each wire clamp in the image, and then the color image is segmented according to the area to obtain a color image containing only a single wire clamp, and then the posture estimation of the color image is performed to obtain the posture information corresponding to each wire clamp, and then the manipulator is controlled according to the posture of the wire clamp to realize the disassembly and assembly operation of the wire clamp, so as to realize the automatic perception of the wire clamp.

[0014] As a preferred example, the step of acquiring the first color image and the first point cloud data of the collected target operation position, and performing image segmentation on the first color image to obtain a mask image includes:

[0015] Determine the position of the operating guide wire by comparing the first color image and the first point cloud data;

[0016] Selecting a wire point from the first point cloud data according to the position, and obtaining the three-dimensional coordinates corresponding to the wire point, and generating a point cloud frame corresponding to the operating wire based on the preset point cloud frame size information with the three-dimensional coordinates as the center; wherein the point cloud frame includes the center point of the point cloud frame and the length, width and height of the point cloud frame;

[0017] The first color image is segmented by a preset deep learning algorithm to generate the mask image.

[0018] The present invention first determines the operating conductor by comparing the color image and the point cloud data, so as to grab the operating conductor to ensure the stability of the wire clamp disassembly and assembly task. Further, a point cloud frame is generated to intercept the point cloud data of the operating conductor, which is convenient for the subsequent completion of the posture perception of the conductor. Further, the color image is segmented to extract only the mask image corresponding to the wire clamp included in the color image, which is convenient for the subsequent posture perception of the wire clamp.

[0019] As a preferred example, the second point cloud data corresponding to the working guide wire is intercepted from the first point cloud data, and contour extraction is performed on the mask image to obtain the first contour corresponding to each wire clamp in the target working position, including:

[0020] intercepting second point cloud data within the point cloud frame from the first point cloud data;

[0021] A contour area formed by each of the first contours is acquired, and each of the first contours is numbered according to the size of the contour area.

[0022] The present invention intercepts the point cloud data of the point cloud frame to calculate the position and posture of the wire, and at the same time obtains the contour area formed by the first contour, so that the contour can be numbered according to the size of the contour area and the principle of "near is big and far is small" of camera projection, so as to simply and clearly indicate the positional relationship between the wire clamp and the robotic arm, thereby facilitating the selection of the operating wire clamp.

[0023] As a preferred example, the step of calculating the direction vector corresponding to the second point cloud data and the geometric information of the bounding box corresponding to each of the first contours includes:

[0024] According to the second point cloud data, a direction vector of a straight line is fitted by a preset principal component analysis algorithm;

[0025] The circumscribed circle of each of the first contours is calculated, and the center point and radius corresponding to the circumscribed circle are obtained.

[0026] The present invention calculates the direction vector of the point cloud data to improve the accuracy of wire posture perception based on the direction vector combined with the point cloud data, and simultaneously calculates the circumscribed circle corresponding to each contour to ensure the integrity of the wire clamp included in the image after interception, thereby improving the accuracy of wire clamp posture perception.

[0027] As a preferred example, the first color image is intercepted according to the geometric information to obtain a second color image corresponding to each wire clip, including:

[0028] Obtaining the serial number of the work line clamp, and searching for the circumscribed circle corresponding to the work line clamp according to the serial number;

[0029] The first color image is intercepted according to the center and radius of the circumscribed circle corresponding to the operation line clamp to generate a second color image corresponding to the operation line clamp.

[0030] The present invention uses numbers to present the positional relationship between the wire clamp and the robotic arm, and uses the number of the working wire clamp to query the area of ​​the working wire clamp in the first color image, and then intercepts the color image according to the area, which facilitates subsequent perception of posture and improves the accuracy of posture.

[0031] As a preferred example, the step of performing posture estimation on the second color image to obtain second posture information corresponding to each wire clip includes:

[0032] Processing the second color image by a preset deep learning algorithm to obtain the rotation angle and translation information corresponding to the working line clamp;

[0033] According to the rotation angle and translation information, in combination with a pre-built 3D model of the wire clamp, second posture information corresponding to the operating wire clamp is calculated.

[0034] The present invention combines the image and the pre-constructed 3D model to realize automatic perception of the wire clamp posture, and then realizes automatic execution of the wire clamp disassembly and assembly task according to the perceived posture information.

[0035] On the other hand, the present invention discloses a robot arm control system for performing wire clamp disassembly and assembly tasks, including a camera assembly, an edge computing module, an interactive terminal and a main control machine;

[0036] The camera assembly is connected to the edge computing module by signal, and is used to collect a first color image and a first point cloud data corresponding to a target operation position, and send the first color image and the first point cloud data to the edge computing module, so that the edge computing module sends the first color image and the first point cloud data to the interactive terminal;

[0037] The interactive terminal is connected to the edge computing module by signal, and is used to receive the first color image and the first point cloud data, and determine the operation wire by comparing the first color image and the first point cloud data, and send the operation wire to the edge computing module, so that the edge computing module executes the robot arm control method for performing the wire clamp disassembly and assembly task:

[0038] Acquire a first color image and first point cloud data of the collected target working position, and perform image segmentation on the first color image to obtain a mask image;

[0039] Extracting second point cloud data corresponding to the operating conductor from the first point cloud data, and performing contour extraction on the mask image to obtain first contours corresponding to each wire clamp in the target operating position; wherein the operating conductor is determined by comparing the first color image and the first point cloud data;

[0040] Calculating a direction vector corresponding to the second point cloud data and geometric information of a bounding box corresponding to each of the first contours;

[0041] Performing posture estimation on the second point cloud data according to the direction vector to obtain the first posture information corresponding to the working guide wire, and intercepting the first color image according to the geometric information to obtain the second color image corresponding to each wire clip;

[0042] Performing posture estimation on the second color image to obtain second posture information corresponding to each wire clip;

[0043] An action plan of the robot arm is generated according to the first posture information and the second posture information, so as to control the robot arm to perform a grasping operation of the wire and an assembly and disassembly operation of the wire clamp according to the action plan.

[0044] The edge computing module is connected to the main control machine signal, and is used to send the first posture information and the second posture information to the main control machine, so that the main control machine generates an action plan for the robotic arm according to the first posture information and the second posture information, so that the robotic arm performs the wire grabbing operation and the wire clamp disassembly and assembly operation according to the action plan.

[0045] The present invention discloses a robotic arm control system for performing wire clamp disassembly and assembly operations. The control system first obtains a first color image and first point cloud data corresponding to the position where the operation is performed, determines the operation wire and the operation wire clamp according to the image and point cloud data, and realizes automatic perception of the wire clamp and the wire. Then, the second point cloud data corresponding to the operation wire is intercepted from the first point cloud data to calculate the direction vector in the point cloud data, and then the direction vector and the point cloud data are combined to perform wire posture fitting, so as to control the robotic arm according to the posture of the wire to realize the wire grabbing operation.

[0046] Furthermore, while performing posture perception of the conductor, the first color image is segmented to extract a mask image composed of multiple wire clips contained in the current image, and then the area of ​​each wire clip in the image can be extracted by extracting the contour of the mask image, and then the color image is segmented according to the area to obtain a color image containing only a single wire clip, and then the posture of the color image is estimated to obtain the posture information corresponding to each wire clip, and then the robotic arm is controlled according to the posture of the wire clip to realize the disassembly and assembly operation of the wire clip, thereby realizing automatic perception of the wire clip.

[0047] As a preferred example, the interactive terminal determines the position of the operating wire by comparing the first color image and the first point cloud data; selects a wire point from the first point cloud data according to the position, and obtains the three-dimensional coordinates corresponding to the wire point, and generates a point cloud frame corresponding to the operating wire according to the preset point cloud frame size information with the three-dimensional coordinates as the center and sends the point cloud frame to the edge computing module; wherein the point cloud frame includes the center point of the point cloud frame and the length, width and height of the point cloud frame.

[0048] The present invention first determines the operating wire by comparing the color image and the point cloud data, so that the operating wire is grasped to ensure the stability of the wire clamp disassembly and assembly task. Further, a point cloud frame is generated to intercept the point cloud data of the operating wire, which is convenient for subsequent completion of the posture perception of the wire, so that the user can select the area to grasp the wire to control the robot arm to automatically grasp the wire.

[0049] As a preferred example, the edge computing module further includes a wire posture sensing unit, an image processing unit and a wire clamp posture sensing unit;

[0050] The wire posture sensing unit is connected to the interactive terminal signal, and is used to receive the point cloud frame and intercept the second point cloud data in the point cloud frame from the first point cloud data; according to the second point cloud data, the direction vector of the straight line is fitted by a preset principal component analysis algorithm;

[0051] The image processing unit is connected to the interactive terminal signal, and is used to perform image segmentation on the color image through a preset deep learning algorithm to generate the mask image; obtain the contour area formed by each first contour, and number each first contour according to the size of the contour area and send the number to the interactive terminal; calculate the circumscribed circle of each first contour, and obtain the center and radius corresponding to the circumscribed circle;

[0052] The wire clamp posture perception unit is connected to the interactive terminal signal, and is used to obtain the number of the operation wire clamp sent by the interactive terminal, and search for the circumscribed circle corresponding to the operation wire clamp according to the number; intercept the first color image according to the center and radius of the circumscribed circle corresponding to the operation wire clamp, and generate a second color image corresponding to the operation wire clamp; process the second color image through a preset deep learning algorithm to obtain the rotation angle and translation information corresponding to the operation wire clamp; calculate the second posture information corresponding to the operation wire clamp according to the rotation angle and translation information, combined with a pre-built wire clamp 3D model.

[0053] The present invention utilizes the edge computing module to establish a signal connection with the interactive terminal, so that the user can select the area to grab the wire and the wire clamp to operate, and the edge computing module calculates to complete the perception of the posture of the wire and the wire clamp, providing the necessary perception information for the robot arm to disassemble and assemble the wire clamp, and then controlling the robot arm to perform the corresponding task.

[0054] As a preferred example, the camera component includes a depth camera; the first color image and the first point cloud data are collected by the depth camera.

[0055] The present invention provides accurate depth perception for the operation execution of the robotic arm based on the depth camera, thereby improving the accuracy of the robotic arm control. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 : is a flow chart of a method for controlling a robot arm for performing a wire clamp disassembly and assembly operation disclosed in an embodiment of the present invention;

[0057] Figure 2 : It is a structural schematic diagram of a mechanical arm control system for performing wire clamp disassembly and assembly operations disclosed in an embodiment of the present invention;

[0058] Figure 3 : is a structural schematic diagram of a wire clamp sensing device disclosed in another embodiment of the present invention;

[0059] Figure 4 : A schematic diagram of information interaction between a wire clamp sensing device and a robot disclosed in another embodiment of the present invention;

[0060] Figure 5 : A flowchart of a method for controlling a robot to grasp a wire based on the wire clamp sensing device disclosed in another embodiment of the present invention;

[0061] Figure 6 : A flowchart of a method for controlling a robot to disassemble and assemble a wire clamp based on the wire clamp sensing device disclosed in another embodiment of the present invention;

[0062] Among them, 201, camera component; 202, edge computing module; 203, interactive terminal; 204, main control machine; 301, depth camera; 302, computing module; 303, mechanical structure; 304, connecting parts; 401, robot interactive terminal; 402, robot control center. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Embodiment 1

[0065] This embodiment discloses a method for controlling a robot arm for performing a wire clamp disassembly and assembly operation. For the specific implementation process of the control method, please refer to Figure 1 , mainly including steps 101 to 107, the steps are:

[0066] Step 101: Acquire a first color image and first point cloud data of a collected target working position, and perform image segmentation on the first color image to obtain a mask image.

[0067] In this embodiment, this step mainly includes: determining the position of the operating wire by comparing the first color image and the first point cloud data; selecting a wire point from the first point cloud data according to the position, and obtaining the three-dimensional coordinates corresponding to the wire point, and generating a point cloud frame corresponding to the operating wire based on the preset point cloud frame size information with the three-dimensional coordinates as the center; wherein the point cloud frame includes the center point of the point cloud frame and the length, width and height of the point cloud frame; performing image segmentation on the first color image through a preset deep learning algorithm to generate the mask image.

[0068] In this embodiment, this step first determines the operating conductor by comparing the color image and the point cloud data, so that the operating conductor is grasped to ensure the stability of the wire clamp disassembly and assembly task. Further, a point cloud frame is generated to intercept the point cloud data of the operating conductor, which is convenient for the subsequent completion of the posture perception of the conductor. Further, the color image is segmented to extract only the mask image corresponding to the wire clamp included in the color image, which is convenient for the subsequent posture perception of the wire clamp.

[0069] Step 102: intercepting second point cloud data corresponding to the operating conductor from the first point cloud data, and performing contour extraction on the mask image to obtain first contours corresponding to each wire clamp in the target operating position; wherein the operating conductor is determined by comparing the first color image and the first point cloud data.

[0070] In this embodiment, the step mainly includes: intercepting the second point cloud data within the point cloud frame from the first point cloud data; obtaining the contour area formed by each first contour, and numbering each first contour according to the size of the contour area.

[0071] In this embodiment, this step intercepts the point cloud data of the point cloud frame to calculate the position and posture of the wire, and at the same time obtains the contour area formed by the first contour, so that the contours are numbered according to the size of the contour area and the principle of "near big and far small" of camera projection, so as to simply and clearly indicate the positional relationship between the wire clamp and the robotic arm, thereby facilitating the selection of the operating wire clamp.

[0072] Step 103: Calculate the direction vector corresponding to the second point cloud data and the geometric information of the bounding box corresponding to each of the first contours.

[0073] In this embodiment, this step mainly includes: fitting the direction vector of the straight line through a preset principal component analysis algorithm according to the second point cloud data; calculating the circumscribed circle of each of the first contours, and obtaining the center and radius corresponding to the circumscribed circle.

[0074] In this embodiment, this step calculates the direction vector of the point cloud data to improve the accuracy of wire posture perception based on the direction vector combined with the point cloud data, and at the same time calculates the circumscribed circle corresponding to each contour to ensure the integrity of the wire clamp included in the image after interception, thereby improving the accuracy of wire clamp posture perception.

[0075] Step 104: performing posture estimation on the second point cloud data according to the direction vector to obtain the first posture information corresponding to the working guide wire, and intercepting the first color image according to the geometric information to obtain the second color image corresponding to each wire clamp.

[0076] In this embodiment, this step mainly includes: obtaining the number of the work line clamp, and searching for the circumscribed circle corresponding to the work line clamp according to the number; intercepting the first color image according to the center and radius of the circumscribed circle corresponding to the work line clamp, and generating a second color image corresponding to the work line clamp.

[0077] In this embodiment, this step uses numbers to present the positional relationship between the wire clamp and the robotic arm, and uses the number of the working wire clamp to query the area of ​​the working wire clamp in the first color image, and then intercepts the color image according to the area, so as to facilitate subsequent perception of posture and improve the accuracy of posture.

[0078] Step 105: performing posture estimation on the second color image to obtain second posture information corresponding to each wire clip.

[0079] In this embodiment, this step mainly includes: processing the second color image through a preset deep learning algorithm to obtain the rotation angle and translation information corresponding to the operating wire clamp; according to the rotation angle and translation information, combined with a pre-built wire clamp 3D model, calculating the second posture information corresponding to the operating wire clamp.

[0080] In this embodiment, this step combines the image and the pre-constructed 3D model to realize automatic perception of the wire clamp posture, and then automatically executes the wire clamp disassembly and assembly task according to the perceived posture information.

[0081] Step 106: Generate an action plan for the robot arm according to the first posture information and the second posture information, so as to control the robot arm to perform a grasping operation of the wire and an assembly and disassembly operation of the wire clamp according to the action plan.

[0082] On the other hand, this embodiment also discloses a robot arm control system for performing the task of disassembling and assembling the wire clamp. For the specific structure of the control system, please refer to Figure 2 , including a camera component 201, an edge computing module 202, an interactive terminal 203 and a main control machine 204.

[0083] The camera component 201 is signal-connected to the edge computing module 202, and is used to collect the first color image and the first point cloud data corresponding to the target working position, and send the first color image and the first point cloud data to the edge computing module 202, so that the edge computing module 202 sends the first color image and the first point cloud data to the interactive terminal 203.

[0084] The interactive terminal 203 is connected to the edge computing module 202 by signal, and is used to receive the first color image and the first point cloud data, determine the operation guide by comparing the first color image and the first point cloud data, and send the operation guide to the edge computing module 202, so that the edge computing module 202 performs the following operations:

[0085] Acquire a first color image and first point cloud data of the collected target working position, and perform image segmentation on the first color image to obtain a mask image;

[0086] Extracting second point cloud data corresponding to the operating conductor from the first point cloud data, and performing contour extraction on the mask image to obtain first contours corresponding to each wire clamp in the target operating position; wherein the operating conductor is determined by comparing the first color image and the first point cloud data;

[0087] Calculating a direction vector corresponding to the second point cloud data and geometric information of a bounding box corresponding to each of the first contours;

[0088] Performing posture estimation on the second point cloud data according to the direction vector to obtain the first posture information corresponding to the working guide wire, and intercepting the first color image according to the geometric information to obtain the second color image corresponding to each wire clip;

[0089] Performing posture estimation on the second color image to obtain second posture information corresponding to each wire clip;

[0090] The edge computing module 202 is connected to the main control machine 204 by signal, and is used to send the first posture information and the second posture information to the main control machine 204, so that the main control machine 204 generates an action plan for the robotic arm according to the first posture information and the second posture information, so that the robotic arm performs the wire grabbing operation and the wire clamp disassembly and assembly operation according to the action plan.

[0091] In this embodiment, the interactive terminal 203 determines the position of the operating wire by comparing the first color image and the first point cloud data; selects a wire point from the first point cloud data according to the position, and obtains the three-dimensional coordinates corresponding to the wire point, and generates a point cloud frame corresponding to the operating wire based on the preset point cloud frame size information with the three-dimensional coordinates as the center and sends the point cloud frame to the edge computing module 202; wherein the point cloud frame includes the center point of the point cloud frame and the length, width and height of the point cloud frame.

[0092] In this embodiment, the edge computing module 202 also includes a wire posture sensing unit, an image processing unit and a wire clamp posture sensing unit.

[0093] The wire posture perception unit is connected to the interactive terminal 203 by signal, and is used to receive the point cloud frame and intercept second point cloud data in the point cloud frame from the first point cloud data; according to the second point cloud data, a direction vector of a straight line is fitted by a preset principal component analysis algorithm.

[0094] The image processing unit is signal-connected to the interactive terminal 203, and is used to perform image segmentation on the color image through a preset deep learning algorithm to generate the mask image; obtain the contour area formed by each first contour, and number each first contour according to the size of the contour area and send the number to the interactive terminal 203; calculate the circumscribed circle of each first contour, and obtain the center and radius corresponding to the circumscribed circle.

[0095] The wire clamp posture perception unit is connected to the interactive terminal 203 by signal, and is used to obtain the number of the operation wire clamp sent by the interactive terminal, and search for the circumscribed circle corresponding to the operation wire clamp according to the number; intercept the first color image according to the center and radius of the circumscribed circle corresponding to the operation wire clamp, and generate a second color image corresponding to the operation wire clamp; process the second color image through a preset deep learning algorithm to obtain the rotation angle and translation information corresponding to the operation wire clamp; calculate the second posture information corresponding to the operation wire clamp according to the rotation angle and translation information, combined with a pre-built wire clamp 3D model.

[0096] In this embodiment, the camera component 201 includes a depth camera; the first color image and the first point cloud data are collected by the depth camera.

[0097] Embodiment 2

[0098] When performing the disassembly and assembly of the wire clamp, it is necessary to sense the wire on which the wire clamp is located and the position and posture of the wire clamp, so as to facilitate the disassembly and assembly of the wire clamp according to the sensed position and posture information. Specifically, this embodiment provides a wire clamp sensing device for sensing the position and posture of the wire and the position and posture of the wire clamp, wherein the specific structure of the wire clamp sensing device refers to Figure 3 , including a depth camera 301, a computing module 302, a mechanical structure 303, and a connecting part 304.

[0099] Reference Figure 3 The depth camera 301 serves as a sensor and can obtain a depth map and a color map in front of the working robot.

[0100] The calculation module 302 is mainly used to estimate the posture of the distribution network conductors and wire clamps.

[0101] The mechanical structure 303 includes a mechanical support member and a housing, which form various components of the wire clamp sensing device into a solid whole.

[0102] The connecting member 304 is used to physically connect with the body of the working robot to fix the device on the robot.

[0103] In one embodiment, Figure 3The wire clamp sensing device shown is installed on a working robot and establishes a signal connection with the working robot, and is used to control the mechanical arm of the working robot to perform the task of disassembling and assembling the wire clamp according to the postures of the wire and the wire clamp calculated by the wire clamp sensing device.

[0104] Optional, such as Figure 3 The information interaction process between the wire clamp sensing device and the robot can be referred to Figure 4 , wherein the wire clamp sensing device is connected to the robot interaction terminal 401 and the robot control center 402 by signal.

[0105] Further, see Figure 4 The depth camera 301 and the computing module 302 are powered by connecting to the robot body power supply to ensure the normal operation of the device.

[0106] Among them, the robot interactive terminal 401 is used to manually select the working conductor and the working wire clamp.

[0107] The robot control center 402 receives the wire and wire clamp positions transmitted by the sensing device, performs motion planning, and controls the robot to automatically complete the subsequent wire clamp disassembly and assembly tasks.

[0108] Optional, such as Figure 3 The sensing device and Figure 4 The interactive terminals and robot control center will form a local area network through switches and wireless APs to transmit data through the network.

[0109] In one embodiment, when a robot equipped with the wire clamp sensing device performs a wire clamp disassembly and assembly task, the working robot performs the wire clamp disassembly and assembly task in two steps: grabbing the wire and disassembling the wire clamp; further, before performing the wire clamp disassembly and assembly task, the working robot is first moved to a predetermined working point to ensure that the wire is directly in front of the robot and within the operating range of the robot arm.

[0110] Furthermore, in the two steps of the task of disassembling and assembling the wire clamp, the method of controlling the robot to grab the wire can refer to Figure 5 , mainly including step 501 to step 504, the steps are:

[0111] Step 501: The depth camera transmits the depth map and the color map to the computing module, and the computing module sends a message to the interactive terminal through the local area network, so that the interactive terminal displays the wire point cloud and the visible light image.

[0112] Step 502: Select the wire point cloud of the robot grasping position through the interactive terminal, so that the interactive terminal sends the point cloud frame information, including the center point (x, y, z) of the point cloud frame, and the length, width and height (l, w, h) of the point cloud frame to the calculation module for calculation.

[0113] In this step, the interactive terminal is provided with a touch screen, through which the point cloud image and the visible light image are displayed, and the point cloud image can be dragged, enlarged and rotated to find the best viewing angle for observing the point cloud. The user will compare the point cloud and the visible light image, find the location of the wire point cloud, click on the wire point cloud (click on any point on the wire point cloud), and a wire point cloud frame (direction perpendicular to the coordinate axis) will be displayed at the point location. In addition, the interactive terminal is provided with an option to adjust the length, width and height of the point cloud frame. The user ensures that the point cloud frame contains enough point cloud data (the default value is greater than 10 valid points) to fit.

[0114] During the fitting process, the three-dimensional coordinates of the clicked wire point are obtained, and a point cloud frame of a default length and size appears with the point coordinates as the center. The user controls the length, width and height of the point cloud frame by pressing keys to generate a point cloud frame of the point cloud at the captured position.

[0115] Step 503: intercept the point cloud within the point cloud frame, and use the PCA algorithm to calculate the direction vector of the straight line.

[0116] In this step, in addition to the PCA algorithm, the least squares method and the RANSAC algorithm can also be used for wire fitting in the point cloud frame, and the process of using the PCA algorithm to calculate the direction vector of the straight line is:

[0117] First, construct the data matrix: According to the point cloud frame data (x b ,y b ,z b ,l,w,h), intercept The point cloud data is represented as a matrix to obtain an n×3 data matrix X.

[0118] Secondly, perform data normalization: get the mean of the coordinate dimension d: Where i = 1 to n, representing the i-th point; j represents the coordinate dimension. Subtract the mean to obtain a new centered data matrix Z ij =X ij -μ j .

[0119] Furthermore, the covariance matrix is ​​calculated: Where Z T represents the transpose of Z, and C is a 3×3 matrix.

[0120] Finally, calculate the eigenvalues ​​and eigenvectors: solve the eigenvalue problem of C and find the eigenvalue λ of the covariance matrix C i and the corresponding eigenvector v i Usually, λ i represents the main variance, v iIndicates the direction of the main variance. Select the eigenvector with the largest eigenvalue, which represents the main direction in the point cloud, that is, the direction of the fitted line.

[0121] Step 504: Send the direction vector and point cloud frame information to the robot control center, which completes the motion planning of the robot arm.

[0122] In this step, the three-dimensional coordinate space of the point cloud and the robot coordinate space will be aligned according to the positional relationship in the physical space. The position and posture of the fitted wire can be obtained through the combination of the point cloud frame and the direction vector. Path planning can be performed based on the position and posture of the wire, and the results of path planning can be converted into instructions that can be executed by the robot arm to guide how each joint of the robot arm moves.

[0123] In this embodiment, in the process of sensing the position of the wire and then controlling the robot to grab the wire, the robot may encounter a situation in which multiple wire clamps of the same shape appear in the picture taken by the camera in the distribution network operation environment. In this case, the operation target is determined by automatically numbering and manually selecting the captured wire clamps to eliminate interference from other wire clamps. Specifically, the method of controlling the robot to perform the disassembly and assembly of the wire clamp can be referred to Figure 6 , mainly including step 601 to step 607, the steps are:

[0124] Step 601: The depth camera acquires a depth image and a color image, and uses PVNet to segment the color image to obtain a mask image.

[0125] In this step, the image segmentation can use Mask R-CNN and other algorithms instead of PVNet to perform image segmentation on the color image I, obtain the mask image M, and set the pixel values ​​of the area with line clips to (255, 255, 255), and the remaining pixels to (0, 0, 0).

[0126] Step 602: extract contours from the mask image and number the contour areas from large to small.

[0127] In this step, the contour of the mask image M is extracted, and the contour area is numbered according to the principle of camera projection that "near is big and far is small", that is, the closer the wire clip is to the camera, the smaller the number is.

[0128] Step 603: Calculate the circumscribed circle of each contour, and obtain the center and radius of the circumscribed circle.

[0129] Step 604: All circumscribed circle information and corresponding wire clamp numbers are sent to the interactive terminal for display. The interactive terminal sends the wire clamp number corresponding to the working wire clamp to be disassembled to the calculation module.

[0130] Step 605: For the input color image, cut out the area of ​​the circumscribed circle where the operation line clamp is located to obtain a processed color image.

[0131] In this step, for the input image I, intercept (u k , v k ) is the center of the circle, 1.5r k The area with radius is formed, and the pixel values ​​outside the area are set to (0, 0, 0), and the processed image I is obtained. sub Wherein, k is the clamp number corresponding to the operation clamp, and (u k , v k ) is the center of the circumscribed circle, and r k is the radius of the circumscribed circle.

[0132] Step 606: Based on the processed color image, use the PVNet posture estimation algorithm to obtain the six-degree-of-freedom posture information of the wire clamp.

[0133] In this step, other algorithms such as ICP can be used to replace the PVNet posture estimation algorithm to obtain the six-degree-of-freedom posture information of the wire clamp.

[0134] Step 607: Send the six-degree-of-freedom posture information to the robot control center so that the robot control center can complete the subsequent motion planning of the robot arm.

[0135] In this step, the six-degree-of-freedom posture information is translation (x, y, z) and rotation (roll, pitch, yaw); when the robot is planning an action, it needs to know the position and posture of the target object (that is, the six degrees of freedom). Since the wire clamp is a rigid object in a limited area, a 3D model of the wire clamp can be constructed, and then the position and posture of the wire clamp in the image can be inferred through the feature relationship between the image and the 3D model.

[0136] In this embodiment, the wire clamp sensing device and the robot can interact with each other, and the user can select the area to grab the wire and the wire clamp to operate, and the edge computing module can calculate to complete the perception of the position of the wire and the wire clamp, providing the robot with necessary perception information for disassembling and assembling the wire clamp.

[0137] Furthermore, the wire clamp sensing device selects point cloud information by the user, uses PCA for fitting, and obtains the wire posture. In the case of multiple wire clamps, the area of ​​each wire clamp is first calculated through image segmentation, and the information is sent to the interactive interface. The user selects the wire clamp to be operated, and then the posture of the wire clamp is calculated through a deep neural network algorithm, so as to accurately obtain the perception information of the wire and wire clamp that need to be manipulated.

[0138] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for controlling a robot arm for performing a wire clamp disassembly and assembly operation, characterized in that: include: Acquire a first color image and first point cloud data of the collected target working position, and perform image segmentation on the first color image to obtain a mask image; Extracting second point cloud data corresponding to the operating conductor from the first point cloud data, and performing contour extraction on the mask image to obtain first contours corresponding to each wire clamp in the target operating position; wherein the operating conductor is determined by comparing the first color image and the first point cloud data; Calculating a direction vector corresponding to the second point cloud data and geometric information of a bounding box corresponding to each of the first contours; Performing posture estimation on the second point cloud data according to the direction vector to obtain the first posture information corresponding to the working guide wire, and intercepting the first color image according to the geometric information to obtain the second color image corresponding to each wire clip; Performing posture estimation on the second color image to obtain second posture information corresponding to each wire clip; An action plan of the robot arm is generated according to the first posture information and the second posture information, so as to control the robot arm to perform a grasping operation of the wire and an assembly and disassembly operation of the wire clamp according to the action plan.

2. A method for controlling a robot arm for performing a wire clamp disassembly and assembly operation according to claim 1, characterized in that: The step of acquiring the first color image and the first point cloud data of the collected target operation position, and performing image segmentation on the first color image to obtain a mask image includes: Comparing the first color image with the first point cloud data to determine the position of the operating guide wire; Selecting a wire point from the first point cloud data according to the position, obtaining the three-dimensional coordinates corresponding to the wire point, and generating a point cloud frame corresponding to the operating wire based on the preset point cloud frame size information with the three-dimensional coordinates as the center; wherein the point cloud frame includes the center point of the point cloud frame and the length, width and height of the point cloud frame; The first color image is segmented by a preset deep learning algorithm to generate the mask image.

3. A method for controlling a robot arm for performing a wire clamp disassembly and assembly operation according to claim 2, characterized in that: The method of intercepting the second point cloud data corresponding to the working conductor from the first point cloud data and performing contour extraction on the mask image to obtain the first contour corresponding to each wire clamp in the target working position includes: intercepting second point cloud data within the point cloud frame from the first point cloud data; A contour area formed by each of the first contours is acquired, and each of the first contours is numbered according to the size of the contour area.

4. A method for controlling a robot arm for performing a wire clamp disassembly and assembly operation according to claim 1, characterized in that: The calculating the direction vector corresponding to the second point cloud data and the geometric information of the bounding box corresponding to each of the first contours includes: According to the second point cloud data, a direction vector of a straight line is fitted by a preset principal component analysis algorithm; The circumscribed circle of each of the first contours is calculated, and the center point and radius corresponding to the circumscribed circle are obtained.

5. A method for controlling a robot arm for performing a wire clamp disassembly and assembly operation according to claim 4, characterized in that: The step of intercepting the first color image according to the geometric information to obtain a second color image corresponding to each wire clip includes: Obtaining the serial number of the work line clamp, and searching for the circumscribed circle corresponding to the work line clamp according to the serial number; The first color image is intercepted according to the center and radius of the circumscribed circle corresponding to the operation line clamp to generate a second color image corresponding to the operation line clamp.

6. A method for controlling a robot arm for performing a wire clamp disassembly and assembly operation according to claim 5, characterized in that: The step of performing posture estimation on the second color image to obtain second posture information corresponding to each wire clip includes: Processing the second color image by a preset deep learning algorithm to obtain the rotation angle and translation information corresponding to the working line clamp; According to the rotation angle and translation information, in combination with a pre-built 3D model of the wire clamp, second posture information corresponding to the operating wire clamp is calculated.

7. A robot arm control system for performing wire clamp disassembly and assembly tasks, characterized in that: Including camera components, edge computing modules, interactive terminals and main control machines; The camera assembly is connected to the edge computing module by signal, and is used to collect a first color image and a first point cloud data corresponding to a target operation position, and send the first color image and the first point cloud data to the edge computing module, so that the edge computing module sends the first color image and the first point cloud data to the interactive terminal; The interactive terminal is connected to the edge computing module by signal, and is used to receive the first color image and the first point cloud data, and determine the operation wire by comparing the first color image and the first point cloud data, and send the operation wire to the edge computing module, so that the edge computing module executes the robot arm control method for performing the wire clamp disassembly and assembly task according to any one of claims 1 to 6: Acquire a first color image and first point cloud data of the collected target working position, and perform image segmentation on the first color image to obtain a mask image; Extracting second point cloud data corresponding to the operating conductor from the first point cloud data, and performing contour extraction on the mask image to obtain first contours corresponding to each wire clamp in the target operating position; wherein the operating conductor is determined by comparing the first color image and the first point cloud data; Calculating a direction vector corresponding to the second point cloud data and geometric information of a bounding box corresponding to each of the first contours; Performing posture estimation on the second point cloud data according to the direction vector to obtain the first posture information corresponding to the working guide wire, and intercepting the first color image according to the geometric information to obtain the second color image corresponding to each wire clip; Performing posture estimation on the second color image to obtain second posture information corresponding to each wire clip; The edge computing module is connected to the main control machine signal, and is used to send the first posture information and the second posture information to the main control machine, so that the main control machine generates an action plan for the robotic arm according to the first posture information and the second posture information, so that the robotic arm performs the wire grabbing operation and the wire clamp disassembly and assembly operation according to the action plan.

8. A robot arm control system for performing wire clamp disassembly and assembly tasks according to claim 7, characterized in that: The interactive terminal determines the position of the operating wire by comparing the first color image and the first point cloud data; selects a wire point from the first point cloud data according to the position, and obtains the three-dimensional coordinates corresponding to the wire point, and generates a point cloud frame corresponding to the operating wire based on the preset point cloud frame size information with the three-dimensional coordinates as the center, and sends the point cloud frame to the edge computing module; wherein the point cloud frame includes the center point of the point cloud frame and the length, width and height of the point cloud frame.

9. A robot arm control system for performing wire clamp disassembly and assembly tasks according to claim 8, characterized in that: The edge computing module also includes a wire posture sensing unit, an image processing unit and a wire clamp posture sensing unit; The wire posture sensing unit is connected to the interactive terminal signal, and is used to receive the point cloud frame and intercept the second point cloud data in the point cloud frame from the first point cloud data; According to the second point cloud data, a direction vector of a straight line is fitted by a preset principal component analysis algorithm; The image processing unit is connected to the interactive terminal signal, and is used to perform image segmentation on the color image through a preset deep learning algorithm to generate the mask image; obtain the contour area formed by each first contour, and number each first contour according to the size of the contour area and send the number to the interactive terminal; calculate the circumscribed circle of each first contour, and obtain the center and radius corresponding to the circumscribed circle; The wire clamp posture perception unit is connected to the interactive terminal signal, and is used to obtain the number of the operation wire clamp sent by the interactive terminal, and search for the circumscribed circle corresponding to the operation wire clamp according to the number; intercept the first color image according to the center and radius of the circumscribed circle corresponding to the operation wire clamp, and generate a second color image corresponding to the operation wire clamp; process the second color image through a preset deep learning algorithm to obtain the rotation angle and translation information corresponding to the operation wire clamp; calculate the second posture information corresponding to the operation wire clamp according to the rotation angle and translation information, combined with a pre-built wire clamp 3D model.

10. A robot arm control system for performing wire clamp disassembly and assembly tasks according to claim 7, characterized in that: The camera component includes a depth camera; the first color image and the first point cloud data are collected by the depth camera.

Images