Robotic arm control system and method containing tool

Abstract

The present invention discloses a control system and method for a mechanical arm containing a tool, the system comprising: a perception layer, a decision layer and an execution layer; the perception layer comprises an image acquisition module and a tool recognition module, the image acquisition module is used to acquire RGB images and depth images, the tool recognition module is used to recognize the purpose of the tool and estimate the length of the tool; the decision layer comprises an end position determination module and a contact point evaluation module, the end position of the tool is obtained by the end position determination module, and the contact point evaluation module is used to evaluate the optimal contact point between the tool and the target object; the execution layer comprises a joint angle prediction module and a mechanical arm motion module, the joint angle change of the mechanical arm in the next step is predicted by the joint angle prediction module, and the mechanical arm motion module is used to control the joint rotation of the mechanical arm according to the predicted joint angle in the next step, and drive the mechanical arm to move. The present invention can complete more tasks beyond the working range without changing the structure of the mechanical arm.

Description

Technical Field

[0001] The present invention relates to the field of robot technology, and in particular to a control system and method for a robot arm containing a tool. Background Art

[0002] Nowadays, the demand for quality home services is increasing, but China's working-age population is decreasing year by year, and the labor cost of the service industry is growing. With the development of robot technology, robots are increasingly appearing in people's lives, playing an increasingly important role in the execution of various service tasks in the home environment. Most traditional service robots are designed for specific tasks, and the fixed design structure limits the functions of the robot to one or two. In order to meet human needs in actual home use scenarios, it is often necessary to configure multiple service robots with different functions, which greatly increases the purchase cost of service robots in human living environments. Therefore, the design of allowing a single service robot to have multiple working functions is very promising.

[0003] In real life, humans usually hold different tools to complete different types of work. The robot arm was originally designed to imitate the human arm, and the end gripper of the robot arm is equivalent to the human hand. Therefore, it is feasible for the robot arm to grasp tools like humans. Inspired by this, the robot arm can use only the gripper as the end effector, and by holding any tool, it can expand its operating range on the basis of the original robot arm, and complete more tasks beyond the working range without changing the structure of the robot arm.

[0004] When the robot arm uses the tool through the end gripper to complete the task beyond the working range, the robot arm needs to select tools of different lengths according to the requirements of the task because the working range that needs to be considered for different tasks may be different. At the same time, in order to successfully complete the operation task, it is necessary to select the appropriate contact point between the tool and the target object according to the task requirements. Summary of the invention

[0005] The purpose of the present invention is to solve the technical problem of being able to complete more tasks beyond the working scope without changing the structure of the robot arm, and to propose a robot arm control system and method containing a tool.

[0006] The technical problem of the present invention is solved by the following technical solutions:

[0007] On the one hand, the present invention discloses a robot control system containing a tool, comprising: a perception layer, a decision layer and an execution layer; the perception layer comprises an image acquisition module and a tool recognition module, in the perception layer, RGB images and depth images are collected by the image acquisition module, and the purpose of the tool and the length of the tool are recognized by the tool recognition module; the decision layer comprises an end position determination module and a contact point evaluation module, in the decision layer, the end position of the tool when the robot arm holds the tool is obtained by the end position determination module, and the optimal contact point between the tool and the target object is evaluated by the contact point evaluation module; the execution layer comprises a joint angle prediction module and a robot arm motion module, in the execution layer, the joint angle change of the robot arm in the next step is predicted based on a recurrent neural network, and the robot arm motion module controls the joint rotation of the robot arm according to the predicted joint angle of the next step, and drives the robot arm to move.

[0008] In some embodiments, the image acquisition module includes a first RGB-D camera and a second RGB-D camera. The first RGB-D camera is installed on the wrist of the robotic arm and is used to acquire RGB images and depth images of the selected tool before using the tool. The second RGB-D camera is installed above the working area and is used to obtain RGB images and depth images in real time during the process of the robotic arm using the tool.

[0009] In some embodiments, the tool recognition module predicts the collected RGB image of the candidate tool through the target detection network, outputs the bounding box of the tool and the reference object in the image and the confidence of the tool type, and judges the highest confidence as the type of tool; uses the open source computer vision library to draw the outline of the tool and the reference object obtained by the target detection network, and obtains the length of the tool through the pixel ratio calculation method.

[0010] In some embodiments, the end position determination module regards the robot arm and the tool as a robot arm-tool combination according to the length of the tool and the DH parameters of the robot arm, and obtains the end position of the robot arm-tool combination through a homogeneous transformation matrix.

[0011] In some embodiments, the calculation formula for the end position of the robot arm-tool assembly is:

[0012] P xyz =f(T D-H ,L G ,L T )

[0013] Among them, P xyz represents the end position of the robot-tool assembly, T D-H Indicates the DH parameter table of the robot arm, L GIndicates the distance between the end position of the robot arm and the center of the gripper, L T Indicates the distance between the center of the gripper and the end of the current tool.

[0014] In some embodiments, the contact point evaluation module outputs the RGB image acquired in real time during the process of the robotic arm using a tool as a segmentation map through a semantic segmentation network, and inputs the segmentation map into the contact point generation network. The contact point generation network predicts multiple candidate contact points and contact force directions on the object contour, and then inputs the RGB image and the information of multiple candidate contact points into the contact point evaluation network, which outputs the position of an optimal contact point and the corresponding contact force direction.

[0015] In some embodiments, the joint angle prediction module uses a variational autoencoder to output the RGB image acquired in real time during the process of the robotic arm using a tool as a compressed latent space representation, inputs the optimal contact point information, latent space representation and the joint angle of the robotic arm in the previous step output by the contact point evaluation module into a multi-time recurrent neural network, and then predicts the joint angle of the robotic arm in the next step.

[0016] Another aspect of the present invention discloses a method for controlling a robot arm including a tool, comprising the following steps:

[0017] S1, RGB image and depth image of acquisition tool and current working area;

[0018] S2, extracting the features of the tool according to the collected RGB image and depth image, and obtaining the length and type of the tool;

[0019] S3, collecting real-time images of the working area;

[0020] S4, obtaining the end position of the robot arm-tool combination according to the posture of the robot arm holding the tool and the length of the tool;

[0021] S5. Extract the features of the object based on the real-time image of the working area, combine it with the target position that the object needs to reach, and evaluate the information of the optimal contact point between the tool and the object.

[0022] S6, collecting the current image and the joint angle of the robot in the previous step in real time during the movement of the robot arm, and obtaining the joint angle of the next step according to the current image, the joint angle of the robot in the previous step and the optimal contact point information;

[0023] S7. Control the joint rotation of the robot arm according to the predicted joint angle of the next step, and drive the robot arm to move.

[0024] In some embodiments, step S2 specifically includes: predicting the collected RGB image of the candidate tool through the target detection network, outputting the bounding box of the tool and the reference object in the image and the confidence of the tool type, and judging the highest confidence as the type of the tool; using the open source computer vision library to draw the outline of the tool and the reference object identified by the target detection network, and obtaining the length of the tool through the pixel ratio calculation method.

[0025] In some embodiments, step S5 specifically includes: outputting the real-time image of the working area as a segmentation map through a semantic segmentation network, inputting the segmentation map into a contact point generation network, the contact point generation network predicts multiple candidate contact points and contact force directions on the object contour, and then inputting the real-time image of the working area and the information of multiple candidate contact points into a contact point evaluation network, and outputting the position of an optimal contact point and the corresponding contact force direction.

[0026] The beneficial effects of the present invention compared with the prior art include:

[0027] The tool-containing robotic arm control system proposed in the present invention can identify the purpose of the tool and estimate the length of the tool, as well as evaluate the optimal contact point between the tool and the target object through the setting of technical features such as a tool identification module and a contact point evaluation module, thereby improving the tool recognition accuracy before the robotic arm selects the tool and the effectiveness of the contact between the tool and the object during the use of the tool, which is beneficial to the decision-making and judgment of the robotic arm, helping the robotic arm to successfully complete the task, so that more tasks beyond the working scope can be completed without changing the structure of the robotic arm. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the structure of a robot arm control system containing a tool according to an embodiment of the present invention.

[0029] Figure 2 It is a schematic diagram of the structure of the perception layer of an embodiment of the present invention.

[0030] Figure 3 is a schematic diagram of a tool identification module according to an embodiment of the present invention.

[0031] Figure 4 It is a schematic diagram of the structure of the decision layer of an embodiment of the present invention.

[0032] Figure 5 It is a schematic diagram of establishing a connecting rod coordinate system in an end position determination module according to an embodiment of the present invention.

[0033] Figure 6 is a schematic diagram of a contact point evaluation module according to an embodiment of the present invention.

[0034] Figure 7 It is a schematic diagram of the structure of the execution layer of an embodiment of the present invention.

[0035] Figure 8 is a schematic diagram of a joint angle prediction module according to an embodiment of the present invention.

[0036] Fig. 9 It is a flow chart of the control method of the robot arm containing the tool of the present invention.

[0037] Fig.10 It is a flow chart of a method for controlling a robot arm including a tool according to an embodiment of the present invention.

[0038] Fig.11a It is an effect diagram of modeling the robot arm-tool combination under Matlab software of an embodiment of the present invention.

[0039] Fig.11b It is an effect diagram of modeling of a robot arm-tool combination in the Coppeliasim simulation system of an embodiment of the present invention.

[0040] Figures 12a to 12i Schematic diagram of the output result of the contact point evaluation module according to an embodiment of the present invention.

[0041] Fig.13a Schematic diagram of a variational autoencoder input image in a joint angle prediction module according to an embodiment of the present invention.

[0042] Fig.13b Schematic diagram of the variational autoencoder output image in the joint angle prediction module of an embodiment of the present invention.

[0043] Fig.14 It is a rendering of the multi-time recurrent neural network in the joint angle prediction module of an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention is further described below with reference to the accompanying drawings and in combination with preferred embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0045] It should be noted that the directional terms such as left, right, up, down, top, and bottom in this embodiment are merely relative concepts or are based on the normal use status of the product, and should not be considered as restrictive.

[0046] The embodiment of the present invention proposes a robot control system with a tool, based on the HCTM architecture. The full name of HCTM is "Hierarchical Control for Tool Manipulation", and the Chinese name is "Hierarchical Control of Tool Manipulation". The HCTM architecture is mainly composed of three parts, namely: perception layer, decision layer, and execution layer. Each layer is responsible for realizing an important function of the system. The three layers work together to realize the complete function of the system. The main idea of ​​the control system is: using the characteristics of high recognition accuracy and effective reduction of data dimensions of convolutional neural networks, feature extraction is performed on the collected images, RGB images and depth images are collected by the image acquisition module in the perception layer, and the length estimation and purpose identification of the tool are realized by the tool recognition module; in the decision layer, the end position of the tool when the robot arm holds the tool is obtained by the end position determination module, and the appropriate contact point between the tool and the target object is found by the contact point evaluation module; the recurrent neural network can be used to predict the next data based on the current state data, the joint angle prediction module in the execution layer controls each step of the robot arm based on the recurrent neural network, and the robot arm movement module is used to drive the rotation of the robot arm joint, thereby improving the intelligence of the robot arm operation.

[0047] The robot arm control system with a tool in the embodiment of the present invention is suitable for the robot arm to use the tool to perform motion planning control. The overall structure diagram is as follows: Figure 1 As shown. The control system includes a perception layer, a decision layer, and an execution layer. The perception layer includes an image acquisition module and a tool recognition module; the decision layer includes an end position determination module and a contact point evaluation module; and the execution layer includes a joint angle prediction module and a robotic arm motion module. The specific structure of the system is as follows:

[0048] 1. Perception layer:

[0049] like Figure 2 As shown, the perception layer in the embodiment of the present invention refers to the process of collecting and analyzing image information to capture images of tools and objects through RGB-D cameras, analyze image features, and mine information about objects and the environment. The perception layer includes an image acquisition module and a tool recognition module.

[0050] 1.1 Image acquisition module:

[0051] The image acquisition module includes a first RGB-D camera and a second RGB-D camera. The first RGB-D camera is installed on the wrist of the robot arm and is responsible for collecting the RGB image and depth image of the selected tool before using the tool; the second RGB-D camera is installed above the working area and is responsible for real-time acquisition of the RGB image and depth image of the robot arm in the process of using the tool.

[0052] 1.2 Tool identification module:

[0053] The tool recognition module uses the object detection network (YOLO v5) and the open source computer vision library (OpenCV) to identify the purpose and estimate the length of the candidate tools. The purpose of identifying the tool is to select the tool that meets the task requirements, and the length of the tool is estimated to select the tool of appropriate specifications. When the object is slightly close to the robotic arm, a short tool is selected, and when the object is far away from the robotic arm, a long tool is required.

[0054] The schematic diagram of the tool identification module is as follows Figure 3 As shown. The input of the tool recognition module is the RGB image captured by the first RGB-D camera. The RGB image is input to the YOLO v5 target detection network. After prediction by the YOLO v5 target detection network, the output is the bounding box of the tool and the reference object in the image and the confidence of the tool type. At this time, the highest confidence is judged as the type of tool. The OpenCV open source computer vision library is used to draw the outlines of the tools and reference objects identified by the YOLO v5 target detection network. The actual size of the reference object is known, and the length of the tool is obtained by the pixel ratio calculation method.

[0055] The principle of the pixel ratio calculation method is to obtain the length ratio of the reference object and the tool by comparing the pixel widths of the reference object and the tool in the RGB image, and calculate the actual length of the tool according to the length ratio between the two when the length of the reference object is known. Assuming that the actual length of the reference object is D, the pixel width of the reference object is P, the pixel width of the tool is T, and the actual length of the tool is L, the formula of the pixel ratio calculation method can be expressed as L = (D*T) / P.

[0056] 2. Decision-making level:

[0057] like Figure 4 As shown, the decision layer in the embodiment of the present invention includes an end position determination module and a contact point evaluation module. The decision layer refers to completing the determination of the current state posture of the robot arm and the analysis of the work task requirements, analyzing the real-time image when the robot arm holds the tool, determining the end position of the robot arm-tool combination, and evaluating the appropriate contact point between the tool and the object according to the target task requirements and the current posture of the object.

[0058] 2.1. End position determination module:

[0059] The end position of the tool is not only the reachable position of the robot after the extended operation, but also the point of contact with the object in some tasks. The function of the end position determination module is to obtain the end position and posture of the tool after collecting the image of the robot arm holding the tool and the current posture of the robot arm.

[0060] When the length of the tool and the DH parameters of the robot are known, the robot and the tool are regarded as a "robot-tool" combination. In the end position determination module, the end position of the robot-tool combination is obtained through the homogeneous transformation matrix, which is expressed by the following formula:

[0061] P xyz =f(T D-H ,L G ,L T )

[0062] Among them, P xyz represents the end position of the robot-tool assembly, T D-H Indicates the DH parameter table of the robot arm, L G Indicates the distance between the end position of the robot arm and the center of the gripper, L T Indicates the distance between the center of the gripper and the end of the current tool.

[0063] The embodiment of the present invention adopts the DH method to perform kinematic modeling on the six-degree-of-freedom robot arm after clamping the tool. The DH method defines the kinematic relationship (relative position and direction) of two continuous links through four parameters, namely, the link length a, the link angle α, the link offset d, and the adjacent link joint angle θ. First, establish the link coordinate system for each joint, such as Figure 5 As shown, the joints are numbered from bottom to top, i represents the i-th joint, Figure 5 Axis in it represents the coordinate system established on the joint, Axis(i-1) represents the coordinate system on the i-1th joint, Axis(i) represents the coordinate system on the i-th joint, Link represents the connecting rod between adjacent joints, Link(i-1) represents the connecting rod between the i-1th joint and the i-th joint, and Link(i) represents the connecting rod between the i-th joint and the i+1th joint.

[0064] The transformation from coordinate system i-1 to coordinate system i is defined as the composite transformation matrix Composite Transformation Matrix The form is as follows:

[0065]

[0066] Among them, represents the length of the common perpendicular line between the coordinate system on the i-1th joint and the coordinate system on the i-th joint; α i represents the angle between the coordinate system on the i-th joint and the coordinate system on the i-1-th joint; d i Represents the distance in the direction of the common axis between the coordinate system on the i-1th joint and the coordinate system on the i-th joint; θ i It represents the angle of rotation along the common axis from the coordinate system on the i-1th joint to the coordinate system on the i-th joint.

[0067] Assuming that the robot arm is a UR5 robot, according to the parameters of the UR5 robot and the definition of the link coordinate system, the DH parameter table T of the tool usage model can be obtained: D-H , as shown in Table 1:

[0068] Table 1 DH parameter table of robot arm-tool combination

[0069] Connecting rod <![CDATA[θ i ]]> <![CDATA[d i ]]> <![CDATA[α i ]]> <![CDATA[α i ]]> 1 <![CDATA[θ1]]> 89.2 0 90 2 <![CDATA[θ2]]> 0 -425 0 3 <![CDATA[θ3]]> 0 -392 0 4 <![CDATA[θ4]]> 109.3 0 90 5 <![CDATA[θ5]]> 94.75 0 -90 6 <![CDATA[θ6]]> 82.5 0 0

[0070] Substitute the DH parameters into Can get By superimposing various equations, we can obtain the homogeneous transformation matrix between the end and base coordinate systems of the UR5 robot:

[0071]

[0072] in,

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] So

[0080]

[0081] Among them, n x 、n y 、n z , o x , o y , o z 、a x 、a y 、a z is the element of the rotation matrix R, which represents the change in the direction of the robot end coordinate system relative to the base coordinate system, n x 、n y 、n z represents the direction cosine of the x-axis of the robot end coordinate system relative to the base coordinate system, o x , o y , o z represents the direction cosine of the y-axis of the robot end coordinate system relative to the base coordinate system, ax 、a y 、a z Represents the direction cosine of the z-axis of the robot end coordinate system relative to the base coordinate system. x 、p y 、p z The position vector representing the origin of the robot end coordinate system relative to the base coordinate system.

[0082] According to the distance from the gripper joint to the center of the gripper, L G mm, the distance between the center of the gripper and the end of the current tool is L T Millimeters, then the homogeneous transformation matrix from the robot end coordinate system to the tool end coordinate system is as follows:

[0083]

[0084] Integrating the above formulas, the homogeneous transformation matrix from the robot base coordinate system to the tool end coordinate system is obtained as:

[0085]

[0086] Among them, p' x 、p' y 、p' z Represents the multiplication of the homogeneous transformation matrix from the robot end coordinate system to the tool end coordinate system The position vector of the tool end relative to the base coordinate system. The rotation matrix R has not changed, so continue to use n x 、n y 、n z , o x , o y , o z 、a x 、a y 、a z Represents the elements of the rotation matrix R.

[0087] 2.2. Contact point evaluation module:

[0088] The core of the contact point evaluation module is an object surface contact point evaluation method based on a semantic segmentation network (DeepLab v3+), a contact point generation network, and a contact point evaluation network. The input data of this method is the output of the second RGB-D camera in the image acquisition module. It analyzes the relationship between the target position, object, and tool in the real-time acquired image, and predicts the success rate of the robot arm in completing the task when the tool contacts the object. This method outputs the contact point position on the object surface with the highest task success rate.

[0089] The schematic diagram of the touchpoint evaluation module is as follows Figure 6As shown. The DeepLab v3+ network is a neural network for semantic segmentation. At the same time, the improved network based on DeepLab v3+ has also achieved good results in image classification and target detection. The input of DeepLab v3+ is an RGB image, and the output is a segmentation map, which will segment the object, target position and tool position. The segmentation map is input into the contact point generation network, which predicts 10 candidate contact points and contact force directions on the object contour, and then inputs the RGB image and the information of the 10 candidate contact points into the contact point evaluation network, which outputs the position of an optimal contact point and the corresponding contact force direction.

[0090] 3. Execution layer:

[0091] like Figure 7 As shown, the execution layer of the embodiment of the present invention refers to driving the robotic arm to complete a task when a tool is used. The execution layer includes a joint angle prediction module and a robotic arm motion module, which solves the changes in joint angles during the movement of the robotic arm using a tool, controls the joint rotation of the robotic arm, and thereby drives the robotic arm to move.

[0092] 3.1. Joint angle prediction module:

[0093] The schematic diagram of the joint angle prediction module is as follows Figure 8 As shown. The joint angle prediction module predicts the change of the joint angle of the robot in the next step through the variational auto-encoders (VAE) and the multi-temporal recurrent neural network (MT-RNN). The real-time image captured by the second RGB-D camera in the image acquisition module is input to the VAE auto-encoder. The intermediate layer of the VAE auto-encoder outputs the compressed latent space representation. The input of the MT-RNN network is the optimal contact point information output by the contact point evaluation module, the latent space representation and the joint angle of the robot in the previous step, and then predicts the joint angle value of the robot in the next step.

[0094] Among them, the predicted joint angle value of the robot arm's next step is obtained through the MT-RNN network. MT-RNN is a recurrent neural network that inputs the position of the optimal contact point, the latent space representation of the image, and the joint angle of the previous step, and outputs the joint angle of the next step. During the training phase, the parameters of the network will be continuously adjusted to make the difference between the output angle value and the actual angle value of the next step as small as possible. The trained MT-RNN network has the ability to predict the joint angle value of the robot arm's next step.

[0095] 3.2. Robotic arm motion module:

[0096] The robot arm motion module rotates the joint angle of the robot arm according to the predicted next joint angle value of the robot arm, drives the robot arm to move, and transmits the current joint angle of the robot arm to the joint angle prediction module to provide data for predicting the next joint angle.

[0097] The Jacobian matrix represents the mapping relationship between the end velocity and joint velocity of the robot-tool assembly. The joint velocity of the robot is defined as The terminal velocity of the robot-tool assembly in the operating space is defined as The Jacobian matrix is ​​J, then the relationship between the three can be expressed as follows:

[0098]

[0099] Assuming that the robot has 6 joints, the Jacobian matrix J(θ) is a 6×6 matrix, and the elements in J(θ) can be expressed as follows:

[0100]

[0101] Among them, J(θ) ij represents the value of the element in the i-th row and j-th column of the Jacobian matrix, The symbol for finding partial derivatives, q i represents the linear and angular displacement of the end of the robot-tool assembly, θ j Represents the rotation angle of the jth revolute joint.

[0102] The first three rows of the Jacobian matrix J represent the transfer ratio J of the end linear velocity of the robot-tool assembly. v The last three lines represent the transfer ratio J of the end angular velocity of the robot-tool assembly. ω ,

[0103]

[0104] During the movement of the robotic arm, it is necessary to accurately calculate the linear mapping relationship between the joint space velocity of the robotic arm and the operation space velocity. Therefore, the establishment of the Jacobian matrix plays an important role in the dexterous operation of the robotic arm.

[0105] The calculation method of the Jacobian matrix J is to partially differentiate the position and posture of the tool end with respect to the position values ​​of each joint of the robot arm. When a rotating joint of the robot arm rotates, the rotation speed will affect the movement speed of the end. The significance of the Jacobian matrix J is to represent the linear mapping relationship between the joint space speed of the robot arm and the operation space speed of the tool end. After obtaining the Jacobian matrix J of the current robot-tool combination through the above method, when the speed values ​​of each joint are known, the speed value of the tool end in the base coordinate system can be calculated.

[0106] like Figures 9 and 10 As shown, an embodiment of the present invention also provides a method for controlling a robot arm including a tool, comprising the following steps:

[0107] S1, collect RGB images and depth images of the tool and current working area.

[0108] S2. Extract the features of the tool based on the collected RGB image and depth image to obtain the length and type of the tool.

[0109] S3. Collect real-time images of the working area.

[0110] S4. According to the posture of the robot arm holding the tool and the length of the tool, the end position of the robot arm-tool combination is obtained.

[0111] S5. Extract the features of the object based on the real-time image of the working area, combine it with the target position that the object needs to reach, and evaluate the information of the optimal contact point between the tool and the object.

[0112] S6. During the movement of the robot arm, the current image and the joint angle of the robot arm in the previous step are collected in real time, and the joint angle of the next step is obtained according to the current image, the joint angle of the robot arm in the previous step and the optimal contact point information.

[0113] S7. Control the joint rotation of the robot arm according to the predicted joint angle of the next step, and drive the robot arm to move.

[0114] Step S2 specifically includes: predicting the collected RGB image of the candidate tool through the target detection network, outputting the bounding box of the tool and the reference object in the image and the confidence of the tool type, and judging the highest confidence as the type of the tool; using the open source computer vision library to draw the outline of the tool and the reference object obtained by the target detection network, and obtaining the length of the tool through the pixel ratio calculation method.

[0115] Step S5 specifically includes: outputting the real-time image of the working area as a segmentation map through the semantic segmentation network, inputting the segmentation map into the contact point generation network, the contact point generation network predicts multiple candidate contact points and contact force directions on the object contour, and then inputting the real-time image of the working area and the information of multiple candidate contact points into the contact point evaluation network, and outputting the position of an optimal contact point and the corresponding contact force direction.

[0116] Experimental example:

[0117] In one embodiment, a UR5 type robot arm with a gripper installed at the end, which can stably grasp the tool; two Kinect v2 model RGB-D cameras, respectively installed on the wrist of the robot arm and above the working area; tools include sticks, hammers, wrenches, knives, etc. Considering the load capacity of the robot arm, some light tools are used; a computer equipped with NVIDIAGEFORCE GTX 1650 graphics card is used for neural network training and the operation of the robot arm dexterous operating system containing tool use. The main goal of the whole system is that the robot arm can realize the operation of the tool through the control system, the robot arm can play the role of the tool, and at the same time, the intelligence of the robot arm control is improved with the help of neural network.

[0118] The effect diagram of the modeling of the mechanical arm-tool assembly under the Matlab software of this embodiment is as follows Fig.11a As shown in the figure, the effect diagram of the robot arm-tool combination modeling in the Coppeliasim simulation system is as follows Fig.11b As shown, it can be seen that the modeling effect of the robot arm-tool combination in the end position determination module is consistent with the effect in the Coppeliasim simulation system.

[0119] The contact point evaluation module outputs the action point of the tool end, the contact point on the object and the target position point of the object during the use of the tool by the robot arm. The schematic diagram of the output result of the contact point evaluation module is shown in the figure. Figures 12a to 12i shown.

[0120] The schematic diagram of the variational autoencoder input image in the joint angle prediction module of this embodiment is as follows: Fig.13a As shown, the schematic diagram of the variational autoencoder output image in the joint angle prediction module is as follows Fig.13b As shown in Figure 1, in the joint angle prediction module, the image restored by the VAE autoencoder is very close to the original image. Therefore, the latent space vector output by the intermediate layer of the VAE autoencoder can well represent the contour features of the image. The effect diagram of the multi-time recurrent neural network in the joint angle prediction module is shown in Figure 1. Fig.14 As shown in the figure, the joint angles of the six rotation joints in the MT-RNN network are normalized between -1 and 1. The solid line represents the joint angle value output by the MT-RNN network, and the dotted line represents the joint angle value during the actual movement of the robot arm. The output joint angle value fits the actual value very well, indicating that the MT-RNN network has learned a series of joint angle changes in the process of the robot arm using tools to complete the task.

[0121] The tool-containing robot arm control system and method according to the embodiments of the present invention have the following beneficial effects:

[0122] (1) Establishing an architecture for extending dexterous manipulation of a robotic arm with tool use.

[0123] The HCTM architecture in the embodiment of the present invention includes a perception layer, a decision layer, and an execution layer. Each layer is responsible for realizing an important function of the system. The three layers work together to realize the complete functions of the robot arm expansion dexterous manipulation system including tool use.

[0124] (2) Improve the accuracy of tool recognition before the robot arm selects the tool.

[0125] The tool recognition module in the embodiment of the present invention can output the length and purpose of the tool based on the RGB image and depth image of the tool through a feature extraction network, thereby improving the robot's ability to recognize the tool and providing a basis for the robot arm to select the appropriate tool.

[0126] (3) It can solve the problem of obtaining the end position of the robot arm-tool combination.

[0127] The embodiment of the present invention uses a method for solving the end position of a robot-tool combination. The end position of the tool can be obtained by inputting the joint angle of the robot into the kinematic equation. This method can accurately obtain the end position of the robot-tool combination.

[0128] (4) Improve the effectiveness of contact between the tool and the object when the robot arm uses the tool.

[0129] To address the contact interaction problem between the tool and the object when the robotic arm uses the tool, an embodiment of the present invention uses a convolutional neural network through a contact point evaluation module to obtain the effective contact points between the tool and the object. This method comprehensively considers the target state and the current state of the object, and can improve the effectiveness of the contact between the tool and the object.

[0130] (5) Intelligent control is achieved for the process of the robotic arm using tools to complete tasks.

[0131] The joint angle prediction module in the embodiment of the present invention predicts the joint angle of the next step according to the current working image of the robotic arm and the joint angle of the previous step during the process of the robotic arm using the tool, thereby realizing intelligent control of the robotic arm's practical tools.

[0132] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A robot arm control system containing a tool, characterized in that: include: Perception layer, decision layer and execution layer; The perception layer includes an image acquisition module and a tool recognition module. In the perception layer, the image acquisition module is used to acquire RGB images and depth images, and the tool recognition module is used to identify the purpose of the tool and estimate the length of the tool. The decision layer includes an end position determination module and a contact point evaluation module. In the decision layer, the end position of the tool when the robot arm holds the tool is obtained by the end position determination module, and the optimal contact point between the tool and the target object is evaluated by the contact point evaluation module; The execution layer includes a joint angle prediction module and a robotic arm motion module. In the execution layer, the joint angle prediction module predicts the next joint angle change of the robotic arm based on a recurrent neural network, and the robotic arm motion module controls the joint rotation of the robotic arm according to the predicted next joint angle to drive the robotic arm to move.

2. The robot arm control system with a tool as claimed in claim 1, characterized in that: The image acquisition module includes a first RGB-D camera and a second RGB-D camera. The first RGB-D camera is installed on the wrist of the robotic arm and is used to collect RGB images and depth images of the selected tool before using the tool. The second RGB-D camera is installed above the working area and is used to obtain RGB images and depth images in real time during the process of the robotic arm using the tool.

3. The tool-containing robot arm control system according to claim 2, characterized in that: The tool recognition module predicts the collected RGB image of the candidate tool through the target detection network, outputs the bounding box of the tool and the reference object in the image and the confidence of the tool type, and determines the highest confidence as the type of the tool; The open source computer vision library is used to draw the contours of the tools and reference objects obtained by the object detection network, and the length of the tool is obtained by the pixel ratio calculation method.

4. The robot arm control system with a tool as claimed in claim 1, characterized in that: The end position determination module regards the robot arm and the tool as a robot arm-tool combination according to the length of the tool and the DH parameters of the robot arm, and obtains the end position of the robot arm-tool combination through a homogeneous transformation matrix.

5. The tool-containing robot arm control system according to claim 4, characterized in that: The calculation formula for the end position of the robot arm-tool assembly is: P xyz =f(T D-H ,L G ,L T ) Among them, P xyz represents the end position of the robot-tool assembly, T D-H Indicates the DH parameter table of the robot arm, L G Indicates the distance between the end position of the robot arm and the center of the gripper, L T Indicates the distance between the center of the gripper and the end of the current tool.

6. The robot arm control system with a tool as claimed in claim 2, characterized in that: The contact point evaluation module outputs the RGB image obtained in real time during the process of the robotic arm using a tool as a segmentation map through a semantic segmentation network, and inputs the segmentation map into a contact point generation network. The contact point generation network predicts multiple candidate contact points and contact force directions on the object contour, and then inputs the RGB image and the information of multiple candidate contact points into the contact point evaluation network, which outputs the position of an optimal contact point and the corresponding contact force direction.

7. The tool-containing robot arm control system according to claim 6, characterized in that: The joint angle prediction module uses a variational autoencoder to output the RGB image of the robot arm in real time during the process of using a tool as a compressed latent space representation, inputs the optimal contact point information, latent space representation and the joint angle of the robot arm in the previous step output by the contact point evaluation module into a multi-time recurrent neural network, and then predicts the joint angle of the robot arm in the next step.

8. A method for controlling a robot arm including a tool, characterized in that: The following steps are involved: S1, RGB image and depth image of acquisition tool and current working area; S2, extracting the features of the tool according to the collected RGB image and depth image, and obtaining the length and type of the tool; S3, collecting real-time images of the working area; S4, obtaining the end position of the robot arm-tool combination according to the posture of the robot arm holding the tool and the length of the tool; S5. Extract the features of the object according to the real-time image of the working area, combine it with the target position that the object needs to reach, and evaluate the information of the optimal contact point between the tool and the object; S6, collecting the current image and the joint angle of the robot in the previous step in real time during the movement of the robot arm, and obtaining the joint angle of the next step according to the current image, the joint angle of the robot in the previous step and the optimal contact point information; S7. Control the joint rotation of the robot arm according to the predicted joint angle of the next step, and drive the robot arm to move.

9. The control method of a robot arm with a tool as claimed in claim 8, characterized in that: Step S2 specifically includes: predicting the collected RGB image of the candidate tool through the target detection network, outputting the bounding box of the tool and the reference object in the image and the confidence of the tool type, and judging the highest confidence as the type of the tool; using the open source computer vision library to draw the outline of the tool and the reference object obtained by the target detection network, and obtaining the length of the tool through the pixel ratio calculation method.

10. The control method of a robot arm with a tool as claimed in claim 8, characterized in that: Step S5 specifically includes: outputting the real-time image of the working area as a segmentation map through the semantic segmentation network, inputting the segmentation map into the contact point generation network, the contact point generation network predicts multiple candidate contact points and contact force directions on the object contour, and then inputting the real-time image of the working area and the information of multiple candidate contact points into the contact point evaluation network, and outputting the position of an optimal contact point and the corresponding contact force direction.

Images