A dual-arm grasping system control method based on multi-vision

Abstract

The present invention relates to a dual-manipulator grasping system control method based on multi-eye vision, including: building an anthropomorphic dual-manipulator, the anthropomorphic dual-manipulator is provided with a plurality of joint steering gears and a development board for controlling the working state of each joint steering gear; building a three-eye vision recognition and analysis system, the three-eye vision recognition and analysis system includes a three-eye vision camera for collecting different visual images of the target object, the three-eye vision camera is connected to a processor, the processor is used to perform image recognition analysis, and calculate and output control instructions of each joint steering gear of the dual-manipulator; the processor transmits the control instruction to the development board, and then controls the dual-manipulator to complete the grasping operation of the target object accordingly. Compared with the prior art, the present invention can realize the precise positioning and identification of the target object, and can control the dual-manipulator to perform efficient and accurate single-arm grasping operation or dual-arm collaborative grasping operation for different targets.

Description

Technical Field

[0001] The present invention relates to the technical field of dual robotic arm control, and in particular to a dual robotic arm grasping system control method based on multi-eye vision. Background Art

[0002] With the rapid development of industrial automation, robotic arms have been widely used in the industrial field, mainly using 3-4 degrees of freedom robotic arms to complete some simple assembly and manufacturing tasks. Industrial robotic arms are mostly used for assembly line operations. Their working process is based on established procedures to complete monotonous and repetitive operations, and they lack autonomy and decision-making.

[0003] Therefore, industrial robotic arms are difficult to apply to complex daily life scenarios, such as domestic service robots. In the prior art, there is a method of positioning through laser ranging to achieve the positioning of the target object, and then combined with the control of the robotic arm, the robotic arm can move to the target position and perform corresponding operations. However, the cost of laser ranging radar is relatively high. The more wire harnesses, the higher the price. For example, a laser radar with 64 wire harnesses costs up to hundreds of thousands of yuan. In addition, the ranging space range of laser ranging radar is limited. In an environment with large dynamic changes, positioning loss and errors are prone to occur. In addition, it is difficult for the robotic arm to perform corresponding operations on different targets, resulting in low operating efficiency and accuracy. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a dual-robotic arm grasping system control method based on multi-eye vision, so as to achieve accurate positioning and identification of the target object, so that the dual robotic arms can operate the target object efficiently and accurately.

[0005] The purpose of the present invention can be achieved by the following technical solution: A dual-manipulator grasping system control method based on multi-vision, comprising the following steps:

[0006] S1. Building an anthropomorphic dual robotic arm, wherein the anthropomorphic dual robotic arm is provided with a plurality of joint servos and a development board for controlling the working state of each joint servo;

[0007] S2. Construct a three-eye visual recognition and analysis system, wherein the three-eye visual recognition and analysis system includes a three-eye visual camera for collecting different visual images of the target object, and the three-eye visual camera is connected to a processor, and the processor is used to perform image recognition and analysis, and calculate and output control instructions for the servos of each joint of the dual robotic arm;

[0008] S3. The processor transmits the control instructions to the development board, and then controls the dual robotic arms accordingly to complete the grasping operation of the target object.

[0009] Furthermore, the step S1 specifically includes the following steps:

[0010] S11. According to the classification of human joints, the design of the robotic arm is divided into shoulder joint, elbow joint and wrist joint;

[0011] In order to imitate the movement of human arms, two joint servos are set at each joint to share two degrees of freedom;

[0012] S12. According to the torque calculation formula and the measured joint length, the torque value corresponding to each joint servo is calculated to complete the selection of the joint servo;

[0013] S13, connecting each joint servo to the development board accordingly to receive corresponding control instructions from the development board.

[0014] Furthermore, the articulated servo specifically adopts a magnetically encoded longitudinal servo.

[0015] Furthermore, the development board is specifically a URT-1 development board.

[0016] Furthermore, the specific working process of the processor in step S2 includes:

[0017] S21, performing instance segmentation on the image captured by the trinocular vision camera to realize the recognition of the target object in the image and extract the mask of the target to be captured;

[0018] S22, for the extracted mask, using the first-order moment of the image to calculate the pixel coordinates of the target material center;

[0019] The main axis and the main axis direction angle of the target object are calculated using the second-order moment of the image;

[0020] S23, using the spatial coordinate conversion relationship of the three-eye stereo vision, converting the pixel coordinates of the target material center into spatial coordinates, and calculating the position of the grasping point that requires the collaborative grasping of the dual robotic arms according to the main axis direction angle;

[0021] S24. Construct an inverse kinematics table, combine it with the master-slave control principle, and calculate the control instructions of the servos of each joint of the dual robotic arms.

[0022] Furthermore, the step S21 specifically uses the Mask-RCNN algorithm to perform instance segmentation on the image collected by the trinocular vision camera:

[0023] The image processed by the Mask-RCNN algorithm shows the confidence box of the target object, the mask branch in Mask-RCNN, and the mask covering the target.

[0024] Export the mask covering the target object in the mask branch, modify the mask matrix color to white 255, and set the non-target object to black 0;

[0025] Export the mask matrix to jpg format;

[0026] Use the Open-cv built-in library to perform image preprocessing operations on the processed jpg images;

[0027] The preprocessed image is gray-scaled and then binarized. By setting the threshold, the mask part of the image is completely connected and there is a clear segmentation between the target object and the background in the image.

[0028] Furthermore, the step S22 specifically includes the following steps:

[0029] S221. Using the image moment, according to the mask that wraps the target object, the center of mass of the target in the two-dimensional plane is obtained:

[0030]

[0031] Among them, x i is the i-th point on the mask, and n is the total number of points on the mask;

[0032] S222, the binary image pixels are comprehensively represented as m00, and the first-order moments m10 and m01 of the target mask image are calculated to determine the grayscale center of the grayscale image:

[0033]

[0034] m00=∑ x ∑ y f(x,y)m10=∑ x ∑ y xf(x,y),m01=∑ x ∑ y yf(x,y)

[0035] S223. For a lying target object, when grasping it, the main axis and the main axis direction angle of the target object are calculated using the second-order moment of the image:

[0036] m20=∑ x ∑ y x 2 f(x,y),m02=∑ x ∑ y y 2 f(x,y),m11=∑ x ∑ y xyf(x,y)

[0037]

[0038] The rotation angle of the end effector of the dual robot arm can be determined by using the main axis and the main axis direction angle of the target object.

[0039] Furthermore, the step S23 specifically includes the following steps:

[0040] S231. Obtain camera internal parameters:

[0041] Collect calibration images, adjust the camera angle so that the camera's field of view can cover the entire area to be captured, and then adjust the camera's focal length and brightness;

[0042] Take multiple images of the calibration plate from three different viewpoints, and ensure that the calibration plate in each image has a set pose difference.

[0043] According to the size value of the real calibration plate, the intrinsic parameter matrix of the camera is obtained;

[0044] S232. After marking the camera internal parameters, use the three-eye stereo vision to perform coordinate transformation:

[0045] The trinocular stereo vision model consists of three cameras, the optical axes of which are at set angles, and are projected on the imaging plane using the three cameras C1, C2, and C3;

[0046] Ideally, the measurement values ​​of the binocular system composed of C1 and C2, the measurement values ​​of the binocular system composed of C1 and C3, and the measurement values ​​of the binocular system composed of C2 and C3 should coincide with the actual coordinates of the measured point P, that is, the point P where O1P1, O2P2, and O3P3 intersect in space is also the measured point P. However, in real scenes, there are certain errors in the binocular system, and the measurement values ​​P1, P2, and P3 do not completely coincide with the actual coordinates of the measured point P, but intersect at three different points in space, that is, the points where O1P1, O2P2, and O3P3 intersect in pairs. Therefore, the real measured point P needs to be obtained through the binocular vision positioning algorithm;

[0047] Based on the error of binocular vision measurement system, the three-dimensional coordinates of the object to be measured are solved by using trinocular vision fusion, and the coordinate transformation expression becomes a homogeneous transformation form:

[0048]

[0049] Assume that the measured coordinates of point P are P = (X w ,Y w ,Z w ), whose optimal estimate satisfies the objective function:

[0050] F=min(‖P-P1‖+‖P-P2‖+‖P‖)

[0051] The coordinates of point P determined in this way are the optimal estimates of the true coordinates of point P.

[0052] Furthermore, the inverse kinematics table in step S24 specifically adopts Craig expression method and combines the mathematical model of the dual robot arm to obtain a DH parameter table, and the DH parameter table can be used to solve the angle that each joint of the target point needs to rotate.

[0053] Furthermore, the step S24 specifically includes the following steps:

[0054] S241, setting one robotic arm as the master arm and the other robotic arm as the slave arm, pre-planning the motion trajectory of the master arm according to the control target, the slave arm follows the movement of the master arm, the master and slave arms satisfy a closed-chain constraint relationship, and deriving the motion trajectory of the slave arm according to the motion constraint relationship;

[0055] S242. According to the coordinate system relationship, the target object is set as Master and the two arms are set as Slaver. The master-slave planning method is adopted. According to the constraint relationship between the ends of the two arms and the target object, the two arms follow the movement of the target object: first, the object trajectory is planned, and according to the constraint equations, the end trajectories of the two robotic arms are obtained respectively, and smoothed in the joint space, and finally the trajectory control instructions are sent to the robotic arms for execution.

[0056] Compared with the prior art, the present invention proposes a dual-robotic arm grasping system based on multi-camera vision and its control method. By building an anthropomorphic dual robotic arm and constructing a three-camera vision recognition and analysis system, the three-camera vision recognition and analysis system is used to collect different visual images of the target object. By recognizing and analyzing the images, on the one hand, the target object can be identified and accurately located, and on the other hand, the control instructions corresponding to the anthropomorphic dual robotic arms can be calculated and output, so that the dual robotic arms can efficiently and accurately realize the grasping operation of the target object, thereby expanding the application field of the robotic arms to more complex daily life service scenarios.

[0057] The present invention first uses the instance segmentation algorithm Mask-RCNN to realize the recognition of the target and the covering of the target mask for the collected image; then the pixel coordinates of the target material center are calculated by the first-order moment of the image, and the main axis and the main axis direction angle of the target are calculated by the second-order moment; then the pixel coordinates of the target are converted into the world coordinates in space by using the three-eye vision space coordinate conversion relationship; after obtaining the world coordinates of the target, the control instructions for single-arm grasping and double-arm collaborative grasping of the target are designed by using inverse kinematics and closed-chain kinematics. This makes the dual robotic arms more autonomous and decisive, and can perform corresponding operations on the identified targets. For example, if a target that can be directly grasped by a single arm is identified, the control adopts a single arm to grasp; if a target that needs to be grasped by two arms in collaboration is identified, the control adopts a collaborative grasping of two arms.

[0058] The present invention performs positioning based on vision, that is, the external sensors are only three low-cost cameras, which greatly reduces the cost compared with the laser radar method; compared with the currently common binocular vision positioning, the present invention adopts three cameras and uses the three-eye vision space conversion relationship to locate the target object, which has a wider viewing angle than binocular vision. In addition, when one of the cameras is damaged, the remaining two cameras can still form a binocular vision system to complete positioning, which is more reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0060] Figure 2a This is a schematic diagram of the human arm joints;

[0061] Figure 2b A schematic diagram of an anthropomorphic robotic arm constructed in an embodiment;

[0062] Figure 3 Schematic diagram of an anthropomorphic dual robotic arm in the embodiment;

[0063] Figure 4 Schematic diagram of instance segmentation detection effect in the embodiment;

[0064] Figure 5 This is a schematic diagram of centroid extraction in the embodiment;

[0065] Figure 6 Schematic diagram of the main axis and the main axis direction angle in the embodiment;

[0066] Figure 7a This is a schematic diagram of a calibration plate image in the embodiment;

[0067] Figure 7b This is a schematic diagram of the camera calibration effect in the embodiment;

[0068] Figure 8 Schematic diagram of the rotation matrix R during the camera intrinsic parameter calibration process in the embodiment;

[0069] Fig. 9 This is a schematic diagram of trinocular stereo vision;

[0070] Fig.10 It is a schematic diagram of the closed-chain kinematic model of double-arm collaboration;

[0071] Fig.11a This is a schematic diagram of the effect of a single robotic arm grasping a lying plastic bottle in an embodiment;

[0072] Fig.11b This is a schematic diagram of the effect of a single robotic arm grasping a plastic bottle placed upright in the embodiment;

[0073] Fig.12a and Figure 12b Schematic diagram of the effect of the double mechanical arms grasping an umbrella in the embodiment;

[0074] Fig.13a and Fig.13b Schematic diagram of the effect of a single robotic arm grasping an egg in an embodiment. DETAILED DESCRIPTION

[0075] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0076] Example

[0077] like Figure 1 As shown, a dual-manipulator grasping system control method based on multi-eye vision includes the following steps:

[0078] S1. Build an anthropomorphic dual robotic arm, wherein the anthropomorphic dual robotic arm is provided with a plurality of joint servos and a development board for controlling the working state of each joint servo;

[0079] S2. Construct a three-eye vision recognition and analysis system, wherein the three-eye vision recognition and analysis system includes a three-eye vision camera for collecting different visual images of the target object, the three-eye vision camera is connected to a processor, and the processor is used to perform image recognition and analysis, and calculate and output control instructions for the servos of each joint of the dual robotic arm;

[0080] S3. The processor transmits the control instructions to the development board, and then controls the dual robotic arms accordingly to complete the grasping operation of the target object.

[0081] This embodiment applies the above technical solution, mainly including:

[0082] 1. Build an anthropomorphic dual robotic arm. This embodiment uses a magnetically encoded bus servo as the joint servo of the robotic arm, and calculates the torque value of each joint servo according to the torque calculation formula to select the joint servo; this embodiment uses the URT-1 development board to realize the control of the entire arm of the robotic arm.

[0083] 2. Build a three-eye vision system. This embodiment uses an instance segmentation algorithm (Mask-RCNN) to achieve target recognition and mask coverage;

[0084] For the extracted mask, the pixel coordinates of the target material centroid are calculated using the first-order moment of the image, and the main axis and the main axis direction angle of the target are calculated using the second-order moment of the image;

[0085] The spatial coordinate conversion relationship of the three-eye stereo vision is used to convert the pixel coordinates of the target material center into spatial coordinates. At the same time, according to the calculated main axis direction angle, the grasping point requiring dual-arm collaborative grasping is further calculated to achieve dual-arm collaborative grasping.

[0086] 3. Based on the constructed robotic arm, construct the DH parameter table of the robotic arm, use inverse kinematics to grasp the target object, and use the double-arm closed-chain kinematics to complete the double-arm coordinated grasping of the target object.

[0087] Specifically, in this embodiment, when building an anthropomorphic dual robotic arm, it completely imitates the human arm (such as Figure 2a The human arm has 6 degrees of freedom excluding the palm part. In this technical solution, the robot arm is divided into shoulder joint, elbow joint and wrist joint according to the joints of the human arm, as shown in 2b. In addition, in order to accurately imitate the movement of the human arm, 2 servos are placed at each joint to share 2 degrees of freedom; and according to the torque calculation formula, the servos of different joints are selected to ensure that the target object can still be grabbed under the condition of extreme torque load. Among them, the torque calculation formula is:

[0088] T=F*D (i.e. torque=force*moment length)

[0089] According to the measurement of joint length and torque calculation, the selection results of torque corresponding to different joint servos can be obtained. In this embodiment, the target object does not exceed 1kg, and it is necessary to ensure that the target object of 500g can still be grasped under the maximum load of the mechanical arm. The actual dual mechanical arm built in this embodiment is as follows Figure 3 shown.

[0090] This embodiment mainly includes the following contents when obtaining the target centroid and the main axis direction angle:

[0091] For the images captured by the trinocular vision camera, Mask-RCNN is used to perform instance segmentation. The mask of the target to be captured is extracted according to the identified category, and the extracted mask is subjected to image preprocessing operations such as binarization and grayscale operations using Open-cv. After processing, the white part in the target image is the connected part, and the centroid of the target object is extracted (the image moment in digital processing is used to extract the centroid of the target object), that is, the centroid of the connected graph part is extracted.

[0092] (1) Figure 4 As shown in the figure, the image processed by the Mask-RCNN algorithm will show the confidence box of the target object, the mask branch in Mask-RCNN, and the mask covering the target. Export the mask covering the target object in the mask branch, modify the mask matrix color to white 255, and set the non-target object to black 0. Export the mask matrix to jpg format for later processing.

[0093] (2) For the processed jpg image, use the Open-cv built-in library to perform image preprocessing. Use the medianBlur() function to grayscale the preprocessed image, and then binarize the image. Set the threshold so that the mask part of the image is completely connected and the target object and the background in the image are clearly separated.

[0094] (3) The extraction of the centroid of connected parts mainly uses the application of image moments in digital image processing and computer vision related fields. Image moments refer to the weighted average (moment) of the grayscale of certain specific pixels in an image, or attributes of an image with similar functions or meanings. Image moments are usually used to describe segmented image objects. Image moments can be used to obtain some properties of the image, including area (overall brightness) and information about the geometric center and direction.

[0095] This technical solution uses image moments to find the center of mass of the target in a two-dimensional plane based on the mask that wraps the target object. Assume that the processed mask consists of n points x i The center of mass of the target object is given by:

[0096]

[0097] The binary image pixels are comprehensively represented as m00, and the first-order moments m10 and m01 of the target mask image are calculated to determine the grayscale center of the grayscale image:

[0098] m00=∑ x ∑ y f(x,y)m10=∑ x ∑ y xf(x,y),m01=∑ x ∑y yf(x,y) (2)

[0099]

[0100] Calculate the coordinates of the target two-dimensional image: When grasping a lying target object with a two-finger gripper, it is necessary to obtain the main axis and main axis angle of the target object to calculate the rotation angle of the end effector. This technical solution uses the second-order moment of the image to calculate the main axis and main axis angle of the target object:

[0101] m20=∑ x ∑ y x 2 f(x,y),m02=∑ x ∑ y y 2 f(x,y),m11=∑ x ∑ y xyf(x,y) (4)

[0102] The inclination angle of the target object (i.e. the main axis direction angle) is calculated as:

[0103]

[0104] The center of mass and direction angle obtained are as follows Figure 5 and Figure 6 shown.

[0105] For target objects that need to be grasped by both arms, the pixel coordinates of the grasping point are calculated based on the center of mass and the main axis direction angle, and then the pixel coordinates are converted into world coordinates through the spatial coordinate conversion relationship for grasping.

[0106] When performing spatial coordinate conversion in this embodiment, it is considered that the role of the camera is to complete the recognition and positioning of the target during the robot arm grasping operation. This technical solution uses three-eye vision, that is, the world coordinates of the grasping point are calculated through camera calibration and three-eye stereo vision. The specific process is as follows:

[0107] This embodiment uses the MATLAB toolkit to calibrate the camera's intrinsic parameters. The camera calibration adopts the Zhang Zhengyou calibration method and uses the calibrateCamera() function to start the camera calibration. The first step is to obtain the camera's intrinsic parameters: collect the calibration image, first manually adjust the camera's angle so that the camera's field of view can cover the entire desktop to be captured, and then adjust the camera's focal length and brightness to achieve a better image quality. Figure 7a and Figure 7bAs shown in the figure, 20 photos of the calibration plate are taken from three cameras of different perspectives. At the same time, the calibration plate in each picture should have the largest possible difference in posture. Then, the intrinsic parameter matrix of the camera is obtained according to the size value of the real calibration plate.

[0108] Export camera intrinsics:

[0109]

[0110] The internal parameters of the three cameras are:

[0111]

[0112]

[0113]

[0114] The external parameter matrix of the camera is obtained by the PNP algorithm, including the rotation matrix R (such as Figure 8 as shown) and the translation matrix T.

[0115] Compared with binocular cameras, trinocular cameras have a larger field of view. At the same time, they can ensure that if one of the cameras is damaged, spatial positioning can still be performed by using the binocular vision formed by the other two cameras.

[0116] After marking the camera internal parameters, the coordinate transformation is performed using trinocular stereo vision. Fig. 9 As shown in the figure, the trinocular stereo vision model is composed of three cameras, and the optical axes of these three cameras are at a certain angle. The trinocular camera stereo vision can also be regarded as three visual models. The three cameras C1, C2, and C3 are used to project on the imaging plane. Ideally, the measurement values ​​of the binocular system composed of C1 and C2, the measurement values ​​of the binocular system composed of C1 and C3, and the measurement values ​​of the binocular system composed of C2 and C3 should coincide with the actual coordinates of the measured point P, that is, the point P in space where O1P1, O2P2, and O3P3 intersect, that is, the measured point P. However, in real scenes, there are certain errors in the binocular system, and the measured values ​​P1, P2, and P3 do not completely coincide with the actual coordinates of the measured point P, but intersect at three different points in space, that is, the points where O1P1, O2P2, and O3P3 intersect in pairs, where the actual measured point P can be obtained by the positioning algorithm of binocular vision.

[0117] Based on the error of binocular vision measurement system, this technical solution uses trinocular vision fusion to solve the three-dimensional coordinates of the object being measured. The coordinate transformation expression becomes a homogeneous transformation form:

[0118]

[0119] Assume that the measured coordinates of point P are P = (X w ,Yw ,Z w ), whose optimal estimate satisfies the objective function:

[0120] F=min(‖P-P1‖+‖P-P2‖+‖P‖) (11)

[0121] The coordinates of point P determined by the above formula are considered to be the optimal estimate of the true coordinates of point P.

[0122] When constructing the inverse kinematics table, this embodiment takes into account that inverse kinematics is the process of determining the parameters of the joint movable object to achieve the required posture. The robot arm is composed of an object connected by joints and a set of rigid segments connected by joints. Changing the angles of the joints can produce infinite shapes. The solution to the forward kinematics problem is the posture of the object given these angles, while the inverse kinematics problem is relatively difficult, mainly because it explains the posture of a given object. Given the three-dimensional coordinates of the terminal effector in space, it is necessary to find the angle that each joint needs to move.

[0123] In this embodiment, based on the abstract mathematical model of the robot arm, DH parameters constructed using Craig expression are shown in the following table.

[0124]

[0125]

[0126] According to the constructed DH parameter table, the angle that each joint needs to rotate to reach the target point can be calculated: (θ1, θ2, θ3, θ4, θ5, θ6).

[0127] In this embodiment, when performing dual-arm collaborative control, a commonly used method for dual-arm coordinated operation is the master-slave control method. The principle of this method is to set one robotic arm as the master arm (Master) and the other robotic arm as the slave arm (Slaver), and then pre-plan the motion trajectory of the master arm according to the control target. The slave arm follows the movement of the master arm, and the master and slave arms satisfy the closed-chain constraint relationship. The motion trajectory of the slave arm is derived according to the motion constraint relationship.

[0128] According to the coordinate system relationship, this embodiment sets the object as the master and the arms as the slave. The master-slave planning method is adopted. According to the constraint relationship between the ends of the arms and the object, the arms follow the movement of the object. First, the object trajectory is planned, and the end trajectories of the left and right arms are obtained according to the constraint equations. Then, the smoothing process is performed in the joint space, and finally the trajectory is sent to the robot arm for execution.

[0129] like Fig.10As shown in the figure, the dual-arm cooperative operation is mainly based on the type of tight-fit operation, that is, when the dual arms grasp a rigid body, the two end grippers and the grasped object do not move relative to each other. A closed kinematic chain will be formed between the dual arms and the grasped object, and the movement of the dual arms must also be subject to corresponding constraints and maintain a certain kinematic relationship.

[0130] According to the coordinate transformation relationship, the transformation matrix of the dual-arm end effector and the material center of the operation target relative to the coordinate origin can be obtained as follows:

[0131]

[0132]

[0133] In the formula, and They are all constant matrices, and the pose constraint relationship between the two end effectors can be obtained:

[0134]

[0135]

[0136] In the formula, and is the position matrix of the two arms’ ends relative to the coordinate origin; and is the rotation matrix of the two arms' ends relative to the coordinate origin.

[0137] Since the relative motion between the two arms is not considered, the speeds of the two arms are equal, and the speed constraint relationship of the two arms can be obtained:

[0138]

[0139]

[0140] In the formula, J(q L ),J(q R ) represent the Jacobian matrices of the left and right arms, respectively, J v (q),J w (q) denote the position Jacobian matrix and attitude Jacobian matrix of the manipulator, respectively. A common method in maintaining coordinated operation of dual-arm systems is the relative Jacobian matrix, which allows the dual-arm system to be considered as a unique redundant manipulator with a number of joints equal to the sum of the relative joints of each manipulator. In addition, the dual-arm system modeled with the relative Jacobian matrix can be controlled with the same algorithm as the single-arm system. The relative Jacobian matrix is ​​then generalized to Jacobian null space protection to achieve collaborative tasks.

[0141] Based on the Jacobian matrix solved above, the rotation angle, speed and acceleration control instructions corresponding to each shutdown servo can be solved. The processor transmits the above control instructions to each joint servo through the development board, and the dual robotic arms can be controlled to complete the grasping operation of the target object.

[0142] In this embodiment, grasping tests are performed on a lying plastic bottle, an upright plastic bottle, an umbrella, and an egg. Fig.11a and Fig.11b As shown in the figure, the calibrated trinocular camera uses Mask-RCNN for instance segmentation. The grasping target is a plastic bottle. A single robotic arm is used to place the plastic bottle at any position within the reach of both arms. After calculating the center of mass and converting it into spatial coordinates, the robotic arm moves to the target object position according to inverse kinematics to achieve the grasping action.

[0143] In daily life, there are some objects that are difficult or impossible to grasp with one hand, such as umbrellas and large packages of tissues. Since one arm cannot grasp the object, both arms are needed to grasp the target object. First, Mask-RCNN is used to perform instance segmentation to detect the umbrella and large package of tissues. Then, the 3D coordinates of the center of mass of the umbrella are obtained. The robot arm selects the grasping points of both arms, and the two arms follow the movement of the target object. The two arms work together to grasp the umbrella. The effect is as follows: Fig.12a and Figure 12b shown.

[0144] For some fragile items that need to control the torque of the gripper, such as eggs. This embodiment attempts to grasp the egg with a single arm based on the feedback of the gripper torque, identifies the spatial position of the egg, controls the torque of the gripper, and grasps the raw egg. Based on the experimental attempt and the grasp of the torque of the gripper egg, the egg is finally grasped successfully (such as Fig.13a and Fig.13b As shown), it also ensures that the eggs will not be crushed due to excessive torque.

[0145] In summary, this technical solution is based on a multi-vision dual-arm grasping system. By building a humanoid 6-DOF robotic arm, the robotic arm is combined with a computer vision algorithm, the target object is identified using an instance segmentation algorithm, and three-eye stereo vision is used for positioning. Different grasping methods are used for different target objects, including single-arm grasping and dual-arm collaborative grasping. This technical solution breaks through the problem that traditional robotic arms can only complete a single repetitive operation. At the same time, the design of an anthropomorphic robotic arm can help users complete some daily operations that are completed by human arms. By combining visual algorithms and robotic arm kinematics, the robotic arm adopts different grasping methods for different targets. The design of the robotic arm in this technical solution is close to that of a real human arm, and it is also the initial prototype of the future anthropomorphic service robot.

[0146] This technical solution can be applied to service robots, including helping humans to complete the grasping and carrying of simple objects. By issuing grasping instructions to the robot, it can grasp a bottle of beverage to a specified location, or grasp an umbrella to a specified location. The robot arm identifies the type and location of the target object based on the recognition algorithm, and makes autonomous decisions to achieve single-arm or double-arm grasping.

Claims

1. A dual-arm grasping system control method based on multi-vision, characterized in that: The following steps are involved: S1. Building an anthropomorphic dual robotic arm, wherein the anthropomorphic dual robotic arm is provided with a plurality of joint servos and a development board for controlling the working state of each joint servo; S2. Construct a three-eye visual recognition and analysis system, wherein the three-eye visual recognition and analysis system includes a three-eye visual camera for collecting different visual images of the target object, and the three-eye visual camera is connected to a processor, and the processor is used to perform image recognition and analysis, and calculate and output control instructions for the servos of each joint of the dual robotic arm; S3, the processor transmits the control instruction to the development board, and then controls the dual robotic arms to complete the grasping operation of the target object; The specific working process of the processor in step S2 includes: S21, performing instance segmentation on the image captured by the trinocular vision camera to realize the recognition of the target object in the image and extract the mask of the target to be captured; S22, for the extracted mask, using the first-order moment of the image to calculate the pixel coordinates of the target material center; The main axis and the main axis direction angle of the target object are calculated using the second-order moment of the image; S23, using the spatial coordinate conversion relationship of the three-eye stereo vision, converting the pixel coordinates of the target material center into spatial coordinates, and calculating the position of the grasping point that requires the collaborative grasping of the dual robotic arms according to the main axis direction angle; S24, constructing an inverse kinematics table, combining the master-slave control principle, and calculating the control instructions of the servos of each joint of the dual robotic arms; The step S21 specifically uses the Mask-RCNN algorithm to perform instance segmentation on the image collected by the trinocular vision camera: The image processed by the Mask-RCNN algorithm shows the confidence box of the target object, the mask branch in Mask-RCNN, and the mask covering the target. Export the mask covering the target object in the mask branch, modify the mask matrix color to white 255, and set the non-target object to black 0; Export the mask matrix to jpg format; Use the Open-cv built-in library to perform image preprocessing operations on the processed jpg images; The preprocessed image is grayed out, and then the gray image is binarized. By setting the threshold, the mask part of the image is completely connected, and at the same time, there is a clear segmentation between the target object and the background in the image. The step S22 specifically includes the following steps: S221. Using the image moment, according to the mask that wraps the target object, the center of mass of the target in the two-dimensional plane is obtained: Among them, x i is the i-th point on the mask, and n is the total number of points on the mask; S222, the binary image pixels are comprehensively represented as m00, and the first-order moments m10 and m01 of the target mask image are calculated to determine the grayscale center of the grayscale image: m00=∑ x ∑ y f(x,y)m10=Σ x S y xf(x,y), m01=∑ x ∑ y yf(x,y) S223. For a lying target object, when grasping it, the main axis and the main axis direction angle of the target object are calculated using the second-order moment of the image: m20=∑ x ∑ y x 2 f(x,y),m02=∑ x ∑ y y 2 f(x,y),m11=∑ x ∑ y xyf(x,y) The rotation angle of the end effector of the dual robot arm can be determined by using the main axis and the main axis direction angle of the target object.

2. According to the multi-vision based dual-arm grasping system control method of claim 1, it is characterized in that: The step S1 specifically includes the following steps: S11. According to the classification of human joints, the design of the robotic arm is divided into shoulder joint, elbow joint and wrist joint; In order to imitate the movement of human arms, two joint servos are set at each joint to share two degrees of freedom; S12. According to the torque calculation formula and the measured joint length, the torque value corresponding to each joint servo is calculated to complete the selection of the joint servo; S13, connecting each joint servo to the development board accordingly to receive corresponding control instructions from the development board.

3. A multi-vision based dual-manipulator grasping system control method according to any one of claims 1 to 2, characterized in that: The joint steering gear specifically adopts a magnetically encoded longitudinal steering gear.

4. A multi-vision based dual-manipulator grasping system control method according to any one of claims 1 to 2, characterized in that: The development board is specifically a URT-1 development board.

5. The method for controlling a dual-manipulator grasping system based on multi-vision according to claim 1, characterized in that: The step S23 specifically includes the following steps: S231. Obtain camera internal parameters: Collect calibration images, adjust the camera angle so that the camera's field of view can cover the entire area to be captured, and then adjust the camera's focal length and brightness; Take multiple images of the calibration plate from three different viewpoints, and ensure that the calibration plate in each image has a set pose difference. According to the size value of the real calibration plate, the intrinsic parameter matrix of the camera is obtained; S232. After marking the camera internal parameters, use the three-eye stereo vision to perform coordinate transformation: The trinocular stereo vision model consists of three cameras, the optical axes of which are at set angles, and are projected on the imaging plane using the three cameras C1, C2, and C3; Ideally, the measurement values ​​of the binocular system composed of C1 and C2, the measurement values ​​of the binocular system composed of C1 and C3, and the measurement values ​​of the binocular system composed of C2 and C3 should coincide with the actual coordinates of the measured point P, that is, the point P where O1P1, O2P2, and O3P3 intersect in space is also the measured point P. However, in real scenes, there are certain errors in the binocular system, and the measurement values ​​P1, P2, and P3 do not completely coincide with the actual coordinates of the measured point P, but intersect at three different points in space, that is, the points where O1P1, O2P2, and O3P3 intersect in pairs. Therefore, the real measured point P needs to be obtained through the binocular vision positioning algorithm; Based on the error of binocular vision measurement system, the three-dimensional coordinates of the object to be measured are solved by using trinocular vision fusion, and the coordinate transformation expression becomes a homogeneous transformation form: Assume that the measured coordinates of point P are P = (X w ,Y w ,Z w ), whose optimal estimate satisfies the objective function: F=min(‖P-P1‖+‖P-P2‖+‖P‖) The coordinates of point P determined in this way are the optimal estimates of the true coordinates of point P.

6. The method for controlling a dual-manipulator grasping system based on multi-vision according to claim 1, characterized in that: The inverse kinematics table in step S24 specifically adopts the Craig expression method and combines the mathematical model of the dual robotic arms to obtain a DH parameter table, and the DH parameter table can be used to calculate the angle that each joint of the target point needs to rotate.

7. The multi-vision based dual-arm grasping system control method according to claim 1, characterized in that: The step S24 specifically The process includes: One robot arm is set as the master arm and the other robot arm is set as the slave arm. According to the control target, the motion trajectory of the master arm is pre-planned. The slave arm follows the movement of the master arm. The master and slave arms satisfy the closed-chain constraint relationship. The motion trajectory of the slave arm is derived according to the motion constraint relationship. According to the coordinate system relationship, the target object is set as Master and the dual arms are set as Slaver. The master-slave planning method is adopted. According to the constraint relationship between the ends of the dual arms and the target object, the dual arms follow the movement of the target object: first, the object trajectory is planned, and according to the constraint equations, the end trajectories of the dual robotic arms are obtained respectively, and smoothed in the joint space. Finally, the trajectory control instructions are sent to the robotic arms for execution.

Images