A method for real-time human single-arm tracking by a six-axis collaborative robotic arm based on a monocular camera.

Abstract

This invention provides a method for a six-axis collaborative robotic arm to real-time follow a human arm based on a monocular camera, belonging to the field of human-computer interaction technology. Addressing the problems of shaky joint detection results and the lack of a fast method for calculating joint angles in existing arm joint detection methods, this invention uses a monocular camera to capture real-time images of the human arm, estimates 2D arm joint angles in real time, and corrects the coordinate positions using arm image segmentation to obtain more accurate and stable joint position. A fast spatial point connection angle calculation method is used to calculate the angles of each joint of the human arm, thereby enabling a six-axis collaborative robotic arm to real-time follow and reproduce the angles and postures of the human arm. This method has the advantages of being fast and effectively reducing detection result shakyness, enabling the collaborative robotic arm to reproduce the angles of each joint of a human arm in real time, with the overall real-time reproduction time being less than 40 milliseconds, meeting the real-time requirements of robotic arms.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction and relates to the technology of 2D human arm joint estimation and human-computer posture one-to-one mapping. Specifically, it relates to a method for a six-axis collaborative robotic arm based on a monocular camera to follow a human single arm in real time. Background Technology

[0002] In some scenarios, such as robotic arm teaching and remote robotic arm control, it is necessary to achieve real-time tracking of the human arm's posture by a collaborative robotic arm. To achieve this tracking, it is necessary to estimate the joint points of the human arm and perform human-machine posture mapping.

[0003] Currently, AI methods are commonly used to estimate 2D human arm joints. For example, the OpenPose method (reference [1]) can detect 2D key points of multiple people simultaneously. Its network input is an RGB image. The detection process first uses a deep learning network to extract the heat map of the area where the joint is located. The position of the joint in the image is determined according to the size of the heat map value. After detecting a single joint, the local graph connection method is used to connect the identified single joints to realize human joint detection. However, directly using the existing AI methods to estimate the joints of the arm has the following problems: because the existing AI model adds random augmentation, the output results of the model for the same input image are slightly different in different number of trials. That is, the joint position estimated by the AI ​​method will jitter. If the random augmentation in the AI ​​model is reduced or removed, the jitter phenomenon can be eliminated, but the robustness and universality of the AI ​​model will be weakened. For some input images, the joint position predicted by the model does not jitter but deviates greatly from the correct position. Therefore, the random augmentation method commonly used in AI models is to avoid the situation where the prediction result deviates too much or even no prediction result occurs by sacrificing a small part of the prediction accuracy. It is a compromise. In real-time tracking and reproduction of robotic arms, the jitter at the joint points is disastrous for the robotic arm, as the robotic arm will also jitter continuously, which can easily damage core components such as motors. References [1] and [2] both propose a method to identify the joint points of the arm from an RGB photo, which is also the most advanced method in the publicly available information. However, the detection results of these methods all have the problem of jitter.

[0004] After obtaining the position information of the joints of the human arm, it is necessary to calculate the joint angles using this information so that the collaborative robotic arm can reproduce the following motion. Currently, a common method for calculating angles from known position information is the vector cosine method, as seen in the mapping between Kinect, Xbox 360, and the six-axis robotic arm. Given the coordinates of points P1, P2, and P3, the magnitude of angle P1P2P3 can be determined by first calculating the cosine of the angle between the vectors, as follows:

[0005]

[0006] In this context, subscripts [0] and [1] represent the x-coordinate and y-coordinate, respectively. Finally, the values ​​of angles P1P2P3 can be obtained by solving the inverse cosine. Directly using the cosine method to solve for the included angle has the following two problems: First, when solving for the inverse cosine, there will be two return values, and it is necessary to determine which angle value to take; second, the computational complexity of formula (1) is high, with a total of seven multiplication and division operations. Directly using this method to calculate the arm angle value is not conducive to the real-time requirements of the robotic arm system.

[0007] In summary, existing technologies need to be improved to meet the requirements of real-time tracking of human arm posture by robotic arms.

[0008] References:

[0009] [1]Cao, Z., Martinez, GH, Simon, T., Wei, S. and Sheikh, YA, 2019. OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. In IEEETransactions on Pattern Analysis and Machine Intelligence.

[0010] [2]Google.Mediapipe.https: / / google.github.io / mediapipe / . Summary of the Invention

[0011] To address the problems existing in the prior art, namely the jitter in the arm joint detection results (i.e., inaccurate and unstable joint detection results) and the lack of an efficient and fast calculation method for the corresponding joint angles of the robotic arm from the arm joint position information, this invention provides a method for a six-axis collaborative robotic arm to follow a human single arm in real time based on a monocular camera. This method enables the real-time capture of the human body and arm using a single ordinary camera (monocular camera), real-time estimation of 2D arm joint angles, and real-time reproduction of the joint angles of the human single arm by the collaborative robotic arm after obtaining the 2D human single arm joint angles. The overall time for real-time reproduction is less than 40 milliseconds.

[0012] This invention relates to a method for a six-axis collaborative robotic arm based on a monocular camera to real-time follow a human single arm, comprising the following steps:

[0013] Step 1: Set up a monocular camera to capture real-time images of a person's right or left arm and input the images into an image processing device.

[0014] Step 2: The image processing device detects the arm contour from the current image and uses the OpenPose network to obtain the position coordinates of the joints of the single arm.

[0015] Step 3: The image processing device corrects the position coordinates of the acquired shoulder joint, elbow joint, and wrist joint.

[0016] Step 4: The image processing device uses the corrected joint positions, combined with the joint positions identified by the OpenPose network, to calculate the angles of the human shoulder, elbow, and wrist joints.

[0017] Step 5: The image processing device sends the calculated angles of the shoulder, elbow, and wrist joints to the six-axis collaborative robotic arm, which then controls the corresponding joints to move to the appropriate angles.

[0018] In step 3, the method for correcting the coordinate positions is as follows: Let the coordinates of the shoulder joint, elbow joint, and wrist joint be P, respectively. a P b and P c (1) First, P b P c The line segment is divided into five equal parts, starting from P. c Take the first division point, and construct P through the first division point. b P c The perpendicular line of the line segment intersects the outline of the arm at point P. i1 and P i2 , obtain P i1 P i2 The midpoint P of the line segment M1 (2) According to the midpoint P M1 The method of obtaining it, and then from P c Divide the end into two equal parts, the second and third, and obtain the midpoint P. M2 and P M3 (3) Passing through the three midpoints P M1 P M2 and P M3 Fit a quadratic curve (4) P a P b The line segment is divided into five equal parts, according to the midpoint P. M1 The method of obtaining it, from P b Divide the end into three equal parts, and obtain the midpoint P. M4 P M5 and P M6 Passing through the three midpoints P M4 P M5 and P M6 Fit a quadratic curve (5) Calculate P c Click The closest point, used as the wrist joint point for correction. calculate and The intersection of these points will be used as the location for correction of the elbow joint. Calculate P a Click The closest point, used as the location for correction of the shoulder joint.

[0019] In step 4, the method for obtaining the angles of the shoulder, elbow, and wrist joints is as follows: Let the positions of the starting joint point, intermediate joint point, and ending joint point of a certain joint angle be P, respectively. s ,P V ,P E Then, according to the following formula, the joint angle f(P) is calculated from the joint point position coordinates. s ,P V ,P E ):

[0020]

[0021] Then the joint angle f(P) in radians s ,P V ,P E Convert to joint angles in degrees Send to the six-axis collaborative robotic arm.

[0022] Compared with the prior art, the advantages of the present invention are:

[0023] (1) The method of this invention proposes an arm image segmentation method to correct the arm joint position obtained by the OpenPose network, making the arm joint position at the current moment more accurate and stable, thus achieving the technical effect of reducing the jitter of the joint detection result. Compared with the continuous frame smoothing filtering method and the AI ​​model modification and retraining / retraining method, this invention has the advantage of quickly and effectively reducing the jitter of the detection result.

[0024] (2) The present invention proposes a method for rapidly calculating the angle of a line connecting spatial points. By converting spatial coordinate positions to vector angles, the angles of each joint of the human arm are calculated, achieving the technical effect of converting position information into angle information. Compared with the traditional method of solving the included angle using the cosine value of a vector, it has the advantage of speed and is also convenient for subsequent follow-up and reproduction of the angles of the corresponding joints of the robotic arm.

[0025] (3) The method of the present invention only requires an ordinary camera, which is low in cost and easy to deploy. It can realize the real-time tracking and reproduction of the angle and posture of the human arm by a six-axis collaborative robotic arm. The overall running time of the robotic arm is less than 40 milliseconds after testing, which meets the design requirements of real-time tracking. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the six-axis collaborative robotic arm based on a monocular camera that can follow a human arm in real time according to the present invention.

[0027] Figure 2 A flowchart illustrating the process of obtaining accurate and stable coordinates of arm joints (taking the right hand as an example) according to an embodiment of the present invention;

[0028] Figure 3 This is a diagram illustrating the labels and names of human joints detected using the OpenPose network;

[0029] Figure 4 This is a schematic diagram of the detection of the captured right arm image according to an embodiment of the present invention; wherein, (a) is a grayscale schematic diagram of the original right arm image captured by the camera; (b) is the right arm contour detection result; and (c) is a schematic diagram of adding the detected right arm joints to the right arm contour detection image.

[0030] Figure 5 This is a schematic diagram of the process for correcting the coordinates of arm joint points (taking the right arm as an example) according to an embodiment of the present invention;

[0031] Figure 6 This is a schematic diagram of auxiliary points and lines used in the coordinate correction of arm joint points (taking the right arm as an example) according to an embodiment of the present invention;

[0032] Figure 7 This is a flowchart illustrating the real-time tracking and reproduction of a six-axis robotic arm according to an embodiment of the present invention;

[0033] Figure 8 The image shows a comparison of the detection results of the elbow joint point on the right arm using the method of the present invention and the existing method; where (a) is the existing method and (b) is the method of the present invention. Detailed Implementation

[0034] The technical solution of the present invention will be described below with reference to the accompanying drawings and embodiments.

[0035] The application scenario of this invention is to use a monocular camera for robotic arm teaching. Robotic arm teaching refers to teaching a robotic arm to demonstrate or perform certain actions through some method. Common teaching methods include computer programming, drag-and-drop replication, etc. This invention uses the posture of a human arm as a template, and uses a regular monocular camera to capture real-time footage of the human arm, allowing a six-axis collaborative robotic arm to follow and replicate the angles of the arm's joints in real time, thereby achieving the teaching purpose.

[0036] In this embodiment of the invention, the system for implementing the method of the present invention is as follows: Figure 1As shown, the invention includes a monocular camera 1, an image processing device 2, and a six-axis collaborative robotic arm 3. The monocular camera 1 captures images of the human arm 4 in real time, transmits the images to the image processing device 2 for joint position detection and angle conversion of the position information to the joints of the arm, and finally controls the joints of the six-axis collaborative robotic arm 3 to move to the same angle as the arm joints. This invention has the advantages of low camera cost, no need for wired sensors for arm teaching, and low hardware requirements. The monocular camera uses a common RGB camera, which is inexpensive. The image processing device can use either a high-end or low-end computing unit, such as an industrial computer or an embedded development board like Nvidia's Jetson Nano. The development language is the cross-platform Python, and the six-axis collaborative robotic arm is lighter and more compact.

[0037] The present invention provides a method for a six-axis collaborative robotic arm based on a monocular camera to real-time follow a human single arm, which mainly includes two processes. The first process is to obtain more accurate and stable arm joint position information. The second process is to more quickly convert the arm position information into arm angle information, thereby enabling the robotic arm to reproduce the angle. The method of the present invention has optimized algorithms for each process, and corresponding hardware requirements.

[0038] The first process is as follows Figure 2 As shown, the following explanation will be provided in three steps.

[0039] Step 1: Position the person's arm radially parallel to the plane of the monocular camera, ensuring the camera can see the arm without significant joint obstruction. Turn on the camera and input the image into the OpenPose network to obtain the coordinates of the arm's joint points.

[0040] Specifically, taking the right arm as an example, the embodiments of the present invention use OpenPose to detect the joints of a single arm, including... Figure 3 The code identifies six joints: 24 (right hip), 12 (right shoulder), 14 (right elbow), 16 (right wrist), 18 (right little finger), and 20 (right index finger). The output is the image coordinates of these six joints. If detecting the left arm, OpenPose will output the following: Figure 3 The image coordinates of six joints: 23 (left hip), 11 (left shoulder), 13 (left elbow), 15 (left wrist), 17 (left little finger), and 19 (left index finger).

[0041] In this embodiment of the invention, the OpenPose network used is the OpenPose network described in reference [1].

[0042] Step 2: Detect the outline of the arm image at the current moment.

[0043] This invention employs an existing method, such as the Canny operator, to detect the arm contour in an image. Other methods capable of detecting arm contours can also be used, as long as the detection accuracy and speed meet the requirements.

[0044] like Figure 4 As shown, the original image of the right arm captured by the camera (a) shows the arm contour detected as shown in (b), and (c) shows the arm contour after adding the right arm joint points detected by OpenPose.

[0045] Step 3: Correct P 12 P 14 and P 16 Position coordinates, to obtain more accurate and stable joint position coordinates. and like Figure 5 and Figure 6 As shown, the position coordinate correction process is as follows:

[0046] S1: Taking the right arm as an example, P 14 P 16 The line segment is divided into five equal parts.

[0047] S2: From P 16 Take the first division point, and construct P through the first division point. 14 P 16 The perpendicular line of the line segment intersects the outline of the arm at point P. i1 and P i2 Take P i1 P i2 The midpoint P of the line segment M1 .

[0048] S3: Similar to operation S2, from P 16 Take the second division point to obtain the midpoint P. M2 .

[0049] S4: Similar to operation S2, from P 16 Take the third equal division point to obtain the midpoint P. M3 .

[0050] S5: Pass P M1 P M2 and P M3 Fit a quadratic curve

[0051] S6: P 12 P 14 The line segment is divided into five equal parts.

[0052] S7: From P 14 End-to-P 12Divide the end into three equal parts, taking the first, second, and third division points respectively. Repeat steps S2, S3, and S4 to obtain the midpoint P. M4 P M5 and P M6 Repeat step S5 to obtain a quadratic fitting curve.

[0053] S8: Calculate P 16 Click The closest point is denoted as

[0054] S9: Calculation and The intersection point is denoted as

[0055] S10: Calculate P 12 Click The closest point is denoted as

[0056] The corrected joint points obtained by the above method have two major advantages: they ensure that the joint points are within the arm, preventing the joint points predicted by the OpenPose network from deviating significantly from the arm; and during minor tremors, this method combines the results of the OpenPose network and the results of arm shape processing, making the current position of the arm joint points more accurate and stable, effectively reducing the problem of arm joint tremors.

[0057] After obtaining the accurate, stable, and minimally shaky position coordinates of the joints of the human arm at the current moment, the method of this invention calculates angle information from the position information, including the angles of the shoulder, elbow, and wrist joints. The method of this invention proposes using the image coordinates P of joint point i... i =(u i ,v i The conversion formula from joint angle to joint angle is as follows:

[0058]

[0059] Among them, P s ,P V ,P E These represent the image coordinates of the starting, intermediate, and ending joints of a certain angle of the arm, respectively. arctan() represents the arctangent function operation, and [0] and [1] represent the x and y coordinates, respectively.

[0060] The joint angle unit obtained by formula (2) is radians. This invention converts it to degrees (°), which makes it easier to set the robot arm joint angle in degrees. The conversion formula is as follows:

[0061]

[0062] The specific steps for solving the angles of the shoulder, elbow, and wrist joints of the left and right single arms are as follows.

[0063] If it is the right arm, calculate the current angle α of the right arm shoulder joint using the following formula. RS :

[0064]

[0065] Calculate the current angle β of the right elbow joint using the following formula. RE :

[0066]

[0067] Calculate the current angle γ of the right wrist joint using the following formula. RW :

[0068]

[0069] Finally, control the joints 2, 3, and 5 of the six-axis collaborative robotic arm to move to α respectively. RS ,β RE and γ RW angle.

[0070] Similarly, for the left arm, the detected left hip joint point P can be obtained according to process 1. 23 Corrected left shoulder joint Left elbow joint and left wrist joint The angle α of the right shoulder joint at the current moment is calculated using the following formula. LS :

[0071]

[0072] Calculate the current angle β of the right elbow joint using the following formula. LE :

[0073]

[0074] Calculate the current angle γ of the right wrist joint using the following formula. LW :

[0075]

[0076] Finally, control the joints 2, 3, and 5 of the six-axis collaborative robotic arm to move to α respectively. LS ,β LE and γ LW angle.

[0077] In the above description, P 24 P 18 P20 The OpenPose network detected the positions of the right hip joint, right little finger, and right index finger joints. 23 P 17 P 19 The locations of the hip joint, little finger, and index finger joints of the left arm, detected by the OpenPose network.

[0078] By repeating the above description of the present invention Figure 2 and Figure 7 The two processes enable a six-axis collaborative robotic arm to follow and reproduce the angle information of a human arm in real time.

[0079] As can be seen from the above description, the angle calculation formula (2) proposed in this invention requires seven multiplication and division operations, while the commonly used cosine angle formula (1) only requires two, which speeds up the calculation efficiency.

[0080] Experiments have shown that, using the method of this invention, the average response time for a six-axis robotic arm to perform a following motion is approximately 21.5 milliseconds. Different image processing units result in different image processing speeds. With a 12th Gen Intel(R) Core(TM) i7-12700F (2.10GHz) CPU, it can process an average of approximately 61 frames per second of RGB images, meaning each image takes approximately 16.39 milliseconds to process. Taking right arm elbow joint detection as an example... Figure 8 The left figure (a) shows the effect of running the method in reference [1] 10 times, and the right figure (b) shows the effect of running the method of the present invention 10 times. It can be seen that the elbow joint point detected by the method of the present invention is more accurate and stable, which is beneficial to reducing the jitter of the detection results.

[0081] As can be seen from the above, the method of this invention can realize the real-time (overall running time <40 milliseconds) tracking and reproduction of the angle and posture of a human arm by a six-axis collaborative robotic arm, meeting the design requirements for real-time tracking of robotic arms. It only requires a common camera, is low-cost and easy to deploy, and can be processed using a general-purpose industrial computer or embedded development board. This method can effectively reduce the jitter in the arm joint detection results, thus effectively avoiding damage to the robotic arm's motors; and it can also solve for joint angles more quickly.

[0082] Except for the technical features described in the specification, all other technologies are known to those skilled in the art. Descriptions of well-known components and technologies are omitted in this invention to avoid redundancy and unnecessary limitation. The embodiments described above do not represent all embodiments consistent with this application. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this invention are still within the protection scope of this invention.

Claims

1. A method for a six-axis collaborative robotic arm based on a monocular camera to follow a human single arm in real time, characterized in that, include: Step 1: Set up a monocular camera to capture real-time images of a person's right or left arm and input the images into an image processing device. Step 2: The image processing device detects the arm contour from the current image and uses the OpenPose network to obtain the position coordinates of the joints of the single arm. Step 3: The image processing device corrects the position coordinates of the acquired shoulder joint, elbow joint, and wrist joint. The correction method is as follows: Let the coordinates of the shoulder joint, elbow joint, and wrist joint be respectively... , and (1) First, will The line segment is divided into five equal parts, from Take the first division point, and then proceed through the first division point. The perpendicular line of the line segment intersects the outline of the arm at... and , obtain midpoint of a line segment (2) According to the midpoint The method of obtaining it, and then from Divide the end into two equal parts, the second and third, and obtain the midpoint. and (3) Passing through three midpoints , and Fit a quadratic curve (4) The line segment is divided into five equal parts, according to the midpoint. The method of obtaining it, from Divide the end into three equal parts, and obtain the midpoint. , and Passing through three midpoints , and Fit a quadratic curve (5) Calculation Up to The closest point, used as the wrist joint point for correction. ;calculate and The intersection of these points will be used as the location for correction of the elbow joint. ;calculate Up to The closest point, used as the location for correction of the shoulder joint. ; Step 4: The image processing device uses the corrected joint positions, combined with the joint positions identified by the OpenPose network, to calculate the angles of the human shoulder, elbow, and wrist joints. Step 5: The image processing device sends the calculated angles of the shoulder, elbow, and wrist joints to the six-axis collaborative robotic arm, which then controls the corresponding joints to move to the appropriate angles.

2. The method according to claim 1, characterized in that, In step 1, OpenPose is used to detect the joints of the right or left arm, including the hip joint, shoulder joint, elbow joint, wrist joint, little finger joint, and index finger joint.

3. The method according to claim 1, characterized in that, In step 1, the arm is positioned radially parallel to the plane of the monocular camera, so that the camera can see the arm without any joint obstruction.

4. The method according to claim 1, characterized in that, In step 4, the method for obtaining the angles of the shoulder joint, elbow joint, and wrist joint is as follows: Let the positions of the starting joint, intermediate joint, and ending joint of a certain joint angle be respectively... Then, according to the following formula, the joint angle f is calculated from the coordinates of the joint point. ): Where arctan() represents the arctangent function operation, and [0] and [1] are the x and y coordinates of the joint position, respectively; The obtained joint angle f( Convert the units of ) to degrees, as shown below: The image processing device will obtain joint angles in degrees. Send to the six-axis collaborative robotic arm.

5. The method according to claim 4, characterized in that, In step 4, the monocular camera captures an image of the person's right arm, and the position of the right hip joint is detected in step 2. Little finger joint position and the joint of the index finger The corrected right shoulder joint position is obtained from step 3. Position of the right elbow joint and right wrist joint Then the angle of the right shoulder joint at the current moment can be calculated. The angle of the right elbow joint The angle of the right wrist joint .

Images