A robot dexterous operation method and system based on visual key point guidance

Abstract

A method and system for dexterous robot manipulation based on visual keypoint guidance is disclosed. The method includes: S1, establishing a base coordinate system, solving for initial keypoints based on the base coordinate system, and locating the dexterous hand based on the initial keypoints to determine the end-effector posture when grasping a tool; S2, the dexterous hand grasps the tool using a pre-established coarse gesture library and based on the end-effector posture when grasping the tool; S3, using the dexterous hand to move the tool, identifying new keypoints after the shift, and calculating the transformation matrix between the tool head coordinate system and the base coordinate system, as well as the transformation matrix between the target object coordinate system and the base coordinate system, based on the new keypoints; S4, calculating the end-effector position of the tool head in operation on the target object, until the dexterous hand completes the operation on the target object. This invention transforms the interaction process between the dexterous hand and the target object into the alignment of keypoints between the object and the hand, conforming to the rules of dexterous hand grasping tools and exhibiting strong generalization ability.

Description

Technical Field

[0001] This invention relates to the field of robotic arm technology, and in particular to a method and system for dexterous robot operation based on visual key point guidance. Background Technology

[0002] Robots are now widely used in many fields such as home, industry, and rescue. Enabling robots to learn human-like dexterity in grasping and performing corresponding tasks is a necessary condition for robots to learn dexterity in complex tasks.

[0003] Currently, most mature grasping algorithms are designed for two-finger grippers. However, two-finger grippers have low degrees of freedom, poor envelope, and a small contact area with the object, which cannot meet the needs of complex and dexterous operations. Five-finger dexterous hands are flexible, human-like, and have more possibilities, enabling them to grasp complex tools. However, they have more degrees of freedom and involve fine-grained hand-object interactions, requiring more sophisticated algorithms. The process of a dexterous hand grasping an object is not simply about holding it firmly; it involves the implementation of functional grasping. For example, when holding a drill, the contact between the index finger and the button, and the contact between the other four fingers and the handle, need to be considered.

[0004] Currently, methods for using dexterous hands to grasp tools and then perform fine manipulations on objects are not mature, and their robustness and accuracy are also low. Summary of the Invention

[0005] This invention provides a method and system for dexterous robot operation based on visual key point guidance to solve the technical problems mentioned in the background art.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] This invention provides a method for dexterous robot manipulation based on visual key points, comprising the following steps:

[0008] S1. Establish a base coordinate system, then solve the initial key points based on the base coordinate system, and locate the dexterous hand based on the initial key points to solve the end-effector posture when grasping the tool.

[0009] S2. The dexterous hand grasps the tool using a pre-established coarse gesture library and based on the end-effector posture when gripping the tool.

[0010] S3. Establish the tool head coordinate system and the target object coordinate system. Use the dexterous hand to move the tool, identify the new key points after the shift, and calculate the transformation matrix between the tool head coordinate system and the base coordinate system and the transformation matrix between the target object coordinate system and the base coordinate system based on the new key points.

[0011] S4. Based on the transformation matrix between the tool head coordinate system and the base coordinate system and the transformation matrix between the target object coordinate system and the base coordinate system, continuously calculate the end position of the tool head's operation on the target object until the dexterous hand holds the tool and completes the operation on the target object.

[0012] Another aspect of the present invention provides a robot dexterous operating system based on visual key point guidance, including a robotic arm, a dexterous hand mounted on the robotic arm, a camera, a control host, a tool rack, tools placed on the tool rack, and a target object. The control host, camera, tool rack, and target object are all placed around the robotic arm, and the dexterous hand, robotic arm, and camera are all electrically connected to the control host.

[0013] With the assistance of a robotic arm and a camera, the dexterous hand manipulates the target object by holding the tool, following the above-mentioned robot dexterity operation method.

[0014] The beneficial effects of this invention are:

[0015] 1. This invention employs a key point detection method and utilizes the key point recognition method CMAKE, which improves the robustness and generalization of the method. It can be applied to various tools and used to achieve different long-term tasks.

[0016] 2. The key point of this method is to transform the interaction process between the dexterous hand and the object into the alignment of key points between the object and the hand, which conforms to the rules of dexterous hand grasping tools and has a strong generalization ability.

[0017] 3. This invention transforms the interaction process between the tool and the target object into the alignment between coordinate systems, which is more direct, efficient, and accurate than other methods. Attached Figure Description

[0018] Figure 1 This is a flowchart of the robot dexterity operation method in this invention;

[0019] Figure 2 This is an operation flowchart in an embodiment of the present invention;

[0020] Figure 3 This is a structural diagram of the robot dexterous operating system in this invention. Detailed Implementation

[0021] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many other different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0022] Reference Figure 1and Figure 2 This application provides a method for dexterous robot operation based on visual key point guidance. Taking a dexterous hand picking up a hammer to hammer a nail as an example, the method includes the following steps:

[0023] S1. Establish a base coordinate system, then solve the initial key points based on the base coordinate system, and locate the dexterous hand based on the initial key points to solve the end posture of the robotic arm when grasping the tool (i.e., the hammer).

[0024] S2. The dexterous hand grasps the tool (hammer) based on a pre-established coarse gesture library and the end-effector posture when grasping the tool (hammer).

[0025] Specifically, once the dexterous hand reaches the designated position, based on our established coarse gesture library, corresponding to "hold_hammer," the dexterous hand will execute the coarse gesture required to grasp the hammer. After the dexterous hand executes the hold coarse gesture, the force experienced by the dexterous hand is read every 0.01 seconds. According to the designed judgment condition for the magnitude of the differential, when the absolute value of the force experienced by a certain finger is less than a certain value, or when the sum of the absolute values ​​of the forces experienced by all fingers is less than a certain threshold, it is considered that the grasp is stable and proceeds to the next step.

[0026] S3. Establish the coordinate system of the tool head (i.e., hammer head) and the coordinate system of the target object (i.e., nail). Use the dexterous hand to move the tool (hammer) and re-identify the new key points after the displacement (due to the displacement of the hammer head key points caused by the gripping process). Calculate the transformation matrix between the tool head (hammer head) coordinate system and the base coordinate system and the transformation matrix between the target object (nail) coordinate system and the base coordinate system based on the new key points.

[0027] S4. Based on the transformation matrix between the tool head (hammer) coordinate system and the base coordinate system and the transformation matrix between the target object (nail) coordinate system and the base coordinate system, continuously calculate the end position of the tool head (hammer) operation on the target object (nail) until the dexterous hand holds the tool (hammer) to complete the operation on the target object (nail) (i.e., the operation of hammering the nail with the hammer).

[0028] This invention discloses a method for dexterous robot manipulation based on visual keypoint guidance, mainly including: using keypoints to locate the dexterous hand; using an established coarse gesture library for dexterous grasping; and the dexterous hand picking up a tool (hammer) and interacting with a target object (nail). The position of the dexterous hand is located using the keypoints on the target object (nail), thereby enabling specific interactions between the dexterous hand and the target object (nail); different grasping methods are implemented for different tools (hammers) under specific tasks based on the established coarse gesture library; once the tool (hammer) is firmly grasped, the dexterous hand can pick it up to perform the specified operation.

[0029] The robot dexterity manipulation method of this invention can be applied to various scenarios, and its key aspects can be refined using different tools (hammers). Therefore, the excellent dexterity manipulation method of this invention enables the use of some tools (hammers) in real-life scenarios, and has good application value.

[0030] Furthermore, this invention employs a key point detection method and utilizes the key point recognition method CMAKE, which improves the robustness and generalization of the method. It can be applied to various tools (hammers) and used to achieve different long-term tasks.

[0031] In some embodiments, S1 specifically includes the following steps:

[0032] S11. Establish a base coordinate system on the base, and use the key point recognition method CMAKE to solve for four initial key points on the tool (hammer) and one initial key point on the target object (nail);

[0033] The four initial key points on the tool (hammer) are the four initial key points when gripping the tool (hammer), namely the index finger key point, the little finger key point, the wrist key point, and the tool head (hammer head) key point; among them, the initial wrist key point is located on the tool (hammer);

[0034] S12. Establish a hand coordinate system on the dexterous hand. Since the key point recognition algorithm CMAKE is used to solve the problem, the hand is located on the tool (hammer), but the actual position of the wrist is a certain distance away from the object. The positioning of the dexterous hand is done through the wrist key points. Therefore, the initial wrist key points need to be adjusted so that the adjusted wrist key points are not located on the tool (hammer). And calculate the coordinates of the adjusted wrist key points.

[0035] S13. Construct a tool (hammer) coordinate system using the key points of the index finger, the key points of the little finger, and the adjusted wrist key points, and calculate the rotation matrix between the tool (hammer) coordinate system and the base coordinate system.

[0036] S14. Calculate the rotation matrix between the end coordinate system and the hand coordinate system. The rotation matrix between the end coordinate system and the hand coordinate system is the HE prior.

[0037] Specifically, before the first grasp, the coordinates of the index finger keypoint, the little finger keypoint, and the wrist keypoint in the base coordinate system are obtained using the teach pendant, and denoted as F. H base L H base W H baseThe method for establishing the hand coordinate system is the same as that for establishing the object coordinate system. The origin W of the hand coordinate system is used. H base By aligning the origin of the end coordinate system with the origin of the end coordinate system, i.e., repeating the steps in S13, we obtain the rotation matrix between the hand coordinate system and the base coordinate system.

[0038] Simultaneously, the position of the end effector in the base coordinate system is recorded, thereby calculating the rotation matrix between the end effector coordinate system and the base coordinate system. Finally, the rotation matrix between the hand coordinate system and the end-effector coordinate system can be calculated:

[0039]

[0040] in, This represents the rotation matrix between the end coordinate system and the hand coordinate system.

[0041] S15. Solve the end-effector posture of the robotic arm when gripping the tool (hammer) using the rotation matrix between the tool (hammer) coordinate system and the base coordinate system, and the rotation matrix between the end-effector coordinate system and the hand coordinate system.

[0042] In some embodiments, S11 specifically includes the following steps:

[0043] S111. Establish a base coordinate system on the base, and determine the transformation matrix between the camera coordinate system and the base coordinate system using the hand-eye calibration method.

[0044] S112. Using the CMAKE key point recognition method, four initial key points on the tool (hammer) and one initial key point N on the target object (nail) are obtained in the camera coordinate system. camera ;

[0045] The four initial key points on the tool (hammer) are the four initial key points for gripping the tool (hammer), namely the index finger key point F. O camera Key point L of the little finger O camera W, the key point of the wrist O ′ camera And the key points of the tool head (hammer head) chuitou camera ;

[0046] S113. Then, transform the five initial key points to the base coordinate system to obtain the initial key points on the tool (hammer) and the target object (nail) in the base coordinate system. The calculation formulas are as follows:

[0047]

[0048] Where, N base This represents the initial key point on the target object (nail) in the base coordinate system; F O base L O base W O ′ base chuitou base These represent the key points of the index finger, little finger, wrist, and tool head (hammer head) when gripping the tool (hammer) in the base coordinate system, respectively; D0, D1, and D2 are the key points of F... O base L O base W O ′ base Abbreviated symbols.

[0049] In some embodiments, S12 specifically includes the following steps:

[0050] S121. Establish a hand coordinate system on a dexterous hand;

[0051] S122. Measure the dimensions of the hand triangle in the palm of the dexterous hand in a hand coordinate system. The three vertices of this triangle in the hand coordinate system are F, L, and W, corresponding to the key points of the index finger, little finger, and adjusted wrist, respectively. Then, obtain the vector based on the dimensions of the hand triangle in the palm of the dexterous hand.

[0052] S123. Because the keypoints obtained by the CMAKE keypoint recognition algorithm are all on the tool (hammer), the positions of the little finger keypoint and wrist keypoint differ from the actual positions when the hand grasps the tool (hammer). Therefore, the little finger keypoint and wrist keypoint need to be adjusted. Calculate the vector between points D0 and D1. Then, point D1 is corrected to obtain point D1', so that the adjusted point D1 is in the correct position; point D1' remains on the tool (hammer); then, based on the vector... Find the vector between point D1' and point D0. vector The formula for calculation is:

[0053]

[0054] Where a, b, and c represent vectors. The three coordinate components in the base coordinate system; |.| represents the magnitude of the vector.

[0055] Based on vector Find the position of point D1', that is, the position of the adjusted little finger key point in the base coordinate system;

[0056] S124. Based on the property that points D0, D1', D2, and W all lie in the same plane M1, and using points D0, D1', and D2, solve for the normal vector of plane M1. The calculation formula is:

[0057]

[0058] Where n1, n2, and n3 represent the normal vectors respectively. The components in the three directions in the base coordinate system. This represents the vector between D0 and D2;

[0059] Then, using points D0, D1', and W, we can solve for normal vectors of different lengths in the same direction. The specific calculation formula is as follows:

[0060]

[0061] Where, vector In the formula, (x F y F , z F ), (x W y W , z W Let (x, y, z) represent the coordinates of points D0 and D2 in the base coordinate system, respectively, and let (x, y, z) represent the coordinates of point W to be solved in the base coordinate system.

[0062] S125. Using the dimensions of the triangle measured in S122, calculate the cosine and sine values ​​of ∠WD0D1'. The calculation formulas are as follows:

[0063]

[0064] in, This represents the vector between point D0 and point W; This represents the vector between point D1' and point W; the angle ∠WD0D1' is an acute angle;

[0065] S126, Solving for the normal vector The modulus, calculated using the following formula:

[0066]

[0067] Where m represents the normal vector The model.

[0068] S127. Based on the coordinates of point D0 in the base coordinate system, establish the following three equations, as follows:

[0069] b*(z--z F)--c*(y - y F ) = m*n1 (1)

[0070] c*(x--x F ) - a*(z - z F ) = m*n2 (2)

[0071] a*(y - y F ) - b*(x--x F ) = m*n3 (3)

[0072] Two equations are established using the lengths of side D0W and side D1'W as follows:

[0073]

[0074] S128. Subtract Equation (4) from Equation (5) to obtain the relationship between the coordinates x, y, and z of point W:

[0075] 2*(x F - x L )*x + 2*(y F - y L )*y + 2*(z F - z L )*z = A (6)

[0076] Where A represents a custom parameter one and satisfies the following equation:

[0077]

[0078] Then use Equation (1) to solve for the relationship between z and y:

[0079]

[0080] And substitute Equation (7) into Equation (6) to obtain the relationship between x and y:

[0081] 2*x*E + F*y = A - G (8)

[0082] Where E, F, and G are custom parameter two, custom parameter three, and custom parameter four respectively, and satisfy the following relationships:

[0083] E = x F - x L

[0084]

[0085] S129. Transform Equation (3) to obtain:

[0086] - b*x + a*y = B (9)

[0087] Where B is a user-defined parameter five, and satisfies the following relationship:

[0088] B = m*n3 + a*y F -b*x F

[0089] Then, combining equation (8) with equation (9), we obtain the analytical expression for y:

[0090]

[0091] Substituting equation (10) into equation (9), we obtain the analytical expression for x:

[0092]

[0093] Substituting equation (10) into equation (7), we obtain the analytical expression for z:

[0094]

[0095] The three coordinate values ​​of point W are obtained through equations (10), (11), and (12). Point W is the adjusted wrist key point, which is the point...

[0096] In some embodiments, S13 specifically includes the following steps:

[0097] S131. Construct a tool (hammer) coordinate system using the index finger keypoint, little finger keypoint, and adjusted wrist keypoint. The three axes of the tool coordinate system are x, y, and sq. o y o z o ;x o The axis is obtained from the unit vector pointing from the adjusted wrist keypoint to the index finger keypoint, z o The axis is determined by x o The result of the cross product of the adjusted unit vector from the wrist keypoint to the little finger keypoint is obtained; y o By z o Axis cross product x o The axis is obtained; the origin of the tool (hammer) coordinate system is W. O base ;

[0098] S132. Representation of the x-axis of the calculation tool (hammer) coordinate system in the base coordinate system. o The calculation formula is:

[0099]

[0100] S133. Representation of the z-axis of the calculation tool (hammer) coordinate system in the base coordinate system.o The calculation formula is:

[0101]

[0102] S134. Representation of the y-axis of the calculation tool (hammer) coordinate system in the base coordinate system. o The calculation formula is:

[0103] y o =z o ×x o

[0104] S135, using x o y o and z o Solving the rotation matrix between the tool (hammer) coordinate system and the base coordinate system The specific calculation formula is as follows:

[0105]

[0106] In some embodiments, the formula for calculating the rotation matrix between the hand coordinate system and the end-effector coordinate system in S14 is:

[0107]

[0108] in, Represents the rotation matrix between the hand coordinate system and the end-effector coordinate system; This represents the rotation matrix between the hand coordinate system and the base coordinate system. This represents the rotation matrix between the end coordinate system and the base coordinate system.

[0109] In some embodiments, S15 specifically includes the following steps:

[0110] S151, with point W O base Establish the end effector coordinate system with the origin as the origin; the end effector coordinate system is the coordinate system of the robotic arm itself, and the directions of the three coordinate axes of the end effector coordinate system are obtained from the teach pendant;

[0111] S152. Based on the rotation matrix between the tool (hammer) coordinate system and the base coordinate system. Rotation matrix between end coordinate system and hand coordinate system Solve for the rotation matrix between the end coordinate system and the base coordinate system. Specifically as follows:

[0112]

[0113] S153, the rotation matrix between the end coordinate system and the base coordinate system. Convert to rotational vector, wrist key point WO base As a position parameter, the end effector posture of the robotic arm when gripping the tool (hammer) is obtained by using the rotation vector and the position parameter.

[0114] In some embodiments, S3 specifically includes the following steps:

[0115] S31. Use a dexterous hand to move the tool (hammer) to the first set position;

[0116] S32. Establish a tool head (hammer) coordinate system based on the actual scenario and calculate the directions of the three coordinate axes of the tool head coordinate system. Represent them using coordinates in the base coordinate system, as follows:

[0117]

[0118] y chuitou =z chuitou ×x chuitou

[0119] Where, x chuitou z chuitou and y chuitou These are the representations of the three coordinate axes of the tool head (hammer head) coordinate system in the base coordinate system;

[0120] Then based on x chuitou Z chuitou and y chuitou Calculate the rotation matrix between the tool head (hammer) coordinate system and the base coordinate system. The specific calculation formula is as follows:

[0121]

[0122] S33. Use the dexterous hand again to move the tool to the second set position, and then use the key point recognition algorithm to solve the new coordinates of the tool head (hammer head) coordinate system in the base coordinate system. Then based on the coordinates Find the transformation matrix between the tool head (hammer) coordinate system and the base coordinate system. The calculation formula is:

[0123]

[0124] S34, Based on point W O base Solve for the transformation matrix between the object coordinate system and the base coordinate system. The specific calculation formula is as follows:

[0125]

[0126] Then, based on the transformation matrix between the object coordinate system and the base coordinate system... Transformation matrix between the tool head (hammer) coordinate system and the base coordinate system Calculate the transformation matrix between the object coordinate system and the tool head (hammer) coordinate system. The calculation formula is:

[0127]

[0128] S35. Based on the actual scenario, establish a coordinate system for the target object (nail), and then define the directions of the three coordinate axes of the target object (nail) coordinate system, representing them using coordinates in the base coordinate system, as follows:

[0129]

[0130] Where, x dingzi y dingzi and z dingzi These represent the three coordinate axes of the target object (nail) coordinate system in the base coordinate system.

[0131] S36, According to x dingzi y dingzi and z dingzi Find the rotation matrix between the target object (nail) coordinate system and the base coordinate system. The specific calculation formula is as follows:

[0132]

[0133] S37. Using a key point recognition algorithm and based on the initial key points N on the target object (nail) in the base coordinate system. base Solve for the transformation matrix between the target object (nail) coordinate system and the base coordinate system. Specifically as follows:

[0134]

[0135] In some embodiments, S4 specifically includes the following steps:

[0136] S41. Based on the transformation matrix between the object coordinate system and the tool head (hammer) coordinate system. Transformation matrix between the target object (nail) coordinate system and the base coordinate system Solve for the transformation matrix between the hand coordinate system and the base coordinate system. The calculation formula is:

[0137]

[0138] This includes two invariants: the invariance of the relative position between the hammer head coordinate system and the hand coordinate system after the dexterous hand picks up the hammer, and the invariance between the hand action coordinate system and the end effector coordinate system; and the coincidence of two pairs of coordinate systems, namely, the coincidence of the hammer head coordinate system and the nail head coordinate system when the hammer head correctly contacts the nail head, and the coincidence of the hand coordinate system and the object coordinate system (hammer coordinate system) when the dexterous hand correctly grasps the hammer.

[0139] S42. Then, based on the transformation matrix between the hand coordinate system and the base coordinate system... Solve for the rotation matrix between the hand coordinate system and the base coordinate system. and the wrist key points in the hand coordinate system and the wrist key points in the base coordinate system.

[0140] S43. Utilize the rotation matrix between the hand coordinate system and the base coordinate system. Solve for the rotation matrix between the current position end coordinate system and the base coordinate system. The calculation formula is as follows:

[0141]

[0142] S44. The rotation matrix between the current end coordinate system and the base coordinate system... Converted to rotation vectors, used as attitude parameters, using wrist keypoints. As a position parameter;

[0143] S45. Control the robotic arm according to the attitude parameters and position parameters in S44, so that the dexterous hand on the robotic arm moves to the specified position;

[0144] S46. Detect the position of the tool head (hammer) again, and return to S3. Calculate the transformation matrix between the current object coordinate system and the tool head (hammer) coordinate system. and the rotation matrix between the current position end coordinate system and the base coordinate system.

[0145] S47, repeat S46 until the tool head (hammer head) is used to complete the operation on the target object (nail).

[0146] Reference Figure 3 In another aspect, the present invention provides a robot dexterous operating system based on visual key point guidance, including a robotic arm, a dexterous hand mounted on the robotic arm, a camera, a control host, a tool rack, tools (hammers) placed on the tool (hammer) rack, and a target object (nail). The control host, camera, tool rack, and target object (nail) are all placed around the robotic arm, and the dexterous hand, robotic arm, and camera are all electrically connected to the control host.

[0147] Among them, the preferred robotic arm is the UR5 robotic arm; the preferred dexterous hand is the time-dependent dexterous hand; the preferred camera is the Intel D435iRGB-D camera; the robot's dexterous operating system also includes Aruco codes, which are used for position correction.

[0148] With the assistance of a robotic arm and a camera, the dexterous hand uses a tool (hammer) to manipulate the target object (nail) according to the above-mentioned robot dexterity operation method.

[0149] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for dexterous robot manipulation based on visual keypoint guidance, characterized in that, Includes the following steps: S1. Establish a base coordinate system, then solve the initial key points based on the base coordinate system, and locate the dexterous hand based on the initial key points to solve the end-effector posture when grasping the tool. S2. The dexterous hand grasps the tool using a pre-established coarse gesture library and based on the end-effector posture when gripping the tool. S3. Establish the tool head coordinate system and the target object coordinate system. Use the dexterous hand to move the tool, identify the new key points after the shift, and calculate the transformation matrix between the tool head coordinate system and the base coordinate system and the transformation matrix between the target object coordinate system and the base coordinate system based on the new key points. S4. Based on the transformation matrix between the tool head coordinate system and the base coordinate system and the transformation matrix between the target object coordinate system and the base coordinate system, continuously calculate the end position of the tool head in operation on the target object until the dexterous hand holds the tool to complete the operation on the target object. S1 specifically includes the following steps: S11. Establish a base coordinate system on the base, and use the key point recognition method CMAKE to solve for four initial key points on the tool and one initial key point on the target object; The four initial key points on the tool are the four initial key points when gripping the tool, namely the index finger key point, the little finger key point, the wrist key point, and the tool head key point; among them, the initial wrist key point is located on the tool. S12. Establish a hand coordinate system on the dexterous hand; then adjust the initial wrist key points so that the adjusted wrist key points are not located on the tool; and calculate the coordinates of the adjusted wrist key points. S13. Construct a tool coordinate system using the key points of the index finger, the key point of the little finger, and the adjusted wrist key points, and calculate the rotation matrix between the tool coordinate system and the base coordinate system. S14. Calculate the rotation matrix between the end coordinate system and the hand coordinate system; S15. Solve the end-effector posture of the robotic arm when gripping the tool using the rotation matrix between the tool coordinate system and the base coordinate system, and the rotation matrix between the end-effector coordinate system and the hand coordinate system.

2. The method for dexterous robot manipulation based on visual key point guidance according to claim 1, characterized in that, S11 specifically includes the following steps: S111. Establish a base coordinate system on the base, and determine the transformation matrix between the camera coordinate system and the base coordinate system using the hand-eye calibration method. ; S112. Using the CMAKE key point recognition method, four initial key points on the tool and one initial key point on the target object are obtained in the camera coordinate system. ; The four initial key points on the tool are the four initial key points for gripping the tool, namely the index finger key point. Key points of the little finger Key points of the wrist and key points of the tool head ; S113. Then, transform the five initial key points to the base coordinate system to obtain the initial key points on the tool and the target object in the base coordinate system. The calculation formulas are as follows: D0 = D1 = D2 = in, This represents the initial key points on the target object in the base coordinate system; , , , These represent the key points of the index finger, little finger, wrist, and tool head when gripping the tool in the base coordinate system, respectively; D0, D1, and D2 are respectively... , , Abbreviated symbols.

3. The robot dexterity manipulation method based on visual key point guidance according to claim 2, characterized in that, S12 specifically includes the following steps: S121. Establish a hand coordinate system on a dexterous hand; S122. Measure the size of the hand triangle in the palm of the dexterous hand on the hand coordinate system. On the base coordinate system, the three vertices of the triangle are F, L, and W, which correspond to the key points of the index finger, the key point of the little finger, and the adjusted wrist key point, respectively. Then, the vector is obtained based on the dimensions of the triangle in the palm of the dexterous hand. ; S123. Calculate the vector between points D0 and D1. Then, point D1 is adjusted to obtain point D1', so that the adjusted point D1 is in the correct position; then, based on the vector... Find the vector between point D1' and point D0. ,vector The formula for calculation is: Where a, b, and c represent vectors. The three coordinate components in the base coordinate system; Represents the magnitude of the vector; Based on vector Find the position of point D1', that is, the position of the adjusted little finger key point in the base coordinate system; S124. Based on the property that points D0, D1', D2, and W all lie in the same plane M1, and using points D0, D1', and D2, solve for the normal vector of plane M1. The calculation formula is: in, , , They represent the normal vectors respectively. Components in the three directions in the base coordinate system; This represents the vector between D0 and D2; Then, using points D0, D1', and W, we can solve for normal vectors of different lengths in the same direction. The specific calculation formula is as follows: Where, vector In the formula, , Let D0 and D2 represent the coordinates of points D0 and D2 in the base coordinate system, respectively. This represents the coordinates of the point W to be solved in the base coordinate system; S125. Solve for the dimensions of the triangle obtained from S122. The cosine and sine values ​​of D0D1' are calculated using the following formulas: in, This represents the vector between point D0 and point W; This represents the vector between point D1' and point W; S126, Solving for the normal vector The modulus, calculated using the following formula: Where m represents the normal vector The model; S127. Based on the coordinates of point D0 in the base coordinate system, establish the following three equations, as follows: Using the lengths of edge D0W and edge D1'W, we can establish the following two equations, respectively: S128. Subtracting equation (4) from equation (5), we obtain the relationship between the coordinates x, y, and z of point W: Where A represents a user-defined parameter and satisfies the following equation: Then, the relationship between z and y is solved using equation (1): Substituting equation (7) into equation (6), we obtain the relationship between x and y: in, , and These are custom parameter two, custom parameter three, and custom parameter four, and they each satisfy the following relationship: S129. Transform equation (3) to obtain: Where B is a user-defined parameter five, and satisfies the following relationship: Then, combining equation (8) with equation (9), we obtain the analytical expression for y: Substituting equation (10) into equation (9), we obtain the analytical expression for x: Substituting equation (10) into equation (7), we obtain the analytical expression for z: The three coordinate values ​​of point W are obtained through equations (10), (11), and (12). Point W is the adjusted wrist key point, which is the point. .

4. The robot dexterity manipulation method based on visual key point guidance according to claim 3, characterized in that, S13 specifically includes the following steps: S131. Construct a tool coordinate system using the index finger keypoint, little finger keypoint, and adjusted wrist keypoint. The three axes of the tool coordinate system are as follows: , ; The axis is obtained from the unit vector pointing from the adjusted wrist key point to the index finger key point. Shaft The result is obtained by multiplying the adjusted unit vector from the wrist keypoint to the little finger keypoint by the axis cross product. Depend on Axis cross product The axis is obtained; the origin of the tool coordinate system is... ; S132. Representation of the x-axis of the calculation tool coordinate system in the base coordinate system The calculation formula is: S133. Representation of the z-axis of the calculation tool coordinate system in the base coordinate system. The calculation formula is: S134. Representation of the y-axis of the calculation tool coordinate system in the base coordinate system. The calculation formula is: S135, Utilization , as well as Solve for the rotation matrix between the tool coordinate system and the base coordinate system. The specific calculation formula is as follows: 。 5. The method for dexterous robot manipulation based on visual key point guidance according to claim 4, characterized in that, The formula for calculating the rotation matrix between the hand coordinate system and the end-effector coordinate system in S14 is: in, Represents the rotation matrix between the hand coordinate system and the end-effector coordinate system; This represents the rotation matrix between the hand coordinate system and the base coordinate system. This represents the rotation matrix between the end coordinate system and the base coordinate system.

6. The method for dexterous robot manipulation based on visual key point guidance according to claim 5, characterized in that, S15 specifically includes the following steps: S151, with points Establish an end-effector coordinate system with the origin as the origin. The end-effector coordinate system is the coordinate system of the robotic arm itself. The directions of the three coordinate axes of the end-effector coordinate system are obtained from the teach pendant. S152, Based on the rotation matrix between the tool coordinate system and the base coordinate system Rotation matrix between the end coordinate system and the hand coordinate system Solve for the rotation matrix between the end coordinate system and the base coordinate system. The details are as follows: ； S153, the rotation matrix between the end coordinate system and the base coordinate system. Convert to rotational vector, wrist key points As a position parameter, the end effector posture of the robotic arm when gripping the tool is obtained by using the rotation vector and the position parameter.

7. The method for dexterous robot manipulation based on visual key point guidance according to claim 6, characterized in that, S3 specifically includes the following steps: S31. Use a dexterous hand to move the tool to the first set position; S32. Establish a tool head coordinate system based on the actual scenario, and calculate the directions of the three coordinate axes of the tool head coordinate system. Represent them using coordinates in the base coordinate system, as follows: in, 、 and These are the representations of the three coordinate axes of the tool head coordinate system in the base coordinate system; Then based on 、 and Calculate the rotation matrix between the head coordinate system and the base coordinate system of the tool. The specific calculation formula is as follows: ； S33. Use the dexterous hand again to move the tool to the second set position, and then use the key point recognition algorithm to solve the coordinates of the new tool head coordinate system in the base coordinate system. Then based on coordinates Find the transformation matrix between the tool head coordinate system and the base coordinate system. The calculation formula is: ； S34, Basis Point Solve for the transformation matrix between the object coordinate system and the base coordinate system. The specific calculation formula is as follows: ； Then, based on the transformation matrix between the object coordinate system and the base coordinate system... Transformation matrix between tool head coordinate system and base coordinate system Calculate the transformation matrix between the object coordinate system and the tool head coordinate system. The calculation formula is: ； S35. Establish a target object coordinate system based on the actual scenario, and then define the directions of the three coordinate axes of the target object coordinate system, representing them using coordinates in the base coordinate system, as follows: in, 、 as well as These represent the three coordinate axes of the target object's coordinate system in the base coordinate system. S36, according to 、 as well as Solve for the rotation matrix between the target object coordinate system and the base coordinate system. The specific calculation formula is as follows: ； S37. Using a key point recognition algorithm and based on the initial key points on the target object in the base coordinate system. Solve for the transformation matrix between the target object coordinate system and the base coordinate system. The details are as follows: 。 8. The method for dexterous robot manipulation based on visual key point guidance according to claim 7, characterized in that, S4 specifically includes the following steps: S41. Based on the transformation matrix between the object coordinate system and the tool head coordinate system Transformation matrix between the target object coordinate system and the base coordinate system Solve for the transformation matrix between the hand coordinate system and the base coordinate system. The calculation formula is: ； S42. Then, based on the transformation matrix between the hand coordinate system and the base coordinate system... Solve for the rotation matrix between the hand coordinate system and the base coordinate system. and the wrist key points in the hand coordinate system and the wrist key points in the base coordinate system. ; S43. Utilize the rotation matrix between the hand coordinate system and the base coordinate system. Solve for the rotation matrix between the current position end coordinate system and the base coordinate system. The calculation formula is as follows: S44. The rotation matrix between the current end coordinate system and the base coordinate system... Converted to rotation vectors, used as attitude parameters, using wrist keypoints. As a position parameter; S45. Control the robotic arm according to the attitude parameters and position parameters in S44, so that the dexterous hand on the robotic arm moves to the specified position; S46. Detect the tool head position again and return to S3. Calculate the transformation matrix between the current object coordinate system and the tool head coordinate system. and the rotation matrix between the current position end coordinate system and the base coordinate system. ; S47, repeat S46 until the tool head is used to complete the operation on the target object.

9. A robot dexterous operating system based on visual keypoint guidance, characterized in that, It includes a robotic arm, a dexterous hand mounted on the robotic arm, a camera, a control unit, a tool rack, tools placed on the tool rack, and a target object. The control unit, camera, tool rack, and target object are all placed around the robotic arm, and the dexterous hand, robotic arm, and camera are all electrically connected to the control unit. With the assistance of a robotic arm and a camera, a dexterous hand manipulates a target object by holding a tool, in accordance with the robot dexterous manipulation method according to any one of claims 1 to 8.

Images