Robot smart operation method and system based on visual key point guidance

Abstract

The invention discloses a robot dexterous operation method and system based on visual key point guidance, and the method comprises the steps: S1, building a base coordinate system, solving an initial key point according to the base coordinate system, positioning a dexterous hand based on the initial key point, and solving the posture of the tail end of a mechanical arm when a tool is grabbed; s2, the dexterous hand grabs the tool through a pre-established coarse gesture library according to the posture of the tail end of the mechanical arm when the tool is grabbed; s3, using the dexterous hand to move the tool, identifying new key points after displacement again, and calculating a transformation matrix between the tool head coordinate system and the base coordinate system and a transformation matrix between the target object coordinate system and the base coordinate system according to the new key points; and S4, calculating the tail end position of the operation of the tool head on the target object until the dexterous hand holds the tool to complete the operation on the target object. According to the method, the interaction process of the dexterous hand and the target object is converted into alignment of the object and the key point of the hand, the rule when the dexterous hand grabs a tool is met, and extremely high generalization is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of robotic arms, and particularly to a method and system for dexterous operation of a robot guided by visual key points. Background Art

[0002] Robots are currently widely used in multiple fields such as home, industry, and rescue. Enabling robots to learn human-like dexterous grasping and perform corresponding tasks is a necessary condition for robots to learn dexterous operation for complex tasks.

[0003] Currently, the relatively mature grasping algorithms are mainly designed for two-finger grippers. However, two-finger grippers have low degrees of freedom, poor enveloping performance, and a small contact surface with objects, and thus cannot meet the requirements of complex dexterous operation. Five-finger dexterous hands are flexible, similar to human hands, and have more possibilities, and can complete the grasping requirements of complex tools. However, they have more degrees of freedom, involve fine-grained hand-object interaction, and require higher algorithms. The process of a dexterous hand grasping an object is not simply about grasping firmly. It more involves the realization of functional grasping. For example, when grasping a drill, it is necessary to consider the contact between the index finger and the button, and the contact between the other four fingers and the handle.

[0004] Currently, the method of using a dexterous hand to grasp a tool and then perform fine operation on an object is not mature, and has low robustness and low accuracy. Summary of the Invention

[0005] The present invention provides a method and system for dexterous operation of a robot guided by visual key points to solve the technical problems mentioned in the background art.

[0006] To achieve the above object, the technical solution of the present invention is realized as follows:

[0007] The present invention provides a method for dexterous operation of a robot guided by visual key points, including the following steps:

[0008] S1. Establish a base coordinate system, then solve the initial key points based on the base coordinate system, and position the dexterous hand based on the initial key points, and solve the pose of the end of the robotic arm when grasping a tool;

[0009] S2. The dexterous hand grasps the tool according to the pre-established coarse gesture library and based on the pose of the end of the robotic arm when grasping the tool;

[0010] S3. Establish a tool head coordinate system and a target object coordinate system, use the dexterous hand to move the tool, re-identify the new key points after displacement, and calculate the transformation matrix between the tool head coordinate system and the base coordinate system and the transformation matrix between the target object coordinate system and the base coordinate system based on the new key points;

[0011] S4. Continuously calculate the end position of the tool head operating on the target object based on the transformation matrix between the tool head coordinate system and the base coordinate system, and the transformation matrix between the target object coordinate system and the base coordinate system, until the dexterous hand holds the tool to complete the operation on the target object.

[0012] On the other hand, the present invention also provides a robot dexterous operation system based on visual key point guidance, including a robotic arm, a dexterous hand mounted on the robotic arm, a camera, a control host, a tool rack, tools placed on the tool rack, and a target object. The control host, the camera, the tool rack, and the target object are all placed on the periphery of the robotic arm. The dexterous hand, the robotic arm, and the camera are all electrically connected to the control host.

[0013] With the assistance of the robotic arm and the camera, the dexterous hand holds the tool to operate on the target object according to the above robot dexterous operation method.

[0014] Advantages of the present invention:

[0015] 1. The present invention uses the key point detection method and utilizes the key point recognition method CMAKE, which improves the robustness and generalization of the method. It can be applied to a variety of tools and use these tools to achieve different long-range tasks.

[0016] 2. The key points used in this method convert the interaction process between the dexterous hand and the object into the alignment of the object and the key points of the hand, which conforms to the law when the dexterous hand grasps the tool and has strong generalization.

[0017] 3. The present invention converts the interaction process between the tool and the target object into the alignment between coordinate systems, which is more direct, efficient, and accurate compared to other methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the robot dexterous operation method in the present invention;

[0019] Figure 2 is the operation flow chart in the embodiment of the present invention;

[0020] Figure 3 is the structure diagram of the robot dexterous operation system in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] To facilitate the understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. The preferred embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0022] Refer to Figure 1and Figure 2 , an embodiment of the present application provides a robot dexterous operation method guided by visual key points. Taking a dexterous hand picking up a hammer to hammer a nail as an example, the method includes the following steps:

[0023] S1. Establish a base coordinate system, then solve the initial key points according to the base coordinate system, and position the dexterous hand based on the initial key points, and solve the pose of the end of the robotic arm when grasping the tool (i.e., the hammer);

[0024] S2. The dexterous hand grasps the tool (hammer) through a pre-established coarse gesture library and according to the pose of the end of the robotic arm when grasping the tool (hammer);

[0025] Specifically, when the dexterous hand reaches the specified position, according to the coarse gesture library we established, corresponding to "hold_hammer", the dexterous hand will execute the coarse gesture required to grasp the hammer. After the dexterous hand executes the hold coarse gesture, the force received by the dexterous hand is read every 0.01 seconds. According to the designed determination condition for the differential magnitude, that is, when the absolute value of the force received by a certain finger is less than a certain value, or when the sum of the absolute values of the forces received by all fingers is less than a certain threshold, it is considered that the current grasp is stable and the next operation is entered.

[0026] S3. Establish a coordinate system for the tool head (i.e., the hammer head) and a coordinate system for the target object (i.e., the nail), use the dexterous hand to move the tool (hammer), and re-identify the new key points after displacement (due to the offset of the key points of the hammer head during the grasping process), and calculate the transformation matrix between the coordinate system of the tool head (hammer head) and the base coordinate system and the transformation matrix between the coordinate system of the target object (nail) and the base coordinate system according to the new key points;

[0027] S4. Continuously calculate the end position of the tool head (hammer head) operating on the target object (nail) according to the transformation matrix between the coordinate system of the tool head (hammer head) and the base coordinate system and the transformation matrix between the coordinate system of the target object (nail) and the base coordinate system until the dexterous hand holds the tool (hammer) to complete the operation on the target object (nail) (i.e., the operation of hammering the nail with the hammer head).

[0028] A robot dexterous operation method disclosed by the present invention mainly includes using key points to position the dexterous hand; using the established coarse gesture library for dexterous grasping; the dexterous hand picks up the tool (hammer) to interact with the target object (nail). Using the position of the key points on the target object (nail) to position the dexterous hand, so as to realize the specific interaction between the dexterous hand and the target object (nail); realizing different grasping methods for different tools (hammers) under specific tasks according to the established coarse gesture library; after grasping the tool (hammer) firmly, the dexterous hand can pick up the tool (hammer) to realize the operations we specified.

[0029] The robot dexterous operation method in the present invention can be transferred to various scenarios, and the key points therein can be further refined for operation on different tools (hammers). Therefore, the excellent dexterous operation method in the present invention can realize the use of some tools (hammers) based on the life scenario and has good application value.

[0030] In addition, the present invention uses the key point detection method and utilizes the key point recognition method CMAKE, which improves the robustness and generalization of the method, can be applied to various tools (hammers), and uses these tools (hammers) to achieve different long-range tasks.

[0031] In some embodiments, S1 specifically includes the following steps:

[0032] S11. Establish a base coordinate system on the base, and use the key point recognition method CMAKE to solve for four initial key points on the tool (hammer) and one initial key point on the target object (nail).

[0033] The four initial key points on the tool (hammer) are the four initial key points when grasping the tool (hammer), namely the index finger key point, the little finger key point, the wrist key point, and the tool head (hammer head) key point; among them, the initial wrist key point is located on the tool (hammer).

[0034] S12. Establish a hand coordinate system on the dexterous hand; since the key point recognition algorithm CMAKE is used to solve for the key points on the tool (hammer), but in fact, the position of the wrist is at a certain distance from the object, and the positioning of the dexterous hand is carried out through the wrist key point. Therefore, the initial wrist key point needs to be adjusted so that the adjusted wrist key point is not located on the tool (hammer); and calculate the coordinates of the adjusted wrist key point.

[0035] S13. Use the index finger key point, the little finger key point, and the adjusted wrist key point to construct the tool (hammer) coordinate system, and calculate the rotation matrix between the tool (hammer) coordinate system and the base coordinate system.

[0036] S14. Calculate the rotation matrix between the end coordinate system and the hand coordinate system, and the rotation matrix between the end coordinate system and the hand coordinate system is the H-E prior.

[0037] Specifically, before the first grasping, use the teach pendant to obtain the coordinates of the index finger key point, the little finger key point, and the wrist key point in the base coordinate system at this time, which are respectively denoted as F H base , L H base , W H base, the method for establishing the hand coordinate system is the same as that for establishing the object coordinate system, and the origin W used in the hand coordinate system H base coincides with the origin of the end coordinate system, that is, repeat the steps in S13 to obtain the rotation matrix between the hand coordinate system and the base coordinate system

[0038] At the same time, record the position of the end in the base coordinate system in the teach pendant at this time, so as to calculate the rotation matrix between the end coordinate system and the base coordinate system Finally, the rotation matrix between the hand coordinate system and the end coordinate system can be calculated:

[0039]

[0040] Among them, represents the rotation matrix between the end coordinate system and the hand coordinate system.

[0041] S15. Use the rotation matrix between the tool (hammer) coordinate system and the base coordinate system, and the rotation matrix between the end coordinate system and the hand coordinate system to solve the posture of the end of the robotic arm when grasping the tool (hammer).

[0042] In some embodiments, S11 specifically includes the following steps:

[0043] S111. Establish a base coordinate system on the base, and use the hand-eye calibration method to determine the transformation matrix between the camera coordinate system and the base coordinate system

[0044] S112. Use the key point recognition method CMAKE to solve and obtain four initial key points on the tool (hammer) and one initial key point N on the target object (nail) in the camera coordinate system camera ;

[0045] The four initial key points on the tool (hammer) are the four initial key points when grasping the tool (hammer), which are the index finger key point F O camera , the little finger key point L O camera , the wrist key point W O ′ camera and the tool head (hammer head) key point chuitou camera ;

[0046] S113. Then transform the five initial key points to the base coordinate system respectively to obtain the initial key points on the tool (hammer) and the target object (nail) in the base coordinate system. The calculation formulas are as follows:

[0047]

[0048] Among them, N base represents the initial key point on the target object (nail) in the base coordinate system; F O base 、L O base 、W O ′ base 、chuitou base respectively represent the index finger key point, little finger key point, wrist key point, and tool head (hammer head) key point when grasping the tool (hammer) in the base coordinate system; D0, D1, and D2 are the abbreviated symbols of F O base 、L O base 、W O ′ base respectively.

[0049] In some embodiments, S12 specifically includes the following steps:

[0050] S121. Establish a hand coordinate system on the dexterous hand;

[0051] S122. Measure the size of the hand triangle in the palm of the dexterous hand in the hand coordinate system. The three vertices of this triangle in the hand coordinate system are F, L, and W, corresponding to the index finger key point, little finger key point, and the adjusted wrist key point respectively; then obtain the vector according to the size of the hand triangle in the palm of the dexterous hand

[0052] S123. Since the key points obtained by the key point recognition algorithm CMAKE are all on the tool (hammer), and there are discrepancies between the little finger key point and the wrist key point and the actual positions when the hand grasps the tool (hammer), it is necessary to adjust the little finger key point and the wrist key point. Calculate the vector between point D0 and point D1 Then correct point D1 to obtain point D1', so that the adjusted point D1 is in the correct position; point D1' is still on the tool (hammer); then, according to the vector Solve for the vector between point D1' and point D0 The vector The calculation formula is:

[0053]

[0054] Among them, a, b, c represent the three coordinate components of the vector in the base coordinate system; |.| represents the modulus of the vector.

[0055] According to the vector Solve for the position of point D1', that is, the position of the adjusted little finger key point in the base coordinate system;

[0056] S124. Based on the property that points D0, D1', D2, and W are coplanar in the same plane M1, and using points D0, D1', and D2, solve for the normal vector of plane M1. The calculation formula is:

[0057]

[0058] Among them, n1, n2, and n3 respectively represent the components of the normal vector in three directions in the base coordinate system. represents the vector between D0 and D2;

[0059] Then, use points D0, D1', and W to solve for the normal vectors with the same direction but different lengths. The specific calculation formula is:

[0060]

[0061] Among them, the vector In the formula, (x F , y F , z F ), (x W , y W , z W ) respectively represent the coordinates of points D0 and D2 in the base coordinate system, and (x, y, z) represents the coordinates of the point W to be solved in the base coordinate system;

[0062] S125. Solve for the cosine value and sine value of ∠WD0D1' through the dimensions of the triangle measured in S122. The calculation formulas are as follows:

[0063]

[0064] Among them, represents the vector between point D0 and point W; represents the vector between point D1' and point W; the angle of ∠WD0D1' is acute;

[0065] S126. Solve for the modulus of the normal vector The specific calculation formula is as follows:

[0066]

[0067] Among them, m represents the modulus of the normal vector .

[0068] S127. Based on the coordinates of point D0 in the base coordinate system, establish the following three equalities, specifically as follows:

[0069] b*(z--z F ​)--c*(y - y F ) = m*n1 (1)

[0070] c*(x--x F ) - a*(z - z F ) = m*n2 (2)

[0071] a*(y - y F ) - b*(x--x F ) = m*n3 (3)

[0072] Two equations are established using the lengths of side D0W and side D1'W as follows:

[0073]

[0074] S128. Subtract Equation (4) from Equation (5) to obtain the relational expression among the coordinates x, y, and z of point W:

[0075] 2*(x F - x L )*x + 2*(y F - y L )*y + 2*(z F - z L )*z = A (6)

[0076] Where A represents the first custom parameter and satisfies the following equation:

[0077]

[0078] Then, use Equation (1) to solve for the relationship between z and y:

[0079]

[0080] And substitute Equation (7) into Equation (6) to obtain the relationship between x and y:

[0081] 2*x*E + F*y = A - G (8)

[0082] Where E, F, and G are the second, third, and fourth custom parameters respectively, and satisfy the following relational expressions:

[0083] E = x F - x L

[0084]

[0085] S129. Transform Equation (3) to obtain:

[0086] - b*x + a*y = B (9)

[0087] Among them, B is the fifth custom parameter and satisfies the following relational expression:

[0088] B = m * n3 + a * y F - b * x F

[0089] Then, combining Equation (8) with Equation (9), the analytical formula of y is obtained:

[0090]

[0091] Substituting Equation (10) into Equation (9), the analytical formula of x is obtained:

[0092]

[0093] Substituting Equation (10) into Equation (7), the analytical formula of z is obtained:

[0094]

[0095] The three coordinate values of point W are obtained through Equations (10), (11), and (12). Point W is the adjusted wrist key point, and the adjusted wrist key point is point

[0096] In some embodiments, S13 specifically includes the following steps:

[0097] S131. Construct a tool (hammer) coordinate system using the index finger key point, the little finger key point, and the adjusted wrist key point. Among them, the three axes of the tool coordinate system are x o , y o , z o ; The x o axis is obtained from the unit vector pointing from the adjusted wrist key point to the index finger key point, and the z o axis is obtained from the result of the cross product of the x o axis and the unit vector pointing from the adjusted wrist key point to the little finger key point; The y o is obtained from the cross product of the z o axis and the x o axis; The origin of the tool (hammer) coordinate system is W O base ;

[0098] S132. Calculate the representation x o of the x-axis of the tool (hammer) coordinate system in the base coordinate system. The calculation formula is:

[0099]

[0100] S133. Calculate the representation z of the z-axis of the tool (hammer) coordinate system in the base coordinate systemo , the calculation formula is:

[0101]

[0102] S134. The representation of the y-axis of the calculation tool (hammer) coordinate system in the base coordinate system is y o , the calculation formula is:

[0103] y o = z o × x o

[0104] S135. Using x o , y o and z o to solve the rotation matrix between the tool (hammer) coordinate system and the base coordinate system The specific calculation formula is as follows:

[0105]

[0106] In some embodiments, the calculation formula of the rotation matrix between the hand coordinate system and the end coordinate system in S14 is:

[0107]

[0108] Wherein, represents the rotation matrix between the hand coordinate system and the end coordinate system; represents the rotation matrix between the hand coordinate system and the base coordinate system; represents the rotation matrix between the end coordinate system and the base coordinate system.

[0109] In some embodiments, S15 specifically includes the following steps:

[0110] S151. Taking the point W O base as the origin, establish the end coordinate system; the end coordinate system is the coordinate system of the robotic arm itself, and the directions of the three coordinate axes of the end coordinate system are obtained from the teach pendant;

[0111] S152. According to the rotation matrix between the tool (hammer) coordinate system and the base coordinate system and the rotation matrix between the end coordinate system and the hand coordinate system to solve the rotation matrix between the end coordinate system and the base coordinate system Specifically as follows:

[0112]

[0113] S153. Convert the rotation matrix between the end coordinate system and the base coordinate system into a rotation vector, the wrist key point WO base As a position parameter, the attitude of the end of the robotic arm when obtaining the grasping tool (hammer) through the rotation vector and the position parameter.

[0114] In some embodiments, step S3 specifically includes the following steps:

[0115] S31. Use the dexterous hand to move the tool (hammer) to the first set position;

[0116] S32. Set up the coordinate system of the tool head (hammer head) according to the actual scenario and calculate the directions of the three coordinate axes of the tool head coordinate system, and represent them with the coordinates in the base coordinate system, specifically as follows:

[0117]

[0118] y chuitou = z chuitou × x chuitou

[0119] Where x chuitou 、z chuitou and y chuitou are respectively the representations of the three coordinate axes of the tool head (hammer head) coordinate system in the base coordinate system;

[0120] Then, based on x chuitou 、Z chuitou and y chuitou calculate the rotation matrix between the tool head (hammer head) coordinate system and the base coordinate system The calculation formula is specifically as follows:

[0121]

[0122] S33. Use the dexterous hand to move the tool to the second set position again, and then solve the coordinates of the new tool head (hammer head) coordinate system in the base coordinate system through the key point recognition algorithm Then, based on the coordinates find the transformation matrix between the tool head (hammer head) coordinate system and the base coordinate system The calculation formula is:

[0123]

[0124] S34. Based on point W O base Solve the transformation matrix between the object coordinate system and the base coordinate system The calculation formula is specifically as follows:

[0125]

[0126] Then, based on the transformation matrix between the object coordinate system and the base coordinate system The transformation matrix between the coordinate system of the tool head (hammer head) and the base coordinate system Calculate the transformation matrix between the object coordinate system and the coordinate system of the tool head (hammer head) The calculation formula is as follows:

[0127]

[0128] S35. Establish the coordinate system of the target object (nail) according to the actual scenario, and then set the directions of the three coordinate axes of the coordinate system of the target object (nail), and represent them with the coordinates in the base coordinate system, as follows:

[0129]

[0130] Among them, x dingzi 、y dingzi and z dingzi respectively represent the representations of the three coordinate axes of the coordinate system of the target object (nail) in the base coordinate system;

[0131] S36. Solve the rotation matrix between the coordinate system of the target object (nail) and the base coordinate system according to x dingzi 、y dingzi and z dingzi The calculation formula is as follows: The calculation formula is specifically as follows:

[0132]

[0133] S37. Through the key point recognition algorithm and based on the initial key point N on the target object (nail) in the base coordinate system base Solve the transformation matrix between the coordinate system of the target object (nail) and the base coordinate system Specifically as follows:

[0134]

[0135] In some embodiments, the specific steps of S4 are as follows:

[0136] S41. Based on the transformation matrix between the object coordinate system and the coordinate system of the tool head (hammer head) The transformation matrix between the coordinate system of the target object (nail) and the base coordinate system Solve the transformation matrix between the hand coordinate system and the base coordinate system The calculation formula is as follows:

[0137]

[0138] This includes these two invariances, namely the invariance of the relative position between the hammer head coordinate system and the hand coordinate system after the dexterous hand picks up the hammer, and the invariance between the hand-made coordinate system and the end coordinate system, as well as the coincidence of two pairs of coordinate systems, that is, the coincidence of the hammer head coordinate system and the nail head coordinate system when the hammer head correctly contacts the nail head, and the coincidence between the hand coordinate system and the object coordinate system (hammer coordinate system) when the dexterous hand correctly grasps the hammer:

[0139] S42. Then, according to the transformation matrix between the hand coordinate system and the base coordinate system Solve the rotation matrix between the hand coordinate system and the base coordinate system And the wrist key points in the hand coordinate system in the base coordinate system

[0140] S43. Use the rotation matrix between the hand coordinate system and the base coordinate system Solve the rotation matrix between the end coordinate system at the current position and the base coordinate system The calculation formula is as follows:

[0141]

[0142] S44. Convert the rotation matrix between the end coordinate system and the base coordinate system at the current moment Into a rotation vector as the attitude parameter, and use the wrist key points As the position parameter;

[0143] S45. Control the manipulator according to the attitude parameter and position parameter in S44, so that the dexterous hand on the manipulator moves to the specified position;

[0144] S46. Detect the position of the tool head (hammer head) again, and return to S3 to calculate the transformation matrix between the object coordinate system and the tool head (hammer head) coordinate system at the current position And the rotation matrix between the end coordinate system and the base coordinate system at the current position

[0145] S47. Loop S46 until the operation on the target object (nail) is completed using the tool head (hammer head).

[0146] Referring to Figure 3 , on the other hand, the present invention also provides a robot dexterous operation system based on visual key point guidance, including a manipulator, a dexterous hand installed on the manipulator, a camera, a control host, a tool rack, a tool (hammer) placed on the tool (hammer) rack, and a target object (nail). The control host, the camera, the tool rack, and the target object (nail) are all placed on the periphery of the manipulator. The dexterous hand, the manipulator, and the camera are all electrically connected to the control host;

[0147] Among them, the robotic arm is preferably a UR5 robotic arm; the dexterous hand is preferably a YINSHI dexterous hand; the camera is preferably an intel D435i RGB-D camera; the robotic dexterous operating system also includes Aruco codes, which are used for position correction.

[0148] With the assistance of the robotic arm and the camera, and according to the above robotic dexterous operation method, the dexterous hand holds a tool (hammer) to operate on a target object (nail).

[0149] As mentioned above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should be covered within the protection scope of the present invention. Moreover, the technical solutions between various embodiments of the present invention can be combined with each other, but it must be based on the fact that those skilled in the art can implement it. When the combination of technical solutions conflicts with each other or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A robot dexterous operation method guided by visual key points, characterized in that, It includes the following steps: S1. Establish a base coordinate system, then solve for the initial key points based on the base coordinate system, and position the dexterous hand based on the initial key points, and solve for the pose of the end of the robotic arm when grasping the tool; S2. The dexterous hand grasps the tool through a pre-established coarse gesture library and based on the pose of the end of the robotic arm when grasping the tool; S3. Establish a tool head coordinate system and a target object coordinate system, use the dexterous hand to move the tool, re-identify the new key points after displacement, and calculate the transformation matrix between the tool head coordinate system and the base coordinate system and the transformation matrix between the target object coordinate system and the base coordinate system based on the new key points; S4. Continuously calculate the end position of the tool head operating on the target object based on the transformation matrix between the tool head coordinate system and the base coordinate system and the transformation matrix between the target object coordinate system and the base coordinate system until the dexterous hand holds the tool to complete the operation on the target object.

2. The robot dexterous operation method based on visual key point guidance according to claim 1, characterized in that The specific steps of S1 are as follows: S11. Establish a base coordinate system on the base, and use the key point recognition method CMAKE to solve for four initial key points on the tool and one initial key point on the target object; The four initial key points on the tool are the four initial key points when grasping the tool, namely the index finger key point, the little finger key point, the wrist key point, and the tool head key point; among them, the initial wrist key point is located on the tool; S12. Establish a hand coordinate system on the dexterous hand; then adjust the initial wrist key point so that the adjusted wrist key point is not located on the tool; and calculate the coordinates of the adjusted wrist key point; S13. Use the index finger key point, the little finger key point, and the adjusted wrist key point to construct a tool coordinate system, and calculate the rotation matrix between the tool coordinate system and the base coordinate system; S14. Calculate the rotation matrix between the end coordinate system and the hand coordinate system; S15. Use the rotation matrix between the tool coordinate system and the base coordinate system and the rotation matrix between the end coordinate system and the hand coordinate system to solve for the pose of the end of the robotic arm when grasping the tool.

3. The robot dexterous operation method based on visual key point guidance according to claim 1, characterized in that, The specific steps of S11 are as follows: S111. Establish a base coordinate system on the base, and use the hand-eye calibration method to determine the transformation matrix between the camera coordinate system and the base coordinate system S112. Use the key-point recognition method CMAKE to solve and obtain four initial key points on the tool in the camera coordinate system and an initial key point N on the target object camera ; The four initial key points on the tool are the four initial key points when grasping the tool, namely the index finger key point F O camera , the little finger key point L O camera , the wrist key point W O ′ camera and the tool head key point chuitou camera ; S113. Then convert the five initial key points to the base coordinate system respectively to obtain the initial key points on the tool and the target object in the base coordinate system, and the calculation formulas are as follows: Among them, N base represents the initial key point on the target object in the base coordinate system; F O base , L O base , W O ′ base , and chuitou base respectively represent the index finger key point, little finger key point, wrist key point, and tool head key point when the tool is grasped in the base coordinate system; D0, D1, and D2 are the abbreviated symbols for F O base , L O base , W0' base .

4. The method for robot dexterous operation guided by visual key points according to claim 3, wherein The specific steps of S12 are as follows: S121. Establish a hand coordinate system on the dexterous hand; S122. Measure the size of the hand triangle in the palm of the dexterous hand in the hand coordinate system. The three vertices of this triangle in the base coordinate system are F, L, and W, corresponding to the index finger key point, the little finger key point, and the adjusted wrist key point respectively; Then, a vector is obtained based on the size of the hand triangle in the dexterous hand palm S123. Calculate the vector between point D0 and point D1 Then adjust point D1 to obtain point D1' so that the adjusted point D1 is in the correct position; then based on the vector Solve for the vector between point D1' and point D0 Vector The calculation formula of is: where a, b, and c represent vectors and the three coordinate components in the base coordinate system; |.| represents the magnitude of the vector; According to the vector Solve for the position of point D1', that is, the position of the adjusted little finger key point in the base coordinate system; S124. According to the property that point D0, point D1', point D2 and point W are coplanar in the same plane M1, and use point D0, point D1', point D2 to solve the normal vector of plane M1 The calculation formula is: where n1, n2, and n3 respectively represent the components of the normal vector in three directions in the base coordinate system; in the base coordinate system; represents the vector between D0 and D2; Then, the normal vectors with the same direction but different lengths are solved by using point D0, point D1', and point W The specific calculation formula is as follows: Among them, the vector In the formula, (x F , y F , z F ), (x W , y W , z W ) respectively represent the coordinates of point D0 and point D2 in the base coordinate system, and (x, y, z) represents the coordinates of the point W to be solved in the base coordinate system; S125. Solve for the cosine value and sine value of ∠WD0D1’ through the size of the triangle measured in S122, and the calculation formulas are as follows: Among them, represents the vector between point D0 and point W; represents the vector between point D1’ and point W; S126. Solve the normal vector Calculate its modulus. The specific calculation formula is as follows: where m represents the magnitude of the normal vector ; S127. Establish the following three equalities based on the coordinates of point D0 in the base coordinate system, specifically as follows: b*(z - z F ) - c*(y - y F ) = m*n1 (1) c*(x - x F ) - a*(z - z F ) = m*n2 (2) a*(y - y F ) - b*(x - x F ) = m*n3 (3) Establish the following two equalities using the lengths of side D0W and side D1’W, respectively as follows: S128. Subtract Equation (4) from Equation (5) to obtain the relational expression between the coordinates x, y, z of point W: 2*(x F -x L )*x+2*(y F -y L )*y+2*(z F -z L )*z = A (6) Among them, A represents a custom parameter one, and satisfies the following equality: Then, the relationship between z and y is solved using Equation (1): Substitute Equation (7) into Equation (6) to obtain the relationship between x and y: 2*x*E + F*y = A - G (8) where E, F, and G are respectively Custom Parameter Two, Custom Parameter Three, and Custom Parameter Four, and respectively satisfy the following relationships: E = x F -x L S129. Transform Equation (3) to obtain: -b*x + a*y = B (9) where B is Custom Parameter Five and satisfies the following relationship: B = m * n3 + a * y F -b * X F Then, combining Equation (8) with Equation (9), the analytical formula for y is obtained: Substitute Equation (10) into Equation (9) to obtain the analytical formula for x: Substitute Equation (10) into Equation (7) to obtain the analytical formula for z: The three coordinate values of point W are obtained through equations (10), (11), and (12). Point W is the adjusted wrist key point, and the adjusted wrist key point is point 5. The method for dexterous operation of a robot guided by visual key points according to claim 4, characterized in that The specific steps of S13 are as follows: S131. Construct a tool coordinate system using the key points of the index finger, the key points of the little finger, and the adjusted wrist key points. The three axes of the tool coordinate system are x o , y o , z o ; The x o axis is obtained from the unit vector pointing from the adjusted wrist key point to the key point of the index finger, and the z o axis is obtained from the result of the cross product of the x o axis and the unit vector pointing from the adjusted wrist key point to the key point of the little finger; The y o is obtained from the cross product of the z o axis and the x0 axis; The origin of the tool coordinate system is W O base ; S132. Calculate the representation x of the x-axis of the tool coordinate system in the base coordinate system. o , and the calculation formula is: S133. Calculate the representation z of the z-axis of the tool coordinate system in the base coordinate system. o , and the calculation formula is: S134. Calculate the representation y of the y-axis of the tool coordinate system in the base coordinate system o , and the calculation formula is: y o = z o × x o S135. Use x o , y o , and z o to solve for the rotation matrix between the tool coordinate system and the base coordinate system The calculation formula is as follows:

6. The method for dexterous robot operation guided by visual key points according to claim 5, wherein In S14, the calculation formula for the rotation matrix between the hand coordinate system and the end coordinate system is: Among them, represents the rotation matrix between the hand coordinate system and the end - effector coordinate system; represents the rotation matrix between the hand coordinate system and the base coordinate system; represents the rotation matrix between the end - effector coordinate system and the base coordinate system.

7. The method for robot dexterous operation guided by visual key points according to claim 6, characterized in that, The specific steps of S15 are as follows: S151. With point W O base as the origin, establish an end coordinate system. The end coordinate system is the coordinate system of the robotic arm itself, and the directions of the three coordinate axes of the end coordinate system are obtained from the teach pendant; S152. According to the rotation matrix between the tool coordinate system and the base coordinate system The rotation matrix between the end - effector coordinate system and the hand coordinate system Solve the rotation matrix between the end - effector coordinate system and the base coordinate system Specifically as follows: S153. Convert the rotation matrix between the end - effector coordinate system and the base coordinate system into a rotation vector, and use the wrist key point W O base as the position parameter. Obtain the attitude of the end - effector of the manipulator when grasping the tool through the rotation vector and the position parameter.

8. The method for robot dexterous operation guided by visual key points according to claim 7, wherein The specific steps of S3 are as follows: S31. Use the dexterous hand to move the tool to the first set position; S32. Set up the tool head coordinate system according to the actual scenario, and calculate the directions of the three coordinate axes of the tool head coordinate system, and represent them with the coordinates in the base coordinate system, specifically as follows: y chuitou = z chuitou × x chuitou where x chuitou , z chuitou and y chuitou are respectively the representations of the three coordinate axes of the tool head coordinate system in the base coordinate system; Then, based on x chuitou , z chuitou and y chuitou calculate the rotation matrix between the coordinate system of the tool head and the coordinate system of the base The specific calculation formula is as follows: S33. Use the dexterous hand again to move the tool to the second set position, and then solve the coordinates of the new tool head coordinate system in the base coordinate system through the key point recognition algorithm. Then, based on the coordinates calculate the transformation matrix between the tool head coordinate system and the base coordinate system. The calculation formula is: S34. Based on point W O base Solve the transformation matrix between the object coordinate system and the base coordinate system The specific calculation formula is as follows: Then, based on the transformation matrix between the object coordinate system and the base coordinate system the transformation matrix between the tool head coordinate system and the base coordinate system calculate the transformation matrix between the object coordinate system and the tool head coordinate system The calculation formula is as follows: S35. Establish the target object coordinate system according to the actual scenario, and then set up the directions of the three coordinate axes of the target object coordinate system, and represent them with the coordinates in the base coordinate system, specifically as follows: Among them, x dingzi , y dingzi and z dingzi respectively represent the representations of the three coordinate axes of the target object coordinate system in the base coordinate system; S36. According to x dingzi , y dingzi and z dingzi Solve the rotation matrix between the coordinate system of the target object and the base coordinate system The calculation formula is as follows: S37. Solve the transformation matrix between the target object coordinate system and the base coordinate system through the key point recognition algorithm and based on the initial key point N on the target object in the base coordinate system base Solve the transformation matrix between the target object coordinate system and the base coordinate system Specifically as follows:

9. The method for dexterous robot operation guided by visual key points according to claim 8, characterized in that The specific steps of S4 are as follows: S41. According to the transformation matrix between the object coordinate system and the tool head coordinate system The transformation matrix between the target object coordinate system and the base coordinate system Solve the transformation matrix between the hand coordinate system and the base coordinate system The calculation formula is as follows: S42. Then, based on the transformation matrix between the hand coordinate system and the base coordinate system Solve the rotation matrix between the hand coordinate system and the base coordinate system And the wrist key points in the hand coordinate system in the base coordinate system S43. Use the rotation matrix between the hand coordinate system and the base coordinate system Solve the rotation matrix between the end coordinate system and the base coordinate system at the current position The calculation formula is as follows: S44. Convert the rotation matrix between the end - effector coordinate system and the base coordinate system at the current moment into a rotation vector as the attitude parameter, and use the wrist key point as the position parameter; S45. Control the robotic arm according to the pose parameters and position parameters in S44, so that the dexterous hand on the robotic arm moves to the specified position; S46. Detect the position of the tool head again, return to S3, and calculate the transformation matrix between the object coordinate system at the current position and the tool head coordinate system and the rotation matrix between the end coordinate system and the base coordinate system at the current position S47. Loop S46 until the target object is operated using the tool head.

10. A robot dexterous operation system guided by visual key points, characterized in that, It includes a robotic arm, a dexterous hand installed on the robotic arm, a camera, a control host, a tool rack, a tool placed on the tool rack, and a target object. The control host, the camera, the tool rack, and the target object are all placed on the periphery of the robotic arm. The dexterous hand, the robotic arm, and the camera are all electrically connected to the control host; The dexterous hand, with the assistance of the robotic arm and the camera, holds the tool and operates the target object according to the robotic dexterous operation method described in any one of Claims 1 to 9.

Images