Mechanical arm and dexterous hand system tool-screw intelligent alignment method based on six-dimensional pose tracking and two-dimensional visual servo

Abstract

The invention relates to the technical field of intelligent robots, in particular to a mechanical arm and dexterous hand system tool-screw intelligent alignment method based on six-dimensional pose tracking and two-dimensional visual servo. Comprising the steps of generating a grasping configuration; an auxiliary mechanical arm is arranged on one side of the mechanical arm dexterous hand system; the auxiliary mechanical arm is moved to the position above the to-be-screwed screw, a camera is used for collecting three-dimensional coordinates of the screw, and a first observation pose and a second observation pose of the auxiliary mechanical arm are determined based on two-dimensional detection; a mechanical arm dexterous hand system is controlled to grab a tool, and the tool is moved to a pre-alignment state; servo tracking adjustment based on the six-dimensional attitude is carried out; and the auxiliary mechanical arm is moved to the first observation pose and the second observation pose respectively, and accurate adjustment is conducted according to the pixel deviation, obtained by the camera, of the center point pixel coordinates of the tool head detection frame and the center point pixel coordinates of the screw detection frame. According to the scheme, when the mechanical arm dexterous hand system carries the tool to align the screw, the tool can be rapidly and accurately moved to align the screw.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and in particular to a tool-screw intelligent alignment method of a robotic arm and a dexterous hand system based on six-dimensional posture tracking and two-dimensional visual servoing. Background Art

[0002] At present, with the development of robot and artificial intelligence technology, the manipulator and dexterous hand system have become the application carrier of intelligent control technology in various tasks due to its human-like configuration. It has the advantages of performing a variety of tasks, strong environmental adaptability, and high operation control accuracy. For the typical task of screwing with electric tools, the alignment process using the manipulator and dexterous hand system faces difficulties in measurement and planning. In terms of measurement, the commonly used fixed position observation faces the contradiction between observation range and observation accuracy. The wrist camera is blocked by the hand during the alignment process, and it is difficult to achieve large-scale observation; in terms of planning, the measurement information based on the six-dimensional posture can achieve a large range of adjustment, but the accuracy is usually difficult to meet the requirements, and the dexterous hand and the tool may move relative to each other during the movement. Adjustment based on two-dimensional vision can achieve fine alignment, but the measurement range is limited. Therefore, in order to solve the above problems, it is of great significance to study a tool-screw intelligent alignment method for a manipulator and dexterous hand system based on six-dimensional posture tracking and two-dimensional visual servoing. Summary of the invention

[0003] An embodiment of the present invention provides a tool-screw intelligent alignment method for a robotic arm and dexterous hand system based on six-dimensional posture tracking and two-dimensional visual servoing, which can quickly and accurately move the tool alignment screw when the robotic arm and dexterous hand system carries the tool alignment screw.

[0004] The embodiment of the present invention provides a tool-screw intelligent alignment method for a robotic arm and a dexterous hand system based on six-dimensional posture tracking and two-dimensional visual servoing, comprising:

[0005] Generate a grasping configuration based on a model of a robotic dexterous hand system and a tool for screwdriving;

[0006] An auxiliary robotic arm is configured on one side of the robotic arm dexterous hand system; wherein the auxiliary robotic arm is equipped with a camera;

[0007] Move the auxiliary robot arm to above the screw to be tightened, use the camera to collect the three-dimensional coordinates of the screw, and determine the first observation posture and the second observation posture of the auxiliary robot arm based on two-dimensional detection; wherein the first observation posture and the second observation posture are different single-degree-of-freedom observation dimensions;

[0008] Controlling the robot arm dexterous hand system to grasp the tool and move the tool to a pre-aligned state;

[0009] In the pre-alignment state, a servo tracking adjustment based on six-dimensional posture is performed to achieve rough alignment;

[0010] The auxiliary robot arm is moved to the first observation posture and the second observation posture respectively, and the pixel deviation of the pixel coordinates of the center point of the tool head detection frame and the pixel coordinates of the center point of the screw detection frame obtained by the camera is precisely adjusted to achieve alignment of the tool and the screw.

[0011] Optionally, the process of controlling the manipulator arm dexterous hand system to grasp a tool and move the tool to a pre-alignment state includes:

[0012] The camera captures an image of the tool and estimates the position and posture of the tool;

[0013] The auxiliary robot arm is moved to track the tool in real time so that the tool is located in the middle of the camera field of view.

[0014] Optionally, when the tool deviates from the middle of the camera field of view, the movement of the manipulator arm dexterous hand system is stopped, and the posture of the tool is recalculated and estimated.

[0015] Optionally, before controlling the manipulator arm dexterous hand system to grasp the tool and move the tool to the pre-alignment state, it also includes:

[0016] The position and posture of the tool relative to the manipulator coordinate system in the manipulator dexterous hand system is calculated based on the system calibration data.

[0017] Optionally, the z-axis of the screw's coordinate system is outward along the screw mounting surface, the x-axis is the projection of the x-axis of the robotic arm dexterous hand system coordinate system onto a plane perpendicular to the z-axis, the y-axis is obtained by cross product calculation using the right-hand rule, the observation point of the first observation posture is in the yz plane, and the observation point of the second observation posture is in the xz plane.

[0018] Optionally, the robotic arm dexterous hand system and the auxiliary robotic arm both have 6 active degrees of freedom, including 3 translations and 3 rotations. The dexterous hand includes five fingers, and the fingers have joints to enable grasping of electric tools. The tools include electric drills, and the camera is an RGBD camera.

[0019] Optionally, the grasping configuration includes the position of the dexterous hand base relative to the tool and the angles of each joint of the dexterous hand.

[0020] Optionally, controlling the manipulator arm dexterous hand system to grasp a tool comprises:

[0021] The tool grasping path is planned and controlled based on the grasping configuration and the posture of the tool.

[0022] Optionally, the step of moving the tool to a pre-alignment state comprises:

[0023] And based on the current scene information, the obstacle avoidance planning of the robotic arm dexterous hand system is implemented.

[0024] Optionally, the performing servo tracking adjustment based on six-dimensional posture to achieve coarse alignment includes:

[0025] Based on the actual posture of the manipulator end coordinate system in the current manipulator dexterous hand system, the actual posture of the tool, and the expected posture of the tool, the expected posture of the manipulator end coordinate system is calculated in real time to complete the rough alignment.

[0026] Compared with the prior art, the present invention has at least the following beneficial effects:

[0027] (1) The present invention performs servo adjustment based on six-dimensional posture estimation and tracking. Compared with the existing methods, it can compensate for the relative movement between the dexterous hand and the tool during large-scale movement and improve the coarse alignment accuracy.

[0028] (2) Based on the adjustment of the six-dimensional posture, the present invention comprehensively utilizes the small-range high-precision information of the two-dimensional image for precise alignment, which solves the contradiction between accuracy and range compared with the existing methods;

[0029] (3) The present invention adopts a movable visual observation method, which can achieve large-scale observation and fine measurement compared with existing methods, and provide measurement input for coarse and fine adjustment;

[0030] (4) The present invention designs a real-time safety monitoring method for observation results. Compared with existing methods, it can effectively deal with anomalies in the posture tracking process and avoid dangers such as collisions caused by posture errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 This is a flow chart of the tool-screw intelligent alignment method of the robot arm and dexterous hand system based on six-dimensional posture tracking and two-dimensional visual servoing of the present invention;

[0033] Figure 2 A configuration image of a dexterous hand grasping a power tool generated by an embodiment of the present invention;

[0034] Figure 3An image of a robotic arm and an auxiliary robotic arm system in a robotic arm dexterous hand system configured for an embodiment of the present invention;

[0035] Figure 4 An image of the screw posture estimation performed in the embodiment of the present invention;

[0036] Figure 5 This is an image of the power tool posture estimation performed in an embodiment of the present invention.

[0037] Figure 6 An image of the pre-alignment process performed in an embodiment of the present invention.

[0038] Figure 7 An image of the coarse alignment adjustment process performed in an embodiment of the present invention.

[0039] Figure 8 This is an image of the fine motion adjustment process performed by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0041] In the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance; unless otherwise specified or explained, the term "multiple" refers to two or more; the terms "connected" and "fixed" should be understood in a broad sense, for example, "connected" can be a fixed connection, a detachable connection, an integral connection, or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0042] In the description of this specification, it should be understood that the directional words such as "upper" and "lower" described in the embodiments of the present invention are described at the angles shown in the drawings and should not be understood as limiting the embodiments of the present invention. In addition, in the context, it should also be understood that when it is mentioned that an element is connected to another element "upper" or "lower", it can not only be directly connected to another element "upper" or "lower", but also indirectly connected to another element "upper" or "lower" through an intermediate element.

[0043] like Figure 1As shown, an embodiment of the present invention provides a tool-screw intelligent alignment method for a robotic arm and a dexterous hand system based on six-dimensional posture tracking and two-dimensional visual servo, comprising:

[0044] S1, the model generation of the grasping configuration based on the robotic arm dexterous hand system and the tool for screw-driving;

[0045] S2, configuring an auxiliary robotic arm on one side of the robotic arm dexterous hand system; wherein the auxiliary robotic arm is equipped with a camera;

[0046] S3, moving the auxiliary robot arm to the top of the screw to be tightened, using the camera to collect the three-dimensional coordinates of the screw, and determining the first observation posture and the second observation posture of the auxiliary robot arm based on the two-dimensional detection; wherein the first observation posture and the second observation posture are different single-degree-of-freedom observation dimensions;

[0047] S4, controlling the robot arm dexterous hand system to grasp the tool and move the tool to a pre-alignment state;

[0048] S5, in the pre-alignment state, performing servo tracking adjustment based on six-dimensional posture to achieve rough alignment;

[0049] S6, respectively move the auxiliary robot arm to the first observation posture and the second observation posture, and accurately adjust the pixel deviation of the pixel coordinates of the center point of the tool head detection frame and the pixel coordinates of the center point of the screw detection frame obtained by the camera to achieve alignment between the tool and the screw.

[0050] For S1, this embodiment uses the robotic arm, five-finger dexterous hand and hand drill power tool in the UR5e robotic arm dexterous hand system. Perform the following steps respectively:

[0051] (1) The models of the robotic arm and the dexterous hand use URDF files to describe their kinematic models. The URDF of the dexterous hand needs to be appropriately simplified. Its motion chain is motor-joint-Cartesian space position and posture. In this embodiment, the joint space is set as the input degree of freedom, and the simulation from the motor to the joint space is simplified to avoid the problem that the kinematics of complex mechanisms are difficult to solve;

[0052] (2) In the Mujoco simulation environment, the URDF model of the dexterous hand and the 3D Mesh file of the power tool are configured, the posture of the dexterous hand and the position of each joint are manually adjusted, grasping simulation is performed, and the configuration with better grasping effect is selected and saved;

[0053] (3) In a physical environment, a robotic arm and dexterous hand system is used to make a grasping attempt based on the angles obtained in step (2), and the dexterous hand posture and joint angles are further manually fine-tuned to determine a reasonable grasping configuration, such as Figure 2 As shown;

[0054] For S2, the system configured in this embodiment is as follows Figure 3 As shown in the figure, the UR5e auxiliary robot is used as the auxiliary robot. The robot arm installation position in the robot arm dexterous hand system is defined as T1, and the auxiliary robot arm installation position is defined as T2. The posture relationship between the two robots can be obtained through multi-robot calibration. The RGBD camera uses a ZED Mini camera, which is installed at the end of the auxiliary robot arm. The relative posture relationship can be obtained through hand-eye calibration and is defined as T 2c This embodiment adopts the ROS system, and the real-time position and posture of the center coordinate system of the flange at the end of the auxiliary robot arm can be obtained in real time through the ROS system, which is defined as T r2 .

[0055] For S3, the auxiliary robot arm is moved to achieve close-range two-dimensional detection of the screws, and the position and posture of the screws are calculated based on the two-dimensional detection results combined with the three-dimensional point cloud. The auxiliary robot arm is used to observe the first and second postures at close range based on the screw posture calculation;

[0056] The result of pose estimation in this embodiment is as follows Figure 4 As shown, the specific method is as follows:

[0057] (1) Move the auxiliary robot arm to the top of the screw and make the xy plane of the camera coordinate system parallel to the screw installation plane, observe the screw at close range, and obtain the RGB image and depth map;

[0058] (2) The YOLOv8 network is trained by manual sample annotation. Screw detection is performed based on the YOLOv8 method. The detection results are the centers of the detection boxes for the four screws, and the image coordinates are u 1～4 、v 1～4 , get the mean value u of the coordinates of the four screws c 、v c , and based on this point, polar coordinate transformation is performed, and the transformation result is ρ 1～4 ,θ 1～4 , based on the angle value θ 1～4 Sort and define the ID number of each screw.

[0059] (3) For each screw and the image coordinates of the center of its detection frame, obtain its three-dimensional position x according to the camera intrinsic parameters 1～4 ,y 1～4 、z 1～4 .

[0060] (4) For each screw’s posture, its coordinate system z-axis is defined as outward along the screw installation surface, the x-axis is the projection of the x-axis of the manipulator coordinate system in the manipulator dexterous hand system onto the plane perpendicular to the z-axis, and the y-axis is obtained by cross product calculation using the right-hand rule; for screws with ids 1 to 4, their posture relative to the camera coordinate system is obtained as T cs1～cs4 ;

[0061] (5) Calculate the position T of each screw relative to the auxiliary robot arm r2s1～r2s4 =T r2 T 2c T cs1～cs4 , based on this pose, the auxiliary manipulator is calculated to observe the first pose at close range. Taking the screw with id 1 as an example, the observation point is located in its yz plane, and the parameter θ s1-1 and ρ s1-1 Set the observed elevation angle and distance to obtain the vector p s1-1 And the coordinates of the observation point in the yz plane, and then the three-dimensional coordinates of the first observation posture are obtained; in terms of posture, the x-axis of the first observation posture is in the same direction as the x-axis of the screw, and the z-axis is in the same direction as the vector p s1-1 In the reverse direction, the y-axis is obtained by the right-hand rule cross product calculation;

[0062] (6) Calculate the second pose of the auxiliary manipulator for close-range observation. Taking the screw with ID 1 as an example, the observation point is located in its xz plane, and the parameter θ is used to calculate the second pose of the auxiliary manipulator for close-range observation. s1-2 and ρ s1-2 Set the observed elevation angle and distance to obtain the vector p s1-2 And the coordinates of the observation point in the xz plane, and then the three-dimensional coordinates of the second observed posture are obtained; in terms of posture, the x-axis of the second observed posture is in the same direction as the y-axis of the screw, and the z-axis is in the same direction as the vector p s1-2 Conversely, the y-axis is obtained by the right-hand rule cross product calculation.

[0063] Controlling the robot arm dexterous hand system grasping tools in S4, including:

[0064] (1) Combined with the tool pose T obtained in step (5) 1t , the relative position T of the dexterous hand and the tool in the dexterous hand grasping configuration obtained from step (1) th And the relative position T between the dexterous hand coordinate system and the manipulator end coordinate system in the manipulator dexterous hand system h1 , calculate and obtain the robot arm grasping posture T g =T 1t T th T h1 , and calculate the pre-grasp pose T pg , which is defined as the grasping posture T g Translate 15cm along the negative direction of its own x-axis;

[0065] (2) Using the ROS Moveit! plugin, the robot arm in the dexterous hand system is planned to move from the current state to the pose T without collision. pg , and then move to T g ;

[0066] (3) Turn on the dexterous hand and drive each axis joint to move to the angle value of each joint of the dexterous hand in the dexterous hand grasping configuration obtained from step (1) to complete the grasping.

[0067] For S4, the pre-alignment process is as follows Figure 6 As shown, the pre-alignment and posture validity detection are as follows:

[0068] (1) Based on the position information T of the screw to be aligned relative to the manipulator in the manipulator dexterous hand system s1 , and the ideal position T of the screw relative to the power tool when the screw is correctly inserted into the hole s1-t , calculate the expected moving posture T of the power tool tt =T s1 T s1-t , and further obtain the expected moving posture T of the manipulator in the manipulator dexterous hand system t1 =T tt T th T h1

[0069] (2) Using the ROS Moveit! plugin, the robot arm in the dexterous hand system is planned to move from the current state to the pose T without collision. t1 ;

[0070] (3) Calculate the real-time position T of the power tool relative to the robot end coordinate system in the robot arm dexterous hand system rt1 , and calculate T rt1 With T th T h1 When the position value or Euler angle deviation value of the relative posture exceeds a certain threshold, it is judged that the posture tracking fails and the moving operation is stopped.

[0071] With respect to S4, in some embodiments of the present invention, in the process of controlling the manipulator arm dexterous hand system to grasp the tool and move the tool to the pre-alignment state, the process includes:

[0072] The camera captures the image of the tool and estimates the tool’s pose;

[0073] The mobile auxiliary robot tracks the tool in real time so that the tool is in the middle of the camera's field of view.

[0074] Specifically, the mobile auxiliary robot arm estimates the tool’s pose and performs real-time tracking based on the RGBD camera feedback information;

[0075] In this embodiment, the pose estimation result is as follows: Figure 5 As shown, the specific steps are as follows:

[0076] (1) The auxiliary robot moves to the observation position T r22, whose field of view includes power tools;

[0077] (2) Based on the YOLOv8 combined with FoundationPose method, the pose of the power tool is obtained and tracked in real time:

[0078] (2.1) Obtain the detection frame of the power tool in the current image based on YOLOv8, and generate a mask image of the current tool based on the detection frame;

[0079] (2.2) Based on the three-dimensional model of the power tool, the real-time RGB image, the depth map information, and the mask image obtained in step (2.1), FoundationPose is used to perform multi-view sampling, simulated image and point cloud rendering, pose optimization, sorting and scoring on the power tool to achieve pose estimation of the power tool;

[0080] (2.3) Using the pose optimization network of FoundationPose, based on the pose estimation result of the previous frame, the tool pose of the current frame is obtained to achieve pose tracking of the power tool;

[0081] (3) Based on the current posture feedback of the power tool, PID control is used to realize the tracking and observation of the power tool by the auxiliary robot arm. To ensure safety, in this embodiment, the angle of rotation around the z-axis of the end of the auxiliary robot arm is controlled so that the x value of the power tool posture is controlled near 0, that is, to ensure that the tool is located in the middle of the camera field of view.

[0082] In some embodiments of the present invention, when the tool deviates from the middle of the camera field of view, the moving robot arm dexterous hand system is stopped and the posture of the tool is recalculated and estimated.

[0083] In some embodiments of the present invention, before controlling the manipulator arm dexterous hand system to grasp the tool and move the tool to the pre-alignment state, it also includes:

[0084] The position and posture of the tool relative to the manipulator coordinate system in the manipulator dexterous hand system is calculated based on the system calibration data.

[0085] In this embodiment, the calculation steps are as follows:

[0086] (1) The system calibration data includes the position and posture relationship T between the manipulator and the auxiliary manipulator in the manipulator dexterous hand system. 12 And the position relationship T of the camera relative to the end of the auxiliary robot arm 2c ;

[0087] (2) Based on the tool’s position T relative to the camera 2t , calculate the position T of the tool relative to the manipulator in the manipulator dexterous hand system 1t =T 12 Tr2 T 2c T 2t .

[0088] For S5, the rough alignment in this implementation is as follows Figure 7 , the specific process is as follows:

[0089] (1) The actual position of the current power tool is T t ;

[0090] (2) Obtaining the desired posture T of the power tool tt ;

[0091] (3) The main source of the error caused by steps (1) and (2) is the relative movement between the power tool and the hand during the large-scale movement. ser =T th T h1 The actual value of has a certain deviation from the grasping configuration generated by S1; iterative adjustment is performed to calculate T sera =T r1 - 1 T t , where T r1 is the real-time position of the end of the manipulator in the manipulator dexterous hand system, and further according to the expected position of the power tool T tt Calculate the desired position T of the end of the manipulator in the manipulator dexterous hand system r1t =T tt T sear -1 , and perform multiple iterations until T esr reached the threshold range.

[0092] For S6, the fine motion adjustment process of the first pose in this implementation is as follows:

[0093] (1) According to the coordinate system of the screw obtained in the above steps, obtain the fine motion adjustment direction vector, specifically the x-axis of the coordinate system;

[0094] (2) Based on YOLOv8, the pixel coordinates u of the center point of the tool head detection frame are identified t1 、v t1 When the tool head and the screw are close, recognition failure is likely to occur. At this time, the pixel coordinates u of the center point of the overall detection frame of the tool head and the rod above it are identified. t2 、v t2 ;

[0095] (3) Identify the pixel coordinates u of the center point of the screw detection frame based on YOLOv8 t0 、v t0 ,For the case of multiple screw identification, the detection box closest to the image center is selected;

[0096] (4) Calculate pixel deviation Δu = u t1 -u t0 , the proportional control method is used to perform iterative motion adjustment, and the direction obtained in the direction step (1) is adjusted until the deviation is less than a certain threshold and the adjustment is completed.

[0097] The fine adjustment in the second position is basically the same as that in the first position. According to the obtained coordinate system of the screw, the fine motion adjustment direction vector is obtained, specifically the y-axis of the coordinate system. The subsequent steps are the same as steps (2) to (4). The process of observation and fine motion adjustment in two directions is as follows: Figure 8 shown.

[0098] In some embodiments of the present invention, the z-axis of the screw's coordinate system is outward along the screw installation surface, the x-axis is the projection of the x-axis of the robotic arm dexterous hand system coordinate system onto a plane perpendicular to the z-axis, the y-axis is obtained by cross product calculation using the right-hand rule, the observation point of the first observation posture is in the yz plane, and the observation point of the second observation posture is in the xz plane.

[0099] In some embodiments of the present invention, the robotic arm dexterous hand system and the auxiliary robotic arm both have 6 active degrees of freedom, including 3 translations and 3 rotations. The dexterous hand includes five fingers, and the fingers have joints to enable grasping of electric tools. The tools include an electric drill, and the camera is an RGBD camera.

[0100] In some embodiments of the present invention, the grasping configuration includes the position of the base of the dexterous hand relative to the tool and the angles of each joint of the dexterous hand.

[0101] In some embodiments of the present invention, controlling a robotic arm dexterous hand system to grasp a tool includes:

[0102] The tool grasping path is planned and controlled based on the grasping configuration and the tool posture.

[0103] In some embodiments of the present invention, when moving the tool to the pre-alignment state, it includes:

[0104] And based on the current scene information, the obstacle avoidance planning of the robotic arm dexterous hand system is realized.

[0105] In some embodiments of the present invention, a servo tracking adjustment based on six-dimensional posture is performed to achieve coarse alignment, including:

[0106] Based on the actual posture of the manipulator end coordinate system, the actual posture of the tool, and the expected posture of the tool in the current manipulator dexterous hand system, the expected posture of the manipulator end coordinate system motion is calculated in real time to complete the rough alignment.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A tool-screw intelligent alignment method for a robotic arm and dexterous hand system based on six-dimensional posture tracking and two-dimensional visual servoing, characterized in that: include: Generate a grasping configuration based on a model of a robotic dexterous hand system and a tool for screwdriving; An auxiliary robotic arm is configured on one side of the robotic arm dexterous hand system; wherein the auxiliary robotic arm is equipped with a camera; Move the auxiliary robot arm to above the screw to be tightened, use the camera to collect the three-dimensional coordinates of the screw, and determine the first observation posture and the second observation posture of the auxiliary robot arm based on two-dimensional detection; wherein the first observation posture and the second observation posture are different single-degree-of-freedom observation dimensions; Controlling the robot arm dexterous hand system to grasp the tool and move the tool to a pre-aligned state; In the pre-alignment state, a servo tracking adjustment based on six-dimensional posture is performed to achieve rough alignment; The auxiliary robot arm is moved to the first observation posture and the second observation posture respectively, and the pixel deviation of the pixel coordinates of the center point of the tool head detection frame and the pixel coordinates of the center point of the screw detection frame obtained by the camera is precisely adjusted to achieve alignment of the tool and the screw.

2. The method according to claim 1, characterized in that: The process of controlling the manipulator arm dexterous hand system to grasp a tool and move the tool to a pre-alignment state includes: The camera captures an image of the tool and estimates the position and posture of the tool; The auxiliary robot arm is moved to track the tool in real time so that the tool is located in the middle of the camera field of view.

3. The method according to claim 2, characterized in that When the tool deviates from the middle of the camera field of view, the movement of the manipulator arm dexterous hand system is stopped, and the posture of the tool is recalculated and estimated.

4. The method according to claim 1, characterized in that Before controlling the manipulator arm dexterous hand system to grasp the tool and move the tool to the pre-alignment state, the method further includes: The position and posture of the tool relative to the manipulator coordinate system in the manipulator dexterous hand system is calculated based on the system calibration data.

5. The method according to claim 1, characterized in that The z-axis of the screw's coordinate system is outward along the screw installation surface, the x-axis is the projection of the x-axis of the robotic arm dexterous hand system coordinate system onto a plane perpendicular to the z-axis, the y-axis is obtained by cross product calculation using the right-hand rule, the observation point of the first observation posture is in the yz plane, and the observation point of the second observation posture is in the xz plane.

6. The method according to claim 1, characterized in that The robotic arm dexterous hand system and the auxiliary robotic arm both have 6 active degrees of freedom, including 3 translations and 3 rotations. The dexterous hand includes five fingers, and the fingers have joints to enable grasping of electric tools. The tools include electric drills, and the camera is an RGBD camera.

7. The method according to claim 1, characterized in that The grasping configuration includes the posture of the dexterous hand base relative to the tool and the angles of each joint of the dexterous hand.

8. The method according to claim 1, characterized in that: The control of the manipulator arm dexterous hand system to grasp a tool comprises: The tool grasping path is planned and controlled based on the grasping configuration and the posture of the tool.

9. The method according to claim 1, characterized in that: The step of moving the tool to a pre-alignment state comprises: And based on the current scene information, the obstacle avoidance planning of the robotic arm dexterous hand system is implemented.

10. The method according to claim 1, characterized in that The servo tracking adjustment based on the six-dimensional posture to achieve rough alignment includes: Based on the actual posture of the manipulator end coordinate system in the current manipulator dexterous hand system, the actual posture of the tool, and the expected posture of the tool, the expected posture of the manipulator end coordinate system is calculated in real time to complete the rough alignment.

Images