A Method for Intelligent Alignment of Tools - Screws in a Manipulator and Dexterous Hand System Based on Six - Dimensional Pose Tracking and Two - Dimensional Visual Servo

Abstract

The present invention relates to the technical field of intelligent robots, and particularly relates to a method for intelligent alignment of a tool - screw for a robotic arm and dexterous hand system based on six - dimensional pose tracking and two - dimensional visual servoing. The method includes: generating a grasping configuration; configuring an auxiliary robotic arm on one side of the robotic arm dexterous hand system; moving the auxiliary robotic arm above the screw to be tightened, collecting the three - dimensional coordinates of the screw by using a camera, and determining the first observed pose and the second observed pose of the auxiliary robotic arm based on two - dimensional detection; controlling the robotic arm dexterous hand system to grasp the tool and move the tool to a pre - alignment state; performing servo tracking adjustment based on six - dimensional pose; respectively moving the auxiliary robotic arm to the first observed pose and the second observed pose, and performing precise adjustment according to the pixel deviation between the center point pixel coordinates of the tool head detection frame and the center point pixel coordinates of the screw detection frame obtained by the camera. This solution can quickly and accurately move the tool to align with the screw when the robotic arm dexterous hand system carries the tool to align with the screw.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent robots, and particularly relates to a method for intelligent alignment of a tool - screw for a robotic arm and dexterous hand system based on six - dimensional pose tracking and two - dimensional visual servoing. Background Art

[0002] Currently, with the development of robot and artificial intelligence technologies, the robotic arm and dexterous hand system has become an application carrier of intelligent control technology in various tasks due to its human - like configuration, and has advantages such as a wide variety of tasks to be performed, strong environmental adaptability, and high operation and control accuracy. For the typical task of using a power tool to screw a screw, the process of alignment using a robotic arm and dexterous hand system faces difficulties in measurement and planning. In terms of measurement, the common fixed - position observation faces the contradiction between the observation range and the observation accuracy. The wrist camera is blocked by the hand during the alignment process and it is difficult to achieve a large - range observation. In terms of planning, the measurement information based on six - dimensional pose can achieve large - range adjustment, but the accuracy usually fails to meet the requirements, and relative movement may occur between the dexterous hand and the tool during the movement. The adjustment based on two - dimensional vision can achieve fine alignment, but the measurement range is limited. Therefore, to solve the above problems, it is of great significance to study a method for intelligent alignment of a tool - screw for a robotic arm and dexterous hand system based on six - dimensional pose tracking and two - dimensional visual servoing. Summary of the Invention

[0003] An embodiment of the present invention provides a method for intelligent alignment of a tool - screw for a robotic arm and dexterous hand system based on six - dimensional pose tracking and two - dimensional visual servoing, which can quickly and accurately move the tool to align with the screw when the robotic arm and dexterous hand system carry the tool to align with the screw.

[0004] An embodiment of the present invention provides a method for intelligent alignment of a tool - screw for a robotic arm and dexterous hand system based on six - dimensional pose tracking and two - dimensional visual servoing, including:

[0005] Generating a grasping configuration based on the models of the robotic arm and dexterous hand system and the tool for screwing the screw;

[0006] Configuring an auxiliary robotic arm on one side of the robotic arm and dexterous hand system; wherein, a camera is installed on the auxiliary robotic arm;

[0007] Moving the auxiliary robotic arm above the screw to be screwed, collecting the three - dimensional coordinates of the screw by using the camera, and determining the first observation pose and the second observation pose of the auxiliary robotic arm based on two - dimensional detection; wherein, the first observation pose and the second observation pose are different single - degree - of - freedom observation dimensions;

[0008] Controlling the robotic arm and dexterous hand system to grasp the tool and move the tool to a pre - alignment state;

[0009] In the pre-alignment state, servo tracking adjustment based on six-degree-of-freedom pose is performed to achieve rough alignment;

[0010] The auxiliary robotic arm is respectively moved to the first observation pose and the second observation pose, and precise adjustment is performed according to the pixel deviation between the center pixel coordinates of the tool head detection frame and the center pixel coordinates of the screw detection frame obtained by the camera, so as to achieve the alignment of the tool and the screw.

[0011] Optionally, in the process of the control robotic arm dexterous hand system grasping the tool and moving the tool to the pre-alignment state, it includes:

[0012] The camera collects an image of the tool and estimates the pose of the tool;

[0013] The auxiliary robotic arm is moved to track the tool in real time, so that the tool is located in the middle of the camera's field of view.

[0014] Optionally, when the tool deviates from the middle of the camera's field of view, the movement of the robotic arm dexterous hand system is stopped, and the pose of the tool is re-estimated.

[0015] Optionally, before the control robotic arm dexterous hand system grasps the tool and moves the tool to the pre-alignment state, it further includes:

[0016] Calculating the pose of the tool relative to the robotic arm coordinate system in the robotic arm dexterous hand system based on the system calibration data.

[0017] Optionally, the z-axis of the coordinate system of the screw is outward along the screw installation plane, the x-axis is the projection of the x-axis direction of the robotic arm dexterous hand system coordinate system on the plane perpendicular to the z-axis, the y-axis is obtained by cross product calculation using the right-hand rule, the observation point of the first observation pose is the y-z plane, and the observation point of the second observation pose is located in the x-z plane.

[0018] Optionally, both the robotic arm dexterous hand system and the auxiliary robotic arm have 6 active degrees of freedom, among which, 3 are translational and 3 are rotational. The dexterous hand includes five fingers, and the fingers have joints to achieve the grasping of the electric tool. The tool includes an electric drill, and the camera is an RGBD camera.

[0019] Optionally, the grasping configuration includes the pose of the dexterous hand base relative to the tool and the angles of each joint of the dexterous hand.

[0020] Optionally, the control robotic arm dexterous hand system grasping the tool includes:

[0021] Planning and controlling the tool grasping path based on the grasping configuration and the pose of the tool.

[0022] Optionally, when moving the tool to the pre-alignment state, it includes:

[0023] And based on the current scene information, implement obstacle avoidance planning for the robotic arm and dexterous hand system.

[0024] Optionally, when performing servo tracking adjustment based on six-dimensional pose to achieve rough alignment, it includes:

[0025] Based on the actual pose of the end coordinate system of the robotic arm, the actual pose of the tool, and the desired pose of the tool in the current robotic arm and dexterous hand system, calculate the desired pose of the movement of the end coordinate system of the robotic arm in real time to complete rough alignment.

[0026] The present invention has at least the following beneficial effects compared with the prior art:

[0027] (1) The present invention performs servo adjustment based on six-dimensional pose estimation and tracking. Compared with the existing methods, it can compensate for the relative movement between the dexterous hand and the tool during large-range movement, and improve the rough alignment accuracy;

[0028] (2) Based on the six-dimensional pose adjustment, the present invention comprehensively uses the small-range and high-precision information of the two-dimensional image for fine alignment. Compared with the existing methods, it solves the contradiction problem between accuracy and range;

[0029] (3) The present invention adopts a movable vision observation method. Compared with the existing methods, it can achieve large-range observation and fine measurement, and provide measurement input for the rough and fine two-level adjustments;

[0030] (4) The present invention designs a real-time safety monitoring method for the observation results. Compared with the existing methods, it can effectively handle abnormalities during pose tracking and avoid dangers such as collisions caused by incorrect poses. Description of the Drawings

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0032] Figure 1 It is a flowchart of the tool-screw intelligent alignment method for the robotic arm and dexterous hand system based on six-dimensional pose tracking and two-dimensional visual servo of the present invention;

[0033] Figure 2 It is an image of the configuration of the dexterous hand grasping an electric tool generated by an embodiment of the present invention;

[0034] Figure 3Images of the robotic arm and the auxiliary robotic arm system in the robotic arm dexterous hand system configured for the embodiments of the present invention;

[0035] Figure 4 Images for the screw pose estimation in the embodiments of the present invention;

[0036] Figure 5 Images for the power tool pose estimation in the embodiments of the present invention.

[0037] Figure 6 Images for the pre-alignment process in the embodiments of the present invention.

[0038] Figure 7 Images for the rough alignment adjustment process in the embodiments of the present invention.

[0039] Figure 8 Images for the fine motion adjustment process in the embodiments of the present invention. Detailed implementation manners

[0040] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0041] In the description of the embodiments of the present invention, unless otherwise clearly specified and defined, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance; unless otherwise specified or stated, the term "plurality" means two or more; the terms "connection", "fixation", etc. shall be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, an integral connection, or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0042] In the description of this specification, it should be understood that the orientation terms such as "upper" and "lower" described in the embodiments of the present invention are described from the angles shown in the accompanying drawings and should not be construed as limiting the embodiments of the present invention. In addition, in the context, it should also be understood that when it is mentioned that an element is connected "above" or "below" another element, it can not only be directly connected "above" or "below" another element, but also be indirectly connected "above" or "below" another element through an intermediate element.

[0043] Such as Figure 1As shown, an intelligent alignment method for a tool - screw of a robotic arm and dexterous hand system based on six - degree - of - freedom pose tracking and two - dimensional visual servoing provided by an embodiment of the present invention includes:

[0044] S1, generating a grasping configuration based on the models of the robotic arm - dexterous hand system and the tool for screwing the screw;

[0045] S2, configuring an auxiliary robotic arm on one side of the robotic arm - dexterous hand system; wherein, a camera is installed on the auxiliary robotic arm;

[0046] S3, moving the auxiliary robotic arm above the screw to be screwed, collecting the three - dimensional coordinates of the screw by using the camera, and determining the first observation pose and the second observation pose of the auxiliary robotic arm based on two - dimensional detection; wherein, the first observation pose and the second observation pose are different single - degree - of - freedom observation dimensions;

[0047] S4, controlling the robotic arm - dexterous hand system to grasp the tool and move the tool to a pre - alignment state;

[0048] S5, performing servo tracking adjustment based on six - dimensional pose in the pre - alignment state to achieve rough alignment;

[0049] S6, respectively moving the auxiliary robotic arm to the first observation pose and the second observation pose, and performing precise adjustment according to the pixel deviation between the center point pixel coordinates of the tool head detection frame and the center point pixel coordinates of the screw detection frame obtained by the camera to achieve the alignment of the tool and the screw.

[0050] For S1, in this embodiment, the robotic arm, five - finger dexterous hand, and electric hand drill tool in the UR5e robotic arm - dexterous hand system are adopted. The following steps are carried out respectively:

[0051] (1) The models of the robotic arm and the dexterous hand describe their kinematic models in the form of URDF files. The URDF of the dexterous hand needs to be appropriately simplified. Its motion chain is motor - joint - Cartesian space position and pose. In this embodiment, the joint space is set as the input degree of freedom, and the simulation from the motor to the joint space is simplified to avoid the problem that the kinematics of complex mechanisms is difficult to solve;

[0052] (2) In the Mujoco simulation environment, configure according to the URDF model of the dexterous hand and the three - dimensional Mesh file of the electric tool, manually adjust the pose of the dexterous hand and the positions of each joint, perform grasping simulation, and select the configuration with better grasping effect for saving;

[0053] (3) In the physical environment, adopt the robotic arm - dexterous hand system, and perform grasping attempts based on the angles obtained in step (2), and further manually fine - tune the pose of the dexterous hand and the joint angles to determine a reasonable grasping configuration, as Figure 2 shown;

[0054] For S2, the system configured in this embodiment is as follows Figure 3 As shown, the UR5e robotic arm is used as the auxiliary robotic arm. Define the installation position of the robotic arm in the robotic arm dexterous hand system as T 1 , and the installation position of the auxiliary robotic arm as T 2 . The pose relationship between the two robotic arms can be obtained through multi-robot calibration. The RGBD camera uses the ZED Mini camera, which is installed at the end of the auxiliary robotic arm. The relative pose relationship can be obtained through hand-eye calibration and is defined as T 2c . In this embodiment, the ROS system is adopted. The real-time pose of the center coordinate system of the end flange of the auxiliary robotic arm can be obtained in real time through the ROS system and is defined as T r2 .

[0055] For S3, move the auxiliary robotic arm to achieve close-range two-dimensional detection of the screw, and calculate the pose of the screw based on the two-dimensional detection results combined with the three-dimensional point cloud. Calculate the first pose and the second pose of the auxiliary robotic arm for close-range observation based on the screw pose;

[0056] The results of pose estimation in this embodiment are as follows Figure 4 As shown, the specific method is as follows:

[0057] (1) Move the auxiliary robotic arm above the screw and make the x-y plane of the camera coordinate system parallel to the screw installation plane. Observe the screw at close range to obtain the RGB image and the depth image;

[0058] (2) Train the YOLOv8 network by means of manual sample annotation. Detect the screw based on the YOLOv8 method. The detection result is the center of the detection frame of 4 screw recognitions, and the image coordinates are u 1～4 , v 1～4 . Obtain the mean values u c , v c of the coordinates of 4 screws, and perform polar coordinate transformation based on this point. The transformation results are ρ 1～4 , θ 1～4 . Sort based on the angle value θ 1～4 and define the id numbers of each screw.

[0059] (3) For each screw and its detection frame center image coordinates, obtain its three-dimensional position x 1～4 , y 1～4 , z 1～4 according to the camera internal parameters.

[0060] (4) For the attitude of each screw, define the z-axis of its coordinate system as outward along the screw installation plane, the x-axis as the projection of the x-axis of the robotic arm coordinate system in the robotic arm dexterous hand system onto the plane perpendicular to the z-axis, and the y-axis is obtained by cross-multiplication calculation using the right-hand rule; for the screws with IDs from 1 to 4, obtain their poses relative to the camera coordinate system as T cs1～cs4 ;

[0061] (5) Calculate the pose T r2s1～r2s4 = T r2 T 2c T cs1～cs4 of each screw relative to the auxiliary robotic arm. Based on this pose, calculate the close-range observation first pose of the auxiliary robotic arm. Taking the screw with ID = 1 as an example, the observation point is located in its y-z plane. By parameters θ s1-1 and ρ s1-1 , set the elevation angle and distance of the observation, obtain the vector p s1-1 and the coordinates of the point where the observation point is located in the y-z plane, and then obtain the three-dimensional coordinates of the observation first pose; in terms of attitude, the x-axis of the observation first pose is in the same direction as the x-axis of the screw, the z-axis is opposite to the vector p s1-1 , and the y-axis is obtained by cross-multiplication calculation using the right-hand rule;

[0062] (6) Calculate the close-range observation second pose of the auxiliary robotic arm. Taking the screw with ID = 1 as an example, the observation point is located in its x-z plane. By parameters θ s1-2 and ρ s1-2 , set the elevation angle and distance of the observation, obtain the vector p s1-2 and the coordinates of the point where the observation point is located in the x-z plane, and then obtain the three-dimensional coordinates of the observation second pose; in terms of attitude, the x-axis of the observation second pose is in the same direction as the y-axis of the screw, the z-axis is opposite to the vector p s1-2 , and the y-axis is obtained by cross-multiplication calculation using the right-hand rule.

[0063] Regarding the control of the robotic arm dexterous hand system to grasp the tool in S4, it includes:

[0064] (1) Combine the tool pose T 1t obtained in step (5), the relative pose T th between the dexterous hand and the tool in the dexterous hand grasping configuration obtained from step (1), and the relative pose T h1 between the dexterous hand coordinate system and the robotic arm end coordinate system in the robotic arm dexterous hand system, calculate and obtain the robotic arm grasping pose T g = T 1t T th T h1 , and calculate the pre-grasping pose T pg , which is defined as the grasping pose T g translated 15 cm in the negative direction of its own x-axis;

[0065] (2) Use the ROS Moveit! plugin to plan the collision-free movement of the robotic arm in the robotic arm and dexterous hand system from the current state to the pose T pg , and then move to T g ;

[0066] (3) Activate the dexterous hand and drive each axis joint to move to the angle values of each joint of the dexterous hand in the dexterous hand grasping configuration obtained from step (1) to complete the grasping.

[0067] For S4, the pre-alignment process is as Figure 6 shown, and the pre-alignment and pose validity detection are as follows:

[0068] (1) Based on the pose information T s1 of the screw to be aligned relative to the robotic arm in the robotic arm and dexterous hand system, and the ideal pose T s1-t of the screw relative to the power tool under the condition of correct screw insertion, calculate the expected movement pose T tt = T s1 T s1-t of the power tool, and further obtain the expected movement pose T t1 = T tt T th T h1

[0069] (2) Use the ROS Moveit! plugin to plan the collision-free movement of the robotic arm in the robotic arm and dexterous hand system from the current state to the pose T t1 ;

[0070] (3) Calculate the real-time pose T rt1 of the power tool relative to the end coordinate system of the robotic arm in the robotic arm and dexterous hand system, and calculate the relative pose of T rt1 and T th T h1 . When the position value or Euler angle deviation value of this pose exceeds a certain threshold, judge that the pose tracking fails and stop the movement operation.

[0071] For S4, in some embodiments of the present invention, in the process of controlling the robotic arm and dexterous hand system to grasp the tool and move the tool to the pre-alignment state, it includes:

[0072] The camera collects the image of the tool and estimates the pose of the tool;

[0073] The mobile auxiliary robotic arm tracks the tool in real time so that the tool is located in the middle of the camera field of view.

[0074] Specifically, the mobile auxiliary robotic arm estimates the pose of the tool and performs real-time tracking based on the feedback information of the RGBD camera;

[0075] In this embodiment, the pose estimation result is as follows: Figure 5 As shown, the specific steps are as follows:

[0076] (1) The auxiliary robot moves to the observation position T r22 , whose field of view includes power tools;

[0077] (2) Based on the YOLOv8 combined with FoundationPose method, the pose of the power tool is obtained and tracked in real time:

[0078] (2.1) Obtain the detection frame of the power tool in the current image based on YOLOv8, and generate a mask image of the current tool based on the detection frame;

[0079] (2.2) Based on the three-dimensional model of the power tool, the real-time RGB image, the depth map information, and the mask image obtained in step (2.1), FoundationPose is used to perform multi-view sampling, simulated image and point cloud rendering, pose optimization, sorting and scoring on the power tool to achieve pose estimation of the power tool;

[0080] (2.3) Using the pose optimization network of FoundationPose, based on the pose estimation result of the previous frame, the tool pose of the current frame is obtained to achieve pose tracking of the power tool;

[0081] (3) Based on the current posture feedback of the power tool, PID control is used to realize the tracking and observation of the power tool by the auxiliary robot arm. To ensure safety, in this embodiment, the angle of rotation around the z-axis of the end of the auxiliary robot arm is controlled so that the x value of the power tool posture is controlled near 0, that is, to ensure that the tool is located in the middle of the camera field of view.

[0082] In some embodiments of the present invention, when the tool deviates from the middle of the camera field of view, the moving robot arm dexterous hand system is stopped and the posture of the tool is recalculated and estimated.

[0083] In some embodiments of the present invention, before controlling the manipulator arm dexterous hand system to grasp the tool and move the tool to the pre-alignment state, it also includes:

[0084] The position and posture of the tool relative to the manipulator coordinate system in the manipulator dexterous hand system is calculated based on the system calibration data.

[0085] In this embodiment, the calculation steps are as follows:

[0086] (1) The system calibration data includes the position and posture relationship T between the manipulator and the auxiliary manipulator in the manipulator dexterous hand system. 12 And the position relationship T of the camera relative to the end of the auxiliary robot arm2c ;

[0087] (2) Calculate the pose T of the tool relative to the robotic arm in the robotic arm dexterous hand system based on the pose T of the tool relative to the camera 2t , and calculate the pose T of the tool relative to the robotic arm in the robotic arm dexterous hand system 1t = T 12 T r2 T 2c T 2t .

[0088] For S5, the rough alignment in this implementation is as follows Figure 7 , and the specific process is as follows:

[0089] (1) The actual pose of the current power tool is T t ;

[0090] (2) Obtain the desired pose T of the power tool tt ;

[0091] (3) The main sources of error caused by steps (1) and (2) are the relative movement between the power tool and the hand during the large-scale movement. That is, T ser = T th T h1 has a certain deviation from the grasping configuration generated in S1; perform iterative adjustment and calculate T sera = T r1 - 1 T t , where T r1 is the real-time pose of the end of the robotic arm in the robotic arm dexterous hand system. Further, calculate the desired pose T of the end of the robotic arm in the robotic arm dexterous hand system according to the desired pose T of the power tool tt T r1t = T tt T sear -1 , and perform multiple iterations until T esr reaches the threshold range.

[0092] For S6, the fine motion adjustment process of the first pose in this implementation is as follows:

[0093] (1) According to the coordinate system of the screw obtained in the above steps, obtain the fine motion adjustment direction vector, specifically the x-axis of the coordinate system;

[0094] (2) Based on the pixel coordinates u t1 and v t1 of the center point of the detection box of the tool head recognized by YOLOv8, when the tool head is relatively close to the screw, it is easy to have a recognition failure. At this time, obtain the pixel coordinates u t2 and v t2 of the center point of the overall detection box of the recognized tool head and the rod above it;

[0095] (3) Based on YOLOv8, identify the pixel coordinates u of the center point of the screw detection frame t0 , v t0 . For the case of multiple screw identifications, select the detection frame closest to the center of the image;

[0096] (4) Calculate the pixel deviation Δu = u t1 - u t0 , and use the method of proportional control to perform iterative motion adjustment, adjusting the direction obtained in step (1) until the deviation is less than a certain threshold and then completing the adjustment.

[0097] The fine adjustment in the second pose is basically the same as that in the first pose. According to the obtained coordinate system of the screw, obtain the fine motion adjustment direction vector, specifically the y-axis of the coordinate system. The subsequent steps are the same as steps (2) to (4). The processes of observation and fine motion adjustment in the two directions are as Figure 8 shown.

[0098] In some embodiments of the present invention, the z-axis of the coordinate system of the screw is outward along the screw installation surface, the x-axis is the projection of the x-axis of the robotic arm dexterous hand system coordinate system on the plane perpendicular to the z-axis, the y-axis is obtained by cross-product calculation using the right-hand rule, the observation point of the first observation pose is the y-z plane, and the observation point of the second observation pose is located in the x-z plane.

[0099] In some embodiments of the present invention, both the robotic arm dexterous hand system and the auxiliary robotic arm have 6 active degrees of freedom, among which, 3 are translational and 3 are rotational. The dexterous hand includes five fingers, and the fingers have joints to achieve the grasping of an electric tool. The tool includes an electric drill, and the camera is an RGBD camera.

[0100] In some embodiments of the present invention, the grasping configuration includes the pose of the dexterous hand base relative to the tool and the angles of each joint of the dexterous hand.

[0101] In some embodiments of the present invention, controlling the robotic arm dexterous hand system to grasp the tool includes:

[0102] Planning and controlling the tool grasping path based on the grasping configuration and the pose of the tool.

[0103] In some embodiments of the present invention, when moving the tool to the pre-alignment state, it includes:

[0104] And implementing obstacle avoidance planning for the robotic arm dexterous hand system based on the current scene information.

[0105] In some embodiments of the present invention, performing servo tracking adjustment based on six-dimensional pose to achieve rough alignment, including:

[0106] Based on the actual pose of the end coordinate system of the robotic arm, the actual pose of the tool, and the desired pose of the tool in the current robotic arm dexterous hand system, the desired pose of the end coordinate system of the robotic arm is calculated in real time to complete the rough alignment.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A tool-screw intelligent alignment method for a robotic arm and dexterous hand system based on six-dimensional posture tracking and two-dimensional visual servoing, characterized in that: include: Generate a grasping configuration based on a model of a robotic dexterous hand system and a tool for screwdriving; An auxiliary robotic arm is configured on one side of the robotic arm dexterous hand system; wherein the auxiliary robotic arm is equipped with a camera; Move the auxiliary robot arm to above the screw to be tightened, use the camera to collect the three-dimensional coordinates of the screw, and determine the first observation posture and the second observation posture of the auxiliary robot arm based on two-dimensional detection; wherein the first observation posture and the second observation posture are different single-degree-of-freedom observation dimensions; Controlling the robot arm dexterous hand system to grasp the tool and move the tool to a pre-aligned state; In the pre-alignment state, a servo tracking adjustment based on six-dimensional posture is performed to achieve rough alignment; The auxiliary robot arm is moved to the first observation posture and the second observation posture respectively, and the pixel deviation of the pixel coordinates of the center point of the tool head detection frame and the pixel coordinates of the center point of the screw detection frame obtained by the camera is precisely adjusted to achieve alignment between the tool and the screw; The process of controlling the manipulator arm dexterous hand system to grasp a tool and move the tool to a pre-alignment state includes: The camera captures an image of the tool and estimates the position and posture of the tool; Moving the auxiliary robot arm to track the tool in real time so that the tool is located in the middle of the camera field of view; When the tool deviates from the middle of the camera field of view, the movement of the manipulator arm dexterous hand system is stopped, and the posture of the tool is recalculated and estimated; Before controlling the manipulator arm dexterous hand system to grasp the tool and move the tool to the pre-alignment state, the method further includes: The position and posture of the tool relative to the manipulator coordinate system in the manipulator dexterous hand system is calculated based on the system calibration data.

2. The method according to claim 1, characterized in that The z-axis of the screw's coordinate system is outward along the screw installation surface, the x-axis is the projection of the x-axis of the robotic arm dexterous hand system coordinate system onto a plane perpendicular to the z-axis, the y-axis is obtained by cross product calculation using the right-hand rule, the observation point of the first observation posture is in the yz plane, and the observation point of the second observation posture is in the xz plane.

3. The method according to claim 1, characterized in that The robotic arm dexterous hand system and the auxiliary robotic arm both have 6 active degrees of freedom, including 3 translations and 3 rotations. The dexterous hand includes five fingers, and the fingers have joints to enable grasping of electric tools. The tools include electric drills, and the camera is an RGBD camera.

4. The method according to claim 1, characterized in that The grasping configuration includes the posture of the dexterous hand base relative to the tool and the angles of each joint of the dexterous hand.

5. The method according to claim 1, characterized in that The control of the manipulator arm dexterous hand system to grasp a tool comprises: The tool grasping path is planned and controlled based on the grasping configuration and the posture of the tool.

6. The method according to claim 1, characterized in that The step of moving the tool to a pre-alignment state comprises: And based on the current scene information, the obstacle avoidance planning of the robotic arm dexterous hand system is implemented.

7. The method according to claim 1, characterized in that The servo tracking adjustment based on the six-dimensional posture to achieve rough alignment includes: Based on the actual posture of the manipulator end coordinate system in the current manipulator dexterous hand system, the actual posture of the tool, and the expected posture of the tool, the expected posture of the manipulator end coordinate system is calculated in real time to complete the rough alignment.

Images