Dexterous hand teleoperation device based on RGBD camera and external parameter calibration and operation method

Abstract

The present application relates to the technical field of robot remote operation, and in particular to a dexterous hand remote device based on an RGBD camera and external parameter calibration and an operation method. The operation method comprises: taking a desktop as a reference object, and obtaining a transformation matrix of the RGBD camera relative to a desktop coordinate system; taking a desktop corner point as a calibration reference point, and teaching to obtain a position of an end of a mechanical arm in a mechanical arm coordinate system, and converting to the desktop coordinate system in combination with the transformation matrix to solve an external parameter calibration matrix of the mechanical arm coordinate system to the desktop coordinate system; the RGBD camera collects an image of a hand of an operator, extracts a position of a hand key point, converts to a camera coordinate system, and then converts to the desktop coordinate system through the transformation matrix to obtain a wrist pose; the wrist pose is converted into a target pose of the end of the mechanical arm, joint angles of the mechanical arm are solved, and the hand key point is remapped as joint angles of a dexterous hand. The technical scheme can improve the comfort, accuracy and safety of remote operation, and meet the demand of embodied intelligence for high-quality remote operation data collection.

Description

Technical Field

[0001] This invention relates to the field of robot teleoperation technology, specifically to a dexterous hand telescopic device and its operation method based on an RGBD camera and extrinsic parameter calibration. Background Technology

[0002] Robot teleoperation technology is a core means of human-robot collaborative work, widely used in hazardous environments, precision manufacturing, medical surgery, and nuclear industry maintenance. With the rapid development of embodied intelligence technology, teleoperation-based data acquisition has become a key data source for training robot strategy models, placing higher demands on the intuitiveness, comfort, and data quality of teleoperation systems.

[0003] However, existing teleoperation technologies still have significant limitations in practical applications, making it difficult to meet the demands for efficient, natural, and precise remote control. Currently, mainstream teleoperation technologies can be mainly divided into three categories: data acquisition solutions based on wearable devices, teleoperation solutions based on VR devices, and markerless gesture recognition solutions based on computer vision.

[0004] Wearable device-based data acquisition solutions obtain hand posture and joint angle information through data gloves, inertial measurement units, or force feedback gloves. While these solutions can directly collect joint data, they have inherent drawbacks: operators must wear specialized equipment for extended periods, and even with lightweight equipment, continuous data acquisition for several hours can still lead to hand fatigue and discomfort, significantly impacting operational comfort and data acquisition quality. Furthermore, data gloves typically have only a dozen or so degrees of freedom, which differs significantly from the high degrees of freedom of a real hand, making it difficult to fully reproduce fine hand movements. Additionally, the high cost of specialized equipment further increases the barrier to data acquisition.

[0005] Remote control solutions based on VR devices achieve remote operation through head-mounted displays combined with controllers or gesture recognition. While they can provide a certain level of immersion, they still require wearing VR headsets and controllers. VR headsets typically weigh 500-800 grams, and prolonged use can easily cause dizziness, leading to a decline in the operator's physical and mental state, and a significant decrease in data acquisition quality over time. In addition, the controller's pose mapping method for controlling the movement of the robotic arm lacks the intuitiveness of natural interaction and is difficult to accurately match the complex movements of the human hand.

[0006] Label-free gesture recognition solutions based on computer vision leverage deep learning and RGBD camera technology (such as open-source frameworks like MediaPipe and FrankMocap) to achieve hand keypoint detection using a monocular RGB or RGBD camera. This eliminates the need for wearing devices and offers better natural interaction characteristics. However, this solution faces the following challenges when applied to robotic arm teleoperation: First, there is a lack of precise calibration between the key points of the hand and the coordinate system of the robotic arm. Existing vision solutions can only acquire the pose of the hand in the camera coordinate system, while the motion control of the robotic arm needs to be based on its own coordinate system. The spatial transformation relationship between the two has not been effectively established, making it difficult to accurately map hand movements to the end effector of the robotic arm.

[0007] Secondly, relative pose control strategies lead to error accumulation. Existing solutions mostly adopt a relative control method that superimposes two pose offsets. In long-term tasks, errors accumulate over time, and the deviation between the actual pose of the robotic arm and the initial pose gradually increases, requiring frequent recalibration, which affects the continuity of operation and data quality.

[0008] Third, RGBD camera extrinsic parameter calibration is complex. Traditional extrinsic parameter calibration relies on specific calibration boards or auxiliary tools (such as right-angle blocks, cubes, etc.), requires additional equipment support, and the calibration process is cumbersome. Furthermore, when the camera and the robotic arm are in different spaces, the traditional hand-eye calibration method cannot be used, further increasing the calibration difficulty and maintenance costs.

[0009] Fourth, the operating range lacks an intuitive limitation mechanism. Due to the use of relative pose control, the system cannot intuitively limit the operating range of the robotic arm based on the natural interaction space. Novice operators are prone to misoperation, causing the robotic arm to exceed the safe range and leading to collision risks. Summary of the Invention

[0010] The purpose of this invention is to propose a dexterous hand remote control device and operation method based on an RGBD camera and extrinsic parameter calibration. This technical solution can lower the threshold of remote operation, improve the comfort, accuracy and safety of remote operation, and meet the needs of embodied intelligence for high-quality remote operation data acquisition.

[0011] To achieve the above objectives, in a first aspect, the present invention proposes a dexterous hand teleoperation method based on an RGBD camera and extrinsic parameter calibration, comprising: Using the desktop as a reference, obtain the transformation matrix of the RGBD camera relative to the desktop coordinate system; Using the corner of the desktop as the calibration reference point, the position of the end effector of the robotic arm in the robotic arm coordinate system is obtained through teaching. The transformation matrix is ​​then used to transform the position to the desktop coordinate system, and the extrinsic parameter calibration matrix from the robotic arm coordinate system to the desktop coordinate system is solved. The operator's hand image is acquired by an RGBD camera, the key points of the hand are extracted, the image is transformed to the camera coordinate system, and then transformed to the desktop coordinate system by the transformation matrix to obtain the wrist pose. The wrist pose is converted into the target pose of the robotic arm end effector, and the joint angles of the robotic arm are calculated. At the same time, the key points of the human hand are remapped to the joint angles of the dexterous hand. The joint angles of the robotic arm and the dexterous hand are sent to the robotic arm and the dexterous hand to realize remote operation.

[0012] As a feasible and preferred approach, the transformation matrix of the RGBD camera relative to the desktop coordinate system is obtained using the desktop as a reference, including: Define a camera coordinate system and a desktop coordinate system. Define the x-axis of the camera coordinate system along the direction of the camera lens, the y-axis along the direction of the camera body, and the z-axis by the right-hand rule. The origin of the desktop coordinate system is directly below the RGBD camera, and the XY plane is parallel to the desktop. Obtain point cloud data from an RGBD camera and obtain the point cloud in desktop coordinates based on the initial transformation matrix; Extract the point cloud representing the desktop and solve the plane equation; Based on the plane equation, the rotation matrix and translation vector are calculated to obtain the transformation matrix from the camera coordinate system to the desktop coordinate system.

[0013] As a feasible and preferred approach, the PCL library is used to fit the point cloud. The equation of the plane is obtained by finding the equation of the plane. ,and ,remember ; Calculate the transformation matrix, including: Define the rotation matrix R as the rotation relationship from the camera coordinate system to the desktop coordinate system: like ,

[0014] like ,if ,but ,if ,but ; Define the translation vector T as the coordinates of the origin of the camera coordinate system in the desktop coordinate system: ; The transformation matrix from the camera coordinate system to the desktop coordinate system is: ; For any point in the camera coordinate system If you need to convert to the desktop coordinate system, then: .

[0015] As a feasible and preferred option, the robotic arm coordinate system moves along the x-direction via the desktop coordinate system. After rotating 180°, the transformation matrix from the robotic arm coordinate system to the desktop coordinate system is: : .

[0016] As a feasible and preferred approach, the extrinsic parameter calibration matrix from the robotic arm coordinate system to the desktop coordinate system is solved, specifically including the following: Calculate the centroid and deviation, including: Calculate the centroid of the measurement point: ; ; Decentrifugation: ; ; The center of mass of the theory is ; Calculate intermediate variables: ; ; ; ; ; ; ; ; Solve for the extrinsic calibration matrix: .

[0017] As a feasible and preferred solution, images of the operator's hand are acquired using an RGBD camera, and the locations of key points on the hand are extracted, including: The pre-trained model extracts the pixel coordinates of key hand points from RGB images, obtains the depth value from the depth map based on the pixel coordinates, and transforms it to the camera coordinate system using RGBD camera intrinsics.

[0018] As a feasible and preferred approach, obtaining wrist pose also includes: By transforming the coordinates, the wrist position is changed to the coordinate system of the desktop, and the position of the wrist in the desktop coordinate system is defined. Regarding the position of the wrist after the conversion Make a judgment, make a judgment Does it exceed the area of ​​the table, restrictions? and ; If the conditions are not met, wait for the next hand keypoint detection; otherwise, after obtaining the wrist position, obtain the wrist pose using the PnP algorithm. .

[0019] As a feasible preferred solution, the inverse kinematics of the robotic arm is solved using the Pinocchio library.

[0020] As a feasible and preferred solution, finger keypoint remapping employs the Dex Retargeting algorithm, transforming the remapping of human hand keypoints into an optimization problem. This problem minimizes the total deviation between the dexterity hand keypoints and the human hand keypoints, while incorporating joint angle boundary constraints and temporal smoothing constraints. The optimization equation is as follows: ,at the same time

[0021] in, Indicates in Always be able to dexterously adjust the angles of each joint of the hand. Indicates the attenuation factor. This indicates the number of people at time t. One key point, express Use joint angles at all times As input, forward kinematics yields the first... One key point, Indicates the penalty coefficient. and These are the lower and upper boundaries of the joint, respectively.

[0022] Secondly, the present invention also provides a dexterous hand remote control device based on an RGBD camera and extrinsic parameter calibration, applied to the aforementioned dexterous hand remote control operation method based on an RGBD camera and extrinsic parameter calibration, comprising: The manual remote operation part includes a first table, an RGBD camera and an RGBD camera fixing structure. The RGBD camera fixing structure is used to fix the RGBD camera to one end of the first table, so that the RGBD camera can completely capture the image of the first table. The robotic arm operation section includes a second table, multiple RGB cameras, a robotic arm, and a robotic arm fixing structure. The RGB cameras are used to collect data, and the robotic arm fixing structure is used to fix the robotic arm. The first table and the second table have the same length and width.

[0023] The beneficial effects of this solution are as follows: This technical solution can complete the acquisition of wrist pose and key hand points using only an RGBD camera, eliminating the need for specialized wearable hardware such as VR devices and data gloves that traditional teleoperation relies on. This significantly reduces hardware procurement and deployment costs, while also reducing the complexity of the equipment's structure. It avoids the pressure, stuffiness, and discomfort caused by wearing VR / motion-sensing devices for extended periods, and prevents data distortion and decreased control precision due to wearer fatigue. The complete absence of wearable constraints enhances the naturalness of teleoperation and the stability of continuous operation.

[0024] Using the natural desktop as a calibration reference, RGBD camera extrinsic parameter calibration can be completed using desktop point clouds. Extrinsic parameter solutions between the robotic arm coordinate system and the desktop coordinate system can be achieved using desktop corner points, eliminating the need for additional calibration boards, markers, or other specialized auxiliary tools. One-click automated calibration is supported, making it easy and quick to learn. Furthermore, even if the camera's installation position changes, as long as the desktop remains within the camera's field of view, extrinsic parameter calibration can be quickly completed, demonstrating strong environmental adaptability and flexibly adapting to different operational scenarios.

[0025] This technical solution directly maps the hand keypoint pose identified by RGBD to the target pose of the robotic arm end effector after multi-coordinate system transformation. Employing an absolute pose control mode, it avoids the error accumulation problem of traditional incremental control. This effectively prevents position drift of the robotic arm and dexterous hand caused by error superposition during long-term teleoperation, eliminating the need for frequent resetting of the control origin and interruption of the work process, thus ensuring pose control accuracy and operational continuity throughout the entire teleoperation process.

[0026] By extracting key hand points and accurately converting multiple coordinate systems, it can not only calculate the joint angles of the robotic arm to achieve end-effector following, but also remap the key hand poses to the joint angles of the dexterous hand, enabling synchronous linkage and accurate replication of robotic arm movement and dexterous hand gestures. The remote control is highly smooth and can meet the needs of dexterous operation scenarios such as fine work and complex gesture grasping. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of a dexterous hand remote control based on an RGBD camera and extrinsic parameter calibration.

[0028] Figure 2 This is a logic diagram of a dexterous hand teleoperation method based on an RGBD camera and extrinsic parameter calibration. Figure 3 A schematic diagram defining the coordinate system for the manual teleoperation part.

[0029] Figure 4 A schematic diagram defining the coordinate system for the robotic arm's operating part.

[0030] Figure 5 This is a schematic diagram of the hand pose acquisition process. Detailed Implementation

[0031] To make the technical solution and advantages of this application clearer, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only some embodiments of the present invention, and are only used to explain this application, not to limit it. It should be noted that the technical features or combinations of technical features described in the following embodiments should not be considered isolated; they can be combined with each other to achieve better technical effects. The same reference numerals appearing in the accompanying drawings of the following embodiments represent the same features or components, and can be applied to different embodiments.

[0032] Furthermore, unless otherwise defined, the technical or scientific terms used in this invention description shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains.

[0033] The present invention will now be described in further detail with reference to the accompanying drawings.

[0034] Reference Figure 1 This disclosure provides a dexterous hand teleoperation device based on an RGBD camera and extrinsic parameter calibration, including a manual teleoperation part and a robotic arm operation part.

[0035] Figure 1 The left image shows the manual remote operation component, including the table, the RGBD camera, and the RGBD camera mounting structure. The camera mounting structure is used to secure the RGBD camera and can be adjustable or non-adjustable, ensuring a stable and reliable operation during data acquisition.

[0036] Figure 1 The right figure shows the robotic arm operation section, which includes the same table as the manual remote operation section, data acquisition equipment (multiple RGB cameras), robotic arm and robotic arm fixing structure for data acquisition. The RGB cameras can be added or removed as needed for data acquisition. In this embodiment, two RGB cameras are used, one in front and one on the left.

[0037] Reference Figure 2 This disclosure provides a dexterous hand teleoperation method based on an RGBD camera and extrinsic parameter calibration, including the following steps.

[0038] Step S100, RGBD camera extrinsic parameter calibration, includes: Step S101: Preparation. Ensure the desktop is clean and tidy, free of clutter. Using the desktop as a reference for external parameter calibration requires knowing its length and width beforehand. Define the length as... , width is Fix the RGBD camera to one end of the table, ensuring the camera can fully view the entire desktop, and measure the height of the camera above the desktop. .

[0039] Step S102, define the coordinate system, including: Reference Figure 3 The camera coordinate system is defined with its x-axis aligned with the camera lens, its y-axis aligned with the camera body, and its z-axis defined using the right-hand rule, pointing upwards. The origin of the desktop coordinate system is defined directly below the RGBD camera, and the XY plane is parallel to the desktop. Furthermore, it is stipulated that the RGBD camera can fully view the entire desktop to increase the detection field of view.

[0040] In this embodiment, Orbbec's Femto Bolt is used as the RGBD camera for gesture acquisition. This RGBD camera is equipped with an IMU, and the roll angle of the RGBD camera reference attitude can be obtained by reading the three-axis acceleration values ​​when the IMU is stationary. Pitch angle Yaw angle Because the accelerometer in the IMU has not been calibrated and can only be used as a reference value, the precise attitude will be obtained later through the point cloud obtained by the camera.

[0041] Define the initial rotation matrix as Then, the transformation from the RGBD camera coordinate system to the desktop coordinate system is defined as follows: ,in: , .

[0042] Step S103: Obtain the point cloud data of the RGBD camera at this time. Based on the defined transformation matrix Obtain the RGBD camera point cloud in desktop coordinate system. , .

[0043] Step S104, extract the z-value in The point cloud between them, this part of the point cloud represents the point cloud of the desktop, in which Indicates the threshold, which is used in this embodiment. The value is 0.1 meters. A point cloud meeting the conditions is defined as... Theoretically Most of the points in the middle are already close to 0, but due to the measurement Inaccuracies may lead to overall offset. Using this portion of the point cloud, a calibration matrix from the RGBD camera coordinate system to the desktop coordinate system is calculated. .

[0044] Step S105: Use the PCL library (point cloud processing library) to fit the point cloud. The equation of the plane, assuming the obtained plane equation is... ,and ,remember .

[0045] Step S106, calculate the transformation matrix, including: Define the rotation matrix R as the rotation relationship from the camera coordinate system to the desktop coordinate system: like ,

[0046] like ,if ,but ,if ,but .

[0047] Define the translation vector T as the coordinates of the origin of the camera coordinate system in the desktop coordinate system: .

[0048] The transformation matrix from the camera coordinate system to the desktop coordinate system is: .

[0049] For any point in the camera coordinate system If you need to convert to the desktop coordinate system, then: .

[0050] Step S200: Robotic arm extrinsic parameter calibration. In this embodiment, it is assumed that the robotic arm is horizontal and installed near the center of the long side of the table. However, since the installation position may not be centered and there are installation errors, calibration is required using the following method. It is certain, however, that the robotic arm's error only involves offsets in the x and y directions. and yaw angle direction deviation ,include: Step S201: Determine the installation location. Assume that the robotic arm is installed horizontally on the long side of the table near the middle.

[0051] Step S202: Select calibration reference points. Use the four corner points of the desktop as calibration reference points. In the desktop coordinate system, the coordinates of the four points in counterclockwise order are as follows: , , , () is used as a point for calibrating the extrinsic parameters of the robotic arm.

[0052] Step S203: Using the teaching method, drag the midpoint of the end of the robotic arm to the four corners of the table, and read the position of the robotic arm in the robotic arm coordinate system at the four corners. This value can usually be read directly from the interface provided by the robot manufacturer.

[0053] Step S204: Transform the coordinates to the desktop coordinate system using a transformation matrix. For example... Figure 4 By definition, the robotic arm's coordinate system can move along the x-direction using the desktop coordinate system. After rotating 180°, the transformation matrix from the robotic arm coordinate system to the desktop coordinate system is: : .

[0054] pass Convert to the desktop coordinate system.

[0055] Define the four points to be transformed to the desktop coordinate system as follows: .

[0056] Step S205, calculate the centroid and deviation, including: Calculate the centroid of the measurement point: ; ; Decentrifugation: ; ; according to Figure 4 The coordinate system is defined such that the centroid of the theoretical point is... .

[0057] The centroid of the theoretical point is the center point of the desktop coordinate system.

[0058] Calculate intermediate variables: ; ; ; ; ; ; ; ; Solve for the extrinsic calibration matrix: .

[0059] Step S206: Complete calibration for points in the robotic arm coordinate system. If you need to convert to the desktop coordinate system, then you need to .

[0060] Step S300: Hand pose acquisition, mainly focusing on the positions of key points on the fingers. and wrist position The definition is consistent with commonly used hand key point detection methods, refer to Figure 5 ,include: Step S301: Acquire image and depth information by using an RGBD camera to acquire RGB images and depth information of the operator's hand.

[0061] Step S302: Extract the positions of hand joints. Using a pre-trained model, input an RGB image and output the pixel coordinates of each key point of the hand on the image. In this embodiment, the Mediapipe framework is used to directly output the information of hand key points, including the positions of each key point of the fingers and the wrist pose, from an input RGB image.

[0062] Step S303: Obtain depth values. Based on the pixel coordinates of the hand key points, obtain the corresponding depth values ​​on the depth map. Then, based on the intrinsic parameters of RGBD, transform the key points to the camera coordinate system to obtain the positions of each key point on the hand. The hand key points include the wrist position.

[0063] Step S304: After the RGBD camera acquires the wrist position, it needs to transform the wrist position to the coordinate system of the desktop through coordinate transformation, and define the position of the wrist in the desktop coordinate system. Regarding the position of the wrist after the conversion. Make a judgment, make a judgment This checks whether the robotic arm exceeds the table's boundaries. The purpose of this is to prevent the robotic arm from going beyond the tabletop area and colliding with the operator or other objects outside the table. Only the pose is determined here. and ,limit and .

[0064] If the conditions are not met, wait for the next hand keypoint detection. Otherwise, after obtaining the wrist position, use the PnP (Perspective-n-Point) algorithm to obtain the wrist pose. Position of the wrist in the tabletop coordinate system As a pose transformation of the robotic arm's end effector, the target joint angle of the robotic arm is obtained and used as a control command input to the robotic arm.

[0065] Step S400, remapping of the robotic arm inverse kinematics with the finger key points, includes: Step S401: Use the Pinocchio library to perform inverse kinematics calculations for the robotic arm. This library meets the requirements of practical engineering in terms of both accuracy and time consumption. The joint angles of the robotic arm are then obtained. θ arm The command is sent to the robotic arm, which then autonomously controls itself to reach the target joint position.

[0066] Step S402: Remap the key points of the human hand to the angles of the dexterous hand joints. θ hand In this invention, the DexRetargeting algorithm is used. The remapping is transformed into an optimization problem, minimizing the total deviation between the dexterous hand keypoints and the human hand keypoints. The optimization equation is as follows: ,at the same time

[0067] in, Indicates in Always be able to dexterously adjust the angles of each joint of the hand. Indicates the attenuation factor. This indicates the number of people at time t. One key point, express Use joint angles at all times As input, forward kinematics yields the first... A key point. This represents the penalty coefficient, which is used to make the actions smoother between two consecutive actions. and These are the lower and upper boundaries of the joint, respectively.

[0068] Step S403, adjust the joint angle of the robotic arm. θ arm and dexterity hand joint angle θ hand It is then distributed to robotic arms and dexterous hands to achieve precise remote operation.

[0069] The above content is merely an embodiment of the present invention. Commonly known structures and characteristics of the solutions are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can improve and implement this solution based on the guidance provided in this application and their own capabilities. Typical well-known structures or operating methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A dexterous hand-operated method based on RGBD camera and extrinsic parameter calibration, characterized in that, include: Using the desktop as a reference, obtain the transformation matrix of the RGBD camera relative to the desktop coordinate system; Using the corner of the desktop as the calibration reference point, the position of the end effector of the robotic arm in the robotic arm coordinate system is obtained through teaching. The transformation matrix is ​​then used to transform the position to the desktop coordinate system, and the extrinsic parameter calibration matrix from the robotic arm coordinate system to the desktop coordinate system is solved. The operator's hand image is acquired by an RGBD camera, the key points of the hand are extracted, the image is transformed to the camera coordinate system, and then transformed to the desktop coordinate system by the transformation matrix to obtain the wrist pose. The wrist pose is converted into the target pose of the robotic arm end effector, and the joint angles of the robotic arm are calculated. At the same time, the key points of the human hand are remapped to the joint angles of the dexterous hand. The joint angles of the robotic arm and the dexterous hand are sent to the robotic arm and the dexterous hand to realize remote operation.

2. The dexterous hand teleoperation method based on RGBD camera and extrinsic parameter calibration according to claim 1, characterized in that, Using the desktop as a reference, obtain the transformation matrix of the RGBD camera relative to the desktop coordinate system, including: Define a camera coordinate system and a desktop coordinate system. Define the x-axis of the camera coordinate system along the direction of the camera lens, the y-axis along the direction of the camera body, and the z-axis by the right-hand rule. The origin of the desktop coordinate system is directly below the RGBD camera, and the XY plane is parallel to the desktop. Obtain point cloud data from an RGBD camera and obtain the point cloud in desktop coordinates based on the initial transformation matrix; Extract the point cloud representing the desktop and solve the plane equation; Based on the plane equation, the rotation matrix and translation vector are calculated to obtain the transformation matrix from the camera coordinate system to the desktop coordinate system.

3. The dexterous hand teleoperation method based on RGBD camera and extrinsic parameter calibration according to claim 2, characterized in that, Use the PCL library to fit the point cloud. The equation of the plane is obtained by finding the equation of the plane. ,and ,remember ; Calculate the transformation matrix, including: Define the rotation matrix R as the rotation relationship from the camera coordinate system to the desktop coordinate system: like , like ,if ,but ,if ,but ; Define the translation vector T as the coordinates of the origin of the camera coordinate system in the desktop coordinate system: ； The transformation matrix from the camera coordinate system to the desktop coordinate system is: ； For any point in the camera coordinate system If you need to convert to the desktop coordinate system, then: 。 4. The dexterous hand teleoperation method based on RGBD camera and extrinsic parameter calibration according to claim 1, characterized in that, The robotic arm's coordinate system moves along the x-direction through the desktop coordinate system. After rotating 180°, the transformation matrix from the robotic arm coordinate system to the desktop coordinate system is: : 。 5. The dexterous hand teleoperation method based on RGBD camera and extrinsic parameter calibration according to claim 4, characterized in that, Solve for the extrinsic calibration matrix from the robotic arm coordinate system to the desktop coordinate system, specifically including the following: Calculate the centroid and deviation, including: Calculate the centroid of the measurement point: ； ； Subtract the center of mass: ； ； The centroid of the theoretical point is ; Calculate intermediate variables: ； ； ； ； ； ； ； ； Solve for the extrinsic calibration matrix: 。 6. The dexterous hand teleoperation method based on RGBD camera and extrinsic parameter calibration according to claim 1, characterized in that, Images of the operator's hand are captured using an RGBD camera, and key hand points are extracted, including: The pre-trained model extracts the pixel coordinates of key hand points from RGB images, obtains the depth value from the depth map based on the pixel coordinates, and transforms it to the camera coordinate system using RGBD camera intrinsics.

7. The dexterous hand teleoperation method based on RGBD camera and extrinsic parameter calibration according to claim 1, characterized in that, Obtaining wrist position also includes: By transforming the coordinates, the wrist position is changed to the coordinate system of the desktop, and the position of the wrist in the desktop coordinate system is defined. Regarding the position of the wrist after the conversion Make a judgment, make a judgment Does it exceed the area of ​​the table, restrictions? and ; If the conditions are not met, wait for the next hand keypoint detection; otherwise, after obtaining the wrist position, obtain the wrist pose using the PnP algorithm. .

8. The dexterous hand teleoperation method based on RGBD camera and extrinsic parameter calibration according to claim 1, characterized in that, The inverse kinematics solution for the robotic arm is performed using the Pinocchio library.

9. The dexterous hand teleoperation method based on RGBD camera and extrinsic parameter calibration according to claim 1, characterized in that, Finger keypoint remapping employs the Dex Retargeting algorithm, transforming the remapping of human hand keypoints into an optimization problem. This minimizes the total deviation between the dexterity hand keypoints and the human hand keypoints, while incorporating joint angle boundary constraints and temporal smoothing constraints. The optimization equation is as follows: ,at the same time in, Indicates in Always be able to dexterously adjust the angles of each joint of the hand. Indicates the attenuation factor. This indicates the number of people at time t. One key point, express Use joint angles at all times As input, forward kinematics yields the first... One key point, Indicates the penalty coefficient. and These are the lower and upper boundaries of the joint, respectively.

10. A dexterous hand-held device based on an RGBD camera and extrinsic parameter calibration, characterized in that, The method for dexterous hand operation based on RGBD camera and extrinsic parameter calibration as described in any one of claims 1-9 includes: The manual remote operation part includes a first table, an RGBD camera and an RGBD camera fixing structure. The RGBD camera fixing structure is used to fix the RGBD camera to one end of the first table, so that the RGBD camera can completely capture the image of the first table. The robotic arm operation section includes a second table, multiple RGB cameras, a robotic arm, and a robotic arm fixing structure. The RGB cameras are used to collect data, and the robotic arm fixing structure is used to fix the robotic arm. The first table and the second table have the same length and width.

Images