Dexterous hand in-palm operation control method based on reinforcement learning and real-time pose feedback

Abstract

The application discloses a kind of based on reinforcement learning and real-time pose feedback nimble palm internal operation control method, the experimental scene of underactuated nimble palm internal operation is built, and data acquisition is carried out in simulation environment;For underactuated nimble palm internal operation task, reinforcement learning algorithm framework is built, and according to task requirement and underactuated nimble hand structure characteristics design fine state space, action space, reward function;Configuration IsaacLab environment realizes the efficient interaction of reinforcement learning algorithm and physical simulation platform;Simulation training is carried out in the environment, and the optimal strategy is learned;Different task scenarios are set, pure state input is used to drive decision, and the autonomous adaptability and generalization control ability of underactuated nimble hand are improved.The application can solve the problems that current robot underactuated nimble hand has insufficient control flexibility in complex environment, is difficult to adapt to unstructured environment and reinforcement learning training results are difficult to migrate from simulation to reality.

Description

Technical Field

[0001] This invention relates to a dexterous hand operation control method, and more particularly to a dexterous hand in-palm operation control method based on reinforcement learning and real-time pose feedback. Background Technology

[0002] Faced with the growing demand for automation and the trend of intelligent development, the application of robotic dexterous hands is gradually expanding to a wider range of fields. As an important actuator for precision operations, robotic dexterous hands possess flexible multi-finger coordination capabilities, enabling them to complete complex tasks such as grasping and palmar manipulation, and have significant application value in aerospace, intelligent manufacturing, medical rehabilitation, and other fields.

[0003] However, current dexterous hand control methods still mainly rely on hard-coded control. Hard-coded control depends on preset logic rules, which is difficult to adapt to dynamically changing task requirements. This control method has significant limitations when facing complex and ever-changing unstructured environments.

[0004] Therefore, there is an urgent need for a robot dexterous hand control method based on reinforcement learning and real-time pose feedback, which can be efficiently trained in a simulation environment and ensure its stability and adaptability in the real environment through Sim-to-Real transfer technology, so as to improve the intelligence level of the robot dexterous hand and expand its application scope in industrial production and daily life services. Summary of the Invention

[0005] Purpose of the invention: This invention aims to solve the problems of insufficient control flexibility of underactuated dexterous hands in complex environments, difficulty in adapting to unstructured environments, and difficulty in transferring reinforcement learning training results from simulation to reality. It provides a dexterous hand in-palm operation control method based on reinforcement learning and real-time pose feedback.

[0006] Technical solution: The present invention provides a dexterous palmar manipulation control method based on reinforcement learning and real-time pose feedback, which specifically includes the following steps:

[0007] (1) Construct an experimental scenario for in-palm manipulation of an underactuated dexterous hand, deploy an underactuated dexterous hand model, an in-palm object model, and an interactive environment in a simulation environment, and collect training data;

[0008] (2) For in-palm manipulation tasks of underactuated dexterous hands, construct an in-palm manipulation reinforcement learning policy network based on PPO or DDPG algorithms, and design a refined state space, action space and reward function according to task requirements and structural characteristics of underactuated dexterous hands.

[0009] (3) Use the training data collected in the simulation environment in step (1) and the reinforcement learning policy network established in step (2) for simulation training to learn the optimal policy; write a manager-based reinforcement learning configuration file and task environment configuration file to realize efficient interaction between the reinforcement learning algorithm and the physical simulation platform.

[0010] (4) Using the reinforcement learning strategy network trained in step (3), the pose information of the in-palm object output by FoundationPose and the dexterous hand joint information obtained directly from the dexterous hand machine are used as state inputs to calculate and output the optimal control strategy for dexterous hand motion control.

[0011] Further, the training data in step (1) includes the state information of the underactuated dexterous hand and the state information of the object in the palm; the state information of the underactuated dexterous hand includes the angle of each joint, the velocity of each joint, the acceleration of each joint, and the position of each joint; the state information of the object in the palm includes position information and posture information.

[0012] Furthermore, the specific steps of step (1) are as follows:

[0013] (11) Using the NVIDIA Isaac Sim physics simulation platform, we built an experimental scenario for in-palm operation of an underactuated dexterous hand and deployed an underactuated dexterous hand model and interactive objects in the scenario to provide a high-fidelity simulation environment.

[0014] (12) To address the joint actuation characteristics of the underactuated dexterous hand, modify its URDF model file, adjust the constraints and dynamic parameters of each joint, etc., to ensure that the dexterous hand in the simulation environment can truly reflect its motion characteristics in actual operation.

[0015] (13) Based on the modified URDF model file, construct the USD model of the dexterous hand to ensure that the dexterous hand can be correctly loaded in the USD scene and can interact normally with other objects.

[0016] (14) Import the completed USD model into the NVIDIA Isaac Sim physics simulation environment to verify the interactivity of the dexterous hand in the simulation environment, ensure the physical consistency and stability of the simulation results, and provide a reliable simulation environment for subsequent reinforcement learning training; and collect training data in the simulation environment.

[0017] Furthermore, the reinforcement learning policy network in step (2) includes a state space, an action space, and a reward function; the state space includes the joint angles, joint velocities, joint accelerations, joint position information of the dexterous hand, and the 6D pose information of the object in the palm; the action space includes all control strategies for each joint of the dexterous hand, adopting a continuous action control method to ensure that the dexterous hand can perform operations smoothly; the reward function is based on a reward mechanism for task completion, configuring reward items to help the agent evaluate the quality of the current state and actions, and continuously optimize the strategy accordingly.

[0018] Furthermore, the reward function is:

[0019] R=λ1R success +λ2R pos +λ3R ori

[0020] In this function, λ1 controls the weight of the success reward, while λ2 and λ3 control the weight of the position tracking reward and the direction tracking reward, respectively. In the combined reward function, the success reward provides the final goal, while the position and direction rewards guide the exploration.

[0021] Success reward R success for:

[0022] dθ=||quat error magnitude(s o ,s g )||

[0023]

[0024] Among them, s g It is the target state of the object in the palm, s o It is the current state of the object in the palm. The state information is represented by a homogeneous matrix of quaternions and position coordinates. quat_error_magnitude calculates the error between the two states. τ is the error threshold between the current direction and the target direction.

[0025] Location tracking reward R pos for:

[0026] R pos =-||p o -p g || 2

[0027] Where, p g It is the target position of the object in the palm, p o It is the current position of the object in the palm. The Euclidean length between the current position and the target position is calculated using the L2 norm. Taking the negative value helps to maximize the reward in subsequent reinforcement learning training.

[0028] Directional tracking reward R ori for:

[0029] β=||q o -q g || 2

[0030]

[0031] Where, q g The target direction of the object in the palm, q o The current orientation of the object in the palm is ∈, which is a small positive number to prevent division by zero errors. The L2 norm is used to calculate the error between the object's current position and the target position, and the reciprocal is taken to maximize the reward in subsequent reinforcement learning training.

[0032] Further, the manager-based reinforcement learning configuration file in step (3) includes a command manager, an action manager, an observation manager, an event manager, a termination manager, and a reward manager; the command manager defines high-level task commands to place the target object into the dexterous hand in each training round; the action manager defines the basic operations that the dexterous hand can perform in the experimental scenario; the observation manager defines the observation items of the dexterous hand during the task, including joint state information and target object posture information, to provide perceptual data for reinforcement learning decisions; the event manager defines and manages events in different modes to help manage state switching during training; the termination manager defines task termination conditions to determine whether the current training round should be terminated, including object falling, operation failure, or reaching the target state; the reward manager designs a reward function, configures reward items, and sets weight coefficients according to task requirements to encourage the agent to achieve the task goal and punish movements that have a negative impact on the task, thereby improving the stability and efficiency of the strategy.

[0033] Furthermore, the non-task environment configuration file described in step (3) adopts pure state input-driven decision-making and performs reinforcement learning training in various task scenarios, so that the reinforcement learning strategy can adapt to different object types, initial poses and environmental constraints, thereby improving the generalization ability in multiple scenarios and enhancing the autonomous adaptability and generalization control ability of the underactuated dexterous hand.

[0034] Furthermore, the specific steps of step (4) are as follows:

[0035] (41) Use Matlab to calibrate the RGBD camera: fix the position of the depth camera and move the camera calibration board. Use the Matlab camera calibration tool to calibrate the acquired calibration board image to obtain the camera intrinsic parameter matrix.

[0036] (42) Construct the OBJ model file of the object to be estimated as a reference to perform pose estimation; if the CAD model file of the object cannot be obtained, use the generalization ability of FoundationPose large-scale synthetic training to complete the pose estimation of the object.

[0037] (43) Use a depth camera to acquire RGB and depth images in the experimental environment, select the first frame of RGB image, and use SAM for mask annotation to optimize feature extraction and pose estimation of the target object.

[0038] (44) The Mask R-CNN detector is used to obtain the 2D bounding box of the object. The initial position of the object is determined by the translation initialization through the median of the 3D points in the bounding box. Multiple rotation angles are uniformly sampled with the target object as the center to generate a rotation matrix for global pose initialization as the initial pose. The image data collected by the RGBD camera is input, and FoundationPose outputs the pose update information of the object in the palm.

[0039] (45) The reinforcement learning strategy network obtained by training is used to calculate and output the optimal control strategy using the current state information as input, so as to achieve precise motion control of the dexterous hand.

[0040] (46) 6D pose information of the in-palm object output by the FoundationPose algorithm, namely the position and orientation information of the in-palm object relative to the camera; real-time joint information of the dexterous hand directly collected from the actual dexterous hand machine, including joint angle, joint velocity, and end position.

[0041] (47) A reinforcement learning strategy network is used to calculate the optimal action strategy based on the current state information. The optimal control strategy output by the reinforcement learning strategy network is converted into a low-level control signal and sent to the dexterous hand actuator; the dexterous hand is controlled to perform fine movements and complete the expected in-palm operation task.

[0042] (48) During the execution process, the real-time state of the dexterous hand and the pose of the object in the palm change in real time. The policy network updates the state observation information at each time step, dynamically adjusts and continuously outputs the control policy.

[0043] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] 1. This invention employs reinforcement learning through a trial-and-error, reward-driven mechanism, enabling the dexterous hand to autonomously explore and optimize control strategies during continuous interaction, thereby improving its adaptability and intelligence level. Through the reinforcement learning method of proximal strategy optimization, the dexterous hand can autonomously learn appropriate operation strategies in different task scenarios, reducing human intervention and improving the stability and generalization ability of task execution.

[0045] 2. Since the movement of underactuated dexterous hands involves continuous control of multiple degrees of freedom and multiple joints sharing a single actuation source, the action space is complex. Traditional reinforcement learning algorithms struggle to efficiently search for the optimal policy. Therefore, a probability distribution is used to represent the policy, and actions are selected through sampling, rather than searching for the best action in a finite set of actions. This approach can naturally handle the continuous action space. Thus, this invention employs a reinforcement learning policy network based on PPO or DDPG algorithms for policy output. The policy distribution is optimized using the policy gradient method, ensuring that the new policy does not deviate too far from the old policy. This helps maintain the stability of learning, reduces the complexity of policy search, and improves training efficiency.

[0046] 3. Since reinforcement learning training requires a large amount of trial and error data, deploying training on a physical machine would be too costly in terms of time and money. Therefore, this invention adopts a method of parallel reinforcement learning training by building an experimental scenario in a physical simulation platform, which not only greatly improves training efficiency but also saves economic costs. After the RGBD camera acquires the RGB image and depth image information of the environment, and the deep learning network outputs the 6D pose information of the object in the hand, the pose information only needs to be input into the reinforcement learning policy network trained in the simulation task scenario to obtain the optimal control policy of the dexterous hand. The optimal control policy output by the reinforcement learning policy network is converted into a low-level control signal to drive the dexterous hand to complete the task autonomously. Attached Figure Description

[0047] Figure 1 This is a flowchart of the present invention;

[0048] Figure 2 This is a schematic diagram of the reinforcement learning training process of the present invention. Detailed Implementation

[0049] The present invention will now be described in further detail with reference to the accompanying drawings.

[0050] like Figure 1 As shown, this invention proposes a dexterous in-palm manipulation control method based on reinforcement learning and real-time pose feedback, comprising the following steps:

[0051] Step 1: Using the NVIDIA Isaac Sim physics simulation platform, an experimental scenario for in-palm manipulation of an underactuated dexterous hand was built. An underactuated dexterous hand model, an in-palm object model, and an interactive environment were deployed within the simulation environment. Parameters such as collision detection and dynamic constraints were set based on the physics engine to ensure high fidelity of the simulation environment. Data was collected in the simulation environment, including real-time state information of the dexterous hand such as joint angles, joint velocities, and end-effector positions, as well as the position and pose information (6D pose information) of the target object.

[0052] An experimental scenario for in-palm manipulation of an underactuated dexterous hand was built using the NVIDIA Isaac Sim physics simulation platform. An underactuated dexterous hand model and interactive objects were deployed within this scenario to construct a high-fidelity simulation environment. The URDF model file of the underactuated dexterous hand was adjusted to optimize joint constraints and dynamic parameters, ensuring that the hand's motion characteristics in the simulation environment accurately reflect actual manipulation. Based on the modified URDF model file, a corresponding USD model was constructed, enabling the dexterous hand to load correctly in the USD scenario and interact normally with other objects. Subsequently, the completed USD model was imported into the NVIDIA Isaac Sim physics simulation environment to verify its interactive performance, including key indicators such as collision detection and dynamic response. This ensured that the simulation results achieved high standards in terms of physical consistency and stability, thus providing reliable data support for subsequent reinforcement learning training.

[0053] To address the structural characteristics and operational requirements of underactuated dexterous hands, a high-precision physics engine built into the simulation platform was used to systematically configure the scene, including collision detection, friction coefficient settings, and rigid body properties, to ensure the realism of the simulation environment and achieve simulation effects that closely resemble real-world scenarios. This provides reliable environmental support for reinforcement learning training. Systematic data acquisition was conducted within the constructed simulation environment. The collected training data was divided into two categories: the first category was the dexterous hand's own state information s. a The first category includes real-time angles, angular velocities, and angular accelerations of each joint, as well as dynamic information such as the position and orientation of the dexterous hand's end effector. The second category is the state information of the object within the hand. b It includes three-dimensional position coordinate information and rotational attitude information, which are used to reflect the dynamic changes and feedback information of the object in the palm during the operation process.

[0054] Step 2: Addressing the characteristics of underactuated dexterous hand in-palm manipulation tasks, such as high state dimensionality, strong dynamic constraints, and complex interactions, the following proposals are put forward: Figure 2 The method shown is based on reinforcement learning to construct an efficient operation decision network. The policy network outputs actions that are applied to the simulation environment. State and reward information from the simulation environment is obtained and stored as experience. The evaluation function is used to assess the value function of the current action, and the dominance function is used to determine whether the current action network is superior to the old one. The gradient of the loss with respect to the weights is calculated, and the policy network parameters are updated to improve the dexterity of the hand's operational autonomy and adaptability in the simulation environment. Reinforcement learning enables dynamic decision-making and action planning for the dexterity of the hand in complex in-palm manipulation tasks, overcoming the problem that traditional hard-coded methods struggle to adapt to the random changes in complex environments.

[0055] Considering the unique structure of underactuated dexterous hands, characterized by multiple joints sharing a single actuation source and a continuous action space with high degrees of freedom, traditional reinforcement learning methods suffer from high action search complexity and slow convergence speed when dealing with continuous action spaces, making it difficult to achieve efficient decision-making. To address this, a reinforcement learning algorithm framework based on Proximal Policy Optimization (PPO) or Deep Deterministic Policy Gradient (DDPG) is constructed to reduce the search complexity of the action space and improve learning efficiency and decision quality.

[0056] A reinforcement learning policy network for underactuated dexterous hand manipulation tasks is proposed, including the design of a refined state space, action space, and reward function. State space S: S∈R n This includes information such as joint angles, joint velocities, joint accelerations, and joint positions of the dexterous hand, as well as the 6D pose information of the object within the palm; motion space A: A∈R n All control strategies for each joint of the dexterous hand employ a continuous motion control approach to ensure smooth operation. Reward function R: A reward mechanism based on task completion is designed, configuring three reward items for in-palm operations to help the agent evaluate the current state and the quality of actions, and continuously optimize the strategy accordingly. The reward function configuration is as follows:

[0057] Success reward R success :

[0058] dθ=||quaterror magnitude(s o ,s g )||

[0059]

[0060] Among them, s g It is the target state of the object in the palm, s o This represents the current state of the object within the palm of the hand. The state information is represented by a homogeneous matrix of quaternions and position coordinates. `quat_error_magnitude` calculates the error between two states, and `τ` is the error threshold between the current direction and the target direction. The success reward objective is clear and does not rely on complex error calculations, making training more efficient. If the agent successfully reaches the goal, it receives a fixed high reward, which helps stabilize learning.

[0061] Location tracking reward R pos :

[0062] R pos =-||p o -p g || 2

[0063] Where, p g It is the target position of the object in the palm, p o The current position of the object in the palm is used. The Euclidean length between the current position and the target position is calculated using the L2 norm. Taking the negative value helps to maximize the reward in subsequent reinforcement learning training. Position tracking reward allows the agent to obtain continuous feedback and obtain rewards without fully reaching the target, thus improving the stability of training.

[0064] Directional tracking reward R ori :

[0065] β=||q o -q g || 2

[0066]

[0067] Where, q g The target direction of the object in the palm, q o It represents the current orientation of the object within the palm of the hand. ∈ is a small positive number to prevent division by zero errors. The L2 norm is used to calculate the error between the object's current position and the target position. Taking the reciprocal helps maximize the reward in subsequent reinforcement learning training. The orientation tracking reward encourages the agent to minimize rotational errors and quickly adjust to the correct orientation, making it particularly suitable for tasks involving dexterous hand manipulation.

[0068] Final reward function:

[0069] R=λ1R success +λ2R pos +λ3R ori

[0070] In this function, λ1 controls the weight of the success reward (which is generally large); λ2 and λ3 control the weight of the position tracking reward and the direction tracking reward, respectively. In the combined reward function, the success reward provides the final goal, while the position and direction rewards guide the exploration, taking into account both stability and convergence speed.

[0071] Step 3: Achieve efficient interaction between reinforcement learning algorithms and physical simulation platforms.

[0072] A reinforcement learning configuration file based on a manager architecture was developed. This file integrates multiple functional modules, including a command manager, action manager, and observation manager, to simulate different task conditions, improving the flexibility and adaptability of training. In this environment, training data collected in the simulation environment of step 1 and the reinforcement learning policy network established in step 2 are used for simulation training to learn the optimal policy. Specifically, the command manager defines high-level task instructions, initializes the task scenario in each training round, and places the target object into the dexterous hand; the action manager defines the basic operations of the dexterous hand in the experimental scenario, ensuring that it can execute reasonable control policies; the observation manager is responsible for collecting the dexterous hand's sensory data, including joint state information and target object posture information, providing accurate environmental feedback for reinforcement learning decisions; the event manager defines and manages events in different modes, ensuring reasonable state switching during training; the termination manager sets task termination conditions, such as object falling, operation failure, or task goal achievement, to determine whether the current training round should end; and the reward manager designs reward functions according to task requirements and configures different reward items and weight coefficients to encourage the agent to perform operations beneficial to the task goal while penalizing ineffective or harmful behaviors, thereby improving the stability and efficiency of the policy. During training, the optimal policy is guided by maximizing rewards.

[0073] Furthermore, by setting different task scenarios and writing corresponding task environment configuration files, reinforcement learning training is carried out under various environmental conditions. Pure state input drives decision-making, enabling the reinforcement learning strategy to adapt to different object types, initial poses, and environmental constraints, thereby improving the generalization ability in multiple scenarios and enhancing the autonomous adaptability and generalization control ability of the underactuated dexterous hand.

[0074] Step 4: Use the 6D pose estimation algorithm FoundationPose to perform in-hand operation migration based on pose estimation.

[0075] RGBD camera calibration is performed using Matlab. The depth camera position is fixed while the calibration board is moved. The acquired calibration board images are processed using Matlab's camera calibration tools to obtain the camera intrinsic parameter matrix. An OBJ model file of the object in the palm of the hand is constructed as a reference to support subsequent pose estimation. If a CAD model of the object is unavailable, pose estimation can be achieved using a small number of reference images by leveraging FoundationPose's large-scale synthetic training capabilities.

[0076] In the experimental environment, RGB and depth images were acquired using a depth camera. The first frame of the RGB image was selected, and the SegmentAnything Model (SAM) was used for masking to optimize feature extraction and pose estimation of the target object. A MaskR-CNN detector was used to obtain the object's 2D bounding box, and the median of the 3D points within the box was used for translation initialization to determine the object's initial position. Simultaneously, multiple rotation angles were uniformly sampled around the target object to generate a global pose initialization rotation matrix as the initial pose. The image data acquired by the RGBD camera was input, and FoundationPose output the pose update information of the object within the palm of the hand.

[0077] Using the reinforcement learning policy network trained in step 3, and taking the current state information as input, including the 6D pose information of the in-palm object output by FoundationPose and the dexterous hand joint information directly collected from the actual machine, the reinforcement learning policy network trained in step 3 calculates and outputs the optimal action, which is then converted into a low-level control signal and sent to the dexterous hand actuator to realize fine movements such as fingertip manipulation, grasping, rotation, and pushing, thereby completing the in-palm manipulation task.

[0078] During execution, the state of the dexterous hand and the pose of the object in the palm change in real time. The policy network updates the state observation information at each time step and dynamically adjusts the control strategy to ensure that the dexterous hand adapts to the ever-changing operating environment.

[0079] This invention employs a reinforcement learning policy network, enabling underactuated dexterous hands to overcome the limitations of traditional hard-coded control. This allows them to autonomously learn and adapt to different in-hand operation tasks, improving the intelligence level of decision-making. The reinforcement learning policy network is trained in a simulation environment and then deployed to a real machine, reducing the cost and risk of real-world training and improving learning efficiency. It can be applied to precision operation scenarios such as industrial assembly, medical rehabilitation, and service robots, enhancing the application value of robots in complex, unstructured environments and providing technical support for human-computer interaction and intelligent manufacturing.

[0080] The present invention has been described in detail above with reference to specific embodiments; however, these descriptions should not be construed as limiting the present invention. Those skilled in the art will understand that various equivalent substitutions, modifications, or improvements can be made to the technical solutions and embodiments of the present invention without departing from the spirit and scope of the invention, and all such modifications and improvements fall within the scope of the present invention. The scope of protection of the present invention is defined by the appended claims.

Claims

1. A dexterous palmar manipulation control method based on reinforcement learning and real-time pose feedback, characterized in that, Includes the following steps: (1) Construct an experimental scenario for in-palm manipulation of an underactuated dexterous hand, deploy an underactuated dexterous hand model, an in-palm object model, and an interactive environment in the simulation environment, and collect training data; (2) For in-palm manipulation tasks of underactuated dexterous hands, construct an in-palm manipulation reinforcement learning policy network based on PPO or DDPG algorithms, and design a refined state space, action space and reward function according to task requirements and structural characteristics of underactuated dexterous hands. (3) Use the training data collected in the simulation environment in step (1) and the reinforcement learning policy network established in step (2) for simulation training to learn the optimal policy; write a manager-based reinforcement learning configuration file and task environment configuration file to realize efficient interaction between reinforcement learning algorithm and physical simulation platform; (4) Using the reinforcement learning strategy network trained in step (3), the pose information of the in-palm object output by FoundationPose and the dexterous hand joint information obtained directly from the dexterous hand machine are used as state inputs to calculate and output the optimal control strategy for dexterous hand motion control. The reinforcement learning policy network in step (2) includes a state space, an action space, and a reward function. The state space includes the joint angles, joint velocities, joint accelerations, joint position information of the dexterous hand, and the 6D pose information of the object in the palm. The action space includes all control strategies for each joint of the dexterous hand, using a continuous action control method to ensure that the dexterous hand can perform operations smoothly. The reward function is based on a reward mechanism for task completion, configuring reward items to help the agent evaluate the quality of the current state and actions, and continuously optimize the strategy accordingly. The reward function is: in, Control the weight of success rewards. , The weights of position tracking rewards and orientation tracking rewards are controlled separately. In the combined reward function, the success reward provides the final goal, while the position and orientation rewards guide the exploration. Success reward for: in, It refers to the target state of the object within the palm of your hand. This represents the current state of the object within the palm of your hand. The state information is represented using a homogeneous matrix of quaternions and position coordinates. `quat_error_magnitude` calculates the error between two states. It is the error threshold between the current direction and the target direction; Location tracking rewards for: in, It is the target position of the object within the palm. It is the current position of the object in the palm. The Euclidean length between the current position and the target position is calculated using the L2 norm. Taking the negative value helps to maximize the reward in subsequent reinforcement learning training. Directional tracking reward for: in, It is the target direction of the object in the palm. It is the current orientation of the object in the palm. It is a small positive number to prevent division by zero errors. The L2 norm is used to calculate the error between the current position of the object and the target position. Taking the reciprocal helps to maximize the reward in subsequent reinforcement learning training.

2. The dexterous palmar manipulation control method based on reinforcement learning and real-time pose feedback according to claim 1, characterized in that, The training data in step (1) includes the state information of the underactuated dexterous hand and the state information of the object in the palm; the state information of the underactuated dexterous hand includes the angle of each joint, the velocity of each joint, the acceleration of each joint, and the position of each joint; the state information of the object in the palm includes position information and posture information.

3. The dexterous palmar manipulation control method based on reinforcement learning and real-time pose feedback according to claim 1, characterized in that, The specific steps of step (1) are as follows: (11) Using the NVIDIA Isaac Sim physics simulation platform, we built an experimental scenario for in-palm manipulation of an underactuated dexterous hand and deployed an underactuated dexterous hand model and interactive objects in the scenario to provide a high-fidelity simulation environment. (12) To address the joint actuation characteristics of the underactuated dexterous hand, modify its URDF model file and adjust the constraints and dynamic parameters of each joint to ensure that the dexterous hand in the simulation environment can truly reflect its motion characteristics in actual operation. (13) Based on the modified URDF model file, construct the USD model of the dexterous hand to ensure that the dexterous hand can be correctly loaded in the USD scene and can interact normally with other objects; (14) Import the completed USD model into the NVIDIA Isaac Sim physics simulation environment to verify the interactivity of the dexterous hand in the simulation environment, ensure the physical consistency and stability of the simulation results, and provide a reliable simulation environment for subsequent reinforcement learning training. Training data was collected in a simulation environment.

4. The dexterous palmar manipulation control method based on reinforcement learning and real-time pose feedback according to claim 1, characterized in that, The manager-based reinforcement learning configuration file in step (3) includes a command manager, an action manager, an observation manager, an event manager, a termination manager, and a reward manager; the command manager defines high-level task commands to place the target object into the dexterous hand in each training round; the action manager defines the basic operations that the dexterous hand can perform in the experimental scenario; the observation manager defines the observations of the dexterous hand during the task, including joint state information and target object posture information, to provide perceptual data for reinforcement learning decisions; the event manager defines and manages events in different modes to help manage state switching during training. The termination manager defines task termination conditions to determine whether the current training round should be terminated, including object falling, operation failure, or reaching the target state. The reward manager designs reward functions, configures reward items, and sets weight coefficients according to task requirements, encouraging agents to achieve task goals and penalizing actions that have a negative impact on the task, thereby improving the stability and efficiency of the strategy.

5. The dexterous palmar manipulation control method based on reinforcement learning and real-time pose feedback according to claim 1, characterized in that, The task environment configuration file described in step (3) adopts pure state input-driven decision-making and performs reinforcement learning training in various task scenarios, so that the reinforcement learning strategy can adapt to different object types, initial poses and environmental constraints, thereby improving the generalization ability in multiple scenarios and enhancing the autonomous adaptability and generalization control ability of the underactuated dexterous hand.

6. The dexterous palmar manipulation control method based on reinforcement learning and real-time pose feedback according to claim 1, characterized in that, The specific steps for step (4) are as follows: (41) Use Matlab to calibrate the RGBD camera: fix the position of the depth camera and move the camera calibration board. Use the Matlab camera calibration tool to calibrate the acquired calibration board image to obtain the camera intrinsic parameter matrix. (42) Construct the OBJ model file of the object to be estimated as a reference and perform pose estimation; If the CAD model file of the object cannot be obtained, the pose estimation of the object is completed by utilizing the generalization ability of FoundationPose's large-scale synthetic training. (43) Use a depth camera to acquire RGB and depth images in the experimental environment, select the first frame of RGB image, and use SAM for mask annotation to optimize feature extraction and pose estimation of the target object; (44) The Mask R-CNN detector is used to obtain the 2D bounding box of the object. The translation is initialized by the median of the 3D points in the bounding box to determine the initial position of the object. Multiple rotation angles are uniformly sampled with the target object as the center to generate a rotation matrix for global pose initialization as the initial pose. The image data collected by the RGBD camera is input, and FoundationPose outputs the pose update information of the object in the palm. (45) The reinforcement learning policy network obtained by training is used to calculate and output the optimal control policy using the current state information as input, so as to achieve precise motion control of the dexterous hand; (46) 6D pose information of the in-palm object output by the FoundationPose algorithm, namely the position and orientation information of the in-palm object relative to the camera; real-time joint information of the dexterous hand directly collected from the actual dexterous hand machine, including joint angle, joint velocity, and end position. (47) A reinforcement learning strategy network is used to calculate the optimal action strategy based on the current state information. The optimal control strategy output by the reinforcement learning strategy network is converted into a low-level control signal and sent to the dexterous hand actuator; the dexterous hand is controlled to perform fine movements and complete the expected in-palm operation task. (48) During the execution process, the real-time state of the dexterous hand and the pose of the object in the palm change in real time. The policy network updates the state observation information at each time step, dynamically adjusts and continuously outputs the control policy.

Images