Progressive motion-power alignment smart operation rapid migration method and system

Abstract

The invention provides a progressive motion-power alignment smart operation rapid migration method and system. Relates to the technical field of robot operation control. The method comprises the following steps: extracting hand-object movement and interaction information from an RGB hand-object interaction video; based on kinematics matching, mapping the hand operation track into a primary control signal of the target dexterous hand; performing dynamic optimization on the primary control signal in combination with a multi-level reward mechanism under a reinforcement learning framework, and generating a stable grabbing control strategy; a dexterous wrist control signal is synthesized according to the object target trajectory, and the semantic consistency of operation tasks is kept; and through a multi-dexterous-hand configuration adaptation mechanism, a control signal is migrated to dexterous hands of different configurations. According to the method, the human hand operation track can be automatically converted into control instructions of multiple types of dexterous hands, so that the dexterous hands have the capability of executing complex tasks such as grabbing, moving and operation, the training data acquisition cost of the dexterous hands is reduced, and the cross-equipment operation reusability and generalization capability are improved.

Description

Technical Field

[0001] The present invention relates to the field of robot operation control technology, and in particular to a method and system for rapid migration of dexterous operations with progressive motion-dynamic alignment. Background Art

[0002] As the core executor of the robot's fine manipulation tasks, the dexterous manipulator (hereinafter referred to as the "dexterous hand") usually has a bionic structure with multiple joints and high degrees of freedom, and is capable of completing complex tasks such as grasping, handling, and assembly. Its control relies on accurate kinematic and dynamic models. Traditional methods require manual teaching or offline programming to generate control instructions, but there are problems such as complex modeling and poor generalization. In recent years, data-driven learning technologies (such as imitation learning and reinforcement learning) have become the key path to breaking through bottlenecks. The core idea is to directly drive the dexterous hand through human operation data (such as videos and sensor signals), reduce dependence on accurate models, and improve task generalization capabilities; Existing smart operation migration methods are mainly divided into three categories: Wearable device solutions, such as data gloves, mixed reality devices, and motion capture systems, use sensors to capture real-time hand movements and map them to dexterous hand joints. These methods offer high accuracy but require specialized equipment, are costly, and restrict operator freedom. Visual perception solutions, such as Dexpilot, use monocular / multi-cameras to track bare hand movements and directly control the dexterous hand through visual algorithms. While this approach reduces costs, it requires synchronized operation between the human hand and the dexterous hand, which consumes a lot of manpower and is easily affected by ambient lighting. Pre-trained generative model solutions: For example, 3D Diffusion Policy synthesizes robotic hand motions through pre-trained models, but relies on high-quality dedicated datasets, has narrow scenario coverage, and is difficult to generalize to new tasks. In addition, motion redirection algorithms (such as DexMV) extract the position of human hand joints from videos and map them to dexterous hands, but lack contact state feedback and have low grasping success rates. DexTransfer combines local trajectory optimization to improve grasping stability, but random exploration may generate physically unfeasible postures. Although reinforcement learning methods can handle complex contact tasks, the high-dimensional motion space leads to low training efficiency and does not consider the continuity of post-grasp operations. Therefore, from the above description, it can be seen that the prior art has the following defects: High data acquisition costs: Wearable devices rely on expensive sensors, vision solutions require synchronous operation, and pre-trained models require dedicated datasets, all of which lead to high barriers to entry and long data acquisition cycles. Limited generalization: Action redirection lacks contact feedback, DexTransfer's random exploration easily violates physical laws, and reinforcement learning fails to incorporate operational semantics, resulting in low transfer success rates (e.g., only 6% for DexMV) and difficulty adapting to continuous interactive tasks. Loss of operational semantics: Existing methods only focus on grasping posture transfer, without considering the coordinated control of the object trajectory and wrist movement after grasping, resulting in discontinuous operational actions and incomplete transmission of task intent; In summary, there is an urgent need for a low-cost, highly generalized, and semantically-preserving dexterous operation migration method to solve key problems in existing technologies, such as data scarcity, low success rate, and incoherent operations. Summary of the Invention

[0003] In response to the above technical problems, a method and system for rapid dexterous manipulation migration with progressive motion-dynamic alignment is provided. The present invention can utilize information extracted from videos of human hand-object interactions to complete dexterous manipulation migration mapping across dexterous hands with a high success rate.

[0004] The technical means adopted in the present invention are as follows: A method for rapid transfer of dexterous manipulation with progressive motion-dynamic alignment, comprising: S1. Extract hand-object motion and interaction information from RGB hand-object interaction videos, including hand gesture sequences, object trajectories, and hand-object contact information. S2, based on kinematic matching, maps the human hand operation trajectory into the primary control signal of the target dexterous hand; S3. Under the reinforcement learning framework, the primary control signal is dynamically optimized with a multi-level reward mechanism to generate a stable grasping control strategy. S4, synthesize dexterous wrist control signals according to the target object trajectory to maintain semantic consistency of the operation task; S5. Through the multi-dexterous hand configuration adaptation mechanism, the control signal is transferred to dexterous hands of different configurations.

[0005] Furthermore, in step S1, the HFL-Net or Hold model is used to extract hand-object motion and interaction information, specifically including: S11. For known object models, HFL-Net is used to predict the object trajectory and hand pose sequence, obtaining the object's translation and rotation information at each time step and the hand's Mano parameters at each time step. The Mano model is then used to reconstruct the hand, and the hand-object contact point is determined by finding the object vertex closest to the fingertip during the operation. S12. For unknown object models, the Hold model is used to reconstruct the object's three-dimensional model and predict the object's trajectory and the human hand's posture sequence. The object's translation information and rotation information at each time step and the human hand's Mano parameters at each time step are obtained. The human hand is reconstructed using the Mano model, and the object vertex closest to the human fingertip during the operation is found to determine the hand-object contact point.

[0006] Furthermore, step S2 specifically includes: S21. Optimize the positions of the human hand and dexterous fingertips and the palm direction vectors through a nonlinear minimization method to generate anthropomorphic joint angle sequences; S22. Perform third-order derivative minimization fitting on the joint angle sequence and generate a continuous primary control signal sequence in combination with the inverse dynamics model.

[0007] Furthermore, step S3 specifically includes: S31. Use the reinforcement learning configurator to extract commonalities from the object poses, initial trajectories of the dexterous hand, and the coordinates of the fingertip contact points in different tasks, and establish a unified reinforcement learning configuration system; S32. Design a hierarchical and modular reward function system to guide the reinforcement learning process to generate a stable and coordinated control strategy; S33. Based on the PPO algorithm, residual action fine-tuning is introduced on the basis of the primary control signal to optimize the dynamic grasping stability.

[0008] Furthermore, step S4 specifically includes: S41, based on the fixed relative position of the hand and the object during stable grasping, calculating the target wrist posture through the object trajectory; S42. Use the PD controller to convert the target posture into the underlying motor control signal to keep the finger grasping posture unchanged.

[0009] Furthermore, step S5 specifically includes: S51, establishing a mapping table of finger correspondences between human hands and dexterous hands; S52. In redirection and reward calculation, only the intersection of the actual fingers of the dexterous hand and the corresponding fingers operated by the human is used.

[0010] The present invention also provides a dexterous manipulation rapid migration system based on the above-mentioned dexterous manipulation rapid migration method of progressive motion-dynamic alignment, comprising: a human hand manipulation knowledge extraction unit, a trajectory proposal unit, a grasping optimization unit, a wrist trajectory synthesis unit, and a multi-dexterous hand configuration adaptation unit, wherein: The human hand operation knowledge extraction unit extracts hand-object motion and interaction information from the RGB hand-object interaction video, including human hand posture sequence, object trajectory and hand-object contact information; The trajectory proposal unit maps the human hand operation trajectory into the primary control signal of the target dexterous hand based on kinematic matching; The grasping optimization unit dynamically optimizes the primary control signal in combination with a multi-level reward mechanism under the reinforcement learning framework to generate a stable grasping control strategy; The wrist trajectory synthesis unit synthesizes the dexterous wrist control signal according to the object target trajectory to maintain the semantic consistency of the operation task; The multi-dexterous hand configuration adaptation unit transfers the control signal to dexterous hands of different configurations through the multi-dexterous hand configuration adaptation mechanism.

[0011] Furthermore, the trajectory proposal unit includes a posture redirection module and a primary trajectory control signal generation module, wherein: The posture redirection module achieves accurate fingertip position matching of the dexterous hand and the human hand trajectory by optimizing the global position vectors of each fingertip of the human hand and the dexterous hand, as well as the palm direction vector; The primary trajectory control signal generation module is used to calculate the control signal to control the dexterous hand to reach the joint angle sequence obtained by redirection optimization.

[0012] Furthermore, the grasping optimization unit includes a reinforcement learning configurator, a hierarchical and modular reward function system and a residual strategy training module, wherein: The reinforcement learning configurator extracts commonalities from the object poses, initial trajectories of the dexterous hand, and the coordinates of the fingertip contact points in different tasks to establish a unified reinforcement learning configuration system; The hierarchical and modular reward function system is used to guide the reinforcement learning process to generate a stable and coordinated control strategy, including approach rewards, grasping rewards, and lifting rewards. Among them, the approach reward is used to motivate the fingertips of the dexterous hand to approach the target contact point as quickly as possible; the grasping reward is used to encourage the dexterous hand to contact the object in a multi-finger synchronous manner and maintain a humanoid hand shape; and the lifting reward is used to motivate the object to move to a preset target position and posture. The residual strategy training module introduces residual action fine-tuning based on the primary control signal based on the PPO algorithm to optimize dynamic grasping stability.

[0013] Compared with the prior art, the present invention has the following advantages: 1. This invention uses RGB hand-object interaction videos to extract hand posture, object trajectory, and contact information, and combines kinematic redirection with inverse dynamics calculations to generate primary control signals. This technology can transfer human hand manipulation experience to multiple types of dexterous hands without the need for wearable devices or dedicated datasets. This significantly improves the transfer success rate compared to existing technologies such as DexMV, and significantly reduces data acquisition costs and hardware dependence.

[0014] 2. The present invention introduces a reinforcement learning configurator and a residual strategy network based on the primary control signal. Through the hierarchical design of proximity rewards, grasping rewards, and lifting rewards, it dynamically optimizes the grasping posture and contact force distribution, avoiding the physical infeasibility problem caused by random exploration in the existing technology. Compared with DexTransfer, the grasping stability is significantly improved, and it supports the semantic retention of complex continuous operation tasks (such as dumping and rotation).

[0015] 3. The present invention calculates the control signal of the dexterous wrist based on the target trajectory of the object, and uses a PD controller to generate smooth motion instructions, ensuring the precise coordination of the object trajectory and wrist movement after grasping, solving the defects of loss of operation semantics and incoherent movements in the existing technology, so that the migrated dexterous hand can fully reproduce the intention and function of human hand operation (such as lifting, translation, and multi-axis tilt), and significantly improves the operation continuity and smoothness compared with PGDM.

[0016] In summary, the present invention achieves dexterous operation migration with low cost, high success rate, high generalization and operational semantic integrity, and provides a universal and efficient solution for the application of dexterous hands in complex task scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 The overall framework for the smart operation migration of the present invention.

[0019] Figure 2 The reinforcement learning configurator of the present invention is configured with an initial posture of the dexterous hand and a target posture of the object.

[0020] Figure 3 This is a finger correspondence map provided by an embodiment of the present invention.

[0021] Figure 4 This is a diagram showing the DexYCB migration effect provided by an embodiment of the present invention.

[0022] Figure 5 The migration efficiency of the method provided in the embodiment of the present invention is compared with that of the PGDM and D-grasp algorithms. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or apparatuses.

[0025] like Figure 1 As shown, the present invention provides a method for rapid migration of dexterous manipulation with progressive motion-dynamic alignment, comprising: S1. Extract hand-object motion and interaction information from RGB hand-object interaction videos, including hand gesture sequences, object trajectories, and hand-object contact information. S2, based on kinematic matching, maps the human hand operation trajectory into the primary control signal of the target dexterous hand; S3. Under the reinforcement learning framework, the primary control signal is dynamically optimized with a multi-level reward mechanism to generate a stable grasping control strategy. S4, synthesize dexterous wrist control signals according to the target object trajectory to maintain semantic consistency of the operation task; S5. Through the multi-dexterous hand configuration adaptation mechanism, the control signal is transferred to dexterous hands of different configurations.

[0026] In specific implementation, as a preferred embodiment of the present invention, in step S1, the HFL-Net or Hold model is used to extract hand-object motion and interaction information, which specifically includes: S11. For known object models, HFL-Net is used to predict the object trajectory and hand pose sequence, obtaining the object's translation and rotation information at each time step and the hand's Mano parameters at each time step. The Mano model is then used to reconstruct the hand, and the hand-object contact point is determined by finding the object vertex closest to the fingertip during the operation. S12. For unknown object models, the Hold model is used to reconstruct the object's three-dimensional model and predict the object's trajectory and the human hand's posture sequence. The object's translation information and rotation information at each time step and the human hand's Mano parameters at each time step are obtained. The human hand is reconstructed using the Mano model, and the object vertex closest to the human fingertip during the operation is found to determine the hand-object contact point.

[0027] In specific implementation, as a preferred embodiment of the present invention, step S2 specifically includes: S21. Optimize the positions of the human hand and dexterous fingertips and the palm direction vectors through a nonlinear minimization method to generate anthropomorphic joint angle sequences; In this embodiment, precise fingertip position matching of the dexterous hand and the human hand trajectory is achieved by optimizing the global position vectors of each fingertip and the palm orientation vector of the human hand and the dexterous hand. This optimization process uses a nonlinear minimization method, with the goal of minimizing the spatial position deviation and palm orientation difference between the human hand and the dexterous hand. Using the nonlinear minimization method, the retargeting objective function is as follows:

[0028] in, Indicates the number of staff The spatial vector of the root finger, Dexterity The spatial vector of the root finger, represents the weight coefficient of the fingertip position error, represents the weight coefficient of palm orientation error, represents the weight coefficient of the regularization term, Indicates the minimum geodesic distance the palm faces, Indicates the direction of the palm of the hand. Indicates the direction of the palm of the dexterous hand, represents the primary joint angle sequence; S22. Perform third-order derivative minimization fitting on the joint angle sequence and generate a continuous primary control signal sequence in combination with the inverse dynamics model.

[0029] In this embodiment, the primary joint angle sequence Perform third-order derivative minimization fitting to constrain the derivative of acceleration to be minimum, thereby suppressing motion mutations and generating a continuous trajectory function ; then Analytical derivation to obtain speed With acceleration , input inverse dynamics model Calculate the theoretical joint torque and combine it with the motor characteristics Convert to underlying control signals , and according to the sampling frequency Discretized into primary control signal sequence .

[0030] In specific implementation, as a preferred embodiment of the present invention, step S3 specifically includes: S31. Use the reinforcement learning configurator to extract commonalities from the object poses, initial trajectories of the dexterous hand, and the coordinates of the fingertip contact points in different tasks, and establish a unified reinforcement learning configuration system; In this embodiment, in the cross-task operation scenario, task specificity significantly poses a severe challenge to the unified reinforcement learning training paradigm. Based on this, the present invention designs a reinforcement learning configurator, such as Figure 2As shown in the figure, the initial posture of the dexterous hand for reinforcement learning, the target posture of the object, the number of reinforcement learning steps, and the target hand posture are configured. Specifically, the frame with the smallest Euclidean distance between the thumb tip and the object grasping point in the entire primary trajectory is determined, and the corresponding dexterous hand posture is used as the initial state of reinforcement learning; the target position of the object is the posture when the object deviates from its original position for the first time by 0.1m during the human hand operation. ; Record the time it takes for the hand to move from pre-grasping to lifting the object to a displacement of 0.1m , combined with the simulation step size of the simulator , and converted into simulation steps, As reinforcement learning step configuration; extract primary joint angle sequence The joint angles from the pre-grab time step Time to complete crawling Joint angles in interval , which is used to guide the grasping posture of the dexterous hand in the subsequent reinforcement learning process.

[0031] S32. Design a hierarchical and modular reward function system to guide the reinforcement learning process to generate a stable and coordinated control strategy; In this embodiment, the hierarchical and modular reward function system includes approach rewards, grab rewards, and lift rewards, among which: The proximity reward is used to motivate the fingertips of the dexterous hand to approach the target contact point as quickly as possible. The function expression is as follows:

[0032] in, represents the number of fingers used in the task, Indicates the The Euclidean distance between a fingertip and its corresponding target contact point, Indicates the historical minimum distance in the current training cycle, represents the proximity reward weight. This reward can encourage independent optimization of each fingertip while maintaining target dependency, thereby enhancing the geometric generalization ability of the task.

[0033] Grasping rewards are used to encourage the dexterous hand to contact objects with multiple fingers in a synchronous manner and maintain a humanoid hand shape; all fingertips and corresponding grasping points are within the contact tolerance range. , the activation crawl reward is as follows:

[0034] in, represents the grab reward weight, Indicates the number of hand-object contact points, encouraging multiple hand-object contact. Indicates the current joint angle Target joint angle He constrains the finger postures that are not used in the current task. represents the corresponding weighting coefficient; The lift reward is used to encourage the object to move to the preset target position and posture. The function expression is as follows:

[0035] in, Represents the weight of the lifting reward, encouraging the object to move from its current position Reach the target pose set by the reinforcement learning configurator .

[0036] S33. Based on the PPO algorithm, residual action fine-tuning is introduced on the basis of the primary control signal to optimize the dynamic grasping stability.

[0037] In this embodiment, in order to achieve efficient optimization of the primary control trajectory, the present invention adopts a strategy network architecture based on residual learning. Based on this, residual action is introduced , so that the final control signal is: Based on the existing reasonable initial behavior, fine-tuning and optimization are performed to ensure that the trajectory generated by the redirection module satisfies the kinematic constraints while guiding the strategy to adaptively compensate for physical dynamic errors. The strategy is trained using the PPO algorithm, which uses a [128, 128] multi-layer feedforward neural network. The parameters in the PPO algorithm are set to clipping = 0.3 and discount factor = 0.995.

[0038] In specific implementation, as a preferred embodiment of the present invention, step S4 specifically includes: S41, based on the fixed relative position of the hand and the object during stable grasping, calculating the target wrist posture through the object trajectory; In this embodiment, the posture of the dexterous wrist during stable grasping is , the object trajectory in the operation phase is , then under the condition that the relative position of hand and object is stable, at each moment Target wrist posture It can be calculated by the following formula: .

[0039] S42. Use the PD controller to convert the target posture into the underlying motor control signal to keep the finger grasping posture unchanged.

[0040] In this embodiment, in order to ensure smooth execution and dynamic response of the wrist trajectory, a servo module based on a PD controller is used to adjust the target posture Convert to low-level motor control signals , generating wrist control commands, and the finger joints maintain the grasping posture unchanged during this process.

[0041] In specific implementation, as a preferred embodiment of the present invention, step S5 specifically includes: S51, establishing a mapping table of finger correspondences between human hands and dexterous hands; In this embodiment, in order to make the transfer method applicable to a variety of dexterous hands, the following Figure 3 The finger mapping diagram shown. The fingertip information used in fingertip redirection and reward calculation is given by the actual fingers used by the human hand and this mapping table.

[0042] S52. In redirection and reward calculation, only the intersection of the actual fingers of the dexterous hand and the corresponding fingers operated by the human is used.

[0043] In this embodiment, during the redirection process, for dexterous hands with fewer than five fingertips, only the actual fingers of the dexterous hand are used to match the corresponding fingers of the human hand. During the reward calculation process, the fingers actually involved in the reward calculation are the intersection of the actual fingers of the manipulator and the fingers used by the human operator.

[0044] The present invention also provides a dexterous manipulation rapid migration system based on the above-mentioned dexterous manipulation rapid migration method of progressive motion-dynamic alignment, comprising: a human hand manipulation knowledge extraction unit, a trajectory proposal unit, a grasping optimization unit, a wrist trajectory synthesis unit, and a multi-dexterous hand configuration adaptation unit, wherein: The human hand operation knowledge extraction unit extracts hand-object motion and interaction information from the RGB hand-object interaction video, including human hand posture sequence, object trajectory and hand-object contact information; The trajectory proposal unit maps the human hand operation trajectory into the primary control signal of the target dexterous hand based on kinematic matching; The grasping optimization unit dynamically optimizes the primary control signal in combination with a multi-level reward mechanism under the reinforcement learning framework to generate a stable grasping control strategy; The wrist trajectory synthesis unit synthesizes the dexterous wrist control signal according to the target trajectory of the object to maintain the semantic consistency of the operation task. In this embodiment, this module is designed to enable the dexterous hand to perform functional actions such as dumping, lifting, rotating, etc. along the object trajectory reflected by the human hand operation after completing the grasping, ensuring that the migration behavior can realize the transmission of the operation intention.

[0045] The multi-dexterous hand configuration adaptation unit transfers the control signal to dexterous hands of different configurations through the multi-dexterous hand configuration adaptation mechanism.

[0046] In specific implementation, as a preferred embodiment of the present invention, the trajectory proposal unit includes a posture redirection module and a primary trajectory control signal generation module, wherein: The posture redirection module achieves accurate fingertip position matching of the dexterous hand and the human hand trajectory by optimizing the global position vectors of each fingertip of the human hand and the dexterous hand, as well as the palm direction vector; The primary trajectory control signal generation module is used to calculate the control signal to control the dexterous hand to reach the joint angle sequence obtained by redirection optimization.

[0047] In specific implementation, as a preferred embodiment of the present invention, the grasping optimization unit includes a reinforcement learning configurator, a hierarchical and modular reward function system and a residual strategy training module, wherein: The reinforcement learning configurator extracts commonalities from the object poses, initial trajectories of the dexterous hand, and the coordinates of the fingertip contact points in different tasks to establish a unified reinforcement learning configuration system; The hierarchical and modular reward function system is used to guide the reinforcement learning process to generate a stable and coordinated control strategy, including approach rewards, grasping rewards, and lifting rewards. Among them, the approach reward is used to motivate the fingertips of the dexterous hand to approach the target contact point as quickly as possible; the grasping reward is used to encourage the dexterous hand to contact the object in a multi-finger synchronous manner and maintain a humanoid hand shape; and the lifting reward is used to motivate the object to move to a preset target position and posture. The residual strategy training module introduces residual action fine-tuning based on the primary control signal based on the PPO algorithm to optimize dynamic grasping stability.

[0048] Example The specific implementation details are introduced by introducing the process of migrating tasks in the DexYCB dataset to dexterous hand operation sequences as follows: DexYCB stores data as image sequences and provides object models. First, HFL-Net is used to predict the image sequence, obtaining the object's translation and rotation information at each time step, as well as the Mano parameters of the hand at each time step. The Mano model is then used to reconstruct the hand and locate the object vertex closest to the fingertips during manipulation to determine the hand-object contact point.

[0049] Then, the extracted fingertip positions and wrist orientations will be fed into the posture redirection module frame by frame for optimization, and a dexterous hand joint sequence matching the human hand kinematics will be obtained. The optimized target joint angles are fed into the primary trajectory control signal generation module, which calculates the joint control signals that drive the dexterous hand to move according to the target joint angle sequence. This module is implemented based on the inverse dynamics functions provided by the MuJoCo simulator. Subsequently, an operating environment is built for each task based on the MuJoCo simulator, with the initial positions of the object and the dexterous hand consistent with the data in the dataset. Controlled by the joint control signals, the dexterous hand can achieve an anthropomorphic primary trajectory within the MuJoCo simulation environment.

[0050] To compensate for grasp failures caused by a lack of dynamic information, the primary trajectory is fed into the grasp optimization module for optimization. As the dexterous hand executes the primary trajectory, the reinforcement learning configurator monitors the distance between the thumb tip and the thumb grasp point on the object and selects the state with the closest distance, without collision, as the pre-grasp pose. The target pose and number of environment iterations for reinforcement learning are calculated based on the pose of the object when it first deviates 0.1m from its original position during the grasp and the corresponding human manipulation. The DexYCB dataset records data at 30Hz, while the MuJoCo simulator is set to run at 120Hz. Therefore, for the DexYCB dataset, the number of environment iterations for reinforcement learning is four times the number of human manipulation time steps. Based on this configuration information, this method uses reinforcement learning to train a residual policy, fine-tune the primary control signal, and establish a stable grasping behavior based on the primary trajectory. Through a unified reward design, combined with the PPO algorithm, rapid reinforcement learning convergence is achieved. In implementation, the weights of the approach reward, grasp reward, and lift reward are set to 10, 10, and 20, respectively.

[0051] The grasping optimization module can only achieve stable grasping of objects, and the expression of human operation semantics still depends on the wrist trajectory. In order to further realize the wrist trajectory control of the dexterous hand while retaining the semantic information of the operation. The present invention uses the characteristic that the relative position of the hand and the object is fixed during stable grasping, calculates the target wrist trajectory based on the true value of the object trajectory extracted from DexYCB, and uses the PD controller to generate wrist control instructions to control the dexterous hand to move along the trajectory. During this process, the finger joints maintain the grasping posture unchanged, thereby completing the operation migration of the DexYCB dataset. The migration effect is as follows Figure 4 shown.

[0052] To effectively measure the migration effect, this method uses the following evaluation indicators to evaluate the migration success rate and migration efficiency respectively: Migration success rate: By real-time monitoring of the object's displacement, rotation, and speed, we analyze the key motion feature sequences of the object during the operation (such as stationary, lifting, translation, and multi-axis tilt). If the object's trajectory motion features maintain the same temporal sequence before and after the migration, the two motion patterns are considered matched, indicating a successful migration.

[0053] Transfer efficiency: The number of environment iterations required for successful training convergence.

[0054] The performance of the present invention is compared with the excellent results in recent years, including DexMV, PGDM, and D-grasp. Table 1 shows the migration success rates of different methods. The present invention has a faster iteration rate while having good migration effect and success rate. Figure 5 shown.

[0055] Table 1. Crawl success rates of different methods

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for rapid migration of dexterous manipulation with progressive motion-dynamic alignment, characterized in that: include: S1. Extract hand-object motion and interaction information from RGB hand-object interaction videos, including hand gesture sequences, object trajectories, and hand-object contact information. S2, based on kinematic matching, maps the human hand operation trajectory into the primary control signal of the target dexterous hand; S3. Under the reinforcement learning framework, the primary control signal is dynamically optimized with a multi-level reward mechanism to generate a stable grasping control strategy. S4, synthesize dexterous wrist control signals according to the target object trajectory to maintain semantic consistency of the operation task; S5. Through the multi-dexterous hand configuration adaptation mechanism, the control signal is transferred to dexterous hands of different configurations.

2. The method for rapid migration of dexterous manipulation with progressive motion-dynamic alignment according to claim 1, characterized in that: In step S1, the HFL-Net or Hold model is used to extract hand-object motion and interaction information, specifically including: S11. For known object models, HFL-Net is used to predict the object trajectory and hand pose sequence, obtaining the object's translation and rotation information at each time step and the hand's Mano parameters at each time step. The Mano model is then used to reconstruct the hand, and the hand-object contact point is determined by finding the object vertex closest to the fingertip during the operation. S12. For unknown object models, the Hold model is used to reconstruct the object's three-dimensional model and predict the object's trajectory and the human hand's posture sequence. The object's translation information and rotation information at each time step and the human hand's Mano parameters at each time step are obtained. The human hand is reconstructed using the Mano model, and the object vertex closest to the human fingertip during the operation is found to determine the hand-object contact point.

3. The method for rapid migration of dexterous manipulation with progressive motion-dynamic alignment according to claim 1, characterized in that: Step S2 specifically includes: S21. Optimize the positions of the human hand and dexterous fingertips and the palm direction vectors through a nonlinear minimization method to generate anthropomorphic joint angle sequences; S22. Perform third-order derivative minimization fitting on the joint angle sequence and generate a continuous primary control signal sequence in combination with the inverse dynamics model.

4. The method for rapid migration of dexterous manipulation with progressive motion-dynamic alignment according to claim 1, characterized in that: Step S3 specifically includes: S31. Use the reinforcement learning configurator to extract commonalities from the object poses, initial trajectories of the dexterous hand, and the coordinates of the fingertip contact points in different tasks, and establish a unified reinforcement learning configuration system; S32. Design a hierarchical and modular reward function system to guide the reinforcement learning process to generate a stable and coordinated control strategy; S33. Based on the PPO algorithm, residual action fine-tuning is introduced on the basis of the primary control signal to optimize the dynamic grasping stability.

5. The method for rapid migration of dexterous manipulation with progressive motion-dynamic alignment according to claim 1, characterized in that: Step S4 specifically includes: S41, based on the fixed relative position of the hand and the object during stable grasping, calculating the target wrist posture through the object trajectory; S42. Use the PD controller to convert the target posture into the underlying motor control signal to keep the finger grasping posture unchanged.

6. The method for rapid migration of dexterous manipulation with progressive motion-dynamic alignment according to claim 1, characterized in that: Step S5 specifically includes: S51, establishing a mapping table of finger correspondences between human hands and dexterous hands; S52. In redirection and reward calculation, only the intersection of the actual fingers of the dexterous hand and the corresponding fingers operated by the human is used.

7. A system for rapid migration of dexterous manipulation with gradual motion-dynamic alignment implemented by the method for rapid migration of dexterous manipulation with gradual motion-dynamic alignment as described in any one of claims 1 to 6, characterized in that: include: Human hand operation knowledge extraction unit, trajectory proposal unit, grasping optimization unit, wrist trajectory synthesis unit and multi-dexterous hand configuration adaptation unit, including: The human hand operation knowledge extraction unit extracts hand-object motion and interaction information from the RGB hand-object interaction video, including human hand posture sequence, object trajectory and hand-object contact information; The trajectory proposal unit maps the human hand operation trajectory into the primary control signal of the target dexterous hand based on kinematic matching; The grasping optimization unit dynamically optimizes the primary control signal in combination with a multi-level reward mechanism under the reinforcement learning framework to generate a stable grasping control strategy; The wrist trajectory synthesis unit synthesizes the dexterous wrist control signal according to the object target trajectory to maintain the semantic consistency of the operation task; The multi-dexterous hand configuration adaptation unit transfers the control signal to dexterous hands of different configurations through the multi-dexterous hand configuration adaptation mechanism.

8. The progressive motion-dynamic alignment dexterous manipulation rapid migration system according to claim 7, characterized in that: The trajectory proposal unit includes a posture redirection module and a primary trajectory control signal generation module, wherein: The posture redirection module achieves accurate fingertip position matching of the dexterous hand and the human hand trajectory by optimizing the global position vectors of each fingertip of the human hand and the dexterous hand, as well as the palm direction vector; The primary trajectory control signal generation module is used to calculate the control signal to control the dexterous hand to reach the joint angle sequence obtained by redirection optimization.

9. The progressive motion-dynamic alignment dexterous manipulation rapid migration system according to claim 7, characterized in that: The grasping optimization unit includes a reinforcement learning configurator, a hierarchical and modular reward function system, and a residual strategy training module, wherein: The reinforcement learning configurator extracts commonalities from the object poses, initial trajectories of the dexterous hand, and the coordinates of the fingertip contact points in different tasks to establish a unified reinforcement learning configuration system; The hierarchical and modular reward function system is used to guide the reinforcement learning process to generate a stable and coordinated control strategy, including approach rewards, grasping rewards, and lifting rewards. Among them, the approach reward is used to motivate the fingertips of the dexterous hand to approach the target contact point as quickly as possible; the grasping reward is used to encourage the dexterous hand to contact the object in a multi-finger synchronous manner and maintain a humanoid hand shape; and the lifting reward is used to motivate the object to move to a preset target position and posture. The residual strategy training module introduces residual action fine-tuning based on the primary control signal based on the PPO algorithm to optimize dynamic grasping stability.

Images