A progressive motion-dynamic alignment dexterous manipulation fast transfer method and system

Abstract

The application provides a kind of progressive motion-power alignment dexterous operation fast migration method and system. It is related to the technical field of robot operation control, the method of the application comprises: extracting hand-object motion and interaction information from RGB hand-object interaction video;Based on kinematic matching, the human hand operation trajectory is mapped to the primary control signal of the target dexterous hand;Under the reinforcement learning framework, the primary control signal is dynamically optimized in combination with a multi-level reward mechanism to generate a stable grasping control strategy;According to the object target trajectory, the dexterous hand wrist control signal is synthesized to maintain the consistency of the operation task semantics;Through multi-dexterous hand configuration adaptation mechanism, the control signal is migrated to different configurations of dexterous hand.The application can automatically convert human hand operation trajectory into control instructions of multiple types of dexterous hand, enabling it to perform complex tasks such as grasping, moving and operating, thereby reducing the cost of acquiring training data for dexterous hand and improving the operation reusability and generalization ability across devices.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot operation control, in particular, especially relates to a progressive motion-power alignment dexterous operation fast transfer method and system. BACKGROUND

[0002] Dexterous manipulator (hereinafter referred to as "dexterous hand") as the core executor of robot fine operation task, usually has multi-joint, high degree of freedom bionic structure, can complete complex tasks such as grasping, carrying, assembling. Its control depends on accurate kinematics and dynamics model, traditional method needs to generate control instruction through artificial demonstration or offline programming, but there are problems such as complex modeling and poor generalization. In recent years, learning technology based on data driving (such as imitation learning, reinforcement learning) has become a key path to break through the bottleneck, its core idea is to directly drive the dexterous hand through human operation data (such as video, sensor signal), reduce the dependence on accurate model, and improve the task generalization ability;

[0003] The existing dexterous operation transfer method is mainly divided into three categories:

[0004] Wearable device scheme: such as data gloves, mixed reality devices, motion capture systems, which collect human hand actions in real time through sensors and map them to dexterous hand joints. This method has high precision but needs special equipment, which is costly and limits the freedom of the operator;

[0005] Visual perception scheme: such as Dexpilot uses monocular / multi-view camera to track bare hand actions, and directly controls the dexterous hand through visual algorithm. Although the cost is reduced, the human hand and the dexterous hand need to be operated synchronously, which consumes a lot of manpower and is easily disturbed by environmental light;

[0006] Pre-trained generative model scheme: such as 3D Diffusion Policy synthesizes manipulator actions through pre-trained model, but depends on high-quality special data set, which has narrow scene coverage and is difficult to generalize to new tasks. In addition, the action reorientation algorithm (such as DexMV) extracts the joint position of the human hand from the video and maps it to the dexterous hand, but lacks contact state feedback, and the grasping success rate is low; DexTransfer combines local trajectory optimization to improve grasping stability, but random exploration may generate physically infeasible poses; although the reinforcement learning method can handle complex contact tasks, the high-dimensional action space leads to low training efficiency, and the continuity of post-grasping operation is not considered;

[0007] Therefore, from the above description, it can be seen that the existing technology has the following defects:

[0008] High cost of data acquisition: wearable devices rely on expensive sensors, visual schemes require synchronous operation, and pre-trained models require special data sets, all of which result in high data acquisition threshold and long cycle;

[0009] Limited generalization ability: action reorientation lacks contact feedback, DexTransfer's random exploration violates physical laws, and reinforcement learning does not integrate operation semantics, resulting in low transfer success rate (e.g., DexMV only 6%) and difficulty adapting to continuous interaction tasks;

[0010] Loss of operation semantics: existing methods only focus on grasp pose transfer and do not consider the coordinated control of object trajectory and wrist motion after grasping, resulting in discontinuous operation and incomplete task intent transfer;

[0011] In summary, there is an urgent need for a low-cost, high-generalization, and operation semantics-preserving dexterous operation transfer method to address the key issues of data scarcity, low success rate, and discontinuous operation in existing technologies. SUMMARY

[0012] According to the above technical problems, a progressive motion-power alignment dexterous operation rapid transfer method and system are provided. The present application can utilize the information extracted from human hand-object interaction videos to complete cross-dexterous hand dexterous operation transfer mapping with high success rate.

[0013] The technical means adopted by the present application are as follows:

[0014] A progressive motion-power alignment dexterous operation rapid transfer method, comprising:

[0015] S1, extracting hand-object motion and interaction information from RGB hand-object interaction videos, including hand pose sequence, object trajectory, and hand-object contact information;

[0016] S2, based on kinematic matching, mapping the human hand operation trajectory to the primary control signal of the target dexterous hand;

[0017] S3, under the reinforcement learning framework, combining a multi-level reward mechanism to dynamically optimize the primary control signal and generate a stable grasping control strategy;

[0018] S4, synthesizing the wrist control signal of the dexterous hand according to the object target trajectory to maintain operation task semantic consistency;

[0019] S5, through a multi-dexterous hand configuration adaptation mechanism, transferring the control signal to dexterous hands of different configurations.

[0020] Further, in step S1, HFL-Net or Hold model is used to extract hand-object motion and interaction information, specifically including:

[0021] S11, for the case of known object model, the object trajectory and hand pose sequence are predicted by HFL-Net, the translation information of the object at each time step, the rotation information and the Mano parameters of the hand at each time step are obtained, the hand is reconstructed by using the Mano model, and the hand-object contact point is determined by finding the object vertex closest to the hand fingertip during the operation process;

[0022] S12, for the case of unknown object model, the object three-dimensional model is reconstructed by Hold model, and the object trajectory and hand pose sequence are predicted, the translation information of the object at each time step, the rotation information and the Mano parameters of the hand at each time step are obtained, the hand is reconstructed by using the Mano model, and the hand-object contact point is determined by finding the object vertex closest to the hand fingertip during the operation process.

[0023] Further, step S2 specifically comprises:

[0024] S21, the humanoid joint angle sequence is generated by optimizing the hand and the dexterous hand fingertip position and the palm direction vector through a nonlinear minimization method;

[0025] S22, the joint angle sequence is fitted by third derivative minimization, and the continuous primary control signal sequence is generated in combination with the inverse dynamics model.

[0026] Further, step S3 specifically comprises:

[0027] S31, a unified reinforcement learning configuration system is established by extracting commonalities from object poses, dexterous hand initial trajectories and fingertip contact point coordinates in different tasks by using a reinforcement learning configurator;

[0028] S32, a hierarchical and modular reward function system is designed to guide the reinforcement learning process to generate stable and coordinated control strategies;

[0029] S33, residual action fine-tuning is introduced based on the PPO algorithm on the basis of the primary control signal to optimize the dynamic grasping stability.

[0030] Further, step S4 specifically comprises:

[0031] S41, based on the fixed relative position between the hand and the object during stable grasping, the target wrist posture is calculated by the object trajectory;

[0032] S42, a PD controller is used to convert the target posture into bottom-level motor control signals to keep the finger grasping posture unchanged.

[0033] Further, step S5 specifically comprises:

[0034] S51, a finger mapping table of the hand and the dexterous hand is established;

[0035] S52、In the redirection and reward calculation, only the intersection of the fingers actually existing in the dexterous hand and the corresponding fingers operated by the human is used.

[0036] The application also provides a dexterous operation fast migration system based on the progressive motion-power alignment and implemented by the dexterous operation fast migration method of the progressive motion-power alignment.

[0037] The human hand operation knowledge extraction unit extracts hand-object motion and interaction information from the RGB hand-object interaction video, including a human hand posture sequence, an object trajectory and hand-object contact information.

[0038] The trajectory proposal unit maps the human hand operation trajectory to the primary control signal of the target dexterous hand based on kinematic matching.

[0039] The grasp optimization unit dynamically optimizes the primary control signal under the reinforcement learning framework and in combination with a multi-level reward mechanism to generate a stable grasp control strategy.

[0040] The wrist trajectory synthesis unit synthesizes the dexterous hand wrist control signal according to the object target trajectory to maintain operation task semantic consistency.

[0041] The multi-dexterous hand configuration adaptation unit migrates the control signal to dexterous hands of different configurations through a multi-dexterous hand configuration adaptation mechanism.

[0042] Further, the trajectory proposal unit includes a posture redirection module and a primary trajectory control signal generation module.

[0043] The posture redirection module realizes accurate fingertip position matching of the dexterous hand and the human hand trajectory by optimizing the global position vector of each fingertip of the human hand and the dexterous hand and the palm direction vector.

[0044] The primary trajectory control signal generation module is configured to calculate the control signal to control the dexterous hand to reach the joint angle sequence obtained by the redirection optimization.

[0045] Further, the grasp optimization unit includes a reinforcement learning configurator, a hierarchical and modular reward function system and a residual strategy training module.

[0046] The reinforcement learning configurator extracts commonalities from object poses, initial trajectories of the dexterous hand and fingertip contact point coordinates in different tasks to establish a unified reinforcement learning configuration system.

[0047] The hierarchical and modular reward function system is used to guide the reinforcement learning process to generate a stable and coordinated control strategy, and includes an approaching reward, a grasping reward and a lifting reward, wherein the approaching reward is used to encourage each fingertip of the dexterous hand to approach the target contact point as soon as possible, the grasping reward is used to encourage the dexterous hand to contact the object in a multi-finger synchronous manner and maintain a human-like hand shape, and the lifting reward is used to encourage the object to move to a preset target position and attitude.

[0048] The residual policy training module introduces residual action fine-tuning based on the PPO algorithm on the basis of the primary control signal, and optimizes the dynamic grasping stability.

[0049] Compared with the prior art, the present application has the following advantages:

[0050] 1. The present application extracts human hand posture, object trajectory and contact information from RGB hand-object interaction video, combines kinematic reorientation and inverse dynamics calculation to generate primary control signals, and does not need to wear a device or a special data set, so as to transfer human hand operation experience to multiple types of dexterous hands, significantly improve the migration success rate of the prior art such as DexMV, and significantly reduce data acquisition cost and hardware dependence.

[0051] 2. The present application introduces a reinforcement learning configurator and a residual policy network on the basis of the primary control signal, dynamically optimizes the grasping posture and contact force distribution through the hierarchical design of the approaching reward, the grasping reward and the lifting reward, avoids the physically infeasible problem caused by random exploration in the prior art, significantly improves the grasping stability of DexTransfer, and supports semantic preservation of complex continuous operation tasks (such as pouring and rotating).

[0052] 3. The present application calculates the wrist control signal of the dexterous hand based on the object target trajectory, generates smooth motion instructions using a PD controller, ensures the accurate cooperation of the object trajectory and wrist action after grasping, solves the defects of operation semantic loss and incoherent motion in the prior art, enables the dexterous hand after migration to completely reproduce the intention and function (such as lifting, translation and multi-axis tilting) of human hand operation, and significantly improves the operation continuity and smoothness of PGDM.

[0053] In summary, the present application realizes low-cost, high-success-rate, high-generalization and operation semantic integrity of dexterous operation migration, and provides a universal and efficient solution for dexterous hand application in complex task scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to make the technical scheme of the present application or the prior art clearer, the accompanying drawings needed in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0055] Figure 1 The present application provides a dexterous operation migration framework.

[0056] Figure 2 The present application provides a dexterous hand initial pose and object target pose configuration for the reinforcement learning configurator.

[0057] Figure 3 The present application provides a finger correspondence graph.

[0058] Figure 4 The present application provides a DexYCB migration effect display graph.

[0059] Figure 5 The present application provides a comparison of the migration efficiency of the method of the present application and PGDM and D-grasp algorithm. DETAILED DESCRIPTION

[0060] In order to make the technical scheme of the present application or the prior art clearer, the accompanying drawings needed in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort.

[0061] It should be noted that the terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above-mentioned accompanying drawings are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device containing a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0062] As shown in Figure 1 The present application provides a progressive motion-dynamic alignment dexterous operation rapid migration method, which comprises:

[0063] S1, extracting hand-object motion and interaction information from an RGB hand-object interaction video, including hand pose sequence, object trajectory and hand-object contact information;

[0064] S2, mapping the human hand operation trajectory to the primary control signal of the target dexterous hand based on kinematic matching;

[0065] S3, under the reinforcement learning framework, dynamically optimizing the primary control signal in combination with a multi-level reward mechanism to generate a stable grasping control strategy;

[0066] S4, synthesizing the dexterous hand wrist control signal according to the object target trajectory to maintain operation task semantic consistency;

[0067] S5, migrating the control signal to dexterous hands of different configurations through a multi-dexterous hand configuration adaptation mechanism.

[0068] In specific implementation, as a preferred embodiment of the present application, in step S1, the HFL-Net or Hold model is used to extract hand-object motion and interaction information, which specifically includes:

[0069] S11, for the case of known object model, the HFL-Net is used to predict the object trajectory and human hand pose sequence, to obtain the translation information of the object at each time step, the rotation information and the Mano parameters of the human hand at each time step, to reconstruct the human hand by using the Mano model, and to find the object vertex closest to the human finger tip in the operation process to determine the hand-object contact point;

[0070] S12, for the case of unknown object model, the Hold model is used to reconstruct the object three-dimensional model and predict the object trajectory and human hand pose sequence, to obtain the translation information of the object at each time step, the rotation information and the Mano parameters of the human hand at each time step, to reconstruct the human hand by using the Mano model, and to find the object vertex closest to the human finger tip in the operation process to determine the hand-object contact point.

[0071] In specific implementation, as a preferred embodiment of the present application, step S2 specifically includes:

[0072] S21, by using a nonlinear minimization method, optimizing the global position vector of each finger tip of the human hand and the dexterous hand, and the palm direction vector, to generate the humanoid joint angle sequence;

[0073] In the present embodiment, by optimizing the global position vector of each finger tip of the human hand and the dexterous hand, and the palm direction vector, the accurate finger tip position matching of the dexterous hand and the human hand trajectory is realized. The optimization process adopts a nonlinear minimization method, and the target is to minimize the spatial position deviation and palm direction difference between the human hand and the dexterous hand finger tip. By using the nonlinear minimization method, the target function is reoriented as follows:

[0074]

[0075] wherein, represents the spatial vector of the first finger of the human hand, Indicating dexterity The spatial vector of the root finger, The weighting coefficients represent the fingertip position error. The weighting coefficient represents the error in palm orientation. This represents the weight coefficient of the regularization term. This indicates the minimum geodesic distance when the palm is facing. Indicates the direction the palm of a person's hand is facing. The direction the palm of a dexterous hand faces. Represents the primary joint angle sequence;

[0076] S22. Perform third-order derivative minimization fitting on the joint angle sequence, and generate a continuous primary control signal sequence by combining it with the inverse dynamics model.

[0077] In this embodiment, the primary joint angle sequence A third-order derivative minimization fitting is performed to minimize the derivative of the constrained acceleration, thereby suppressing abrupt changes in motion and generating a continuous trajectory function. ; then on Speed ​​of analytical differentiation With acceleration Input inverse dynamics model Calculate the theoretical joint torque and combine it with the motor characteristics. Converted to low-level control signals and according to sampling frequency Discretize into a primary control signal sequence .

[0078] In a specific implementation, as a preferred embodiment of the present invention, step S3 specifically includes:

[0079] S31. Use a reinforcement learning configurator to extract commonalities from object poses, dexterous hand initial trajectories, and fingertip contact point coordinates in different tasks, and establish a unified reinforcement learning configurator system.

[0080] In this embodiment, the significant task specificity in cross-task operation scenarios poses a severe challenge to the unified reinforcement learning training paradigm. Based on this, the present invention designs a reinforcement learning configurator, such as... Figure 2 As shown, the initial pose of the dexterous hand, the target pose of the object, the number of reinforcement learning steps, and the target hand pose are configured for reinforcement learning. Specifically, the frame with the smallest Euclidean distance between the thumb tip and the object grasping point in the entire primary trajectory is determined, and its corresponding dexterous hand pose is used as the initial state for reinforcement learning; the target object position is taken as the pose when the object first deviates from its original position by 0.1m during the hand operation. Record the time from when the hand pre-grabs the object to when the object is lifted and shifted by 0.1m. , combined with the simulation step length of the simulator , and converted into simulation steps, as the reinforcement learning step number configuration; extract the primary joint angle sequence from the pre-grasp time step joint angle to the completion of the grasp time interval joint angle , used to guide the grasp posture of the dexterous hand in the subsequent reinforcement learning process.

[0081] S32, design a hierarchical and modular reward function system to guide the reinforcement learning process to generate a stable and coordinated control strategy;

[0082] In this embodiment, the hierarchical and modular reward function system includes proximity reward, grasp reward and lifting reward, wherein:

[0083] The proximity reward is used to encourage each fingertip of the dexterous hand to approach the target contact point as soon as possible, and the function expression is as follows:

[0084]

[0085] wherein, represents the number of fingers used in the task, represents the Euclidean distance between the th fingertip and its corresponding target contact point, represents the historical minimum distance in the current training period, represents the proximity reward weight. This reward can encourage independent optimization of each fingertip while maintaining target dependence, to enhance the geometric generalization ability of the task.

[0086] The grasp reward is used to encourage the dexterous hand to contact the object in a multi-finger synchronous manner and maintain a human-like hand shape; when all fingertips and corresponding grasp points enter the contact tolerance range , the grasp reward is activated as follows:

[0087]

[0088] wherein, represents the grasp reward weight, represents the number of hand-object contact points, encouraging multi-point contact between the hand and the object, represents the cosine similarity between the current joint angle and the target joint angle , which constrains the posture of the fingers not used in the current task. represents the corresponding weighting coefficient;

[0089] The lifting reward is used to encourage the object to move to the preset target position and posture, and the function expression is as follows:

[0090]

[0091] wherein, represents the weight of the lifting reward, encouraging the object to reach the target pose set by the reinforcement learning configurator from the current pose . .

[0092] S33, introducing residual action fine-tuning based on the primary control signal based on the PPO algorithm, optimizing dynamic grasping stability.

[0093] In the embodiment, in order to realize efficient optimization of the primary control trajectory, the application adopts a policy network architecture based on residual learning. On the basis of the primary control signal provided by the existing redirection module , residual action is introduced, so that the final control signal is: , which is fine-tuned and optimized on the basis of the existing reasonable initial behavior, not only retains the motion trajectory generated by the redirection module to meet the kinematic constraints, but also guides the policy to adaptively compensate for physical dynamic errors. The policy training adopts the PPO algorithm, and the network structure is a multilayer feedforward neural network of [128, 128]. The parameter setting in the PPO algorithm is clipping=0.3, discountfactor=0.995.

[0094] In specific implementation, as a preferred embodiment of the application, step S4 specifically comprises:

[0095] S41, based on the fixed relative position of the hand and the object during stable grasping, the target wrist pose is calculated through the object trajectory;

[0096] In the embodiment, the pose of the wrist of the dexterous hand during stable grasping is , the object trajectory during the operation stage is , and under the condition that the relative position of the hand and the object is stable, the target wrist pose of each time can be calculated by the following formula:

[0097] .

[0098] S42, using a PD controller to convert the target pose into a bottom-layer motor control signal, keeping the finger grasping pose unchanged.

[0099] In the embodiment, in order to ensure the smooth execution and dynamic response of the wrist trajectory, a servo module based on a PD controller is used to convert the target pose into a bottom-layer motor control signal , generating a wrist control instruction, and the finger joints keep the grasping pose unchanged in this process.

[0100] In a specific implementation, as a preferred embodiment of the present application, step S5 specifically includes:

[0101] S51, establishing a hand-finger mapping table of the human hand and the dexterous hand;

[0102] In this embodiment, in order to make the migration method applicable to various dexterous hands, a hand-finger mapping table as shown in the figure is established. The finger information used in the finger redirection and reward calculation is given by the actual fingers used by the human hand operation and the mapping table. Figure 3

[0103] S52, in the redirection and reward calculation, only the intersection of the fingers actually existing in the dexterous hand and the corresponding fingers of the human hand is used.

[0104] In this embodiment, in the redirection process, for the dexterous hand with less than 5 fingers, only the matching optimization of the fingers actually existing in the dexterous hand and the corresponding fingers of the human hand is used. In the reward calculation process, the fingers actually participating in the reward calculation are the intersection of the fingers actually existing in the dexterous hand and the fingers used by the human hand operation.

[0105] The present application also provides a progressive motion-power alignment dexterous operation fast migration system implemented by the progressive motion-power alignment dexterous operation fast migration method described above, comprising a human hand operation knowledge extraction unit, a trajectory proposal unit, a grasping optimization unit, a wrist trajectory synthesis unit and a multi-dexterous hand configuration adaptation unit, wherein:

[0106] The human hand operation knowledge extraction unit extracts hand-object motion and interaction information from an RGB hand-object interaction video, including a human hand posture sequence, an object trajectory and hand-object contact information.

[0107] The trajectory proposal unit maps the human hand operation trajectory to the primary control signal of the target dexterous hand based on kinematic matching.

[0108] The grasping optimization unit dynamically optimizes the primary control signal under the reinforcement learning framework combined with a multi-level reward mechanism to generate a stable grasping control strategy.

[0109] The wrist trajectory synthesis unit synthesizes the dexterous hand wrist control signal according to the object target trajectory to maintain the consistency of the operation task semantics. In this embodiment, this module aims to make the dexterous hand perform functional actions such as pouring, lifting, rotating, etc. along the object trajectory embodied by the human hand operation after completing the grasping, to ensure that the migration behavior can realize the transmission of the operation intention.

[0110] The multi-dexterous hand configuration adaptation unit migrates the control signal to dexterous hands of different configurations through a multi-dexterous hand configuration adaptation mechanism.

[0111] ​In a specific implementation, as a preferred embodiment of the present application, the trajectory proposal unit comprises a pose reorientation module and a primary trajectory control signal generation module, wherein:

[0112] The pose reorientation module achieves accurate fingertip position matching of the dexterous hand and the human hand trajectory by optimizing the global position vector of each fingertip of the human hand and the palm direction vector of the dexterous hand.

[0113] The primary trajectory control signal generation module is configured to calculate a control signal to control the dexterous hand to reach the reoriented optimized joint angle sequence.

[0114] In a specific implementation, as a preferred embodiment of the present application, the grasp optimization unit comprises a reinforcement learning configurator, a hierarchical and modular reward function system, and a residual policy training module, wherein:

[0115] The reinforcement learning configurator extracts commonalities from object poses in different tasks, initial trajectories of the dexterous hand, and fingertip contact point coordinates to establish a unified reinforcement learning configuration system.

[0116] The hierarchical and modular reward function system is configured to guide the reinforcement learning process to generate a stable and coordinated control strategy, including proximity rewards, grasping rewards, and lifting rewards, wherein the proximity rewards are configured to encourage each fingertip of the dexterous hand to approach the target contact point as quickly as possible, the grasping rewards are configured to encourage the dexterous hand to contact the object in a multi-finger synchronous manner and maintain a human-like hand shape, and the lifting rewards are configured to encourage the object to move to a preset target position and attitude.

[0117] The residual policy training module introduces residual action fine-tuning based on the PPO algorithm on the basis of the primary control signal to optimize dynamic grasp stability.

[0118] Embodiment

[0119] The specific implementation details will be introduced by introducing the process of task migration in the DexYCB dataset to the dexterous hand operation sequence, as follows:

[0120] The data in DexYCB is stored in the form of image sequences, and object models are provided. First, the HFL-Net is used to predict the image sequence to obtain the translation information and rotation information of the object at each time step and the Mano parameters of the human hand at each time step. The human hand is reconstructed using the Mano model, and the object vertex closest to the human hand fingertip during the operation is found to determine the hand-object contact point.

[0121] Then, the extracted human hand fingertip position and wrist orientation will be sent frame by frame to the pose reorientation module for optimization to obtain a dexterous hand joint sequence that matches the human hand kinematics. In this embodiment, are set to 1, 0.01, 0.005 respectively. The target joint angles obtained by optimization are sent to the primary trajectory control signal generation module to calculate the joint control signal for driving the dexterous hand to move according to the target joint angle sequence. This module is implemented based on the inverse dynamics function provided by the MuJoCo simulator. Subsequently, an operating environment is built for each task based on the MuJoCo simulator, and the initial positions of the object and the dexterous hand are consistent with the data in the dataset. The dexterous hand can be controlled by the joint control signal to achieve humanoid primary trajectory in the MuJoCo simulation environment.

[0122] In order to compensate for the failed grasp caused by the lack of dynamic information, the primary trajectory is sent to the grasp optimization module for optimization. During the execution of the primary trajectory by the dexterous hand, the reinforcement learning configurator monitors the distance between the thumb tip and the thumb grasp point on the object, and selects the state with the closest distance between the two under the premise of no collision as the pre-grasp pose. At the same time, according to the pose of the object when it first deviates from the original position by 0.1m during the human hand grasping process and the corresponding human hand operation, the target pose of the reinforcement learning and the number of environment iterations are calculated. The data in the DexYCB dataset is recorded at a frequency of 30HZ, while the MuJoCo simulator is set to run at 120HZ. Therefore, for the DexYCB dataset, the number of environment iterations for reinforcement learning is four times the number of human operation time steps. Based on these configuration information, the method uses reinforcement learning to train a residual policy to fine-tune the primary control signal and establish a stable grasping behavior of the dexterous hand based on the primary trajectory. Through unified reward design, combined with PPO algorithm, the reinforcement learning is realized quickly converges. In specific implementation, the weights of the approach reward, the grasp reward and the lifting reward are set to 10, 10 and 20 respectively.

[0123] The grasp optimization module can only achieve stable grasping of the object, and the expression of human operation semantics also depends on the wrist trajectory. In order to further realize the control of the wrist trajectory of the dexterous hand while preserving the semantic information of the operation, the invention uses the feature that the stable grasping hand object has a fixed relative position, calculates the target wrist trajectory based on the object trajectory true value extracted from DexYCB, and uses a PD controller to generate wrist control instructions to control the dexterous hand to move along the trajectory, while the finger joints remain in the grasping pose unchanged, thereby completing the operation transfer of the DexYCB dataset. The transfer effect is shown in Figure 4 .

[0124] To effectively measure the transfer effect, the following evaluation indicators are used to evaluate the transfer success rate and transfer efficiency respectively:

[0125] Migration success rate: by detecting the displacement, rotation, speed, etc. of the object in real time, the key motion feature sequence of the object in the operation process (such as static, lifting, translation, multi-axis tilt) is analyzed. If the motion features of the object trajectory before and after migration remain consistent in time sequence, it is considered that the motion modes match, that is, the migration is successful.

[0126] Migration efficiency: the number of training converged environment iteration steps required for successful migration.

[0127] The performance of the present application is compared with the excellent results in recent years, including DexMV, PGDM, D-grasp. Table 1 shows the success rate of different methods, and the present application has good migration effect and success rate, and also has faster iteration rate, as shown in Table 1. Figure 5

[0128] Table 1: Success rate of different methods

[0129]

[0130] Finally, it should be pointed out that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.​

Claims

1. A progressive motion-dynamic alignment dexterous manipulation fast transfer method, characterized in that, Using the information extracted from human hand-object interaction videos, cross-dexterous hand dexterous operation transfer mapping is completed with high success rate, including: S1, extracting hand-object motion and interaction information from RGB hand-object interaction videos, including hand posture sequence, object trajectory and hand-object contact information; S2, based on kinematic matching, the human hand operation trajectory is mapped to the primary control signal of the target dexterous hand; S3, under the reinforcement learning framework, the primary control signal is dynamically optimized combined with a multi-level reward mechanism to generate a stable grasping control strategy; the multi-level reward mechanism includes proximity reward, grasping reward and lifting reward, wherein: The proximity reward is used to encourage each fingertip of the dexterous hand to approach the target contact point as soon as possible, and the function expression is as follows: wherein, represents the number of fingers used in the task, represents the Euclidean distance between the th finger tip and its corresponding target contact point, represents the historical minimum distance within the current training epoch, represents the proximity reward weight; the proximity reward encourages individual finger optimization while maintaining target dependency to enhance the geometric generalization capability of the task; The grasp reward is used to encourage the dexterous hand to make contact with the object in a multi-finger synchronous manner and maintain a human-like hand shape; when all the fingertips and the corresponding grasp points are within the contact tolerance range The grasp reward is activated as follows: wherein, represents a grasp reward weight, represents the number of hand-object contact points, encouraging multi-point hand-object contact, represents the cosine similarity between the current joint angles and the target joint angles, constraining the finger pose that is not used for the current task, represents the corresponding weighting coefficient; The lifting reward is used to encourage the object to move to the preset target position and attitude, and the function expression is as follows: wherein, represents a weight of lifting reward, encouraging the object to reach a target pose set by the reinforcement learning configurator from the current pose ; S4, according to the object target trajectory, the wrist control signal of the dexterous hand is synthesized to maintain the semantic consistency of the operation task; S5, through a multi-dexterous hand configuration adaptation mechanism, the control signal is transferred to dexterous hands of different configurations.

2. A progressive motion-dynamic alignment dexterous manipulation fast transfer method according to claim 1, characterized in that, In step S1, HFL-Net or Hold model is used to extract hand-object motion and interaction information, specifically including: S11, for the case of known object model, the object trajectory and hand posture sequence are predicted by HFL-Net to obtain the translation information of the object at each time step, the rotation information and the Mano parameters of the hand at each time step, the hand is reconstructed by Mano model, and the object vertex closest to the fingertip of the hand during operation is found to determine the hand-object contact point; S12, for unknown object model, the object trajectory and hand posture sequence are predicted by Hold model to reconstruct the object three-dimensional model and obtain the translation information of the object at each time step, the rotation information and the Mano parameters of the hand at each time step, the hand is reconstructed by Mano model, and the object vertex closest to the fingertip of the hand during operation is found to determine the hand-object contact point.

3. A progressive motion-dynamic alignment dexterous manipulation fast transfer method according to claim 1, characterized in that, Step S2 specifically includes: S21, through a nonlinear minimization method, the position of the hand and the fingertip of the dexterous hand and the palm direction vector are optimized to generate a humanoid joint angle sequence; S22, the third derivative of the joint angle sequence is minimized and fitted to generate a continuous primary control signal sequence combined with an inverse dynamics model.

4. A progressive motion-dynamic alignment dexterous manipulation fast transfer method according to claim 1, characterized in that, Step S3 specifically includes: S31, using a reinforcement learning configurator to extract commonalities from object poses, initial trajectories of dexterous hands and fingertip contact point coordinates in different tasks to establish a unified reinforcement learning configuration system; S32, designing a hierarchical and modular reward function system to guide the reinforcement learning process to generate stable and coordinated control strategies; S33, based on PPO algorithm, residual action fine-tuning is introduced on the basis of primary control signal to optimize dynamic grasping stability.

5. A progressive motion-dynamic alignment dexterous manipulation fast transfer method according to claim 1, wherein, Step S4 specifically includes: S41, based on the fixed relative position of hand and object during stable grasping, the target wrist attitude is calculated according to the object trajectory; S42, a PD controller is used to convert the target attitude into bottom-level motor control signal to keep the finger grasping attitude unchanged.

6. A progressive motion-dynamic alignment dexterous manipulation fast transfer method according to claim 1, wherein, Step S5 specifically includes: S51, a hand-finger mapping table of human hand and dexterous hand is established; S52、In the redirection and reward calculation, only the intersection of the fingers actually existing in the dexterous hand and the corresponding fingers operated by the human hand is used.

7. A progressive motor-dynamic alignment dexterous operation fast migration system based on the progressive motor-dynamic alignment dexterous operation fast migration method of any one of claims 1-6. Comprise: A human hand operation knowledge extraction unit, a trajectory proposal unit, a grasping optimization unit, a wrist trajectory synthesis unit, and a multi-dexterous hand configuration adaptation unit, wherein: The human hand operation knowledge extraction unit extracts hand-object motion and interaction information from an RGB hand-object interaction video, including a human hand pose sequence, an object trajectory, and hand-object contact information. The trajectory proposal unit maps the human hand operation trajectory to the primary control signal of the target dexterous hand based on kinematic matching. The grasping optimization unit dynamically optimizes the primary control signal under a reinforcement learning framework combined with a multi-level reward mechanism to generate a stable grasping control strategy. The wrist trajectory synthesis unit synthesizes dexterous hand wrist control signals according to the object target trajectory to maintain operation task semantic consistency. The multi-dexterous hand configuration adaptation unit migrates the control signal to dexterous hands of different configurations through a multi-dexterous hand configuration adaptation mechanism.

8. The progressive motion-dynamic aligned dexterous manipulation quick transfer system of claim 7, wherein, The trajectory proposal unit includes a pose redirection module and a primary trajectory control signal generation module, wherein: The pose redirection module achieves accurate fingertip position matching between the dexterous hand and the human hand trajectory by optimizing the global position vector of each fingertip of the human hand and the palm direction vector. The primary trajectory control signal generation module is used to calculate the control signal to control the dexterous hand to reach the joint angle sequence obtained by redirection optimization.

9. The progressive motion-dynamic alignment dexterous manipulation quick transfer system according to claim 7, wherein, The grasping optimization unit includes a reinforcement learning configurator, a hierarchical and modular reward function system, and a residual policy training module, wherein: The reinforcement learning configurator extracts commonalities from object poses, initial trajectories of dexterous hands, and fingertip contact point coordinates in different tasks to establish a unified reinforcement learning configuration system. The hierarchical and modular reward function system is used to guide the reinforcement learning process to generate stable and coordinated control strategies, including proximity rewards, grasping rewards, and lifting rewards, wherein the proximity reward is used to encourage each fingertip of the dexterous hand to approach the target contact point as quickly as possible; the grasping reward is used to encourage the dexterous hand to contact the object in a multi-finger synchronous manner and maintain a human-like hand shape; and the lifting reward is used to encourage the object to move to a pre-set target position and attitude. The residual policy training module introduces residual action fine-tuning based on the PPO algorithm on the basis of the primary control signal to optimize dynamic grasping stability.

Images