Training method, device and computer equipment of motion model based on residual learning

Abstract

The present invention discloses a training method, device and computer equipment for an action model based on residual learning, and belongs to the field of artificial intelligence. The method comprises: obtaining a first reference trajectory of a reference hand and a second reference trajectory of an object; determining an imitation reward based on the first reference trajectory and the wrist posture and finger posture of a mechanical dexterous hand, and performing imitation training on the action model to be trained to obtain an imitation action model; determining a residual reward based on the second reference trajectory and the object interaction state, and performing residual training on the imitation action model to obtain a residual action model; obtaining an action model based on the imitation action model and the residual action model; when the action model is applied, the mechanical dexterous hand imitates the action of the reference hand. The present invention is trained in stages, with the first stage imitating hand movements, and the second stage finely adjusting the interaction constraints of the object interaction task, so as to achieve accurate imitation of the reference hand by the mechanical dexterous hand and dynamic adaptation to complex tasks.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and in particular relates to a training method, device and computer equipment for an action model based on residual learning. Background Art

[0002] In recent years, the field of embodied intelligence (EAI) has developed rapidly, and more and more research is devoted to giving AI-driven embodied systems the ability to interact with physical or virtual environments. Just as human hands are central to interaction, a large amount of research in the field of EAI focuses on mechanical dexterous hand manipulation. Achieving near-human-level manipulation capabilities in complex two-handed tasks is not only of great research value, but also an important step towards general artificial intelligence.

[0003] However, to promote the development of EAI, it has become an urgent need to quickly obtain large-scale, accurate and human-level operation sequences to train data-driven embodied agents. In related technologies, mechanical dexterous hands are trained through exploration and operation generation based on reinforcement learning and teleoperation to collect human-machine aligned data. However, these two methods have their own limitations: traditional reinforcement learning requires carefully designed task-specific reward functions, which usually limits the complexity of the task and may cause unnatural robot movements; while teleoperation methods are costly and inefficient, and the data generated are usually targeted at specific agents and are not universal.

[0004] Therefore, there is an urgent need for a method that can simply and efficiently achieve the transfer of human manipulation skills to dexterous robotic hands. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention proposes a training method, device and computer equipment for a motion model based on residual learning, so as to simply and efficiently realize the precise migration of human two-handed motion to mechanical dexterous hands.

[0006] In a first aspect, the present invention provides a method for training a motion model for a mechanical dexterous hand, the method comprising:

[0007] Acquire a first reference trajectory of a reference hand, and acquire a second reference trajectory of an object when the reference hand performs an object interaction task;

[0008] Based on the first reference trajectory, the wrist posture and the finger posture of the mechanical dexterous hand, an imitation reward is determined, and the action model to be trained is subjected to imitation training to obtain an imitation action model; the imitation reward includes at least a wrist tracking reward and a finger imitation reward;

[0009] Based on the second reference trajectory and the object interaction state, a residual reward is determined, and residual training is performed on the imitation action model to obtain a residual action model; the residual reward includes at least an imitation reward, an object tracking reward, and a contact force reward;

[0010] Based on the imitation motion model and the residual motion model, a motion model is obtained; when the motion model is applied, the mechanical dexterous hand imitates the motion of the reference hand.

[0011] In the above technical solution, by introducing the first reference trajectory and reward function for coarse-grained imitation training, the mechanical dexterous hand can accurately imitate the motion characteristics of the reference hand and achieve highly simulated motion reproduction. Then, by combining the second reference trajectory and the object interaction state for fine-grained optimization training, the mechanical dexterous hand can not only imitate the motion of the reference hand, but also dynamically adapt to the operation tasks of complex objects. In addition, by training in stages, firstly realizing basic imitation through imitating the motion model, and then improving the interactive task performance through the residual model, the training efficiency and model effect can be significantly improved.

[0012] According to one embodiment of the present invention, determining the imitation reward function based on the first reference trajectory, the wrist posture and the finger posture of the mechanical dexterous hand, and performing imitation training on the action model to be trained to obtain the imitation action model includes:

[0013] Obtaining sensory information of the robotic dexterous hand;

[0014] The mechanical dexterous hand imitates the first reference trajectory and the perception information as a first state, and the wrist posture and finger posture are actions, and the first state is input into the action model to be trained to obtain the imitation action output by the action model;

[0015] Determine an imitation reward for the mechanical dexterous hand to perform the imitation action in the first state; the imitation reward at least includes a wrist tracking reward and a finger imitation reward;

[0016] The action model is trained with the maximization of the imitation reward as a training goal to obtain an imitation action model.

[0017] According to an embodiment of the present invention, determining the residual reward based on the second reference trajectory and the object interaction state, and performing residual training on the imitation action model to obtain the residual action model includes:

[0018] determining an object interaction state based on a second reference trajectory of the object when the reference hand performs the object interaction task;

[0019] Taking the first state and the object interaction state as the second state, taking the wrist posture residual and the finger posture residual as the residual action, inputting the second state into the imitation action model to obtain the residual action output by the imitation action model;

[0020] Determining a residual reward for the mechanical dexterous hand to perform the residual action in the second state; the residual reward at least includes an imitation reward, an object tracking reward, and a contact force reward;

[0021] The imitation action model is subjected to residual training with the residual reward maximization as a training objective to obtain a residual action model.

[0022] In a second aspect, the present invention provides a control method for a mechanical dexterous hand, the method comprising:

[0023] Determine the target object interaction task;

[0024] When the target object interaction task is an unlearned task, obtaining a first hand trajectory of a reference hand to be imitated when performing the target object interaction task and a first object trajectory of an object in the target object interaction task;

[0025] Inputting the first hand trajectory into a trained imitation action model to obtain a target imitation action output by the imitation action model;

[0026] Taking residual reward maximization as a training goal, the imitation action model is trained based on the first object trajectory and the object interaction state to obtain a target residual action model adapted to the target object interaction task; the residual reward at least includes an imitation reward, an object tracking reward, and a contact force reward adapted to the imitation action model;

[0027] Inputting the first object trajectory into the target residual motion model to obtain a target residual motion output by the target residual motion model;

[0028] Fusing the target imitation action and the target residual action to obtain a first target action;

[0029] The dexterous robotic hand is controlled according to the first target action to perform the target object interaction task.

[0030] In the above technical solution, when controlling the mechanical dexterous hand, it is determined whether the target object interaction task is a new task that has not been learned, and if it is a new task, the target imitation action is output through the trained imitation action model to perform preliminary imitation and tracking of the reference hand trajectory, and the residual action model is retrained for the specific new task, and the target residual action is output through the retrained residual action model to achieve fine imitation under the new task. Therefore, in the scenario of facing various new tasks, there is no need to design a reward function specifically for each task and conduct specific training on the entire action model. Only a part of the action model, namely the residual action model, needs to be retrained, which greatly improves the efficiency of model training and application, and enables the action model to adapt to the complex and changeable task requirements in the actual environment. In addition, by integrating imitation and residual action, the mechanical dexterous hand can accurately imitate the reference action and meet the task requirements of complex and fine tasks.

[0031] In a third aspect, the present invention provides a training device for a motion model of a mechanical dexterous hand, the device comprising:

[0032] A first acquisition module, used to acquire a first reference trajectory of a reference hand, and to acquire a second reference trajectory of an object when the reference hand performs an object interaction task;

[0033] A first training module is used to determine an imitation reward based on the first reference trajectory, the wrist posture and the finger posture of the mechanical dexterous hand, and perform imitation training on the action model to be trained to obtain an imitation action model; the imitation reward at least includes a wrist tracking reward and a finger imitation reward;

[0034] A second training module is used to determine a residual reward based on the second reference trajectory and the object interaction state, and perform residual training on the imitation action model to obtain a residual action model; the residual reward at least includes an imitation reward, an object tracking reward, and a contact force reward;

[0035] The first determination module is used to obtain a motion model based on the imitation motion model and the residual motion model; when the motion model is applied, the mechanical dexterous hand imitates the motion of the reference hand.

[0036] In the above technical solution, by introducing the first reference trajectory and reward function for coarse-grained imitation training, the mechanical dexterous hand can accurately imitate the motion characteristics of the reference hand and achieve highly simulated motion reproduction. Then, by combining the second reference trajectory and the object interaction state for fine-grained optimization training, the mechanical dexterous hand can not only imitate the motion of the reference hand, but also dynamically adapt to the operation tasks of complex objects. In addition, by training in stages, firstly realizing basic imitation through imitating the motion model, and then improving the interactive task performance through the residual model, the training efficiency and model effect can be significantly improved.

[0037] In a fourth aspect, the present invention provides a control device for a mechanical dexterous hand, the device comprising:

[0038] A second determination module is used to determine the target object interaction task;

[0039] A second acquisition module is used to acquire, when the target object interaction task is an unlearned task, a first hand trajectory of a reference hand to be imitated when performing the target object interaction task and a first object trajectory of an object in the target object interaction task;

[0040] A first input module, used for inputting the first hand trajectory into a trained imitation action model to obtain a target imitation action output by the imitation action model;

[0041] A third training module is used to train the imitation action model based on the first object trajectory and the object interaction state with the residual reward maximization as the training goal, so as to obtain a target residual action model adapted to the target object interaction task; the residual reward at least includes an imitation reward, an object tracking reward and a contact force reward adapted to the imitation action model;

[0042] A second input module, used for inputting the first object trajectory into the target residual motion model to obtain a target residual motion output by the target residual motion model;

[0043] A fusion module, used for fusing the target imitation action and the target residual action to obtain a first target action;

[0044] A control module is used to control the dexterous mechanical hand according to the first target action to perform the target object interaction task.

[0045] In the above technical solution, when controlling the mechanical dexterous hand, it is determined whether the target object interaction task is a new task that has not been learned, and if it is a new task, the target imitation action is output through the trained imitation action model to perform preliminary imitation and tracking of the reference hand trajectory, and the residual action model is retrained for the specific new task, and the target residual action is output through the retrained residual action model to achieve fine imitation under the new task. Therefore, in the scenario of facing various new tasks, there is no need to design a reward function specifically for each task and conduct specific training on the entire action model. Only a part of the action model, namely the residual action model, needs to be retrained, which greatly improves the efficiency of model training and application, and enables the action model to adapt to the complex and changeable task requirements in the actual environment. In addition, by integrating imitation and residual action, the mechanical dexterous hand can accurately imitate the reference action and meet the task requirements of complex and fine tasks.

[0046] In a fifth aspect, the present invention provides a dexterous mechanical hand, comprising the control device of the dexterous mechanical hand as described in the fourth aspect.

[0047] In a sixth aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the training method for the motion model of a mechanical dexterous hand as described in the first aspect or the control method for a mechanical dexterous hand as described in the second aspect is implemented.

[0048] In the seventh aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for the motion model of a mechanical dexterous hand as described in the first aspect or the control method for a mechanical dexterous hand as described in the second aspect.

[0049] In an eighth aspect, the present invention provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instructions to implement the training method for the motion model of a mechanical dexterous hand as described in the first aspect or the control method for the mechanical dexterous hand as described in the second aspect.

[0050] In a ninth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the training method for the motion model of a mechanical dexterous hand as described in the first aspect or the control method of the mechanical dexterous hand as described in the second aspect.

[0051] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0053] Figure 1 is a schematic diagram of an application scenario of a training method for a motion model of a mechanical dexterous hand provided in some embodiments of the present invention;

[0054] Figure 2 is a schematic diagram of a training scenario for an object interaction task provided in some embodiments of the present invention;

[0055] Figure 3 is a schematic diagram of a scenario in which a mechanical dexterous hand provided in some embodiments of the present invention performs an object interaction task;

[0056] Figure 4 is a flow chart of a training method for a motion model of a mechanical dexterous hand provided in some embodiments of the present invention;

[0057] Figure 5 is a schematic diagram of the process of the first stage training provided in some embodiments of the present invention;

[0058] Figure 6 is a schematic diagram of a flow chart of providing second-stage training in some other embodiments of the present invention;

[0059] Figure 7 is a flow chart of a control method of a mechanical dexterous hand provided in some embodiments of the present invention;

[0060] Figure 8 is a schematic structural diagram of a training device for a motion model of a mechanical dexterous hand provided in some embodiments of the present invention;

[0061] Fig. 9 is a schematic structural diagram of a control device for a mechanical dexterous hand provided in some embodiments of the present invention;

[0062] Fig.10 It is a schematic diagram of the structure of a computer device provided in some embodiments of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0064] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by technicians in the technical field of the present invention; the terms used in the specification of the present invention in the application are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present invention or the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order or a primary and secondary relationship.

[0065] Reference to "embodiments" in the present invention means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present invention may be combined with other embodiments.

[0066] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", and "attached" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0067] The term "and / or" in the present invention is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.

[0068] The term "plurality" used in the present invention refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple sheets" refers to more than two sheets (including two sheets).

[0069] When faced with the problem of how to transfer human manipulation actions to dexterous hands in a simulation environment, some studies have achieved this goal through imitation learning. This method has the following advantages: first, imitating human manipulation trajectories can promote more natural interactions between dexterous hands and objects, thereby achieving more smooth and realistic movements; second, a large number of existing motion capture (MoCap) data sets and posture estimation technologies provide convenient conditions for extracting manipulation knowledge from human demonstration videos; finally, the simulation environment provides a low-cost platform for verifying the feasibility and completeness of manipulation tasks, accelerating the process of real robot deployment.

[0070] However, the inventors found that there are still many challenges to achieve accurate and efficient human manipulation motion transfer. Since two-handed dexterous hands have a high degree of freedom (DoF), their morphological characteristics are significantly different from those of human hands, and direct motion redirection often has poor effects. In addition, although motion capture data is usually more accurate, error accumulation may lead to critical failures in high-precision tasks, thereby hindering the accurate strategy execution of physical simulation. At the same time, traditional motion generation methods have difficulty capturing fine finger movements due to smooth transition problems. These factors have caused many cutting-edge explorations to be limited to simple grasping and lifting tasks, while research on complex two-handed tasks such as unscrewing bottle caps and capping pens is still not effective.

[0071] In view of this, an embodiment of the present invention provides a training method for a motion model of a mechanical dexterous hand. The training is divided into two stages. The first stage mainly trains the mechanical dexterous hand to imitate human hand movements. The second stage fine-tunes the mechanical dexterous hand for specific tasks, especially the interaction constraints of object interaction tasks. Based on a reward mechanism that combines general rewards with rewards for specific tasks, the motion model is optimized in stages. Finally, the mechanical dexterous hand achieves accurate imitation of a reference hand and dynamic adaptation to complex tasks.

[0072] Specifically, the embodiment of the present invention uses a universal motion model for reinforcement learning to accurately imitate human hand movements in a noisy environment. On this basis, combined with the residual learning method, the motion model is further incrementally optimized to capture the fine movements of the fingers and the forces applied, so as to further improve the robot's operational precision, thereby enabling the mechanical dexterous hand to adapt to various complex two-handed tasks and perform them with high precision and high fidelity.

[0073] In conjunction with the accompanying drawings, the training method, device, computer equipment, etc. of the motion model based on residual learning provided by the embodiments of the present invention are described in detail through specific embodiments and their application scenarios.

[0074] The training method of the motion model based on residual learning provided by the embodiment of the present invention can be applied to Figure 1 In the application environment shown, the computer device 10 is connected to the mechanical dexterous hand 30 for communication. The computer device 10 can be used as the computing core of the mechanical dexterous hand 30 and can be set on the mechanical dexterous hand 30, such as a CPU set in the mechanical dexterous hand; or the computer device 10 can be set independently of the mechanical dexterous hand 30, and after the training is completed, the corresponding software program or module is configured on the mechanical dexterous hand 30, and so on.

[0075] Take the object interaction task as an example, the interaction task with the pen (including the pen body and pen cap), Figure 2 As shown, the reference hand 20 performs a series of actions in the object interaction task to form a first reference trajectory. The reference hand 20 includes a left hand 201 and a right hand 202. At the same time, when the object 40 interacts with the reference hand 20, a second reference trajectory is formed. Among them, the object 40 includes a pen body 401 and a pen cap 402. The second reference trajectory includes the trajectory of the pen body 201 and the trajectory of the pen cap 202. Furthermore, the mechanical dexterous hand 30 can imitate the movements of the reference hand 10 through training. The mechanical dexterous hand 30 includes a left hand 301 and a right hand 302. Among them, the object interaction task can be an object grasping task, a moving task, etc.

[0076] For example, after the training is completed, Figure 3 As shown, the robotic dexterous hand 30 can be set on the robotic arm 50 to move the object 21 from platform A to platform B.

[0077] It should be noted that a dexterous robotic hand is a robot end effector with multiple independently controlled degrees of freedom, allowing it to move freely in different directions and manipulate objects. Dexterous robotic hands are usually designed with multiple joints and finger segments, enabling them to perform complex operations such as grasping, pinching, rotating and manipulating irregularly shaped objects. Dexterous robotic hands can be set up separately or integrated on robots, such as humanoid robots, robotic arms, etc. Dexterous robotic hands are usually equipped with tactile sensors, force sensors or visual sensors to sense contact force, surface characteristics of objects or relative positions, thereby achieving precise force feedback and position control.

[0078] The training method of the motion model based on residual learning can be applied to a computer device, and can be specifically executed by hardware and / or software in the computer device. The computer device can be a mechanical dexterous hand or other terminal devices.

[0079] Exemplarily, terminal devices include but are not limited to various desktop computers, laptops, smart phones, tablet computers, vehicle terminals, Internet of Things devices, or portable wearable devices. The Internet of Things devices may be one or more of smart speakers, smart TVs, smart air conditioners, or smart vehicle devices. Portable wearable devices may be one or more of smart watches, smart bracelets, or head-mounted devices. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0080] The training method for the action model of a mechanical dexterous hand provided in an embodiment of the present invention can be executed by a functional module or functional entity in a computer device that can implement the method. The training method for the action model of a mechanical dexterous hand provided in an embodiment of the present invention is described below using a computer device as an example of the execution subject.

[0081] like Figure 4 As shown, the training method for the action model of the mechanical dexterous hand includes: steps 410 to 440.

[0082] Step 410: Obtain a first reference trajectory of a reference hand, and obtain a second reference trajectory of an object when the reference hand performs an object interaction task.

[0083] Among them, the reference hand refers to the human hand or another highly flexible robotic hand that serves as the learning target of the robotic dexterous hand in motion imitation training, and is used to provide standard motion trajectory and operation method.

[0084] The first reference trajectory refers to the trajectory data recorded when the reference hand performs an object interaction task, including wrist posture, finger joint angle, wrist speed, finger joint speed, etc. The first reference trajectory includes a left-hand trajectory and a right-hand trajectory. In order to make the training independent of hand-object interaction data, the computer device is trained through a preset hand motion dataset. Since most datasets are dominated by right-hand trajectories, the computer device can generate the corresponding left-hand trajectory through a mirror operation. For static posture datasets, the computer device can interpolate and sample between random posture pairs through the spherical linear interpolation (SLERP) method to enrich the diversity of the dataset and improve the generalization ability of the model.

[0085] Object interaction tasks refer to the operation between the hand and the object, such as grasping, carrying, rotating or precisely placing. In the object interaction task, the motion trajectory of the object is called the second reference trajectory, which usually records the displacement, rotation or deformation of the object. Exemplarily, the object interaction task includes but is not limited to the contact, movement, rotation, grasping, assembly and other processes between the mechanical dexterous hand and the object.

[0086] The computer device can record the first reference trajectory of the reference hand and the second reference trajectory of the object when the reference hand performs the object interaction task through a motion capture system. Need to imitate the reference hand behavior. Among them, and They represent the left and right hands of the robotic dexterous hand, and denote the left and right hands of the reference hand, respectively. In the object interaction task, the object In and denote the objects interacting with the left and right hands of the reference hand, respectively. For example, in the pen cap removal task, one hand of the reference hand holds the pen cap, and the other hand holds the pen body. The first reference trajectory is defined as , the second reference trajectory is defined as , where T represents the total number of frames. The reference trajectory of each reference hand Six degrees of freedom including wrist ,speed ,in represents the linear velocity of the reference hand, Represents the angular velocity of the reference hand and the finger joint positions defined based on the MANO model and its corresponding speed , where F represents the number of finger key points including the fingertips, represents the linear velocity of the finger joint, Indicates the angular velocity of the finger joint. The key points of the hand include at least the point with the largest curve on the midline of the finger. The trajectory of each object Including its DoF pose , and its speed ,in represents the linear velocity of the object, In order to reduce the spatial complexity, the embodiment of the present invention normalizes all translations relative to the wrist position of the mechanical dexterous hand, while keeping the rotation information to maintain the correctness of the gravity direction.

[0087] Among them, the hidden Markov decision process can be expressed as ,in represents the state space, represents the action space, Indicates the state transition dynamics, is the reward function, Based on the definitions of the above parameters, the embodiment of the present invention regards the imitation training task of the robotic dexterous hand on the reference hand as a solution process for a hidden Markov decision process (MDP).

[0088] Step 420: Determine an imitation reward based on the first reference trajectory, the wrist posture and the finger posture of the mechanical dexterous hand, and perform imitation training on the action model to be trained to obtain an imitation action model; the imitation reward includes at least a wrist tracking reward and a finger imitation reward.

[0089] In the first stage of training, the computer device mainly trains the mechanical dexterous hand to preliminarily imitate the movements of human hands. In this stage, the computer device can train the action model based on the reinforcement learning strategy and take the imitation reward as the goal to maximize the imitation reward, so that the imitation movement of the mechanical dexterous hand output by it is close to the movement of the reference hand. Among them, the imitation reward is used to evaluate the similarity between the movement of the mechanical dexterous hand and the movement of the reference hand. The reward function Used to encourage the robotic hand to follow the first reference trajectory , while ensuring the stability and smoothness of the movement.

[0090] In some embodiments, the imitation reward includes at least a wrist following reward and a finger imitation reward. The wrist following reward is represented by , refers to the reward value obtained by evaluating the similarity between the trajectory of the robotic dexterous wrist and the reference wrist trajectory. The wrist tracking reward is used to optimize the overall motion trajectory of the robotic dexterous hand by minimizing and The difference between is used to encourage the dexterous wrist movement to approach the first reference trajectory. The finger imitation reward is expressed as , which encourages the finger joint positions of the dexterous hand to be consistent with the reference trajectory. For example,

[0091] (1)

[0092] in, For the f The weight of each finger key point, is the attenuation coefficient, Indicates the distance between the position of the finger key point and the position of the finger joint on the reference hand.

[0093] To reduce the abruptness of the action, in some embodiments, the computer device also calculates a smoothness reward, expressed as , by penalizing the element-by-element product of joint velocity and torque, the robot hand is trained to move more naturally and smoothly. For example, the imitation reward can be expressed by the following formula (2):

[0094] (2)

[0095] in, Bonus for wrist tracking The corresponding weight, Reward for smoothness The corresponding weight.

[0096] Step 430: Based on the second reference trajectory and the object interaction state, determine the residual reward, and perform residual training on the imitation action model to obtain the residual action model; the residual reward at least includes an imitation reward, an object tracking reward, and a contact force reward.

[0097] In the second stage of training, the computer device mainly fine-tunes the mechanical dexterous hand for specific tasks, especially the interaction constraints of the object interaction task, with the purpose of learning the residual action, that is, the amount of action fine-tuned on the basis of the imitation action. In the second stage, based on the imitation action model obtained through pre-training, the computer device uses residual learning to further optimize the rough imitation action to ensure that it meets the task requirements of the object interaction task. In this stage, the computer device can train the imitation action model obtained through pre-training based on the reinforcement learning strategy, combined with the object interaction state, with the goal of maximizing the residual reward, so that the imitation action of the mechanical dexterous hand output by it is close to the action of the reference hand.

[0098] The object interaction state refers to the dynamic state of the object during the operation of the mechanical dexterous hand, such as whether it is grasped firmly, whether the movement trajectory meets the task requirements, etc. Exemplarily, the object interaction state includes the second reference trajectory of the object when the object is moved by the reference hand, the position and velocity (including linear velocity and angular velocity) of the object relative to the wrist position, the center of mass, gravity, the distance of the object relative to the key points of the fingers, the key points of the object, and the contact force.

[0099] In the second stage of training, the goal is to efficiently transfer human two-handed manipulation skills to dexterous hands in a task-independent manner. Therefore, in addition to the imitation reward in the first stage, the residual reward also includes the object following reward and the contact force reward. The object following reward is expressed as , which minimizes the displacement and velocity differences between the simulated object and its reference trajectory, i.e. and , to encourage precise operation. The contact force reward is expressed as , which aims to encourage appropriate contact force and is used to evaluate the rationality of the contact force during the contact between the robot hand and the object. Reasonable contact force can not only avoid damage to the object, but also ensure the success rate of the grasping task. For example, the contact force reward can be expressed as:

[0100] (3)

[0101] in, Indicates fingertips The minimum distance to the object surface in the current coordinate system, is the indicator function, The contact force of the dexterous fingertips, and are weight and attenuation coefficient respectively, is the hand-object distance threshold. Further, in some embodiments, the residual reward can be expressed as:

[0102] (4)

[0103] in, is the imitation reward obtained in formula (2), Tracking rewards for objects The corresponding weight, Reward for contact The corresponding weight.

[0104] Step 440: Obtain a motion model based on the imitation motion model and the residual motion model; when the motion model is applied, the mechanical dexterous hand imitates the motion of the reference hand.

[0105] The computer device fuses the imitation action model with the residual action model to generate a final action model. The computer device can deploy the action model on a mechanical dexterous hand for task execution in a real environment. For example, in a robot assembly task, the mechanical dexterous hand completes the precise placement of complex parts by imitating human grasping and assembly actions. For another example, in the field of service robots, by imitating human delicate gestures, flexible manipulation of tableware, fragile items, etc. is achieved. For another example, in the field of medical robots, the mechanical dexterous hand performs complex operations by imitating the operations of surgeons, and so on. The computer device fuses the imitation action model with the residual action model, which can be specifically expressed as combining the output of the imitation action model and the output of the residual action model according to a certain strategy to obtain the final action. In some embodiments, the computer device superimposes the imitation action with the residual action to obtain the final action.

[0106] When the dexterous robotic hand performs the action to imitate the reference hand, it is specifically manifested in that the computer device applies a six-dimensional force (corresponding to DoF) to the dexterous robotic hand through the hardware set on the dexterous robotic hand, so as to realize the action of the dexterous robotic hand in space.

[0107] The training method for the motion model of a mechanical dexterous hand provided by the embodiment of the present invention introduces a first reference trajectory and a reward function to perform coarse-grained imitation training, so that the mechanical dexterous hand can accurately imitate the motion characteristics of the reference hand and achieve highly simulated motion reproduction, and then performs fine-grained optimization training by combining a second reference trajectory and the object interaction state, so that the mechanical dexterous hand can not only imitate the motion of the reference hand, but also dynamically adapt to the operation tasks of complex objects. In addition, by training in stages, firstly realizing basic imitation by imitating the motion model, and then improving the interactive task performance by the residual model, the training efficiency and model effect can be significantly improved.

[0108] In the embodiment of the present invention, by decoupling the imitation of human hands by dexterous hands from the physics-based interaction constraints, the complexity of the action space at each stage is significantly reduced, thereby improving the training efficiency. For most hand-object interaction tasks (including single-handed operation of a single object, two-handed operation of a hinged object, two-handed operation of two objects, etc.), smooth and natural dexterous hand movements can be transferred within 20 minutes. Through the staged training method, the first stage is used to train the imitation of hand movements, and the second stage is fine-tuned for the interaction constraints of specific tasks, thereby effectively alleviating the smooth transition problem, being able to capture the fine movements of the fingers, and achieving precise and natural two-handed operation. Moreover, the above training method does not rely on task-specific manually designed reward functions, but is based on the correspondence between human fingertips and mechanical dexterous fingertips to model and stimulate movement and contact, thereby achieving universal cross-entity and cross-task operation skill transfer, so that the method can be extended to dexterous hands of various structures. Since it is often difficult to obtain large-scale hand-object interaction data by imitating hand trajectories, in the embodiment of the present invention, a general model is trained by interpolating and enhancing the existing hand motion data set, so that it can adapt to a variety of operation tasks and be robust to noise; and a physical constraint relaxation mechanism is introduced in the residual learning stage, so that the dexterous hand can firmly grasp the object and quickly align the reference trajectory in the early stage of transfer training, thereby accelerating convergence and preventing model collapse. At the same time, according to the contact termination condition, that is, when the distance from the human hand to the operated object is less than a certain threshold, the dexterous hand must be in strong contact with the surface of the object, otherwise the current reinforcement learning exploration is terminated in advance, thereby obtaining a more stable action transfer result.

[0109] As a highly flexible robot end effector, the mechanical dexterous hand can imitate human hand movements and complete complex tasks. In this process, its performance depends on the comprehensive performance of imitation ability and task adaptability. However, the current imitation learning method has the problems of insufficient motion accuracy and lack of real-time adaptability when dealing with diverse tasks. Therefore, the present invention proposes a training method based on perception information and reference trajectory, which iteratively trains the action model by optimizing the reward function, thereby improving the effect of imitation learning and task execution ability. To this end, in some embodiments, such as Figure 5 As shown, based on the first reference trajectory, the wrist posture and the finger posture of the mechanical dexterous hand, the imitation reward function is determined, and the action model to be trained is subjected to imitation training to obtain an imitation action model, including steps 510 to 540:

[0110] Step 510: Acquire the perception information of the dexterous mechanical hand;

[0111] Step 520: The mechanical dexterous hand imitates the first reference trajectory and the sensed information as a first state, and the wrist posture and the finger posture are actions, and the first state is input into the action model to be trained to obtain the imitation action output by the action model;

[0112] Step 530: determining an imitation reward for the dexterous mechanical hand to perform an imitation action in the first state; the imitation reward at least includes a wrist tracking reward and a finger imitation reward;

[0113] Step 540: Train the action model with maximizing the imitation reward as the training goal to obtain an imitation action model.

[0114] Among them, perception information refers to the data related to the joint state, posture, speed and torque of the mechanical dexterous hand obtained by its own sensors, which is used for real-time feedback and control and is an important basis for the optimization of the motion model. Specifically, in the first stage of training, the computer equipment obtains the real-time perception information of the mechanical dexterous hand through the sensors installed on the mechanical dexterous hand (such as inertial measurement unit (IMU), force sensor and visual camera), including finger joint angle, contact force, wrist position and posture, etc. Combined with the perception information, the computer device can construct a first state with the first reference trajectory. The motion model to be trained takes the first state as input and outputs an imitation action. Among them, the state of each dexterous hand is defined as , which contains the target hand trajectory and current proprioceptive information .here, and Represent the joint angle and wrist position, as well as their corresponding speeds. The purpose of this first phase of training is to train the strategy through reinforcement learning , generate imitation actions The computer device constructs an imitation reward based on the wrist tracking reward and the finger imitation reward, and uses a reinforcement learning algorithm to train the action model with the cumulative value of the imitation reward as the optimization target to generate an imitation action model.

[0115] In the above embodiment, by introducing sensory information, the mechanical dexterous hand can dynamically adapt to environmental changes during the imitation process, thereby improving the accuracy and consistency of the action; and through wrist tracking rewards and finger imitation rewards, the mechanical dexterous hand can accurately imitate the action of the reference hand so that its trajectory is close to the first reference trajectory. By decoupling the dexterous hand's imitation of the human hand from the physics-based interaction constraints, the focus is on imitating hand movements in the first stage, which significantly reduces the complexity of the action space in each stage, thereby improving the training efficiency.

[0116] In complex object interaction tasks, the dexterous mechanical hand needs to have the ability to imitate the reference hand movements and adapt to changes in the object state at the same time. However, it is difficult to cope with the needs of dynamic changes in the object state in the task by relying solely on the initial motion model. In an embodiment of the present invention, in order to fully capture the interaction information between the dexterous mechanical hand and the object, the state space is expanded, and additional information related to the interaction is added on the basis of the hand-related state. In addition, by introducing object interaction states and motion residuals, combined with imitation rewards, object tracking rewards, and contact force rewards, the initial motion model is further trained, thereby improving the adaptability and operation accuracy of the dexterous mechanical hand in object interaction tasks. To this end, in some embodiments, such as Figure 6 As shown, based on the second reference trajectory and the object interaction state, a residual reward is determined, and residual training is performed on the imitation action model to obtain a residual action model, including steps 610 to 640:

[0117] Step 610: determining the object interaction state based on a second reference trajectory of the object when the reference hand performs the object interaction task;

[0118] Step 620: Taking the first state and the object interaction state as the second state, taking the wrist posture residual and the finger posture residual as the residual action, inputting the second state into the imitation action model to obtain the residual action output by the imitation action model;

[0119] Step 630: determine a residual reward for the dexterous mechanical hand to perform a residual action in the second state; the residual reward at least includes an imitation reward, an object tracking reward, and a contact force reward;

[0120] Step 640: Perform residual training on the imitation action model with the residual reward maximization as the training objective to obtain a residual action model.

[0121] The computer device obtains a second reference trajectory of the object when the reference hand performs an action in the object interaction task, and calculates the object interaction state. In some embodiments, the computer device can calculate the object mesh through motion capture data. o and generate collidable objects in a virtual environment (e.g., a simulation environment) In some embodiments, in order to better encode the shape of the object, the computer device may represent the shape of the object through the BPS (sphere distributed point set) representation. BPS is a way to represent a three-dimensional object (or shape) as a point set distributed on a sphere. This method constructs a unified shape representation by spatially sampling the object and normalizing the sampled points onto the sphere.

[0122] In order to make the object trajectory close to the reference trajectory, the following data are also introduced in the embodiment of the present invention: the position of the object relative to the wrist Location and speed , Centroid , and the gravity vector , to enhance the perception of force and torque of the mechanical dexterous hand. In some embodiments, the computer device may set the gravity constant G to zero and the friction coefficient F to a higher value at the beginning of training, so that the mechanical dexterous hand can firmly grasp the object and quickly align with the first reference trajectory. As the training progresses, the computer device may gradually restore the real value of G and reduce F to simulate actual physical conditions.

[0123] In addition, the computer device uses the distance measurement formula Characterizes the spatial relationship between the finger key points and the object position. represents the position of the finger key point in the tth frame, Indicates the position of the object relative to the wrist in frame t In order to capture the tactile feedback in the interaction, the embodiment of the present invention further introduces the contact force C generated by the simulation environment to reflect the interaction between the mechanical dexterous fingertip and the surface of the object. The magnitude of the contact force can be used to measure the magnitude of the force applied by the mechanical dexterous hand to grasp the object in complex tasks.

[0124] Based on this, the object interaction state in the embodiment of the present invention includes the second reference trajectory of the object, the position and speed (including linear speed and angular speed) of the object relative to the wrist position, the center of mass, gravity, the distance of the object relative to the key points of the fingers, the key points of the object, and the contact force when the object is moved by the reference hand. The object interaction state can be expressed as:

[0125] (5)

[0126] in, is the object interaction state at the tth frame, is the second reference trajectory of the object at the tth frame, is the position of the object relative to the wrist at the tth frame location, is the speed of the object at the tth frame, is the center of mass of the object at the tth frame, is the gravity vector of the object at the tth frame, The key points of the object mapped on the sphere, Represents the distance between the finger key point and the object position, is the contact force between the object and the dexterous robotic hand at the tth frame.

[0127] Thus, the computer device constructs a second state based on the first state and the object interaction state. The imitation action model to be optimized and trained takes the second state as input and outputs a residual action. Exemplarily, the second state is, for example, the union of the first state and the object interaction state, that is, The training goal of the second stage is to learn the residual action , optimize the initial imitation action To meet the task requirements. In each step of the operation, the computer equipment first samples the imitation action Then, the computer device corrects the motion sampling residual The final action is calculated as: . The residual action is superimposed on the imitation action by element-by-element addition. Control within the joint range of the dexterous hand to ensure legal movements.

[0128] The computer device constructs an imitation reward based on the imitation reward, object tracking reward and contact force reward, and uses a reinforcement learning algorithm to train the action model with the cumulative value of the residual reward as the optimization target to generate a residual action model. By updating the model parameters and optimizing the residual action, the dexterous robot hand can accurately respond to dynamic object interaction tasks.

[0129] In some embodiments, at the beginning of residual training, the residual motion should be close to zero because the mechanical dexterous hand motion can roughly follow the first reference trajectory. This initialization helps prevent model collapse and accelerate convergence. The computer device can initialize the residual motion model through a zero-mean Gaussian distribution and gradually activate the training using a warm-up strategy.

[0130] In the above embodiment, by introducing the object interaction state and action residual, the model can cope with the dynamic changes of objects in the task and improve the flexibility of operation. The residual training improves the operation accuracy of the mechanical dexterous hand in fine tasks and enhances the stability of the model in complex environments. By decoupling the dexterous hand's imitation of the human hand from the physics-based interaction constraints, the focus in the second stage is on the fine adjustment of the interaction constraints for specific tasks, which significantly reduces the complexity of the action space at each stage, thereby improving the training efficiency.

[0131] In the imitation training of the mechanical dexterous hand, the training effect of the model is crucial to the operation accuracy. Since the mechanical dexterous hand has six degrees of freedom in space, the movement trajectory of the mechanical dexterous hand may deviate from the reference trajectory during the training process. Excessive deviation will affect the learning efficiency and task adaptability of the model. Therefore, during the first stage of training, the computer device also monitors and controls the training process based on the early termination condition. That is, the training of the first stage is subject to the first early termination condition. The first early termination condition refers to: the key point of the dexterous hand deviates from the first reference trajectory by more than the first deviation threshold. That is, in some embodiments, during the imitation training process, the above method also includes: determining the first deviation parameter in the imitation training process; the first deviation parameter characterizes the degree of deviation between the movement trajectory of the mechanical dexterous hand and the first reference trajectory; when the first deviation parameter exceeds the first deviation threshold, restarting the training.

[0132] The first deviation threshold can be set according to the specific task. Generally speaking, for fine grasping tasks, a relatively small threshold is set; while for rough operation tasks, a larger deviation is allowed and a relatively larger threshold is set. The computer device can calculate the key points of the dexterous hand at each step or at fixed time steps. The distance of the deviation from the first reference trajectory exceeds the first deviation threshold. When , the first training phase is terminated in advance and reinitialized. In some embodiments, the computer device can retrain from scratch; or, the computer device can also retain the model parameters of the low deviation phase and only restart the optimization process of the high deviation phase, etc. In some embodiments, the computer device can also gradually reduce the , to encourage extensive exploration of the robotic hand in the early stages of training, and subsequently focus on fine finger control. In some embodiments, the first deviation threshold can be set dynamically. For example, in the early stages of training, a large deviation is allowed, and the computer device can set a loose threshold, and as the training progresses, the computer device gradually tightens the deviation threshold to the target range.

[0133] In the above embodiment, through real-time deviation monitoring, invalid training states can be quickly discovered and corrected to avoid wasting computing resources; by retraining when the deviation exceeds the limit, it helps to ensure that the trajectory accuracy of the final model meets expectations, improve the operational reliability of complex tasks, and reduce the time for repeated trial and error of model parameters, thereby improving training efficiency.

[0134] Similar to the first stage, the computer device also monitors and controls the training process based on the early termination condition. That is, the training of the second stage is subject to the second early termination condition. The second early termination condition refers to: the deviation of the object posture from the second reference trajectory exceeds the second deviation threshold. That is, in some embodiments, during the residual training process, the above method also includes: determining the second deviation parameter in the residual training process; the second deviation parameter characterizes the degree of deviation between the object posture and the second reference trajectory; when the second deviation parameter exceeds the second deviation threshold, restarting the training. The second deviation threshold can be set according to the complexity and tolerance of the task. Generally speaking, for tasks with lower precision requirements such as object grasping, a relatively large threshold is set; and for tasks with higher precision requirements such as fine assembly, a relatively stricter threshold is set. The computer device can calculate the object posture at each step or at fixed time steps. The distance of the deviation from the second reference trajectory exceeds the second deviation threshold. When , the second training phase is terminated early and reinitialized. As the training progresses, the In some embodiments, the computer device may retrain from scratch; alternatively, the computer device may retain the model parameters of the low deviation phase and only restart the optimization process of the high deviation phase, and so on.

[0135] In the above embodiment, through real-time deviation monitoring, invalid training states can be quickly discovered and corrected to avoid wasting computing resources; by retraining when the deviation exceeds the limit, it helps to ensure that the trajectory accuracy of the final model meets expectations, improve the operational reliability of complex tasks, and reduce the time for repeated trial and error of model parameters, thereby improving training efficiency.

[0136] In some embodiments, the second phase of training is also subject to a contact termination condition. The contact termination condition means that when the reference hand is firmly grasping the object, the contact force must not be equal to zero, otherwise the computer device terminates the second training phase in advance and reinitializes. When the reference hand is firmly grasping the object, that is, ,in, Indicates fingertips h f The minimum distance to the surface of the object in the current coordinate system, is the hand-object distance threshold.

[0137] Based on the same inventive concept, an embodiment of the present invention also provides a control method for a mechanical dexterous hand. The execution subject of the control method can be a functional module or functional entity in a computer device that can implement the method. The following takes a computer device as an example of the execution subject to illustrate the control method for a mechanical dexterous hand provided by an embodiment of the present invention. Figure 7 As shown, the control method of the mechanical dexterous hand includes steps 710 to 740:

[0138] Step 710: Determine the target object interaction task.

[0139] Step 720: When the target object interaction task is an unlearned task, obtain a first hand trajectory of a reference hand to be imitated when performing the target object interaction task and a first object trajectory of an object in the target object interaction task.

[0140] Step 730: input the first hand trajectory into the trained imitation action model to obtain the target imitation action output by the imitation action model.

[0141] Step 740: Taking residual reward maximization as the training goal, the imitation action model is trained based on the first object trajectory and the object interaction state to obtain a target residual action model adapted to the target object interaction task; the residual reward at least includes an imitation reward, an object tracking reward, and a contact force reward adapted to the imitation action model.

[0142] Step 750: Input the first object trajectory into the target residual motion model to obtain the target residual motion output by the target residual motion model.

[0143] Step 760: Fuse the target imitation action and the target residual action to obtain a first target action.

[0144] Step 770: Control the dexterous robotic hand according to the first target action to perform the target object interaction task.

[0145] The target object interaction task refers to a task in which a specific form of interaction is performed between a mechanical dexterous hand and an object to achieve the expected goal. The task objectives usually include changing the state of an object (position, posture, shape, mechanical state, etc.) or completing a specific function. The successful completion of the task depends on the mechanical dexterous hand's operating accuracy, force control ability, and adaptability to the task environment. Exemplarily, the target object interaction task includes but is not limited to the contact, movement, rotation, grasping, and assembly of the mechanical dexterous hand and the object. In actual tasks, the mechanical dexterous hand may need to perform unlearned object interaction tasks. In order to achieve rapid adaptation to new tasks, the present invention proposes a method combining imitation learning and residual training. By imitating the trajectory of the reference hand, the trained imitation action model is used to generate preliminary actions, and the residual action model is optimized by residual rewards to adapt to specific tasks. Finally, the imitation action and the residual action are fused to generate the target action to achieve efficient execution of the target task.

[0146] Specifically, the computer device obtains an unlearned target object interaction task, and determines the first hand trajectory of the reference hand when the reference hand performs the target object interaction task, and the first object trajectory of the object in the target object interaction task. The computer device inputs the first hand trajectory into the trained imitation action model to obtain the target imitation action output by the trained imitation action model. Since the target object interaction task is an unlearned object interaction task, the computer device trains the imitation action model based on the current target object interaction task, based on the first object trajectory and the object interaction state as the actual state, to obtain a target residual action model adapted to the target object interaction task. The training process can refer to the aforementioned embodiment, which will not be repeated here. The residual reward at least includes an imitation reward, an object tracking reward, and a contact force reward adapted to the imitation action model. The specific details can refer to the aforementioned embodiment, which will not be repeated here. Furthermore, after residual learning is performed for a specific target object interaction task, the computer device inputs the first object trajectory into the trained target residual action model to obtain the target residual action output by the target residual action model. The computer device fuses the target imitation action with the target residual action to obtain the first target action. Thus, the computer device can control the dexterous mechanical hand to perform the new task according to the first target action.

[0147] According to the control method of the mechanical dexterous hand provided by the embodiment of the present invention, when controlling the mechanical dexterous hand, it is determined whether the target object interaction task is a new task that has not been learned, and when it is a new task, the target imitation action is output through the trained imitation action model to perform preliminary imitation and tracking of the reference hand trajectory, and the residual action model is retrained for the specific new task, and the target residual action is output through the retrained residual action model to achieve fine imitation under the new task. Therefore, in the scenario of facing various new tasks, there is no need to design a reward function specifically for each task and perform specific training on the entire action model. Only a part of the action model, that is, the residual action model, needs to be retrained, which greatly improves the efficiency of model training and application, and enables the action model to adapt to the complex and changeable task requirements in the actual environment. In addition, by integrating imitation and residual action, the mechanical dexterous hand can accurately imitate the reference action and meet the task requirements of complex and fine tasks.

[0148] In the case where the target object interaction task is a learned task, that is, a task that has been trained in the training phase, the above method further includes steps 810 to 830:

[0149] Step 810: when the target object interaction task is a learned task, obtaining a second hand trajectory of the reference hand when the reference hand to be imitated performs the target object interaction task, and a second object trajectory of the object in the target object interaction task;

[0150] Step 820: input the second hand trajectory and the second object trajectory into the trained action model to obtain a second target action output by the action model;

[0151] Step 830: Control the dexterous robotic hand according to the second target action to perform the target object interaction task.

[0152] When performing the learned target object interaction task, the mechanical dexterous hand can use the model and knowledge obtained in the training stage to directly complete the task. By obtaining the trajectory of the reference hand and the trajectory of the object, and inputting these into the trained action model, the target action for task execution can be quickly generated. This method not only reduces the computational overhead of task execution, but also improves the completion efficiency and accuracy of known tasks. Specifically, when the computer device determines that the target object interaction task is a learned task, it records the hand action trajectory of the reference hand when completing the grasping and placing task, that is, the second hand trajectory, and records the motion trajectory of the object during the task, that is, the second object trajectory. Then, the computer device inputs the second hand trajectory and the second object trajectory into the trained action model to obtain the second target action output by the action model. Among them, the computer device inputs the second hand trajectory and the second object trajectory into the trained action model to obtain the second target action output by the action model, which is essentially to output the target imitation action based on the second hand trajectory through the trained imitation action model, and output the target residual action based on the second object trajectory through the trained residual action model, and obtain the final target action based on the fusion of the target imitation action and the target residual action, that is, the second target action.

[0153] In the above embodiment, when the target object interaction task is a learned task, the second hand trajectory and the second object trajectory are directly used as inputs, and the corresponding target action can be output using the trained action model, without the need to design a reward function specifically for the task and conduct specific training on the entire action model, which greatly improves the efficiency of model training and application, and enables the action model to adapt to the complex and changeable task requirements in the actual environment. In addition, by integrating imitation and residual actions, the mechanical dexterous hand can accurately imitate the reference action and meet the task requirements of complex and delicate tasks.

[0154] In complex object interaction tasks, the mechanical dexterous hand not only needs to output reasonable actions, but also must ensure that these actions are physically and operationally feasible. By setting action compliance conditions, the action amplitude of the mechanical dexterous hand is limited so that it is within the operable range of the device, and the applied force matching the action is further determined, thereby achieving precise control. This control strategy can effectively prevent the occurrence of abnormal actions, improve the safety of operations and the stability of task execution. To this end, in some embodiments, the mechanical dexterous hand is controlled according to the first target action, including: determining whether the first target action meets the action compliance conditions; the action compliance conditions are at least used to constrain the action amplitude of the mechanical dexterous hand to be within the operable range; when the first target action meets the action compliance conditions, determining the applied force matching the first target action; applying the applied force to the mechanical dexterous hand to achieve control of the mechanical dexterous hand.

[0155] Among them, the action compliance conditions refer to the standards or rules used to constrain the actions of the mechanical dexterous hand to ensure that the actions it generates meet the operational requirements in terms of physics, function and safety. The main purpose is to limit the movement amplitude, posture, speed, torque and other parameters of the mechanical dexterous hand within the operable range of the mechanical dexterous hand, so as to ensure the smooth completion of the task while avoiding damage to the equipment or target objects. These conditions can be defined by preset parameter thresholds, dynamic real-time feedback or by specific task environments. For example, compliance conditions usually include the rotation angle range of the finger joints, the wrist movement amplitude, the limit on the size of the applied force, etc.

[0156] In some embodiments, the action compliance conditions include, but are not limited to, one or more of kinematic constraints, dynamic constraints, or environmental constraints. Among them, kinematic constraints ensure that the joint angles, positions, and speeds of the mechanical dexterous hand are within the operable range. Dynamic constraints are used to ensure that the generated actions do not exceed the load capacity of the robotic arm or trigger safety limits. Environmental constraints are determined based on the state of the object or the task objectives, such as avoiding interference with environmental objects or violating task rules. Exemplarily, when the fingers of the mechanical dexterous hand perform a grasping action, the rotation angle of the joint must be within the range of 30 to 90 degrees to prevent the mechanical finger joints from being excessively bent and causing damage to the equipment, while ensuring the stability of the grasping action. For example, the rotation amplitude of the mechanical dexterous wrist shall not exceed 60 degrees to ensure the operational stability of the mechanical wrist and prevent joint damage caused by excessive rotation angles. For example, the clamping force applied by the mechanical dexterous hand on the object shall not exceed 10N to avoid damage to the target object due to excessive force and protect the mechanical structure. For example, the end effector of a robotic hand must not move faster than 0.5 m / s during a task, and its acceleration must not exceed 1 m / s. 2 etc., to prevent the grasped object from slipping or the device from vibrating due to too fast movements. For another example, when the mechanical dexterous hand grasps an object, the contact area of ​​the finger must be located on the specified surface of the object to ensure that the grasping force acts on the reasonable force area of ​​the object to avoid grasping failure. If the first target action does not meet the action compliance conditions, in some embodiments, the computer device can regenerate the imitation action and / or residual action. During this period, the computer device can adjust the weight in the reward, etc. After determining that the first target action meets the action compliance conditions, the computer device further determines the applied force that matches the first target action. The applied force is a six-dimensional vector in space, corresponding to the DoF degree of freedom of the mechanical dexterous hand. Furthermore, the computer device generates a control instruction to instruct the hardware in the mechanical dexterous hand to apply the applied force to the mechanical dexterous hand through operation to achieve control of the mechanical dexterous hand.

[0157] In the above embodiments, by setting action compliance conditions, equipment damage or task failure caused by excessive operation of the robotic hand is avoided, and by matching the applied force, the robotic hand can achieve stable operation in different object and task environments, thereby improving the safety and accuracy of the robotic hand in performing complex object interaction tasks.

[0158] The training method for the motion model of a mechanical dexterous hand provided in an embodiment of the present invention may be performed by a training device for the motion model of a mechanical dexterous hand. In the embodiment of the present invention, the training device for the motion model of a mechanical dexterous hand performs the training method for the motion model of a mechanical dexterous hand as an example to illustrate the training device for the motion model of a mechanical dexterous hand provided in an embodiment of the present invention.

[0159] The embodiment of the present invention also provides a training device for a motion model of a mechanical dexterous hand, which is applied to a computer device. Figure 8 As shown, the training device for the action model of the mechanical dexterous hand includes a first acquisition module 801, a first training module 802, a second training module 803 and a first determination module 804. Among them:

[0160] A first acquisition module 801 is used to acquire a first reference trajectory of a reference hand, and to acquire a second reference trajectory of an object when the reference hand performs an object interaction task;

[0161] The first training module 802 is used to determine the imitation reward based on the first reference trajectory, the wrist posture and the finger posture of the mechanical dexterous hand, and perform imitation training on the action model to be trained to obtain an imitation action model; the imitation reward at least includes a wrist tracking reward and a finger imitation reward;

[0162] The second training module 803 is used to determine the residual reward based on the second reference trajectory and the object interaction state, and perform residual training on the imitation action model to obtain the residual action model; the residual reward at least includes an imitation reward, an object tracking reward and a contact force reward;

[0163] The first determination module 804 is used to obtain a motion model based on the imitation motion model and the residual motion model; when the motion model is applied, the mechanical dexterous hand imitates the motion of the reference hand.

[0164] According to the training device for the motion model of the mechanical dexterous hand provided by the embodiment of the present invention, by introducing the first reference trajectory and reward function for coarse-grained imitation training, the mechanical dexterous hand can accurately imitate the motion characteristics of the reference hand and achieve highly simulated motion reproduction, and then by combining the second reference trajectory and the object interaction state for fine-grained optimization training, the mechanical dexterous hand can not only imitate the motion of the reference hand, but also dynamically adapt to the operation tasks of complex objects. In addition, by training in stages, firstly realizing basic imitation by imitating the motion model, and then improving the interactive task performance by the residual model, the training efficiency and model effect can be significantly improved.

[0165] In some embodiments, the first training module is also used to obtain the perception information of the mechanical dexterous hand; take the mechanical dexterous hand imitating the first reference trajectory and perception information as the first state, and take the wrist posture and finger posture as the action, input the first state into the action model to be trained, and obtain the imitation action output by the action model to be trained; determine the imitation reward of the mechanical dexterous hand performing the imitation action in the first state; the imitation reward includes at least a wrist tracking reward and a finger imitation reward; imitate training is performed on the action model to be trained with maximizing the imitation reward as the training goal, and an imitation action model is obtained.

[0166] In some embodiments, the above-mentioned device also includes a first termination module, which is used to determine a first deviation parameter in the imitation training process; the first deviation parameter represents the degree of deviation between the motion trajectory of the mechanical dexterous hand and the first reference trajectory; when the first deviation parameter exceeds the first deviation threshold, the training is restarted.

[0167] In some embodiments, the second training module is also used to determine the object interaction state based on the second reference trajectory of the object when the reference hand performs the object interaction task; take the first state and the object interaction state as the second state, and take the wrist posture residual and the finger posture residual as the residual action, input the second state into the imitation action model, and obtain the residual action output by the imitation action model; determine the residual reward of the mechanical dexterous hand performing the residual action in the second state; the residual reward includes at least imitation reward, object tracking reward and contact force reward; residual training is performed on the imitation action model with the maximization of the residual reward as the training goal to obtain a residual action model.

[0168] In some embodiments, the above-mentioned device also includes a second termination module, which is used to determine a second deviation parameter in the residual training process; the second deviation parameter represents the degree of deviation between the object posture and the second reference trajectory; when the second deviation parameter exceeds the second deviation threshold, the training is restarted.

[0169] The control method of the mechanical dexterous hand provided in the embodiment of the present invention can be executed by the control device of the mechanical dexterous hand. In the embodiment of the present invention, the control device of the mechanical dexterous hand executing the control method of the mechanical dexterous hand is taken as an example to illustrate the control device of the mechanical dexterous hand provided in the embodiment of the present invention.

[0170] The embodiment of the present invention also provides a control device for a mechanical dexterous hand, which is applied to a computer device. Fig. 9 As shown, the control device of the mechanical dexterous hand includes a second determination module 901, a second acquisition module 902, a first input module 903, a third training module 904, a second input module 905, a fusion module 906 and a control module 907. Among them:

[0171] A second determination module 901 is used to determine a target object interaction task;

[0172] A second acquisition module 902 is used to acquire a first hand trajectory of a reference hand to be imitated when performing the target object interaction task and a first object trajectory of an object in the target object interaction task when the target object interaction task is an unlearned task;

[0173] The first input module 903 is used to input the first hand trajectory into the trained imitation action model to obtain the target imitation action output by the imitation action model;

[0174] The third training module 904 is used to train the imitation action model based on the first object trajectory and the object interaction state with the residual reward maximization as the training goal, so as to obtain a target residual action model adapted to the target object interaction task; the residual reward at least includes an imitation reward, an object tracking reward and a contact force reward adapted to the imitation action model;

[0175] A second input module 905 is used to input the first object trajectory into the target residual motion model to obtain the target residual motion output by the target residual motion model;

[0176] A fusion module 906 is used to fuse the target imitation action and the target residual action to obtain a first target action;

[0177] The control module 907 is used to control the dexterous mechanical hand according to the first target action to perform the target object interaction task.

[0178] According to the training device for the motion model of the mechanical dexterous hand provided by the embodiment of the present invention, when controlling the mechanical dexterous hand, it is determined whether the target object interaction task is a new task that has not been learned, and when it is a new task, the target imitation action is output through the trained imitation action model to perform preliminary imitation and tracking of the reference hand trajectory, and for the specific new task, the residual action model is retrained, and the target residual action is output through the retrained residual action model to achieve fine imitation under the new task. Therefore, in the scenario of facing various new tasks, there is no need to design a reward function specifically for each task and perform specific training on the entire action model. Only a part of the action model, that is, the residual action model, needs to be retrained, which greatly improves the efficiency of model training and application, and enables the action model to adapt to the complex and changeable task requirements in the actual environment. In addition, by integrating imitation and residual action, the mechanical dexterous hand can accurately imitate the reference action and meet the task requirements of complex and fine tasks.

[0179] In some embodiments, the second acquisition module is also used to obtain the second hand trajectory of the reference hand to be imitated when the target object interaction task is performed, and the second object trajectory of the object in the target object interaction task when the target object interaction task is a learned task; input the second hand trajectory and the second object trajectory into the trained action model to obtain the second target action output by the action model; and control the mechanical dexterous hand according to the second target action to perform the target object interaction task.

[0180] In some embodiments, the control module is also used to determine whether the first target action satisfies the action compliance condition; the action compliance condition is at least used to constrain the action amplitude of the mechanical dexterous hand to be within the operable range; when the first target action satisfies the action compliance condition, determine the applied force that matches the first target action; and apply the applied force to the mechanical dexterous hand to achieve control of the mechanical dexterous hand.

[0181] The training device for the motion model of the mechanical dexterous hand or the control device of the mechanical dexterous hand in the embodiment of the present invention can be a computer device, or a component in the computer device, such as an integrated circuit or a chip. The computer device can be a terminal device or a server. Exemplarily, the computer device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted computer device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR) / virtual reality (Virtual Reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (Ultra-mobile Personal Computer, UMPC), a netbook or a personal digital assistant (Personal Digital Assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), a television (Television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present invention.

[0182] The embodiment of the present invention further provides a mechanical dexterous hand, the mechanical dexterous hand comprising: Fig. 9 The control device of the mechanical dexterous hand is shown.

[0183] The training device for the motion model of the mechanical dexterous hand or the control device of the mechanical dexterous hand in the embodiment of the present invention may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present invention.

[0184] The training device for the action model of the mechanical dexterous hand provided by the embodiment of the present invention can achieve Figures 4 to 6 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0185] The control device of the mechanical dexterous hand provided by the embodiment of the present invention can realize Figure 7 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0186] In some embodiments, Fig.10As shown, an embodiment of the present invention further provides a computer device 1000, including a processor 1001, a memory 1002, and a computer program stored in the memory 1002 and executable on the processor 1001. When the program is executed by the processor 1001, each process of the above-mentioned method embodiments is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0187] It should be noted that the computer device in the embodiment of the present invention includes the mobile computer device and the non-mobile computer device mentioned above.

[0188] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned training method for the motion model of a mechanical dexterous hand or the control method of a mechanical dexterous hand are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0189] The processor is the processor in the computer device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0190] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned training method for the motion model of a mechanical dexterous hand or the control method of a mechanical dexterous hand.

[0191] The processor is the processor in the computer device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0192] An embodiment of the present invention further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned training method for the motion model of a mechanical dexterous hand or the control method of a mechanical dexterous hand, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0193] It should be understood that the chip mentioned in the embodiment of the present invention may also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0194] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present invention is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0195] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0196] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0197] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0198] If there is no special description, all embodiments and optional embodiments of the present invention can be combined with each other to form a new technical solution. If there is no special description, all technical features and optional technical features of the present invention can be combined with each other to form a new technical solution.

[0199] If not otherwise specified, all steps of the present invention may be performed sequentially or randomly, preferably sequentially. For example, the method includes steps (a) and (b), which means that the method may include steps (a) and (b) performed sequentially, or may include steps (b) and (a) performed sequentially. For example, mentioning that the method may also include step (c) means that step (c) may be added to the method in any order, for example, the method may include steps (a), (b) and (c), or may include steps (a), (c) and (b), or may include steps (c), (a) and (b), etc.

[0200] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A training method for a motion model of a mechanical dexterous hand, characterized in that: The method comprises: Acquire a first reference trajectory of a reference hand, and acquire a second reference trajectory of an object when the reference hand performs an object interaction task; Based on the first reference trajectory, the wrist posture and the finger posture of the mechanical dexterous hand, an imitation reward is determined, and the action model to be trained is subjected to imitation training to obtain an imitation action model; the imitation reward includes at least a wrist tracking reward and a finger imitation reward; Based on the second reference trajectory and the object interaction state, a residual reward is determined, and residual training is performed on the imitation action model to obtain a residual action model; the residual reward includes at least an imitation reward, an object tracking reward, and a contact force reward; Based on the imitation motion model and the residual motion model, a motion model is obtained; when the motion model is applied, the mechanical dexterous hand imitates the motion of the reference hand.

2. The training method for the action model of a mechanical dexterous hand according to claim 1, characterized in that: The step of determining the imitation reward based on the first reference trajectory, the wrist posture and the finger posture of the mechanical dexterous hand, and performing imitation training on the action model to be trained to obtain the imitation action model includes: Obtaining sensory information of the robotic dexterous hand; Taking the mechanical dexterous hand imitating the first reference trajectory and the perception information as a first state, taking the wrist posture and the finger posture as an action, inputting the first state into the action model to be trained, and obtaining the imitation action output by the action model to be trained; Determine an imitation reward for the mechanical dexterous hand to perform the imitation action in the first state; the imitation reward includes at least a wrist tracking reward and a finger imitation reward; The action model to be trained is subjected to imitation training with the maximization of the imitation reward as a training goal to obtain an imitation action model.

3. The training method for the action model of a mechanical dexterous hand according to claim 2, characterized in that: During the imitation training process, the method further comprises: Determining a first deviation parameter during the imitation training process; the first deviation parameter represents a degree of deviation between the motion trajectory of the mechanical dexterous hand and the first reference trajectory; When the first deviation parameter exceeds a first deviation threshold, the training is restarted.

4. The training method for the action model of a mechanical dexterous hand according to claim 2 or 3, characterized in that: The determining of the residual reward based on the second reference trajectory and the object interaction state, and performing residual training on the imitation action model to obtain the residual action model includes: determining an object interaction state based on a second reference trajectory of the object when the reference hand performs the object interaction task; Taking the first state and the object interaction state as the second state, taking the wrist posture residual and the finger posture residual as the residual action, inputting the second state into the imitation action model to obtain the residual action output by the imitation action model; Determining a residual reward for the mechanical dexterous hand to perform the residual action in the second state; the residual reward at least includes an imitation reward, an object tracking reward, and a contact force reward; The imitation action model is subjected to residual training with the residual reward maximization as a training objective to obtain a residual action model.

5. The training method for the action model of a mechanical dexterous hand according to claim 4, characterized in that: During the residual training process, the method further includes: Determine a second deviation parameter in the residual training process; the second deviation parameter represents the degree of deviation between the object posture and the second reference trajectory; When the second deviation parameter exceeds a second deviation threshold, the training is restarted.

6. A control method for a mechanical dexterous hand, characterized in that: The method comprises: Determine the target object interaction task; When the target object interaction task is an unlearned task, obtaining a first hand trajectory of a reference hand to be imitated when performing the target object interaction task and a first object trajectory of an object in the target object interaction task; Inputting the first hand trajectory into a trained imitation action model to obtain a target imitation action output by the imitation action model; Taking residual reward maximization as a training goal, the imitation action model is trained based on the first object trajectory and the object interaction state to obtain a target residual action model adapted to the target object interaction task; the residual reward at least includes an imitation reward, an object tracking reward, and a contact force reward adapted to the imitation action model; Inputting the first object trajectory into the target residual motion model to obtain a target residual motion output by the target residual motion model; Fusing the target imitation action and the target residual action to obtain a first target action; The dexterous robotic hand is controlled according to the first target action to perform the target object interaction task.

7. The control method of the mechanical dexterous hand according to claim 6, characterized in that: The method further comprises: In the case where the target object interaction task is a learned task, obtaining a second hand trajectory of the reference hand when the reference hand to be imitated performs the target object interaction task, and a second object trajectory of the object in the target object interaction task; Inputting the second hand trajectory and the second object trajectory into a trained motion model to obtain a second target motion output by the motion model; The dexterous robotic hand is controlled according to the second target action to perform the target object interaction task.

8. The control method of the mechanical dexterous hand according to claim 6 or 7, characterized in that: The controlling the dexterous mechanical hand according to the first target action comprises: Determining whether the first target action satisfies an action compliance condition; the action compliance condition is at least used to constrain the action amplitude of the mechanical dexterous hand to be within an operable range; In a case where the first target action satisfies an action compliance condition, determining an applied force that matches the first target action; The applied force is applied to the dexterous mechanical hand to achieve control of the dexterous mechanical hand.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for training the motion model of a mechanical dexterous hand according to any one of claims 1 to 5 or the method for controlling a mechanical dexterous hand according to any one of claims 6 to 8 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the training method for the motion model of a mechanical dexterous hand as described in any one of claims 1 to 5 or the control method for a mechanical dexterous hand as described in any one of claims 6 to 8 is implemented.

Images