Dexterous hand action generation method and system
The method and system generate dexterous hand actions using real-time sensory input, addressing the limitations of deep learning by enabling universal applicability across varied objects and environments.
Patent Information
- Application Number
- CN202510617864.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-14
- Publication Date
- 2025-07-15
AI Technical Summary
The existing smart hand movement learning methods can only be targeted at specific scenarios, have poor generalization and require a lot of data training, making it difficult to adapt to items of different shapes.
The pre-constructed image analysis module and mechanical model library are used to generate finger execution modes and joint force trajectories in combination with visual inputs, and dynamically generate dexterous hand movements without relying on deep learning training.
It realizes the generation of generalized actions of skilled hands on different items, solves the problems of generalization and lack of data, and dynamically adapts to various scenarios.
Smart Images
Figure CN120307295A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of robotics, and more particularly, to a method and system for generating motions of a dexterous hand. Background Art
[0002] The dexterous hand is the core of the interaction between a humanoid robot / robot arm and an object. Its motion ability is crucial for the interaction. Currently, the motions of the dexterous hand are mainly completed by using the method of reinforcement learning. The general method is to wear a data acquisition sensor on the human hand, collect the motion data of the human hand, and input it to a neural network for training. After the training is completed, the dexterous hand of the robot can learn this motion. There is a natural problem with this method, that is, each time of learning can only learn a specific motion. For example, through reinforcement learning, the dexterous hand learns the motion of "holding a teacup", but this motion is only applicable to the shape and size of the teacup during the learning. Therefore, the following problems exist:
[0003] 1. If a teacup with a different shape (such as with or without a handle, different handle shapes) or size is changed, the success rate of the dexterous hand's motion of "holding a teacup" will drop significantly, and it is necessary to re-learn to adapt to the new teacup shape. 2. Each time of learning a new scene motion requires a large amount of data, so the problem of data scarcity will be faced. 3. The shapes of teacups are diverse, and other items are even more strange. There are thousands of scenes in life, and it is almost impossible to use the traversal method to learn all the life scenes. As a result, the generalization of the current dexterous hand motions is weak. Summary of the Invention
[0004] In view of the deficiencies of the prior art, the present invention provides a method and system for generating motions of a dexterous hand.
[0005] According to one aspect of the present invention, a method for generating motions of a dexterous hand is provided, including:
[0006] Performing image analysis on the acquired item image by using a pre-constructed image analysis module in combination with a pre-constructed material property table to obtain the item information of the item;
[0007] Using the object feature - mechanical model applicability matrix in a pre-constructed finger degree-of-freedom allocator to determine a feasible mechanical model for combination from a pre-constructed mechanical model library according to the item information to obtain a combined mechanical model;
[0008] Using the finger degree-of-freedom allocator to allocate the degrees of freedom of the fingers to the combined mechanical model to generate a finger execution pattern and a resultant force trajectory;
[0009] Generating finger vectors of each finger according to the finger execution pattern and the resultant force trajectory;
[0010] Generate the actions of the dexterous hand based on the finger vectors of each finger.
[0011] Optionally, the item information includes: the characteristics, material, mass, handle position, operation point position, prohibited force-bearing surface, and visual position information of the item.
[0012] Optionally, the mechanical model library includes multiple sub-model libraries, and the information in the sub-model libraries includes: the names of each mechanical model, the directions of each force in each mechanical model, the requirements for auxiliary actions, and the feasible unit combinations. Among them, in the feasible unit combinations, the thumb takes the phalanx as the unit, and the other fingers take the whole finger as the unit.
[0013] Optionally, generate the finger vectors of each finger according to the finger execution mode and the resultant force trajectory, including:
[0014] Use a force resolver to decompose the resultant force trajectory into the finger force vectors of specific fingers according to the finger execution mode;
[0015] Use a position generator to generate the finger position vectors of each finger according to the finger execution mode and the item information;
[0016] Determine the finger vectors of each finger according to the finger force vectors and the finger position vectors, where the finger vectors include the vectors of each phalanx of the thumb and the end vectors of the fingers other than the thumb.
[0017] Optionally, it further includes: sending the tactile signals collected by the tactile sensor to the force solver to provide end force feedback, forming an end force closed loop.
[0018] Optionally, it further includes: sending the visual position information to the position generator to provide end position feedback, forming an end position closed loop.
[0019] Optionally, generate the actions of the dexterous hand based on the finger vectors of each finger, including:
[0020] Solve the finger vectors to obtain the motion angles and torques of each joint;
[0021] Based on the drive boards of each joint, control the corresponding motor modules to rotate according to the motion angles and torques of each joint, generating the actions of the dexterous hand.
[0022] Optionally, send the real-time motion angles and torques of the motor modules to the drive boards of each joint to provide motor operation feedback, forming a closed loop of motor motion angles and torques.
[0023] According to another aspect of the present invention, there is provided a motion generation system for a dexterous hand, comprising: an image acquisition module, a common material property table module, an image analysis module, a mechanical model library module, a finger degree of freedom allocator module, a force decomposer module, a finger position generator module, a control signal selector module, a motion solver module, a drive board module, and a motor module, wherein
[0024] The image analysis module is used to analyze the item image collected by the image acquisition module in combination with the material property table in the common material property table module to obtain the item information of the item;
[0025] The finger degree of freedom allocator is used to determine a feasible mechanical model from the mechanical model library module according to the item information by using the object feature - mechanical model applicability matrix therein, combine them to obtain a combined mechanical model, and allocate the degrees of freedom of the fingers to the combined mechanical model to generate a finger execution mode and a resultant force trajectory;
[0026] The control signal selector module, the force decomposer module, and the finger position generator module are used to generate finger vectors for each finger according to the finger execution mode and the resultant force trajectory;
[0027] The motion solver module, the drive board module, and the motor module are used to generate the motion of the dexterous hand based on the finger vectors of each finger.
[0028] Optionally, the motion generation system of the dexterous hand further comprises: a tactile sensor module, which sends tactile signals to a force resolver to provide end - force feedback and form an end - force closed - loop.
[0029] Optionally, the image analysis module is further used to send the output item visual position information to the position generator to provide end - position feedback; and the motor module is further used to send real - time torque and position information to the drive board module to provide motor operation feedback and form a closed - loop of the motor motion angle and torque.
[0030] Thus, through the motion generation system of the dexterous hand provided by the present invention, the motion of the fingers will no longer depend on deep - learning training for specific items / scenes, but dynamically generate the motion of the fingers according to real - time input information from sensors such as visual cameras. Furthermore, it truly enables the dexterous hand of the robot to be universal. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] By referring to the following drawings, the exemplary embodiments of the present invention can be more fully understood:
[0032] Figure 1 is a schematic structural diagram of a motion generation system for a dexterous hand provided by an exemplary embodiment of the present invention;
[0033] Figure 2It is a schematic diagram of the implementation of an image analysis module provided by an exemplary embodiment of the present invention;
[0034] Figure 3 It is a schematic diagram of an entry in a mechanical model library provided by an exemplary embodiment of the present invention;
[0035] Figure 4 It is a schematic diagram of an entry in a three - point lever force model provided by an exemplary embodiment of the present invention;
[0036] Figure 5 It is a schematic diagram of an entry in a two - force grasping (grab) model from the top provided by an exemplary embodiment of the present invention;
[0037] Figure 6 It is a schematic diagram of an entry in a pressing model provided by an exemplary embodiment of the present invention;
[0038] Figure 7 It is a schematic diagram of an entry in a single - force hooking model provided by an exemplary embodiment of the present invention;
[0039] Figure 8 It is a schematic diagram of an entry in a three - force holding model provided by an exemplary embodiment of the present invention;
[0040] Figure 9 It is a schematic diagram of an entry in a two - force holding (grasp) model from the side provided by an exemplary embodiment of the present invention;
[0041] Figure 10 It is a schematic flowchart of a method for generating the motion of a dexterous hand provided by an exemplary embodiment of the present invention. Detailed implementation manners
[0042] Next, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments of the present invention. It should be understood that the present invention is not limited by the exemplary embodiments described herein.
[0043] It should be noted that: Unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present invention.
[0044] Those skilled in the art can understand that terms such as "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices, or modules, etc., and neither represent any specific technical meaning nor indicate an inevitable logical order between them.
[0045] It should also be understood that in the embodiments of the present invention, "a plurality of" may refer to two or more, and "at least one" may refer to one, two, or more.
[0046] It should also be understood that for any component, data, or structure mentioned in the embodiments of the present invention, in the absence of a clear definition or contrary implications in the context, it is generally understood as one or more.
[0047] In addition, the term "and / or" in the present invention is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present invention generally represents an "or" relationship between the associated objects before and after.
[0048] It should also be understood that the description of each embodiment of the present invention emphasizes the differences between the embodiments. The same or similar parts can be referred to each other. For the sake of brevity, they will not be described in detail one by one.
[0049] At the same time, it should be understood that for the convenience of description, the dimensions of each part shown in the drawings are not drawn according to the actual proportional relationship.
[0050] The following description of at least one exemplary embodiment is actually only illustrative and in no way limits the present invention and its application or use.
[0051] Technologies, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but in appropriate cases, the technologies, methods, and devices should be regarded as part of the specification.
[0052] It should be noted that similar reference numerals and letters indicate similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.
[0053] Figure 1 is a schematic structural diagram of the motion generation system of the dexterous hand provided by the first aspect of the embodiment of the present invention. As Figure 1 shown, the motion generation system of the dexterous hand includes: an image acquisition module, a common material property table module, an image analysis module, a mechanical model library module, a finger degree-of-freedom allocator module, a force decomposer module, a finger position generator module, a control signal selector module, a motion solver module, a drive board module, and a motor module, where
[0054] The image analysis module is used to analyze the item image collected by the image acquisition module in combination with the material property table in the common material property table module to obtain the item information of the item;
[0055] The finger degree-of-freedom allocator is used to determine a feasible mechanical model from the mechanical model library module according to the item information by using the object feature-mechanical model applicability matrix therein, combine them to obtain a combined mechanical model, and allocate the degrees of freedom of the fingers to the combined mechanical model to generate a finger execution mode and a resultant force trajectory;
[0056] The control signal selector module, the force resolver module, and the finger position generator module are used to generate finger vectors for each finger according to the finger execution mode and the resultant force trajectory;
[0057] The motion solver module, the drive board module, and the motor module are used to generate the actions of the dexterous hand based on the finger vectors of each finger.
[0058] Specifically, the dexterous hand learns actions through reinforcement learning. Essentially, it is a kind of "imitation", which can be analogized to a child who sees an adult doing a certain action in a certain scenario and then follows suit. In this imitation process, the child does not understand the mechanism of action and the intention of each step. The action itself being imitated occurs in a specific scenario (the "scenario" includes all external factors such as the shape, size, position of the object being operated, and the surrounding environment, etc., which are not the robot's own body). Therefore, the child can only "invoke" this action when encountering the same scenario. So, the "imitation" has strict requirements for the scenario. In order to fundamentally solve the generalization problem of the dexterous hand's deep learning of actions, the present invention designs a brand-new generative architecture. The dexterous hand will "generate" actions according to the information such as the shape and size of the item obtained from the visual input, rather than simply "imitating", thus fundamentally solving the generalization problem. At the same time, because there is no need for imitation learning, the problems of data scarcity and inability to traverse all scenarios are naturally solved.
[0059] Furthermore, referring to Figure 1 As shown, the action generation system of the dexterous hand provided by the present invention can generate corresponding hand actions and perform corresponding operations according to the information such as the shape, size, function, material, and position of the item to be operated, in combination with the target task. This architecture is composed of 12 modules / modules, namely an image acquisition module (which can be but is not limited to a binocular camera module, and the present invention uses a binocular camera module for the embodiment), a common material property table module, an image analysis module, a mechanical model library module, a finger degree-of-freedom allocator module (abbreviated as the "allocator module"), a tactile sensor module, a force resolver module, a finger position generator module (abbreviated as the "position generator module"), a control signal selector module, a motion solver module, a drive board module, and a motor module.
[0060] The camera captures an image of an object, and the image analysis module analyzes the image. By combining with the object attribute table, information such as the characteristics, material, mass, handle position, and operation point position of the object is obtained, and the information is sent to the distributor module. According to the relevant information of the object and in combination with the task objective, the distributor selects a feasible mechanical model from the mechanical model library for combination, and assigns the degrees of freedom of the fingers to the selected mechanical model to execute the force application requirements in the mechanical model. The result of the assignment is recorded as the "finger execution mode". The distributor sends the generated finger execution mode and resultant force trajectory to the subsequent force decomposer and position generator. The force decomposer decomposes the resultant force trajectory into finger force vectors of specific fingers according to the finger execution mode and sends them to the subsequent control signal selector. The position generator generates finger position vectors of each finger according to the finger execution mode and in combination with the output information of the aforementioned image analysis module and sends them to the subsequent control signal selector. The control signal selector selects the finger force vector or the finger position vector according to relevant conditions and sends it to the subsequent motion calculation module. The motion calculation module calculates the motion angles and torques of each joint for the input finger vectors and sends them to the drive boards corresponding to the respective joints. The drive boards control the corresponding motor modules to rotate according to the input motion angle and torque information.
[0061] Among them, whether to select the finger force vector or the finger position vector for the finger vector is related to the motion control of the dexterous hand. Before contacting the object, visual feedback is used and the position vector is selected. After contacting the object, force feedback is used and the force vector is selected.
[0062] The tactile signals of the tactile sensors are sent to the force resolver to provide end force feedback and form an end force closed loop. The visual position information of the object output by the image analysis module is sent to the position generator to provide end position feedback. The motor modules send real-time torques and motion angles to the drive boards to provide motor operation feedback and form closed loops for motor torque and position.
[0063] In an embodiment of the present invention, as Figure 2 shown, the image analysis module analyzes the images of the binocular cameras, including object feature recognition; object material recognition, mass estimation; handle and operation point recognition, and prohibited force surface recognition. Information such as the characteristics, material, mass, handle position, and operation point position of the object is obtained.
[0064] Specifically, by analyzing the images of the binocular cameras, the shape and material information of the object are recognized through methods such as neural networks, and then through geometric measurements, the size information of the object is obtained, and thus the volume of the object can be calculated. By referring to the common material property table, the density and surface friction coefficient of the object are obtained, and thus the mass of the object and the magnitude of the finger end force can be calculated.
[0065] Meanwhile, the image analysis module analyzes object features (including but not limited to object shape, size, etc.), and identifies information such as the position of the item handle, the shape of the handle, the position of the operation point, and the shape of the operation point. This provides important reference for the actions generated by the subsequent dispenser.
[0066] As Figure 2 shown, the green module represents a pre-trained neural network that completes specific functions.
[0067] The images captured by the camera are respectively processed through four independent processing lines.
[0068] The link of the first line is: image input - simplify the object - measure the geometric dimensions of the simplified model - calculate the volume of the object;
[0069] The link of the second line is: image input - identify the object material - look up the table to obtain the object density;
[0070] The link of the third line is: image input - analyze the object features - obtain the object features;
[0071] The link of the fourth line is: image input - simultaneously send to the handle recognition network, the operation point recognition network, and the prohibited force surface recognition network - obtain the handle position, the operation point position, and the prohibited force surface of the object.
[0072] Among them, the output of the first line (object volume) and the output of the second line (object density) are combined to calculate the object weight.
[0073] In an embodiment of the present invention, the common material property table pre-records the physical properties of common materials in life, including but not limited to parameters such as density, stiffness, brittleness, and surface friction coefficient. It is used to estimate the mass of the item and the magnitude of the force required during dexterous hand operation.
[0074] In an embodiment of the present invention, in the mechanical model library of the mechanical model library module, it is divided into multiple sub-model libraries according to the number of required forces. In each sub-model library, several mechanical model entries are included.
[0075] Each mechanical model entry includes the following information: mechanical model; the directions of the forces in the mechanical model; auxiliary action requirements; feasible unit combinations, with the thumb taking the phalanx as the unit and the other fingers taking the whole finger as the unit. As Figure 3 shown, the mechanical model library includes a three-force model library, a two-force model library, and a single-force model library. Among them, Figure 4 is the entry information of the three-point lever force model, Figure 5 is the entry information of the two-force grasping (grab) model from the top, Figure 6 is the entry information of the single-force pressing model.
[0076] In an embodiment of the present invention, the finger degree-of-freedom allocator module receives visual information from the previous-stage image analysis module, selects one or more feasible mechanical models from each sub-module in the mechanical model library according to the characteristics of the object such as its shape and texture, and combines them according to the operation requirements (such as the need to pick up / press down). According to the item information of the group, a possible finger allocation scheme is generated, and combined with the relevant information of the item to be operated, the possible finger allocation schemes are evaluated to obtain the optimal finger allocation scheme, denoted as the "finger execution mode", and sent to the subsequent stage.
[0077] Specifically, the specific implementation of the finger degree-of-freedom allocation model is as follows:
[0078] An object can be described from different perspectives, as shown in Table 1.
[0079] Table 1
[0080]
[0081] Combining different descriptions together to form "object features", and then establishing a mapping table (1.0 indicates complete applicability, 0 indicates complete inapplicability) between different features and applicable mechanical models according to the degree of matching. An object feature - mechanical model applicability matrix is formed, as shown in Table 2. The object feature - mechanical model applicability matrix is included in the finger degree-of-freedom allocator module as a sub-module.
[0082] Table 2
[0083]
[0084] In an embodiment of the present invention, the tactile sensor is responsible for converting the pressure signal at the end of the actuator into an electrical signal and directly measuring the pressure at the end of the actuator. It is an important input component in the end torque closed loop.
[0085] In an embodiment of the present invention, the finger degree-of-freedom allocator outputs the finger execution mode to the force decomposer. The finger execution mode includes the mechanical model, the directions of each force in the mechanical model, and the corresponding execution units. The force decomposer decomposes the resultant force trajectory into the component force vectors of each end finger according to this information and sends them to the subsequent stage.
[0086] In an embodiment of the present invention, the position generator receives the finger execution mode generated by the finger degree-of-freedom allocator, and generates the finger position vectors of each finger according to the characteristics of the item such as its shape and size given by the image analysis module, and parameters such as the homography matrix of the current camera view, so as to adapt to the shape of the item and realize the preparatory action of grasping the item.
[0087] In an embodiment of the present invention, the control signal selector selects one of the finger force vectors of each finger and the finger position vectors of each finger according to relevant sensing information as the control source and sends it to the subsequent stage.
[0088] In an embodiment of the present invention, the motion calculation module decomposes the input finger vectors into motion components of each actuator in the series drive and sends them to the corresponding motor drive board.
[0089] In an embodiment of the present invention, the motor drive board receives the control quantity from the motion calculation module to control the rotation of the corresponding motor. At the same time, it receives the motor torque feedback (current feedback) and motor position feedback transmitted back by the motor to form a motor torque and position closed loop, and adjusts the output quantity in real time to achieve the purpose of precise control.
[0090] In an embodiment of the present invention, the motor assembly includes components such as a reducer, a motor, and an encoder. It is responsible for converting the input control signal into mechanical motion. At the same time, it generates a position feedback signal and sends it to the drive board to form a motor position closed loop to achieve precise position control.
[0091] In addition, the core of the present invention responsible for motion generation is the mechanical model library + finger degree-of-freedom allocator. Taking a three-finger dexterous hand as an example, the protection scope is not limited to the three-finger dexterous hand, including dexterous hands with any other degrees of freedom. Taking the four tasks of picking up a paper box, lifting a plastic bag, playing the piano, and using a lighter as examples, the generation process of finger motions under this architecture is described below.
[0092] I. Picking up a paper box:
[0093] 1. The visual recognition module analyzes the input data of the camera to obtain that the object to be operated is a paper box. At the same time, by combining the recognition of the material and volume, the weight of the paper box is estimated, and the above information is sent to the allocator.
[0094] 2. After receiving the requirement of "picking up a paper box", the allocator combines the input item information with the item feature - mechanical model application matrix and finds that the suitable model is: "two-finger grasping (pinch) model from the top".
[0095] 3. The allocator retrieves the table item information of the "two-finger grasping (pinch) model from the top", as Figure 5 shown.
[0096] According to the table item information, through permutation and combination, it is found that there are the following several finger allocation schemes for performing two-finger grasping:
[0097] (1) Grasp with (distal phalanx of the thumb, index finger);
[0098] (2) Grasp with (distal phalanx of the thumb, ring finger);
[0099] The allocator's calculation found that the stability of Solution (1) is the best. Assuming that the distal phalanx of the thumb and the index finger are not occupied, the allocator adopts Solution (1), that is: grasping with (the distal phalanx of the thumb, the index finger).
[0100] The allocation scheme is denoted as the "finger execution mode". The allocator sends the finger execution mode to the subsequent force resolver.
[0101] 4. The force resolver module decomposes the resultant force (trajectory) according to the directions of the respective forces in the finger execution mode to obtain the specific force application magnitude for each finger. The force resolver sends the magnitude and direction of the force to the control signal selector.
[0102] 5. The control signal selector selects a vector signal and sends it backward.
[0103] 6. The motion calculation module decomposes the vector signal into signals for each series joint and sends them to the corresponding drive board.
[0104] 7. The drive board controls the rotation of each motor according to the received control signal.
[0105] It can be seen that the above finger movements are dynamically generated based entirely on real-time sensed information such as the external dimensions of the object and the idle states of the respective fingers, and do not require "training" in a specific scenario in advance. Therefore, the problem of generality can be fundamentally solved. In other words, if the object to be operated changes from a cup to something else, the above architecture and method can still be used to generate suitable finger movements.
[0106] According to the above architecture and process, the following examples illustrate how to generate the actions of "lifting a plastic bag", "playing the piano", and "using a lighter".
[0107] II. Lifting a plastic bag:
[0108] 1. The visual recognition module analyzes the input data of the camera, discovers that the object to be operated is a plastic bag, analyzes that the plastic bag needs to be "lifted", identifies the handle position of the plastic bag, and estimates the weight of the plastic bag in combination with the volume recognition of the plastic bag and the judgment of the contents. And sends the above information to the allocator.
[0109] 2. The allocator receives the requirement for the action of "lifting a plastic bag". According to the input item information and in combination with the item feature - mechanical model application matrix, it discovers that the suitable model is: single-force hook
[0110] 3. The allocator retrieves the table item information of "single-force hook", as Figure 7 shown.
[0111] According to the table item information, through permutation and combination, it is found that there are the following several finger allocation schemes for performing the "lifting" action:
[0112] (1) Hook the handle with (index finger).
[0113] (2) Hook the handle with (ring finger).
[0114] (3) Hook the handle with (index finger and ring finger together).
[0115] The allocator calculates and finds that the load capacity of solution (3) is the best. So it is decided to choose solution (3), that is: hook the handle with (index finger and ring finger together).
[0116] The allocation scheme is recorded as "finger execution mode". The allocator sends the finger execution mode to the subsequent force resolver.
[0117] 4. The force resolver module decomposes the resultant force (trajectory) according to the directions of the respective forces in the finger execution mode to obtain the specific force applied by each finger. The force resolver sends the magnitude and direction of the force to the control signal selector.
[0118] 5. The control signal selector selects a vector signal and sends it backward.
[0119] 6. The motion calculation module decomposes the vector signal into signals for each series joint and sends them to the corresponding drive board.
[0120] 7. The drive board controls the rotation of each motor according to the received control signal.
[0121] III. Playing the piano:
[0122] 1. The visual recognition module analyzes the input data of the camera and finds that the object to be operated is a piano key. After analysis, it is obtained that multiple fingers need to press simultaneously to operate the piano key, and the above information is sent to the allocator.
[0123] 2. The allocator receives the requirement of pressing with multiple fingers simultaneously. According to the input item information and combined with the item feature - mechanical model application matrix, it is found that the suitable model is: "press".
[0124] 3. The allocator finds that each finger can complete the task of pressing with multiple fingers by using the "press" model respectively. So it retrieves the entry information of "press", as Figure 7 shown.
[0125] The allocator combines the entry information with the arrangement order of the piano keys and finds the finger allocation scheme for pressing the piano keys with multiple fingers simultaneously (assuming that it is necessary to press do, mi, so of the C chord) as follows:
[0126] (1) Press do with (distal phalanx of thumb), press mi with index finger, and press so with ring finger;
[0127] The above allocation scheme is denoted as the "finger execution mode". The allocator sends the finger execution mode to the subsequent force resolver.
[0128] 4. According to the directions of the respective forces in the finger execution mode, the force resolver module decomposes the resultant force (trajectory) to obtain the specific force applied by each finger. The force resolver sends the magnitude and direction of the force to the control signal selector.
[0129] 5. The control signal selector selects one vector signal and sends it backward.
[0130] 6. The motion calculation module decomposes the vector signal into signals for each series joint and sends them to the corresponding drive board.
[0131] 7. The drive board controls the rotation of each motor according to the received control signal.
[0132] IV. Using a lighter
[0133] 1. By analyzing the input data of the camera, the visual recognition module discovers that the object to be operated is a lighter, and analyzes that operating the lighter requires "grasping" + "pressing the button". At the same time, by combining the recognition of the material and volume, the weight of the lighter is estimated. And the above information is sent to the allocator.
[0134] 2. The allocator receives the requirements of "grasping" + "pressing the button". According to the input item information, it respectively searches for applicable mechanical models in combination with the item feature - mechanical model application matrix. Among them, the effective operation (pressing the button) is searched first.
[0135] (a) For the pressing operation, the allocator finds that the suitable model is:
[0136] "Press";
[0137] (b) For the grasping operation, the found suitable models are:
[0138] "Grasping with three forces";
[0139] "Grasping (grabbing) with two forces from the top";
[0140] "Grasping (holding) with two forces from the side";
[0141] The allocator finds that for the shape of the lighter, "single - force pressing" and "grasping (grabbing) with two forces from the top"
[0142] are contradictory and cannot be selected simultaneously. Therefore, the remaining possible models are:
[0143] "Grasping with three forces";
[0144] "Grasping (holding) with two forces from the side";
[0145] The pressing operation is implemented using the "press" model, and the gripping operation is implemented using the "three-force grip" or "two-force grip (from the side)".
[0146] 3. The dispenser retrieves the table item information of "press", "three-force grip", and "two-force grip (from the side)", as shown respectively in Figure 6 , Figure 8 and Figure 9 .
[0147] According to the table item information, through permutation and combination, it is found that there are the following several finger assignment schemes for performing "grip" + "press":
[0148] (1) Three-force grip + single-force press: Hold the lighter with (proximal phalanx of the thumb, index finger, ring finger), and press the ignition button with (distal phalanx of the thumb);
[0149] (2) Two-force grip + single-force press: Hold the lighter with (proximal phalanx of the thumb, index finger), and press the ignition button with (distal part of the thumb);
[0150] (3) Two-force grip + single-force press: Hold the lighter with (proximal phalanx of the thumb, ring finger), and press the ignition button with (distal phalanx of the thumb);
[0151] The dispenser calculates and finds that the stability of scheme (1) is the best. So it is decided to choose scheme (1), that is: Hold the lighter with (proximal phalanx of the thumb, index finger, ring finger), and press the ignition button with (distal phalanx of the thumb).
[0152] The assignment scheme is recorded as the "finger execution mode". The dispenser sends the finger execution mode to the subsequent force resolver.
[0153] 4. The force resolver module decomposes the resultant force (trajectory) according to the directions of the forces in the finger execution mode to obtain the specific force applied by each finger. The force resolver sends the magnitude and direction of the force to the control signal selector.
[0154] 5. The control signal selector selects a vector signal and sends it backward.
[0155] 6. The motion settlement module decomposes the vector signal into signals for each series joint and sends them to the corresponding drive board.
[0156] 7. The drive board controls the rotation of each motor according to the received control signal.
[0157] As can be seen from the above examples, the finger motion generation system provided by the present invention innovatively establishes the concept of a mechanical model. Based on common mechanical models (about a dozen in number), the task objective is decomposed into a combination of multiple mechanical models, and then available finger degrees of freedom are allocated to each force in the combination, thereby generating finger motions in various complex and differentiated scenarios (the number is incalculable). In other words, this architecture innovatively establishes the concept of a mechanical model and decouples the selection of the mechanical model from the allocation of finger degrees of freedom. Before the execution of the allocation, there is no pre-trained fixed pattern such as "what kind of motion each finger makes", but rather, according to the input from sensors such as visual cameras, the motion of each finger is generated in real time, which is a brand-new architecture of "motion generation" in the true sense, thus truly enabling the general-purpose use of the robotic dexterous hand.
[0158] Thus, through the motion generation system of the dexterous hand provided by the present invention, the motion of the fingers will no longer depend on deep learning training for specific objects / scenarios, but rather, according to the real-time input information from sensors such as visual cameras, the motion of the fingers is dynamically generated.
[0159] Figure 10 It is a schematic flowchart of the motion generation method of the dexterous hand provided in the second aspect of the embodiment of the present invention. As Figure 10 shown, the motion generation method 100 of the dexterous hand includes the following steps:
[0160] Step 101, perform image analysis on the acquired object image by using a pre-constructed image analysis module in combination with a pre-constructed material property table to obtain object information of the object;
[0161] Step 102, use the object feature - mechanical model applicability matrix in the pre-constructed finger degree-of-freedom allocator to determine a feasible combination of mechanical models from the pre-constructed mechanical model library according to the object information to obtain a combined mechanical model;
[0162] Step 103, use the finger degree-of-freedom allocator to allocate the degrees of freedom of the fingers to the combined mechanical model to generate a finger execution pattern and a resultant force trajectory;
[0163] Step 104, generate finger vectors of each finger according to the finger execution pattern and the resultant force trajectory;
[0164] Step 105, generate the motion of the dexterous hand based on the finger vectors of each finger.
[0165] Specifically, the motion generation method of the dexterous hand provided in the second aspect of the embodiment of the present invention is implemented based on the motion generation system of the dexterous hand described in the first aspect of the embodiment of the present invention. The specific implementation method refers to the content described in the first aspect of the embodiment of the present invention and will not be elaborated here one by one.
[0166] The foregoing description has been presented for purposes of illustration and description. In addition, this description is not intended to limit embodiments of the invention to the form disclosed herein. Although several example aspects and embodiments have been discussed above, those skilled in the art will recognize some variations, modifications, alterations, additions, and subcombinations thereof.
Claims
1. A method for generating motions of a dexterous hand, characterized in that, Comprising: Performing image analysis on the acquired item image by using a pre - constructed image analysis module in combination with a pre - constructed material property table to obtain the item information of the item; Using an object feature - mechanical model applicability matrix in a pre - constructed finger degree - of - freedom allocator to determine a feasible mechanical model for combination from a pre - constructed mechanical model library according to the item information, thereby obtaining a combined mechanical model; Using the finger degree - of - freedom allocator to allocate the degrees of freedom of the fingers to the combined mechanical model, generating a finger execution mode and a resultant force trajectory; Generating finger vectors of each finger according to the finger execution mode and the resultant force trajectory; Generating the motion of the dexterous hand based on the finger vectors of each finger; 2. The method according to claim 1, wherein The item information includes: the characteristics, material, mass, handle position, operation point position, force - prohibited surface, and visual position information of the item; 3. The method according to claim 1, characterized in that, The mechanical model library includes multiple sub - model libraries, and the information in the sub - model libraries includes: the name of each mechanical model, the direction of each force in each mechanical model, the auxiliary motion requirements, and the feasible unit combinations, where in the feasible unit combinations, the thumb takes the phalanx as the unit, and the other fingers take the whole finger as the unit; 4. The method according to claim 2, wherein Generating finger vectors of each finger according to the finger execution mode and the resultant force trajectory, including: Using a force decomposer to decompose the resultant force trajectory into finger force vectors of specific fingers according to the finger execution mode; Using a position generator to generate finger position vectors of each finger according to the finger execution mode and the item information; Determining the finger vectors of each finger according to the finger force vectors and the finger position vectors, where the finger vectors include the vectors of each phalanx of the thumb and the end vectors of the fingers other than the thumb; 5. The method according to claim 4, wherein Also comprising: Sending the tactile signals collected by the tactile sensor to the force resolver to provide end - force feedback, forming an end - force closed - loop; 6. The method according to claim 4, wherein Also comprising: Sending the visual position information to the position generator to provide end - position feedback, forming an end - position closed - loop; 7. The method according to claim 1, wherein Generating the motion of the dexterous hand based on the finger vectors of each finger, including: Calculating the finger vectors to obtain the motion angles and torques of each joint; Based on the drive boards of each joint, controlling the corresponding motor modules to rotate according to the motion angles and torques of each joint, generating the motion of the dexterous hand; 8. The method according to claim 7, characterized in that Sending the real - time motion angles and torques of the motor modules to the drive boards of each joint to provide motor operation feedback, forming a motor motion angle and torque closed - loop; 9. A dexterous hand motion generation system for implementing the dexterous hand motion generation method according to any one of claims 1-8, characterized in that, Comprising: An image acquisition module, a common material property table module, an image analysis module, a mechanical model library module, a finger degree - of - freedom allocator module, a force decomposer module, a finger position generator module, a control signal selector module, a motion calculation module, a drive board module, and a motor module, where The image analysis module is used to analyze the item image collected by the image acquisition module in combination with the material property table in the common material property table module to obtain the item information of the item; The finger degree-of-freedom allocator is used to determine a feasible mechanical model from the mechanical model library module according to the article information by using the object feature-mechanical model applicability matrix therein, combine them to obtain a combined mechanical model, and allocate the degrees of freedom of the fingers to the combined mechanical model to generate a finger execution mode and a resultant force trajectory; The control signal selector module, the force decomposer module, and the finger position generator module are used to generate finger vectors for each finger according to the finger execution mode and the resultant force trajectory; The motion solver module, the driver board module, and the motor module are used to generate the actions of the dexterous hand based on the finger vectors of each finger.
10. The motion generation system of the dexterous hand according to claim 9, wherein, It further includes: A tactile sensor module, which is used to send tactile signals to a force resolver to provide end force feedback and form an end force closed loop.
11. The motion generation system of the dexterous hand according to claim 9, characterized in that, The image analysis module is further used to send the output visual position information of the article to the position generator to provide end position feedback; and the motor module is further used to send real-time torque and position information to the driver board module to provide motor operation feedback and form a closed loop of the motor motion angle and torque.