Method and system for moving objects in hand using multi-finger dexterous hand

Abstract

The present application discloses a method, system, device, and computer-readable storage medium for moving an object in a hand using a multi-finger dexterous hand, which belongs to the field of manipulator control and includes: obtaining first spatial posture data of the multi-finger dexterous hand and second spatial posture data of the object when the grasping state of the multi-finger dexterous hand does not match a preset movement termination condition; using the first spatial posture data and the second spatial posture data as initial values ​​of a control optimization model for the multi-finger dexterous hand to grasp the object, solving the control optimization model to determine the grasping control data of the multi-finger dexterous hand to grasp the object; and controlling the grasping posture of the multi-finger dexterous hand according to the grasping control data. The present application automatically adjusts the grasping posture and optimizes the control to keep the multi-finger dexterous hand stable during the process of moving an object over a large range, thereby solving the problems of insufficient accuracy and low applicability when the multi-finger dexterous hand grasps and moves objects.

Description

Technical Field

[0001] The present application belongs to the field of manipulator control, and specifically relates to a method, system, device and computer-readable storage medium for moving an object in a hand using a multi-fingered dexterous hand. Background Art

[0002] Multi-fingered dexterous hands capable of in-hand object movement have important applications in robotic manipulation, particularly in scenarios requiring complex tasks within confined spaces, such as industrial assembly, medical procedures, and service robotics. In these scenarios, robotic hands must possess high-precision and stable grasping and manipulation capabilities to complete complex tasks. Ensuring that a robotic hand can stably control and move objects within its hand without moving its base, especially during large-scale movements, is a crucial step toward achieving human-like dexterity.

[0003] In the existing technology, multi-finger dexterous hands usually rely on the thumb as the core to move objects in the hand, and pre-collect the geometric information of the object to use kinematic or dynamic models for trajectory optimization.

[0004] However, due to the requirement for an unchanged relative posture between the object and the thumb of the robot, the range of motion of the object in the hand is limited, making it difficult to move accurately over a large range. At the same time, complex adjustments need to be made for different geometric models, making it difficult to meet the needs of actual tasks. Summary of the Invention

[0005] The present application aims to provide a method, system, device and computer-readable storage medium for moving objects in the hand using a multi-finger dexterous hand, at least to solve the problems of insufficient accuracy and low applicability when the multi-finger dexterous hand holds and moves objects.

[0006] In a first aspect, embodiments of the present application disclose a method for moving an object in a hand using a multi-finger dexterous hand, which is applied to a controller of the multi-finger dexterous hand, comprising:

[0007] When the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, obtaining first spatial posture data of the multi-finger dexterous hand and obtaining second spatial posture data of the object grasped by the multi-finger dexterous hand;

[0008] The first spatial posture data and the second spatial posture data are used as initial values ​​of a control optimization model for the multi-finger dexterous hand to grasp the object, and the control optimization model is solved to determine the grasping control data of the multi-finger dexterous hand to grasp the object; the optimization goal of the control optimization model is to minimize the predicted value of the sum of the first posture error data, the movement distance data, and the second posture error data of the multi-finger dexterous hand until the target time; the first posture error data is determined according to the first spatial posture data and the first expected posture data of the multi-finger dexterous hand; the second posture error data is determined according to the second spatial posture data and the second expected posture data of the object;

[0009] The grasping posture of the multi-finger dexterous hand is controlled according to the grasping control data.

[0010] In a second aspect, embodiments of the present application further disclose a system for moving an object in a hand using a multi-finger dexterous hand, comprising:

[0011] A multi-finger dexterous hand and a controller for the multi-finger dexterous hand;

[0012] The multi-finger dexterous hand is used for grasping objects;

[0013] The controller is used to obtain the first spatial posture data of the multi-finger dexterous hand and the second spatial posture data of the object grasped by the multi-finger dexterous hand when the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, and use the first spatial posture data and the second spatial posture data as the initial values ​​of the control optimization model of the multi-finger dexterous hand grasping the object, solve the control optimization model to determine the grasping control data of the multi-finger dexterous hand grasping the object, and control the grasping posture of the multi-finger dexterous hand according to the grasping control data; the optimization goal of the control optimization model is to minimize the predicted value of the sum of the first posture error data, movement distance data, and second posture error data of the multi-finger dexterous hand until the target time; the first posture error data is determined based on the first spatial posture data and the first expected posture data of the multi-finger dexterous hand; the second posture error data is determined based on the second spatial posture data and the second expected posture data of the object.

[0014] In a third aspect, an embodiment of the present application further discloses an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application further discloses a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0016] In summary, in the embodiment of the present application, the obtained spatial posture data is used as the initial value of the control optimization model to solve the control optimization model, and the grasping control data determined automatically adjusts the grasping posture of the multi-finger dexterous hand to ensure the stability and accuracy of the grasping; then the control instructions of the multi-finger dexterous hand are determined by minimizing the posture error of the multi-finger dexterous hand, the motion distance data, and the minimum value of the sum of the posture error of the object as the optimization goal, so as to accurately control the grasping posture and keep the object stable during a large range of movement, thereby avoiding the limited range of motion due to the limitation of the thumb; at the same time, the grasping posture of the multi-finger dexterous hand is repeatedly optimized and executed by the grasping control data, and the grasping posture is continuously adjusted to reduce the actual execution error so that it can be applied to different object shapes. Therefore, based on the method of the embodiment of the present application, by automatically adjusting the grasping posture and optimizing the control, the multi-finger dexterous hand is kept stable during a large range of movement, the grasping accuracy and applicability are improved, and the problems of insufficient accuracy and low applicability when the multi-finger dexterous hand holds objects and moves are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In the attached figure:

[0018] Figure 1 This is a flowchart of the steps of a method for moving an object in a hand using a multi-finger dexterous hand provided in an embodiment of the present application;

[0019] Figure 2 is a schematic diagram illustrating the posture in the embodiment of the present application;

[0020] Figure 3 This is a flowchart of another method for moving an object in a hand using a multi-finger dexterous hand provided by an embodiment of the present application;

[0021] Figure 4 A block diagram of a system for moving an object in a hand using a multi-finger dexterous hand is provided in an embodiment of the present application;

[0022] Figure 5 is a block diagram of an electronic device according to an embodiment of the present application;

[0023] Figure 6 This is a block diagram of an electronic device according to another embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0026] Existing techniques for grasping objects using a multi-fingered dexterous hand assume that the relative position (position and orientation) of the object and the thumb remains unchanged, centered around the thumb. Because the thumb of a multi-fingered dexterous hand typically has no more than four degrees of freedom and limited finger joint angles, this assumption significantly restricts the object's range of motion, preventing it from reaching a desired location at a distance.

[0027] The multi-finger dexterous hand is an advanced robotic device characterized by at least one finger joint on each finger. Through the combined movement of these joints, the multi-finger dexterous hand can achieve efficient and precise grasping operations on objects with multi-degree-of-freedom flexibility. Therefore, the present invention proposes a trajectory optimization method that is completely kinematic and independent of the surface geometry of the object. Under the premise of establishing an initial stable grasp, this trajectory optimization method can be applied to any unknown object in the hand. The fundamental idea is to maintain a stable grasp by constraining the position of the fingertips on the object surface to remain unchanged throughout the entire trajectory. At the same time, by optimizing the error between the expected position of the object and the predicted position, the object can be moved to the expected position as accurately as possible.

[0028] Considering the grasping process within a certain time period 1:T, the specific mathematical description of the trajectory optimization problem can be expressed by the following formula:

[0029] ;

[0030] ;

[0031] ;

[0032] Among them, p represents the position coordinate of the object, R represents the direction of the object, and q represents the motion vector of the finger. Indicates the error between the final position of the object's trajectory and the expected position, Indicates the trajectory The error between the position of the fingertip in the object coordinate system at the moment and the initial time, Indicates the movement distance of the finger in the joint space during the entire trajectory;

[0033] In the constraint function, represents the motion vector of the finger at time t, This means that the joint angle of the finger at any moment does not exceed the joint angle range allowed by the mechanical structure. Indicates the difference between the mutual distances between the calibration points generated based on the calibration points selected on the finger and the minimum value between these mutual distances. It can be used to prevent collisions between fingers.

[0034] By solving the above optimization problem, the action instructions of the multi-finger dexterous hand in each time unit can be obtained. Furthermore, through the process of re-planning and re-execution, the results of the finger movement after the action instructions are applied are fed back to realize closed-loop execution operations, thereby achieving the effect of improving the accuracy of object movement. At the same time, it can be seen that through the above process, the introduction of too many spatial reference points is also avoided, that is, there is no need to examine the specific shape of the object to be grasped. While the model is simplified, the grasping effect is still maintained.

[0035] Based on the above theoretical foundation, Figure 1 As shown, a method for moving an object in a hand using a multi-finger dexterous hand is provided in an embodiment of the present application, and is applied to a controller of the multi-finger dexterous hand.

[0036] The method may include the following steps:

[0037] Step 101: When the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, first spatial posture data of the multi-finger dexterous hand is obtained, and second spatial posture data of the object grasped by the multi-finger dexterous hand is obtained.

[0038] In some embodiments of the present application, in order to ensure that the multi-finger dexterous hand maintains stability and accuracy during the grasping process, it is necessary to obtain the first spatial posture data of the multi-finger dexterous hand and the second spatial posture data of the object grasped by the multi-finger dexterous hand when the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition. By obtaining these data, the actual state of the multi-finger dexterous hand and the grasped object can be monitored and evaluated in real time so as to make timely adjustments and optimizations. The posture data of the multi-finger dexterous hand and the grasped object can be obtained using sensors or measuring equipment. The spatial posture data is used to describe the current position and orientation of the multi-finger dexterous hand and the object. In this way, control optimization can be performed based on the acquired data to ensure stability and accuracy during the grasping process.

[0039] In a specific example, as in Figure 2 As shown in , when using the multi-finger dexterous hand 301 to grasp object X, the system finds that the grasping state does not match the preset movement termination condition. Using the sensors on the device, the first spatial pose data A1 of the multi-finger dexterous hand in the world coordinate system is obtained, and the second spatial pose data B2 of the object in the object coordinate system is obtained through the label or mark O attached to the object. The system uses this data for subsequent optimization and solution to ensure the stability and accuracy of the multi-finger dexterous hand during the grasping process. By obtaining spatial pose data and movement distance data, the operation of the multi-finger dexterous hand can be monitored and adjusted in real time to ensure the stability and accuracy of the grasping process.

[0040] Step 102: Using the first spatial posture data and the second spatial posture data as initial values ​​of a control optimization model for the multi-finger dexterous hand to grasp an object, the control optimization model is solved to determine grasping control data for the multi-finger dexterous hand to grasp an object.

[0041] Among them, the optimization goal of the control optimization model is to minimize the predicted value of the sum of the first pose error data, motion distance data, and second pose error data of the multi-fingered dexterous hand until the target time; the first pose error data is determined according to the first spatial pose data and the first expected pose data of the multi-fingered dexterous hand; the second pose error data is determined according to the second spatial pose data and the second expected pose data of the object.

[0042] In some embodiments of the present application, in order to solve the control optimization model for the multi-finger dexterous hand to grasp an object, it is necessary to input the first spatial pose data and the second spatial pose data as the initial values ​​of the control optimization model. By using these initial value data, it can be ensured that the control optimization model accurately reflects the current state of the multi-finger dexterous hand and the grasped object, thereby optimizing the grasping control data. The process of performing this step is to input the acquired first spatial pose data and second spatial pose data into the control optimization model, and solve the model to determine the grasping control data for the multi-finger dexterous hand to grasp the object. The first spatial pose data describes the current position and orientation of the multi-finger dexterous hand, the second spatial pose data describes the current position and orientation of the grasped object, and the motion distance data records the displacement of the multi-finger dexterous hand during the movement. The optimization goal of the control optimization model is to minimize the predicted value of the sum of the first pose error data, the motion distance data and the second pose error data of the object of the multi-finger dexterous hand until the target time. The optimized gripping control data obtained in this step will be used in the subsequent gripping control process, thereby ensuring the accuracy and stability of the gripping process.

[0043] In a specific example, Figure 2 As shown, when the multi-finger dexterous hand 301 is used to grasp the object X, the first spatial posture data A1 of the multi-finger dexterous hand in the world coordinate system and the corresponding first desired posture data A2, and the second spatial posture data B1 of the grasped object in the object coordinate system and the corresponding second desired posture data B2 are obtained through the sensor. These data are input into the control optimization model and solved to determine the grasping control data. The optimization goal of the control optimization model is to minimize the predicted value of the sum of the posture error and the motion distance data of the multi-finger dexterous hand and the object. Through this process, the optimized grasping control data obtained will be used to control the grasping posture of the multi-finger dexterous hand. By solving the control optimization model, precise control of the grasping process can be achieved, thereby improving the stability and accuracy of the multi-finger dexterous hand in the process of grasping and moving objects.

[0044] Step 103: Control the grasping posture of the multi-finger dexterous hand according to the grasping control data.

[0045] In some embodiments of the present application, in order to achieve precise control of the grasping posture of the multi-finger dexterous hand, it is necessary to operate according to the grasping control data. By using the optimized grasping control data, it is possible to ensure that the multi-finger dexterous hand maintains a stable grasping state in actual operation, thereby improving the accuracy and stability of the grasping. According to the grasping control data obtained previously, the grasping posture of the multi-finger dexterous hand is adjusted in real time to adapt to the actual situation of the grasped object. The grasping control data includes the joint angles of the multi-finger dexterous hand at different time points to ensure the coordinated movements of each joint during the grasping process. The multi-finger dexterous hand can perform grasping operations according to predetermined trajectories and postures, thereby achieving stable movement of the grasped object.

[0046] In a specific example, when using a multi-finger dexterous hand for grasping, the previously solved grasping control data is input into the multi-finger dexterous hand's control system. Based on this data, the system adjusts the joint angles of the multi-finger dexterous hand in real time to ensure that the movements of each joint are coordinated and consistent during the grasping process. Through this process, the multi-finger dexterous hand can stably grasp and move objects according to a predetermined trajectory and posture. By controlling the grasping posture of the multi-finger dexterous hand based on the grasping control data, stable movement of objects can be achieved, improving the accuracy and stability of the grasp.

[0047] In summary, in the embodiment of the present application, the obtained spatial posture data is used as the initial value of the control optimization model to solve the control optimization model, and the grasping control data determined automatically adjusts the grasping posture of the multi-finger dexterous hand to ensure the stability and accuracy of the grasping; then the control instructions of the multi-finger dexterous hand are determined by minimizing the posture error of the multi-finger dexterous hand, the motion distance data, and the minimum value of the sum of the posture error of the object as the optimization goal, so as to accurately control the grasping posture and keep the object stable during a large range of movement, thereby avoiding the limited range of motion due to the limitation of the thumb; at the same time, the grasping posture of the multi-finger dexterous hand is repeatedly optimized and executed by the grasping control data, and the grasping posture is continuously adjusted to reduce the actual execution error so that it can be applied to different object shapes. Therefore, based on the method of the embodiment of the present application, by automatically adjusting the grasping posture and optimizing the control, the multi-finger dexterous hand is kept stable during a large range of movement, the grasping accuracy and applicability are improved, and the problems of insufficient accuracy and low applicability when the multi-finger dexterous hand holds objects and moves are solved.

[0048] Figure 3 This is another method for moving an object in the hand using a multi-finger dexterous hand provided in an embodiment of the present application, and is applied to a controller of the multi-finger dexterous hand.

[0049] The method may include the following steps:

[0050] Step 201: Establish a control optimization model based on the preset first initial expected posture data and the second initial expected posture data.

[0051] In some embodiments of the present application, in order to establish a control optimization model, it is necessary to preset first initial expected posture data and second initial expected posture data. By establishing an accurate control optimization model, an optimization target can be provided for the grasping process of the multi-finger dexterous hand to ensure stability and accuracy during the grasping process. The specific process is to obtain the first initial expected posture data and the second initial expected posture data of the multi-finger dexterous hand, and use these data to establish a control optimization model. The first initial expected posture data and the second initial expected posture data are used to describe the position and orientation of the multi-finger dexterous hand and the object in an ideal state. The control optimization model established in this way will serve as the basis for solving the grasping control data in subsequent steps, thereby realizing optimized control of the grasping process of the multi-finger dexterous hand.

[0052] In one specific example, before initiating the multi-finger dexterous hand, the user pre-defines an ideal gripping state, which includes a first initial desired pose data for the multi-finger dexterous hand and a second initial desired pose data for the object to be grasped. This data is then input into the multi-finger dexterous hand's control system, which uses it to establish a control optimization model, which serves as the basis for subsequent control optimization. By setting the initial desired pose data and establishing the control optimization model, the system can optimize the gripping control data during actual operation, effectively improving the multi-finger dexterous hand's grip stability and precision.

[0053] Step 202: When the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, first spatial posture data of the multi-finger dexterous hand is obtained, and second spatial posture data of the object grasped by the multi-finger dexterous hand is obtained.

[0054] The method shown in this step has been described in step 101 and will not be repeated here.

[0055] Optionally, in order to obtain first spatial pose data of the multi-fingered dexterous hand, step 202 includes the following sub-steps:

[0056] Sub-step 2021, obtaining the joint posture data of each joint of the multi-finger dexterous hand.

[0057] Each joint posture data is used to represent the spatial position and / or bending angle of the joint corresponding to the joint posture data.

[0058] In some embodiments of the present application, in order to obtain the first spatial posture data of the multi-finger dexterous hand, it is necessary to first obtain the joint posture data of each joint of the multi-finger dexterous hand. By obtaining the posture data of each joint, the specific position and state of each joint of the multi-finger dexterous hand during operation can be fully understood. The spatial position and / or bending angle of each joint of the multi-finger dexterous hand is recorded using a sensor or measuring device. The joint posture data is used to describe the position and angle of each joint in the current state. The acquired joint posture data is used as the basis for further calculating the spatial posture vector of the multi-finger dexterous hand, which can provide accurate initial data for subsequent steps.

[0059] In a specific example, when using a multi-finger dexterous hand for grasping, sensors installed at each joint first acquire the joint pose data for each joint. This data includes the joint's spatial position and bending angle. This data is recorded and transmitted to the multi-finger dexterous hand's control system. Through this process, the system can fully understand the specific state of each joint in the multi-finger dexterous hand, providing accurate initial data for subsequent control optimization. By acquiring the joint pose data for each joint, the multi-finger dexterous hand can provide accurate and comprehensive joint state information for operation, thereby improving the precision and stability of the operation.

[0060] In sub-step 2022, the joint pose vectors of the multi-fingered dexterous hand generated based on all the joint pose data are determined as the first spatial pose data.

[0061] In some embodiments of the present application, in order to determine the first spatial pose data of the multi-finger dexterous hand, it is necessary to calculate the joint pose vector of the multi-finger dexterous hand generated based on all the joint pose data. By integrating all the joint pose data, a complete joint pose vector can be generated to accurately describe the overall spatial pose of the multi-finger dexterous hand. The pose data of each joint previously acquired are integrated to generate a joint pose vector of the multi-finger dexterous hand. The joint pose vector is used to describe the overall spatial position and orientation of the multi-finger dexterous hand in the current state. The generated joint pose vector will serve as the first spatial pose data to provide basic data for subsequent grasping control, thereby ensuring the operational accuracy of the multi-finger dexterous hand.

[0062] In a specific example, when using a multi-finger dexterous hand to perform a grasping operation, the joint pose data of each joint is first obtained through the sensor. Subsequently, these joint pose data are input into the system, and the system integrates all the joint pose data to generate a joint pose vector. This joint pose vector describes the overall spatial position and orientation of the multi-finger dexterous hand in the current state. The generated joint pose vector is determined as the first spatial pose data and input into the control optimization model. In this example, by generating and determining the joint pose vector, the overall spatial pose of the multi-finger dexterous hand can be accurately described, providing basic data for subsequent optimization control and improving the accuracy and stability of the operation.

[0063] Optionally, in order to obtain second spatial pose data of an object grasped by the multi-fingered dexterous hand, step 202 includes the following sub-steps:

[0064] Sub-step 2023: Acquire spatial coordinate data of at least one reference point on the object.

[0065] In some embodiments of the present application, in order to obtain the second spatial posture data of an object, it is necessary to first obtain the spatial coordinate data of at least one reference point on the object. By obtaining the spatial coordinate data of the object's reference point, the position and orientation of the object in space can be accurately located, thereby providing basic data for further control optimization. The spatial coordinate data of a specified reference point on the object is measured and recorded using a sensor or measuring device. The spatial coordinate data of the reference point is used to describe the position of the object in its current state. The obtained spatial coordinate data of the reference point is used as the basis for further calculating the second spatial posture data of the object, thereby ensuring the accuracy of the multi-finger dexterous hand's operation on the object.

[0066] In a specific example, when using a multi-fingered dexterous hand for grasping, the spatial coordinate data of at least one reference point on the object can be obtained by setting markers on the object. This data is captured and recorded by a camera installed in the working environment. This coordinate data is input into the control system of the multi-fingered dexterous hand for subsequent calculation and optimization of the object's spatial pose data. In this example, by obtaining the spatial coordinate data of at least one reference point on the object, the position and orientation of the object can be accurately determined, providing an accurate data foundation for further control optimization, thereby improving the operational precision and stability of the multi-fingered dexterous hand.

[0067] Sub-step 2024, determining the second spatial pose data based on all the spatial coordinate data.

[0068] In some embodiments of the present application, in order to determine the second spatial posture data of the object, it is necessary to perform calculations based on all the spatial coordinate data. By integrating the spatial coordinate data of all reference points, the overall position and orientation of the object in space can be accurately described, thereby providing basic data for the control optimization of the multi-finger dexterous hand. The spatial coordinate data of all reference points previously acquired are integrated to generate the second spatial posture data of the object. The second spatial posture data is used to describe the overall position and orientation of the object in the current state. The generated second spatial posture data will serve as the basis for optimizing the control in subsequent steps, thereby ensuring the operational accuracy and stability of the multi-finger dexterous hand.

[0069] In a specific example, when using a multi-finger dexterous hand to perform a grasping operation, the spatial coordinate data of multiple reference points on the object can be obtained by setting marking points on the object. These data can be captured and recorded by a camera installed in the working environment. The experimenter inputs these coordinate data into the system, and the system integrates the spatial coordinate data of all reference points to generate the second spatial posture data of the object. This posture data describes the overall position and orientation of the object in its current state. The generated second spatial posture data is input into the control optimization model for subsequent optimization control. By integrating the spatial coordinate data of the reference points, the overall spatial posture of the object can be accurately described, providing an accurate data basis for the operation of the multi-finger dexterous hand, thereby improving the accuracy and stability of the operation.

[0070] In step 203, the first spatial posture data and the second spatial posture data are used as initial values ​​of a control optimization model for the multi-finger dexterous hand to grasp the object, and the control optimization model is solved to determine grasping control data for the multi-finger dexterous hand to grasp the object.

[0071] Among them, the optimization goal of the control optimization model is to minimize the predicted value of the sum of the first pose error data, motion distance data, and second pose error data of the multi-fingered dexterous hand until the target time; the first pose error data is determined according to the first spatial pose data and the first expected pose data of the multi-fingered dexterous hand; the second pose error data is determined according to the second spatial pose data and the second expected pose data of the object.

[0072] The method shown in this step has been described in step 102 and will not be repeated here.

[0073] Optionally, the control optimization model is established in a motion solver, which is used to solve the grasping control data of the multi-fingered dexterous hand grasping an object. Step 203 includes the following sub-steps:

[0074] Sub-step 2031: input the first spatial pose data, the second spatial pose data, and the motion distance data into the motion solver, and determine the analytical gradient of the control optimization model according to the target hyperparameters of the motion solver.

[0075] In some embodiments of the present application, in order to quickly determine the solution of the control optimization model, it is necessary to input the first spatial pose data, the second spatial pose data and the motion distance data into the motion solver, and calculate according to the target hyperparameters of the motion solver. By inputting these data into the motion solver, its target hyperparameters can be used to determine the analytical gradient of the control optimization model, thereby determining the grasping control data of the multi-fingered dexterous hand. All the acquired pose data and motion distance data are input into the motion solver and solved according to the pre-set target hyperparameters. The target hyperparameters of the motion solver are used to define the optimization parameters used in the solution process. The generated analytical gradient will be used as the basis for judging the accuracy of the grasping control data in the subsequent steps, thereby ensuring the accuracy of the grasping process.

[0076] In a specific example, when using a multi-finger dexterous hand to perform a grasping operation, first spatial posture data, second spatial posture data, and motion distance data are acquired through sensors. These data are input into the motion solver in the control system of the multi-finger dexterous hand. The system performs calculations based on the set target hyperparameters and determines the analytical gradient of the control optimization model. Through this process, an analytical gradient can be obtained for further optimizing the grasping control data, providing an accurate control basis for the operation of the multi-finger dexterous hand. By inputting the posture data and motion distance data into the motion solver, the grasping control data of the multi-finger dexterous hand can be optimized, and the accuracy and stability of the grasping operation can be improved.

[0077] Sub-step 2032 : When the analytical gradient is less than a preset gradient threshold, the output value of the motion solver is determined as the gripping control data.

[0078] In some embodiments of the present application, in order to quickly determine the output value of the motion solver as accurate grasping control data, it is necessary to determine the output value of the motion solver as grasping control data when the analytical gradient is less than a preset gradient threshold. By setting the threshold of the analytical gradient, the calculation accuracy and reliability of the grasping control data can be ensured. After calculating the analytical gradient of the motion solver, the analytical gradient is compared with the preset gradient threshold. When the analytical gradient is less than the threshold, the output value of the motion solver is determined as the grasping control data. The analytical gradient is an indicator that describes the rate of change of the output value during the solution process and is used to evaluate the accuracy of the solution result. The obtained grasping control data will be used for actual grasping operations, thereby ensuring the operational continuity of the multi-finger dexterous hand.

[0079] In a specific example, when operating a multi-finger dexterous hand, the first spatial posture data, the second spatial posture data, and the motion distance data are acquired through the sensor and input into the motion solver for calculation. The system calculates the analytical gradient of the motion solver and compares it with the preset gradient threshold. If the analytical gradient is less than the preset gradient threshold, the output value of the motion solver is determined as the grasping control data and used for the actual grasping operation of the multi-finger dexterous hand. By setting the threshold of the analytical gradient, it is possible to ensure the calculation accuracy of the grasping control data and quickly obtain the output result, thereby improving the operational continuity of the multi-finger dexterous hand.

[0080] Step 204: Control the grasping posture of the multi-finger dexterous hand according to the grasping control data.

[0081] The method shown in this step has been described in step 103 and will not be repeated here.

[0082] Step 205 : When the grasping state of the multi-fingered dexterous hand matches the movement termination condition, the control optimization model is adjusted according to the updated value of the second desired posture data.

[0083] In some embodiments of the present application, in order to adapt to the needs of the subsequent grasping process, it is necessary to adjust the control optimization model according to the updated value of the second expected posture data when the grasping state of the multi-finger dexterous hand matches the movement termination condition. By updating the control optimization model, the latest state requirements of the multi-finger dexterous hand and the grasped object can be reflected in a timely manner, thereby adjusting the grasping control. When the grasping state of the multi-finger dexterous hand reaches the preset movement termination condition, the system will adjust the previously established control optimization model according to the updated value of the second expected posture data currently obtained. Through this adjustment, it can be ensured that the control optimization model matches the new grasping requirements. The adjusted control model will be able to adapt to subsequent grasping targets.

[0084] In a specific example, when operating a multi-finger dexterous hand, when the system detects that the grasping state of the multi-finger dexterous hand matches the preset movement termination condition, the system will obtain updated values ​​for the second desired posture data of the current object. The system adjusts the control optimization model based on these updated values. Through this adjustment, the optimized control model can accurately reflect the latest goal of grasping the object, ensuring that the control data provided for subsequent grasping operations matches the subsequent requirements. By adjusting the control optimization model, the system can achieve the consistency requirements of the grasping process and ensure the long-term stability of the multi-finger dexterous hand during the movement of the grasping object.

[0085] Optionally, in some implementations of the present application, the movement termination condition is one of the following conditions: the actual execution error is less than the ideal error calculated by trajectory optimization, the number of replanning times reaches a preset maximum replanning time threshold, and the replanning time interval reaches a preset maximum replanning time threshold.

[0086] Among them, the actual execution error is used to characterize the difference between the second spatial posture data of the object and the second expected posture data; the number of replanning is used to record the number of executions of the grasping posture of the multi-finger dexterous hand controlled according to the grasping control data from the preset starting time to the target time; and the replanning time is used to record the time interval from the preset starting time to the target time.

[0087] In summary, in the embodiment of the present application, the obtained spatial posture data is used as the initial value of the control optimization model to solve the control optimization model, and the grasping control data determined automatically adjusts the grasping posture of the multi-finger dexterous hand to ensure the stability and accuracy of the grasping; then the control instructions of the multi-finger dexterous hand are determined by minimizing the posture error of the multi-finger dexterous hand, the motion distance data, and the minimum value of the sum of the posture error of the object as the optimization goal, so as to accurately control the grasping posture and keep the object stable during a large range of movement, thereby avoiding the limited range of motion due to the limitation of the thumb; at the same time, the grasping posture of the multi-finger dexterous hand is repeatedly optimized and executed by the grasping control data, and the grasping posture is continuously adjusted to reduce the actual execution error so that it can be applied to different object shapes. Therefore, based on the method of the embodiment of the present application, by automatically adjusting the grasping posture and optimizing the control, the multi-finger dexterous hand is kept stable during a large range of movement, the grasping accuracy and applicability are improved, and the problems of insufficient accuracy and low applicability when the multi-finger dexterous hand holds objects and moves are solved.

[0088] refer to Figure 4 , which shows an in-hand object movement system 30 using a multi-finger dexterous hand provided by an embodiment of the present application, comprising:

[0089] A multi-finger dexterous hand 301 and a controller 302 for the multi-finger dexterous hand;

[0090] The multi-finger dexterous hand 301 is used to grasp objects;

[0091] The controller 302 is used to obtain the first spatial posture data of the multi-finger dexterous hand and the second spatial posture data of the object grasped by the multi-finger dexterous hand when the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, and use the first spatial posture data and the second spatial posture data as the initial values ​​of the control optimization model of the multi-finger dexterous hand grasping the object, solve the control optimization model to determine the grasping control data of the multi-finger dexterous hand grasping the object, and control the grasping posture of the multi-finger dexterous hand according to the grasping control data; the optimization goal of the control optimization model is to minimize the predicted value of the sum of the first posture error data, movement distance data, and second posture error data of the multi-finger dexterous hand until the target time; the first posture error data is determined based on the first spatial posture data and the first expected posture data of the multi-finger dexterous hand; the second posture error data is determined based on the second spatial posture data and the second expected posture data of the object.

[0092] Optionally, the in-hand object movement system 30 using the multi-finger dexterous hand is further used to establish a control optimization model according to preset first initial expected posture data and second initial expected posture data.

[0093] Optionally, the in-hand object movement system 30 using the multi-finger dexterous hand is further used to adjust the control optimization model according to the updated value of the second desired posture data when the grasping state of the multi-finger dexterous hand matches the movement termination condition.

[0094] Optionally, the control optimization model is established on a motion solver, and the motion solver is used to solve the grasping control data of the multi-finger dexterous hand grasping the object; the in-hand object movement system 30 using the multi-finger dexterous hand is also used to use the first spatial posture data and the second spatial posture data as the initial values ​​of the control optimization model of the multi-finger dexterous hand grasping the object, and solve the control optimization model to determine the grasping control data of the multi-finger dexterous hand grasping the object, specifically for inputting the first spatial posture data, the second spatial posture data and the motion distance data into the motion solver, and determining the analytical gradient of the control optimization model according to the target hyperparameters of the motion solver, and when the analytical gradient is less than a preset gradient threshold, determining the output value of the motion solver as the grasping control data.

[0095] Optionally, when using the in-hand object movement system 30 of the multi-finger dexterous hand to obtain the first spatial posture data of the multi-finger dexterous hand, it is specifically used to obtain the joint posture data of each joint of the multi-finger dexterous hand; each joint posture data is used to characterize the spatial position and / or bending angle of the joint corresponding to the joint posture data, and the joint posture vector of the multi-finger dexterous hand generated based on all the joint posture data is determined as the first spatial posture data.

[0096] Optionally, when the in-hand object movement system 30 of the multi-finger dexterous hand obtains the second spatial posture data of an object grasped by the multi-finger dexterous hand, it is specifically used to obtain the spatial coordinate data of at least one reference point on the object, and determine the second spatial posture data based on all the spatial coordinate data.

[0097] In summary, in the embodiment of the present application, the obtained spatial posture data is used as the initial value of the control optimization model to solve the control optimization model, and the grasping control data determined automatically adjusts the grasping posture of the multi-finger dexterous hand to ensure the stability and accuracy of the grasping; then the control instructions of the multi-finger dexterous hand are determined by minimizing the posture error of the multi-finger dexterous hand, the motion distance data, and the minimum value of the sum of the posture error of the object as the optimization goal, so as to accurately control the grasping posture and keep the object stable during a large range of movement, thereby avoiding the limited range of motion due to the limitation of the thumb; at the same time, the grasping posture of the multi-finger dexterous hand is repeatedly optimized and executed by the grasping control data, and the grasping posture is continuously adjusted to reduce the actual execution error so that it can be applied to different object shapes. Therefore, based on the method of the embodiment of the present application, by automatically adjusting the grasping posture and optimizing the control, the multi-finger dexterous hand is kept stable during a large range of movement, the grasping accuracy and applicability are improved, and the problems of insufficient accuracy and low applicability when the multi-finger dexterous hand holds objects and moves are solved.

[0098] Reference Figure 5 , electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0099] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 502 may include one or more modules to facilitate interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate interaction between the multimedia component 508 and the processing component 502.

[0100] The memory 504 is used to store various types of data to support operations on the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, multimedia, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0101] The power supply assembly 506 provides power to the various components of the electronic device 500. The power supply assembly 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0102] The multimedia component 508 includes an interface that provides an output interface between the electronic device 500 and the user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensors can not only sense the demarcation of a touch or slide action, but also detect the duration and pressure associated with the touch or slide action. In some embodiments, the multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When the electronic device 500 is in an operating mode, such as a capture mode or a multimedia mode, the front-facing camera and / or the rear-facing camera can receive external multimedia data. Each front-facing camera and the rear-facing camera can have a fixed optical lens system or have focal length and optical zoom capabilities.

[0103] The audio component 510 is used to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC) that receives external audio signals when the electronic device 500 is in an operating mode, such as a call mode, a recording mode, or a voice recognition mode. The received audio signals may be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0104] The input / output I / O interface 512 provides an interface between the processing component 502 and peripheral interface modules, such as a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0105] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500. The sensor assembly 514 can also detect changes in the position of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and temperature changes of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0106] The communication component 516 is used to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0107] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement the methods provided in the embodiments of the present application.

[0108] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, which can be executed by the processor 520 of the electronic device 500 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0109] Figure 6FIG2 is a block diagram of an electronic device 600 according to another embodiment of the present invention. For example, the electronic device 600 may be provided as a server.

[0110] Reference Figure 6 The electronic device 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632 for storing instructions executable by the processing component 622, such as an application. The application stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute the instructions to perform the method provided in the embodiments of the present application.

[0111] The electronic device 600 may further include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0112] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0113] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for moving an object in a hand using a multi-finger dexterous hand, characterized in that: Controllers for multi-finger dexterous hands include: When the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, obtaining first spatial posture data of the multi-finger dexterous hand and obtaining second spatial posture data of the object grasped by the multi-finger dexterous hand; The first spatial posture data and the second spatial posture data are used as initial values ​​of a control optimization model for the multi-finger dexterous hand to grasp the object, and the control optimization model is solved to determine the grasping control data of the multi-finger dexterous hand to grasp the object; the optimization goal of the control optimization model is to minimize the predicted value of the sum of the first posture error data, the movement distance data, and the second posture error data of the object of the multi-finger dexterous hand until the target time; the first posture error data is determined according to the first spatial posture data and the first expected posture data of the multi-finger dexterous hand; the second posture error data is determined according to the second spatial posture data and the second expected posture data of the object; The control optimization model is established in a motion solver, and the motion solver is used to solve the grasping control data of the multi-finger dexterous hand grasping the object; the first spatial posture data and the second spatial posture data are used as initial values ​​of the control optimization model of the multi-finger dexterous hand grasping the object, and the control optimization model is solved to determine the grasping control data of the multi-finger dexterous hand grasping the object, including: Inputting the first spatial pose data and the second spatial pose data into the motion solver, and determining an analytical gradient of the control optimization model based on target hyperparameters of the motion solver; When the analytical gradient is less than a preset gradient threshold, determining the output value of the motion solver as the grasping control data; controlling the grasping posture of the multi-finger dexterous hand according to the grasping control data; When the grasping state of the multi-fingered dexterous hand matches the movement termination condition, the control optimization model is adjusted according to the updated value of the second desired posture data.

2. The method for moving an object in a hand using a multi-finger dexterous hand according to claim 1, wherein: Before obtaining the first spatial posture data of the multi-finger dexterous hand and obtaining the second spatial posture data of the object grasped by the multi-finger dexterous hand when the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, the method for moving an in-hand object using the multi-finger dexterous hand further includes: The control optimization model is established according to the preset first initial expected posture data and the second initial expected posture data.

3. The method for moving an object in hand using a multi-finger dexterous hand according to claim 1, wherein: The obtaining of first spatial posture data of the multi-fingered dexterous hand comprises: Acquiring joint posture data of each joint of the multi-fingered dexterous hand; each joint posture data is used to represent the spatial position and / or bending angle of the joint corresponding to the joint posture data; The joint pose vectors of the multi-fingered dexterous hand generated based on all the joint pose data are determined as the first spatial pose data.

4. The method for moving an object in hand using a multi-finger dexterous hand according to claim 1, wherein: The obtaining of second spatial pose data of the object grasped by the multi-fingered dexterous hand comprises: Acquiring spatial coordinate data of at least one reference point on the object; The second spatial pose data is determined based on all the spatial coordinate data.

5. The method for moving an object in hand using a multi-finger dexterous hand according to claim 1, wherein: The formula of the control optimization model is: ； ； ； Among them, p represents the position coordinate of the object, R represents the direction of the object, and q represents the motion vector of the finger. Indicates the error between the final position of the object's trajectory and the expected position, Indicates the trajectory The error between the position of the fingertip in the object coordinate system at the moment and the initial time, Indicates the movement distance of the finger in the joint space during the entire trajectory; In the constraint function, represents the motion vector of the finger at time t, This means that the joint angle of the finger at any moment does not exceed the joint angle range allowed by the mechanical structure. Indicates the difference between the mutual distances between the calibration points generated based on the calibration points selected on the finger and the minimum value of these mutual distances.

6. A system for moving objects in the hand using a multi-finger dexterous hand, characterized in that: include: A multi-finger dexterous hand and a controller for the multi-finger dexterous hand; The multi-finger dexterous hand is used for grasping objects; The controller is used to obtain first spatial posture data of the multi-finger dexterous hand and second spatial posture data of the object grasped by the multi-finger dexterous hand when the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, and use the first spatial posture data and the second spatial posture data as initial values ​​of a control optimization model for the multi-finger dexterous hand grasping the object, solve the control optimization model to determine the grasping control data of the multi-finger dexterous hand grasping the object, and control the grasping posture of the multi-finger dexterous hand according to the grasping control data, and solve the control optimization model in the multi-finger dexterous hand. When the grasping state matches the movement termination condition, the control optimization model is adjusted according to the updated value of the second expected posture data; the optimization goal of the control optimization model is to minimize the predicted value of the sum of the first posture error data, the movement distance data, and the second posture error data of the object until the target time; the first posture error data is determined according to the first spatial posture data and the first expected posture data of the multi-fingered dexterous hand; the second posture error data is determined according to the second spatial posture data and the second expected posture data of the object; The control optimization model is established in a motion solver, and the motion solver is used to solve the grasping control data of the multi-finger dexterous hand grasping the object; The controller uses the first spatial posture data and the second spatial posture data as the initial values ​​of the control optimization model of the multi-finger dexterous hand grasping the object to solve the control optimization model to determine the grasping control data of the multi-finger dexterous hand grasping the object. Specifically, the controller is used to input the first spatial posture data and the second spatial posture data into the motion solver, and determine the analytical gradient of the control optimization model according to the target hyperparameters of the motion solver, and when the analytical gradient is less than a preset gradient threshold, determine the output value of the motion solver as the grasping control data.

7. An electronic device, characterized in that: include: a processor, a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for moving an in-hand object using a multi-finger dexterous hand according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for moving an in-hand object using a multi-finger dexterous hand according to any one of claims 1 to 5.

Images