Method and system for moving object in hand by using multi-fingered dexterous hand

Abstract

The invention discloses an in-hand object moving method, system and device using a multi-fingered dexterous hand and a computer readable storage medium, and belongs to the field of manipulator control, and the method comprises the steps that under the condition that the grabbing state of the multi-fingered dexterous hand is not matched with a preset movement termination condition, first space pose data of the multi-fingered dexterous hand is obtained, obtaining second space pose data of the object; the first space pose data and the second space pose data serve as initial values of a control optimization model for the multi-fingered dexterous hand to grab the object, and the control optimization model is solved to determine grabbing control data for the multi-fingered dexterous hand to grab the object; and controlling the grasping posture of the multi-fingered dexterous hand according to the grasping control data. The multi-fingered dexterous hand is kept stable in the large-range object moving process by automatically adjusting the grabbing posture and optimizing control, and the problems that when the multi-fingered dexterous hand holds an object to move, precision is insufficient, and applicability is low are solved.

Description

Technical Field

[0001] The present application belongs to the field of manipulator control, and specifically relates to a method, system, device and computer-readable storage medium for moving an object in a hand using a multi-finger dexterous hand. Background Art

[0002] Multi-fingered dexterous hands for in-hand object movement have important applications in the field of robotic manipulation, especially in scenarios where complex tasks are performed in a small space, such as industrial assembly, medical operations, and service robots. In these scenarios, the manipulator must have high-precision and stable grasping and manipulation capabilities to complete complex tasks. Especially during large-scale movements, ensuring that the manipulator can stably control and move objects in the hand without moving the base is an important step in achieving the dexterity of the human hand.

[0003] In the prior art, multi-finger dexterous hands usually rely on the thumb as the core to move objects in the hand, and collect geometric information of the object in advance to optimize the trajectory using kinematic or dynamic models.

[0004] However, due to the requirement for an unchanged relative position between the object and the thumb of the robot, the range of motion of the object in the hand is limited, making it difficult to move accurately over a large range. At the same time, complex adjustments need to be made for different geometric models, which makes it difficult to meet the needs of actual tasks. Summary of the invention

[0005] The present application aims to provide a method, system, device and computer-readable storage medium for moving objects in the hand using a multi-finger dexterous hand, at least to solve the problems of insufficient accuracy and low applicability when the multi-finger dexterous hand holds and moves objects.

[0006] In a first aspect, an embodiment of the present application discloses a method for moving an object in a hand using a multi-finger dexterous hand, which is applied to a controller of the multi-finger dexterous hand, comprising: When the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, first spatial position data of the multi-finger dexterous hand is obtained, and second spatial position data of the object grasped by the multi-finger dexterous hand is obtained; The first spatial posture data and the second spatial posture data are used as initial values ​​of a control optimization model for the multi-finger dexterous hand to grasp the object, and the control optimization model is solved to determine the grasping control data for the multi-finger dexterous hand to grasp the object; the optimization goal of the control optimization model is to obtain the minimum value of the predicted value of the sum of the first posture error data, the motion distance data, and the second posture error data of the multi-finger dexterous hand until the target time; the first posture error data is determined according to the first spatial posture data and the first expected posture data of the multi-finger dexterous hand; the second posture error data is determined according to the second spatial posture data and the second expected posture data of the object; The grasping posture of the multi-finger dexterous hand is controlled according to the grasping control data.

[0007] In a second aspect, the present application also discloses a system for moving an object in a hand using a multi-finger dexterous hand, comprising: A multi-finger dexterous hand and a controller of the multi-finger dexterous hand; The multi-finger dexterous hand is used to grasp objects; The controller is used to obtain the first spatial posture data of the multi-finger dexterous hand and the second spatial posture data of the object grasped by the multi-finger dexterous hand when the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, and use the first spatial posture data and the second spatial posture data as the initial values ​​of the control optimization model of the multi-finger dexterous hand grasping the object, solve the control optimization model to determine the grasping control data of the multi-finger dexterous hand grasping the object, and control the grasping posture of the multi-finger dexterous hand according to the grasping control data; the optimization goal of the control optimization model is to minimize the predicted value of the sum of the first posture error data, the movement distance data, and the second posture error data of the multi-finger dexterous hand until the target time; the first posture error data is determined according to the first spatial posture data and the first expected posture data of the multi-finger dexterous hand; the second posture error data is determined according to the second spatial posture data and the second expected posture data of the object.

[0008] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application further discloses a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In summary, in the embodiment of the present application, the obtained spatial posture data is used as the initial value of the control optimization model to solve the control optimization model, and the grasping control data determined automatically adjusts the grasping posture of the multi-finger dexterous hand to ensure the stability and accuracy of the grasping; and then the control instructions of the multi-finger dexterous hand are determined by minimizing the posture error of the multi-finger dexterous hand, the motion distance data, and the minimum value of the sum of the posture error of the object as the optimization target, so as to accurately control the grasping posture so that the object remains stable during a large range of movement, thereby avoiding the limited range of motion caused by the restriction of the thumb; at the same time, the grasping posture of the multi-finger dexterous hand is repeatedly optimized and executed by the grasping control data, and the grasping posture is continuously adjusted to reduce the actual execution error, so that it can be applied to different object shapes. Therefore, based on the method of the embodiment of the present application, by automatically adjusting the grasping posture and optimizing the control, the multi-finger dexterous hand is kept stable during a large range of movement, the grasping accuracy and applicability are improved, and the problem of insufficient accuracy and low applicability when the multi-finger dexterous hand holds objects and moves is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the attached picture: Figure 1 is a flowchart of the steps of a method for moving an object in a hand using a multi-finger dexterous hand provided in an embodiment of the present application; Figure 2 is a schematic diagram for explaining the posture in the embodiment of the present application; Figure 3 is a flowchart of another method for moving an object in a hand using a multi-finger dexterous hand provided by an embodiment of the present application; Figure 4 A block diagram of a system for moving an object in a hand using a multi-finger dexterous hand provided in an embodiment of the present application; Figure 5 is a block diagram of an electronic device according to an embodiment of the present application; Figure 6 It is a block diagram of an electronic device of another embodiment provided by the embodiments of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0013] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0014] In the prior art, when a multi-finger dexterous hand grasps an object, the thumb of the multi-finger dexterous hand is used as the core, and it is assumed that the relative posture (position and orientation) between the object and the thumb remains unchanged. Since the thumb of a multi-finger dexterous hand usually has no more than four degrees of freedom and the finger joint angles are limited, this assumption greatly limits the object's movement space, making it impossible for the object to reach a desired position farther away.

[0015] The multi-finger dexterous hand is an advanced manipulator device, which is characterized by at least one finger joint on each finger. Through the combined movement of these joints, the multi-finger dexterous hand can achieve efficient and accurate grasping operations on objects with multi-degree-of-freedom flexibility. Therefore, the present invention proposes a trajectory optimization method that is completely kinematic and does not rely on the geometric information of the object surface. Under the premise of establishing an initial stable grasp, the trajectory optimization method can be applied to any unknown object in the hand. The fundamental idea is to maintain a stable grasp by constraining the position of the fingertips on the object surface to remain unchanged throughout the trajectory. At the same time, by optimizing the error between the expected position and the predicted position of the object, the object can be moved to the expected position as accurately as possible.

[0016] Considering the grasping process within a certain time period 1:T, the specific mathematical description of the trajectory optimization problem can be expressed by the following formula: ; ; ; Among them, p represents the position coordinate of the object, R represents the direction of the object, and q represents the motion vector of the finger. Represents the error between the final position of the object's trajectory and the expected position, Indicates the trajectory The error between the position of the fingertip in the object coordinate system at the moment and the initial time, Indicates the movement distance of the finger in the joint space during the entire trajectory; In the constraint function, represents the motion vector of the finger at time t, It means that the joint angle of the finger at any time does not exceed the joint angle range allowed by the mechanical structure. Indicates the difference between the mutual distances between the calibration points generated based on the calibration points selected on the finger and the minimum value of these mutual distances. It can be used to prevent collisions between fingers.

[0017] By solving the above optimization problem, the action instructions of the multi-finger dexterous hand in each time unit can be obtained, and through the process of re-planning and re-execution, the results of the finger movement after the action instructions are fed back to achieve a closed-loop execution operation, which can achieve the effect of improving the accuracy of object movement; at the same time, it can be seen that through the above process, the introduction of too many spatial reference points is avoided, that is, there is no need to examine the specific shape of the grasped object, and the grasping effect is still maintained while the model is simplified.

[0018] Based on the above theoretical foundation, Figure 1 As shown, a method for moving an object in a hand using a multi-finger dexterous hand is provided in an embodiment of the present application, and is applied to a controller of the multi-finger dexterous hand.

[0019] The method may include the following steps: Step 101, when the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, first spatial posture data of the multi-finger dexterous hand is obtained, and second spatial posture data of the object grasped by the multi-finger dexterous hand is obtained.

[0020] In some embodiments of the present application, in order to ensure that the multi-finger dexterous hand maintains stability and accuracy during the grasping process, it is necessary to obtain the first spatial posture data of the multi-finger dexterous hand when the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, and obtain the second spatial posture data of the object grasped by the multi-finger dexterous hand. By obtaining these data, the actual state of the multi-finger dexterous hand and the grasped object can be monitored and evaluated in real time, so as to make timely adjustments and optimizations. The posture data of the multi-finger dexterous hand and the grasped object can be obtained by using sensors or measuring equipment. The spatial posture data is used to describe the current position and orientation of the multi-finger dexterous hand and the object. In this way, control optimization can be performed based on the acquired data to ensure stability and accuracy during the grasping process.

[0021] In a specific example, as in Figure 2As shown in , when the multi-finger dexterous hand 301 is used to grasp the object X, the system finds that the grasping state does not match the preset movement termination condition. The first spatial posture data A1 of the multi-finger dexterous hand in the world coordinate system is obtained by using the sensor on the device, and the second spatial posture data B2 of the object in the object coordinate system is obtained through the label or mark O attached to the object. The system uses these data for subsequent optimization solutions to ensure the stability and accuracy of the multi-finger dexterous hand during the grasping process. By obtaining the spatial posture data and the movement distance data, the operation of the multi-finger dexterous hand can be monitored and adjusted in real time to ensure the stability and accuracy of the grasping process.

[0022] Step 102: Using the first spatial posture data and the second spatial posture data as initial values ​​of a control optimization model for the multi-finger dexterous hand to grasp an object, the control optimization model is solved to determine grasping control data for the multi-finger dexterous hand to grasp an object.

[0023] Among them, the optimization goal of the control optimization model is to minimize the predicted value of the sum of the first pose error data, the motion distance data, and the second pose error data of the object of the multi-finger dexterous hand until the target time; the first pose error data is determined according to the first spatial pose data and the first expected pose data of the multi-finger dexterous hand; the second pose error data is determined according to the second spatial pose data and the second expected pose data of the object.

[0024] In some embodiments of the present application, in order to solve the control optimization model of the multi-finger dexterous hand grasping an object, the first spatial posture data and the second spatial posture data need to be input as the initial values ​​of the control optimization model. By using these initial value data, it can be ensured that the control optimization model accurately reflects the current state of the multi-finger dexterous hand and the grasped object, thereby optimizing the grasping control data. The process of performing this step is to input the acquired first spatial posture data and second spatial posture data into the control optimization model, and solve the model to determine the grasping control data of the multi-finger dexterous hand grasping the object. The first spatial posture data describes the current position and orientation of the multi-finger dexterous hand, the second spatial posture data describes the current position and orientation of the grasped object, and the motion distance data records the displacement of the multi-finger dexterous hand during the movement. The optimization goal of the control optimization model is to minimize the predicted value of the sum of the first posture error data, the motion distance data and the second posture error data of the multi-finger dexterous hand until the target time. The optimized grasping control data obtained in this step will be used in the subsequent grasping control process, thereby ensuring the accuracy and stability of the grasping process.

[0025] In a specific example, Figure 2As shown, when the multi-finger dexterous hand 301 is used to grasp the object X, the first spatial posture data A1 of the multi-finger dexterous hand in the world coordinate system and the first expected posture data A2 corresponding thereto, and the second spatial posture data B1 of the grasped object in the object coordinate system and the second expected posture data B2 corresponding thereto are obtained through the sensor. These data are input into the control optimization model and solved to determine the grasping control data. The optimization goal of the control optimization model is to minimize the predicted value of the sum of the posture error and the motion distance data of the multi-finger dexterous hand and the object. Through this process, the optimized grasping control data obtained will be used to control the grasping posture of the multi-finger dexterous hand. By solving the control optimization model, accurate control of the grasping process can be achieved, thereby improving the stability and accuracy of the multi-finger dexterous hand in the process of grasping the moving object.

[0026] Step 103: Control the grasping posture of the multi-finger dexterous hand according to the grasping control data.

[0027] In some embodiments of the present application, in order to achieve precise control of the grasping posture of the multi-finger dexterous hand, it is necessary to operate according to the grasping control data. By using the optimized grasping control data, it is possible to ensure that the multi-finger dexterous hand maintains a stable grasping state in actual operation, thereby improving the accuracy and stability of grasping. According to the grasping control data obtained by the previous solution, the grasping posture of the multi-finger dexterous hand is adjusted in real time to adapt to the actual situation of the grasped object. The grasping control data contains the joint angles of the multi-finger dexterous hand at different time points to ensure the coordinated movements of each joint during the grasping process. The multi-finger dexterous hand can perform grasping operations according to a predetermined trajectory and posture, thereby achieving stable movement of the grasped object.

[0028] In a specific example, when using a multi-finger dexterous hand to perform a grasping operation, the grasping control data obtained previously is input into the control system of the multi-finger dexterous hand. Based on these data, the system adjusts the joint angles of the multi-finger dexterous hand in real time to ensure that the movements of each joint are coordinated during the grasping process. Through this process, the multi-finger dexterous hand can stably grasp and move objects according to a predetermined trajectory and posture. By controlling the grasping posture of the multi-finger dexterous hand according to the grasping control data, the stable movement of the object can be achieved, and the accuracy and stability of the grasping can be improved.

[0029] In summary, in the embodiment of the present application, the obtained spatial posture data is used as the initial value of the control optimization model to solve the control optimization model, and the grasping control data determined automatically adjusts the grasping posture of the multi-finger dexterous hand to ensure the stability and accuracy of the grasping; and then the control instructions of the multi-finger dexterous hand are determined by minimizing the posture error of the multi-finger dexterous hand, the motion distance data, and the minimum value of the sum of the posture error of the object as the optimization target, so as to accurately control the grasping posture so that the object remains stable during a large range of movement, thereby avoiding the limited range of motion caused by the restriction of the thumb; at the same time, the grasping posture of the multi-finger dexterous hand is repeatedly optimized and executed by the grasping control data, and the grasping posture is continuously adjusted to reduce the actual execution error, so that it can be applied to different object shapes. Therefore, based on the method of the embodiment of the present application, by automatically adjusting the grasping posture and optimizing the control, the multi-finger dexterous hand is kept stable during a large range of movement, the grasping accuracy and applicability are improved, and the problem of insufficient accuracy and low applicability when the multi-finger dexterous hand holds objects and moves is solved.

[0030] Figure 3 Another method for moving an object in a hand using a multi-finger dexterous hand is provided in an embodiment of the present application and is applied to a controller of the multi-finger dexterous hand.

[0031] The method may include the following steps: Step 201: Establish a control optimization model based on preset first initial expected posture data and second initial expected posture data.

[0032] In some embodiments of the present application, in order to establish a control optimization model, it is necessary to preset the first initial expected posture data and the second initial expected posture data. By establishing an accurate control optimization model, an optimization target can be provided for the grasping process of the multi-finger dexterous hand to ensure stability and accuracy during the grasping process. The specific process is to obtain the first initial expected posture data and the second initial expected posture data of the multi-finger dexterous hand, and use these data to establish a control optimization model. The first initial expected posture data and the second initial expected posture data are used to describe the position and orientation of the multi-finger dexterous hand and the object under ideal conditions. The control optimization model established in this way will serve as the basis for solving the grasping control data in subsequent steps, thereby realizing the optimized control of the grasping process of the multi-finger dexterous hand.

[0033] In a specific example, before starting the operation of the multi-finger dexterous hand, the user pre-sets an ideal grasping state, which includes the first initial expected posture data of the multi-finger dexterous hand and the second initial expected posture data of the object to be grasped. These data are input into the control system of the multi-finger dexterous hand, and the system uses these data to establish a control optimization model, which serves as the basis for subsequent control optimization. By setting the initial expected posture data and establishing the control optimization model, the system can optimize the grasping control data in actual operation, thereby effectively improving the grasping stability and accuracy of the multi-finger dexterous hand.

[0034] Step 202, when the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, first spatial posture data of the multi-finger dexterous hand is obtained, and second spatial posture data of the object grasped by the multi-finger dexterous hand is obtained.

[0035] The method shown in this step has been explained in step 101 and will not be repeated here.

[0036] Optionally, in order to obtain the first spatial posture data of the multi-fingered dexterous hand, step 202 includes the following sub-steps: Sub-step 2021, obtaining the joint posture data of each joint of the multi-finger dexterous hand.

[0037] Each joint posture data is used to characterize the spatial position and / or bending angle of the joint corresponding to the joint posture data.

[0038] In some embodiments of the present application, in order to obtain the first spatial posture data of the multi-finger dexterous hand, it is necessary to first obtain the joint posture data of each joint of the multi-finger dexterous hand. By obtaining the posture data of each joint, the specific position and state of each joint of the multi-finger dexterous hand during operation can be fully understood. The spatial position and / or bending angle of each joint of the multi-finger dexterous hand is recorded using a sensor or a measuring device. The joint posture data is used to describe the position and angle of each joint in the current state. The acquired joint posture data is used as the basis for further calculating the spatial posture vector of the multi-finger dexterous hand, which can provide accurate initial data for subsequent steps.

[0039] In a specific example, when using a multi-finger dexterous hand for grasping operations, the joint posture data of each joint is first obtained through the sensors installed on each joint. These data include the spatial position and bending angle of the joint. These data are recorded and transmitted to the control system of the multi-finger dexterous hand. Through this process, the system can fully understand the specific status of each joint of the multi-finger dexterous hand and provide accurate initial data for subsequent control optimization. By obtaining the joint posture data of each joint, accurate and comprehensive joint status information can be provided for the operation of the multi-finger dexterous hand, thereby improving the accuracy and stability of the operation.

[0040] In sub-step 2022, the joint posture vector of the multi-fingered dexterous hand generated based on all the joint posture data is determined as the first spatial posture data.

[0041] In some embodiments of the present application, in order to determine the first spatial pose data of the multi-finger dexterous hand, it is necessary to calculate the joint pose vector of the multi-finger dexterous hand generated according to all the joint pose data. By integrating all the joint pose data, a complete joint pose vector can be generated to accurately describe the overall spatial pose of the multi-finger dexterous hand. The pose data of each joint previously acquired are integrated to generate a joint pose vector of the multi-finger dexterous hand. The joint pose vector is used to describe the overall spatial position and orientation of the multi-finger dexterous hand in the current state. The generated joint pose vector will be used as the first spatial pose data to provide basic data for subsequent grasping control, thereby ensuring the operational accuracy of the multi-finger dexterous hand.

[0042] In a specific example, when using a multi-finger dexterous hand for grasping operations, the joint posture data of each joint is first obtained through the sensor. Subsequently, these joint posture data are input into the system, and the system integrates all the joint posture data to generate a joint posture vector. This joint posture vector describes the overall spatial position and orientation of the multi-finger dexterous hand in the current state. The generated joint posture vector is determined as the first spatial posture data and input into the control optimization model. In this example, by generating and determining the joint posture vector, the overall spatial posture of the multi-finger dexterous hand can be accurately described, providing basic data for subsequent optimization control and improving the accuracy and stability of the operation.

[0043] Optionally, in order to obtain second spatial position data of an object grasped by the multi-fingered dexterous hand, step 202 includes the following sub-steps: Sub-step 2023, obtaining spatial coordinate data of at least one reference point on the object.

[0044] In some embodiments of the present application, in order to obtain the second spatial posture data of the object, it is necessary to first obtain the spatial coordinate data of at least one reference point on the object. By obtaining the spatial coordinate data of the reference point of the object, the position and orientation of the object in space can be accurately located, thereby providing basic data for further control optimization. The spatial coordinate data of the specified reference point on the object is measured and recorded using a sensor or measuring device. The spatial coordinate data of the reference point is used to describe the position of the object in the current state. The acquired spatial coordinate data of the reference point is used as the basis for further calculating the second spatial posture data of the object, thereby ensuring the accuracy of the multi-finger dexterous hand's operation on the object.

[0045] In a specific example, when using a multi-finger dexterous hand for grasping operations, the spatial coordinate data of at least one reference point on the object can be obtained by setting a mark point on the object. These data are captured and recorded by a camera installed in the working environment. These coordinate data are input into the control system of the multi-finger dexterous hand for subsequent calculation and optimization of the spatial posture data of the object. In this example, by obtaining the spatial coordinate data of at least one reference point on the object, the position and orientation of the object can be accurately located, providing an accurate data basis for further control optimization, thereby improving the operation accuracy and stability of the multi-finger dexterous hand.

[0046] Sub-step 2024, determining the second spatial pose data based on all the spatial coordinate data.

[0047] In some embodiments of the present application, in order to determine the second spatial posture data of the object, it is necessary to perform calculations based on all the spatial coordinate data. By integrating the spatial coordinate data of all reference points, the overall position and orientation of the object in space can be accurately described, thereby providing basic data for the control optimization of the multi-fingered dexterous hand. The spatial coordinate data of all reference points previously acquired are integrated to generate the second spatial posture data of the object. The second spatial posture data is used to describe the overall position and orientation of the object in the current state. The generated second spatial posture data will serve as the basis for optimizing the control in subsequent steps, thereby ensuring the operational accuracy and stability of the multi-fingered dexterous hand.

[0048] In a specific example, when using a multi-finger dexterous hand for grasping operations, the spatial coordinate data of multiple reference points on the object can be obtained by setting marking points on the object. These data can be captured and recorded by a camera installed in the working environment. The experimenter inputs these coordinate data into the system, and the system integrates the spatial coordinate data of all reference points to generate the second spatial posture data of the object. This posture data describes the overall position and orientation of the object in the current state. The generated second spatial posture data is input into the control optimization model for subsequent optimization control. By integrating the spatial coordinate data of the reference points, the overall spatial posture of the object can be accurately described, providing an accurate data basis for the operation of the multi-finger dexterous hand, thereby improving the accuracy and stability of the operation.

[0049] Step 203: Use the first spatial posture data and the second spatial posture data as initial values ​​of a control optimization model for the multi-finger dexterous hand to grasp the object, and solve the control optimization model to determine the grasping control data for the multi-finger dexterous hand to grasp the object.

[0050] Among them, the optimization goal of the control optimization model is to minimize the predicted value of the sum of the first pose error data, the motion distance data, and the second pose error data of the object of the multi-finger dexterous hand until the target time; the first pose error data is determined according to the first spatial pose data and the first expected pose data of the multi-finger dexterous hand; the second pose error data is determined according to the second spatial pose data and the second expected pose data of the object.

[0051] The method shown in this step has been explained in step 102 and will not be repeated here.

[0052] Optionally, the control optimization model is established in a motion solver, and the motion solver is used to solve the grasping control data of the multi-finger dexterous hand grasping the object. Step 203 includes the following sub-steps: Sub-step 2031, inputting the first spatial pose data, the second spatial pose data and the motion distance data into the motion solver, and determining the analytical gradient of the control optimization model according to the target hyperparameters of the motion solver.

[0053] In some embodiments of the present application, in order to quickly determine the solution of the control optimization model, the first spatial posture data, the second spatial posture data and the motion distance data need to be input into the motion solver, and calculated according to the target hyperparameters of the motion solver. By inputting these data into the motion solver, its target hyperparameters can be used to determine the analytical gradient of the control optimization model, thereby determining the grasping control data of the multi-fingered dexterous hand. All the acquired posture data and motion distance data are input into the motion solver, and solved according to the pre-set target hyperparameters. The target hyperparameters of the motion solver are used to define the optimization parameters used in the solution process. The generated analytical gradient will be used as the basis for judging the accuracy of the grasping control data in the subsequent steps, thereby ensuring the accuracy of the grasping process.

[0054] In a specific example, when a multi-finger dexterous hand is used to perform a grasping operation, the first spatial posture data, the second spatial posture data, and the motion distance data are obtained through the sensor. These data are input into the motion solver in the control system of the multi-finger dexterous hand, and the system calculates according to the set target hyperparameters to determine the analytical gradient of the control optimization model. Through this process, the analytical gradient for further optimizing the grasping control data can be obtained, providing an accurate control basis for the operation of the multi-finger dexterous hand. By inputting the posture data and the motion distance data into the motion solver, the grasping control data of the multi-finger dexterous hand can be optimized to improve the accuracy and stability of the grasping operation.

[0055] Sub-step 2032, when the analytical gradient is less than a preset gradient threshold, determining the output value of the motion solver as the grasping control data.

[0056] In some embodiments of the present application, in order to quickly determine the output value of the motion solver as accurate grasping control data, it is necessary to determine the output value of the motion solver as grasping control data when the analytical gradient is less than a preset gradient threshold. By setting the threshold of the analytical gradient, the calculation accuracy and reliability of the grasping control data can be ensured. After calculating the analytical gradient of the motion solver, the analytical gradient is compared with the preset gradient threshold. When the analytical gradient is less than the threshold, the output value of the motion solver is determined as the grasping control data. The analytical gradient is an indicator that describes the rate of change of the output value during the solution process and is used to evaluate the accuracy of the solution result. The obtained grasping control data will be used for actual grasping operations, thereby ensuring the operational continuity of the multi-fingered dexterous hand.

[0057] In a specific example, when operating a multi-finger dexterous hand, the first spatial posture data, the second spatial posture data and the motion distance data are acquired through the sensor and input into the motion solver for calculation. The system calculates the analytical gradient of the motion solver and compares it with the preset gradient threshold. If the analytical gradient is less than the preset gradient threshold, the output value of the motion solver is determined as the grasping control data and used for the actual grasping operation of the multi-finger dexterous hand. By setting the threshold of the analytical gradient, the calculation accuracy of the grasping control data can be ensured, and the output result can be obtained quickly, thereby improving the operational continuity of the multi-finger dexterous hand.

[0058] Step 204: Control the grasping posture of the multi-finger dexterous hand according to the grasping control data.

[0059] The method shown in this step has been explained in step 103 and will not be repeated here.

[0060] Step 205: When the grasping state of the multi-finger dexterous hand matches the movement termination condition, the control optimization model is adjusted according to the updated value of the second desired posture data.

[0061] In some embodiments of the present application, in order to adapt to the needs of the subsequent grasping process, it is necessary to adjust the control optimization model according to the updated value of the second expected posture data when the grasping state of the multi-finger dexterous hand matches the movement termination condition. By updating the control optimization model, the latest state requirements of the multi-finger dexterous hand and the grasped object can be reflected in a timely manner, thereby adjusting the grasping control. When the grasping state of the multi-finger dexterous hand reaches the preset movement termination condition, the system will adjust the previously established control optimization model according to the updated value of the second expected posture data currently obtained. Through this adjustment, it can be ensured that the control optimization model matches the new grasping requirements. The adjusted control model will be able to adapt to subsequent grasping targets.

[0062] In a specific example, when operating a multi-finger dexterous hand, when the system detects that the grasping state of the multi-finger dexterous hand matches the preset movement termination condition, the system obtains the updated value of the second desired posture data of the current object. The system adjusts the control optimization model according to these updated values. Through this adjustment, the optimized control model can accurately reflect the latest goal of grasping the object to ensure that the control data provided for subsequent grasping operations matches the subsequent requirements. By adjusting the control optimization model, the system can achieve the continuity requirements of the grasping process and ensure the long-term stability of the multi-finger dexterous hand during the movement of the grasping object.

[0063] Optionally, in some implementations of the present application, the movement termination condition is one of the following conditions: the actual execution error is less than the ideal error calculated by trajectory optimization, the number of replanning times reaches a preset maximum replanning time threshold, and the replanning time interval reaches a preset maximum replanning time threshold.

[0064] Among them, the actual execution error is used to characterize the difference between the second spatial posture data of the object and the second expected posture data; the number of re-planning is used to record the number of executions of the grasping posture of the multi-finger dexterous hand controlled according to the grasping control data from the preset starting time to the target time; the re-planning time is used to record the time interval from the preset starting time to the target time.

[0065] In summary, in the embodiment of the present application, the obtained spatial posture data is used as the initial value of the control optimization model to solve the control optimization model, and the grasping control data determined automatically adjusts the grasping posture of the multi-finger dexterous hand to ensure the stability and accuracy of the grasping; and then the control instructions of the multi-finger dexterous hand are determined by minimizing the posture error of the multi-finger dexterous hand, the motion distance data, and the minimum value of the sum of the posture error of the object as the optimization target, so as to accurately control the grasping posture so that the object remains stable during a large range of movement, thereby avoiding the limited range of motion caused by the restriction of the thumb; at the same time, the grasping posture of the multi-finger dexterous hand is repeatedly optimized and executed by the grasping control data, and the grasping posture is continuously adjusted to reduce the actual execution error, so that it can be applied to different object shapes. Therefore, based on the method of the embodiment of the present application, by automatically adjusting the grasping posture and optimizing the control, the multi-finger dexterous hand is kept stable during a large range of movement, the grasping accuracy and applicability are improved, and the problem of insufficient accuracy and low applicability when the multi-finger dexterous hand holds objects and moves is solved.

[0066] refer to Figure 4 , which shows a system 30 for moving an object in a hand using a multi-finger dexterous hand provided by an embodiment of the present application, comprising: A multi-finger dexterous hand 301 and a controller 302 of the multi-finger dexterous hand; The multi-finger dexterous hand 301 is used to grasp objects; The controller 302 is used to obtain the first spatial posture data of the multi-finger dexterous hand and the second spatial posture data of the object grasped by the multi-finger dexterous hand when the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, and use the first spatial posture data and the second spatial posture data as the initial values ​​of the control optimization model of the multi-finger dexterous hand grasping the object, solve the control optimization model to determine the grasping control data of the multi-finger dexterous hand grasping the object, and control the grasping posture of the multi-finger dexterous hand according to the grasping control data; the optimization goal of the control optimization model is to minimize the predicted value of the sum of the first posture error data, movement distance data, and second posture error data of the multi-finger dexterous hand until the target time; the first posture error data is determined according to the first spatial posture data and the first expected posture data of the multi-finger dexterous hand; the second posture error data is determined according to the second spatial posture data and the second expected posture data of the object.

[0067] Optionally, the in-hand object movement system 30 using the multi-finger dexterous hand is further used to establish a control optimization model according to preset first initial expected posture data and second initial expected posture data.

[0068] Optionally, the in-hand object movement system 30 using the multi-finger dexterous hand is also used to adjust the control optimization model according to the updated value of the second expected posture data when the grasping state of the multi-finger dexterous hand matches the movement termination condition.

[0069] Optionally, the control optimization model is established on a motion solver, and the motion solver is used to solve the grasping control data of the multi-finger dexterous hand grasping the object; the in-hand object movement system 30 using the multi-finger dexterous hand is also used to use the first spatial posture data and the second spatial posture data as the initial values ​​of the control optimization model of the multi-finger dexterous hand grasping the object, and solve the control optimization model to determine the grasping control data of the multi-finger dexterous hand grasping the object, and specifically input the first spatial posture data, the second spatial posture data and the motion distance data into the motion solver, and determine the analytical gradient of the control optimization model according to the target hyperparameters of the motion solver, and when the analytical gradient is less than a preset gradient threshold, determine the output value of the motion solver as the grasping control data.

[0070] Optionally, when using the in-hand object movement system 30 of the multi-finger dexterous hand to obtain the first spatial posture data of the multi-finger dexterous hand, it is specifically used to obtain the joint posture data of each joint of the multi-finger dexterous hand; each joint posture data is used to characterize the spatial position and / or bending angle of the joint corresponding to the joint posture data, and the joint posture vector of the multi-finger dexterous hand generated according to all the joint posture data is determined as the first spatial posture data.

[0071] Optionally, when the in-hand object movement system 30 of the multi-finger dexterous hand obtains the second spatial posture data of an object grasped by the multi-finger dexterous hand, it is specifically used to obtain the spatial coordinate data of at least one reference point on the object, and determine the second spatial posture data based on all the spatial coordinate data.

[0072] In summary, in the embodiment of the present application, the obtained spatial posture data is used as the initial value of the control optimization model to solve the control optimization model, and the grasping control data determined automatically adjusts the grasping posture of the multi-finger dexterous hand to ensure the stability and accuracy of the grasping; and then the control instructions of the multi-finger dexterous hand are determined by minimizing the posture error of the multi-finger dexterous hand, the motion distance data, and the minimum value of the sum of the posture error of the object as the optimization target, so as to accurately control the grasping posture so that the object remains stable during a large range of movement, thereby avoiding the limited range of motion caused by the restriction of the thumb; at the same time, the grasping posture of the multi-finger dexterous hand is repeatedly optimized and executed by the grasping control data, and the grasping posture is continuously adjusted to reduce the actual execution error, so that it can be applied to different object shapes. Therefore, based on the method of the embodiment of the present application, by automatically adjusting the grasping posture and optimizing the control, the multi-finger dexterous hand is kept stable during a large range of movement, the grasping accuracy and applicability are improved, and the problem of insufficient accuracy and low applicability when the multi-finger dexterous hand holds objects and moves is solved.

[0073] Reference Figure 5 , the electronic device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0074] The processing component 502 generally controls the overall operation of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0075] The memory 504 is used to store various types of data to support the operation of the electronic device 500. Examples of such data include instructions for any application or method operating on the electronic device 500, contact data, phone book data, messages, pictures, multimedia, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0076] The power supply component 506 provides power to the various components of the electronic device 500. The power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 500.

[0077] The multimedia component 508 includes an interface that provides an output interface between the electronic device 500 and the user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 500 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0078] The audio component 510 is used to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the electronic device 500 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0079] The input / output I / O interface 512 provides an interface between the processing component 502 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0080] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 500. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 500, the relative positioning of components, such as the display and keypad of the electronic device 500, and the sensor assembly 514 can also detect the position change of the electronic device 500 or a component of the electronic device 500, the presence or absence of user contact with the electronic device 500, the orientation or acceleration / deceleration of the electronic device 500, and the temperature change of the electronic device 500. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0081] The communication component 516 is used to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0082] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to implement the methods provided in the embodiments of the present application.

[0083] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, and the instructions can be executed by a processor 520 of an electronic device 500 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0084] Figure 6FIG. 6 is a block diagram of an electronic device 600 according to another embodiment of the present invention. For example, the electronic device 600 may be provided as a server.

[0085] Reference Figure 6 , the electronic device 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632, for storing instructions that can be executed by the processing component 622, such as an application. The application stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute instructions to perform the method provided in the embodiment of the present application.

[0086] The electronic device 600 may also include a power supply component 626 configured to perform power management of the electronic device 600, a wired or wireless network interface 650 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 658. The electronic device 600 may operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0087] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0088] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for moving an object in a hand using a multi-finger dexterous hand, characterized in that: Controller for multi-finger dexterous hand, including: When the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, first spatial position data of the multi-finger dexterous hand is obtained, and second spatial position data of the object grasped by the multi-finger dexterous hand is obtained; The first spatial posture data and the second spatial posture data are used as initial values ​​of a control optimization model for the multi-finger dexterous hand to grasp the object, and the control optimization model is solved to determine the grasping control data for the multi-finger dexterous hand to grasp the object; the optimization goal of the control optimization model is to minimize the predicted value of the sum of the first posture error data, the motion distance data, and the second posture error data of the multi-finger dexterous hand until the target time; the first posture error data is determined according to the first spatial posture data and the first expected posture data for the multi-finger dexterous hand; the second posture error data is determined according to the second spatial posture data and the second expected posture data of the object; The grasping posture of the multi-finger dexterous hand is controlled according to the grasping control data.

2. The method for moving an object in a hand using a multi-finger dexterous hand as claimed in claim 1, characterized in that: In the case where the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, before obtaining the first spatial posture data of the multi-finger dexterous hand and obtaining the second spatial posture data of the object grasped by the multi-finger dexterous hand, the method for moving an in-hand object using the multi-finger dexterous hand also includes: The control optimization model is established according to the preset first initial expected posture data and the second initial expected posture data.

3. The method for moving an object in a hand using a multi-finger dexterous hand as claimed in claim 1, characterized in that: The method for moving an object in a hand using a multi-finger dexterous hand also includes: When the grasping state of the multi-finger dexterous hand matches the movement termination condition, the control optimization model is adjusted according to the updated value of the second desired posture data.

4. The method for moving an object in a hand using a multi-finger dexterous hand as claimed in claim 1, characterized in that: The control optimization model is established in a motion solver, and the motion solver is used to solve the grasping control data of the multi-finger dexterous hand grasping the object; the first spatial posture data and the second spatial posture data are used as initial values ​​of the control optimization model of the multi-finger dexterous hand grasping the object, and the control optimization model is solved to determine the grasping control data of the multi-finger dexterous hand grasping the object, including: Inputting the first spatial pose data and the second spatial pose data into the motion solver, and determining an analytical gradient of the control optimization model according to a target hyperparameter of the motion solver; When the analytical gradient is less than a preset gradient threshold, the output value of the motion solver is determined as the grasping control data.

5. The method for moving an object in a hand using a multi-finger dexterous hand as claimed in claim 1, characterized in that: The obtaining of first spatial posture data of the multi-fingered dexterous hand comprises: Acquire joint posture data of each joint of the multi-fingered dexterous hand; each joint posture data is used to characterize the spatial position and / or bending angle of the joint corresponding to the joint posture data; The joint posture vector of the multi-fingered dexterous hand generated based on all the joint posture data is determined as the first spatial posture data.

6. The method for moving an object in a hand using a multi-finger dexterous hand as claimed in claim 1, characterized in that: The obtaining of second spatial position data of the object grasped by the multi-finger dexterous hand comprises: Acquiring spatial coordinate data of at least one reference point on the object; The second spatial position data is determined based on all the spatial coordinate data.

7. The method for moving an object in a hand using a multi-finger dexterous hand as claimed in claim 1, characterized in that: The formula of the control optimization model is: ； ； ； Among them, p represents the position coordinate of the object, R represents the direction of the object, and q represents the motion vector of the finger. Represents the error between the final position of the object's trajectory and the expected position, Indicates the trajectory The error between the position of the fingertip in the object coordinate system at the moment and the initial time, Indicates the movement distance of the finger in the joint space during the entire trajectory; In the constraint function, represents the motion vector of the finger at time t, It means that the joint angle of the finger at any time does not exceed the joint angle range allowed by the mechanical structure. It represents the difference between the mutual distances between the calibration points generated according to the calibration points selected on the finger and the minimum value of these mutual distances.

8. A system for moving objects in a hand using a multi-finger dexterous hand, characterized in that: include: A multi-finger dexterous hand and a controller of the multi-finger dexterous hand; The multi-finger dexterous hand is used to grasp objects; The controller is used to obtain the first spatial posture data of the multi-finger dexterous hand and the second spatial posture data of the object grasped by the multi-finger dexterous hand when the grasping state of the multi-finger dexterous hand does not match the preset movement termination condition, and use the first spatial posture data and the second spatial posture data as the initial values ​​of the control optimization model of the multi-finger dexterous hand grasping the object, solve the control optimization model to determine the grasping control data of the multi-finger dexterous hand grasping the object, and control the grasping posture of the multi-finger dexterous hand according to the grasping control data; the optimization goal of the control optimization model is to minimize the predicted value of the sum of the first posture error data, the movement distance data, and the second posture error data of the multi-finger dexterous hand until the target time; the first posture error data is determined according to the first spatial posture data and the first expected posture data of the multi-finger dexterous hand; the second posture error data is determined according to the second spatial posture data and the second expected posture data of the object.

9. An electronic device, characterized in that: include: A processor, a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for moving an in-hand object using a multi-finger dexterous hand as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for moving an in-hand object using a multi-finger dexterous hand as claimed in any one of claims 1 to 7.

Images