Multi-sensing intelligent anthropomorphic five-finger dexterous hand
By optimizing the path and response of the humanoid dexterous hand through multimodal perception, differential geometry path planning, and reinforcement learning, the problem of insufficient synchronization between perception and response in existing technologies is solved, and efficient and safe grasping operations are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUXI QIANYE TECHNOLOGY CO LTD
- Filing Date
- 2025-11-12
- Publication Date
- 2026-07-31
AI Technical Summary
Existing humanoid dexterous hands have technical defects in perception, path planning, and response synchronization, resulting in localized pressure overload, path redundancy, and excessively long response times during the grasping process, which cannot meet the real-time requirements of industrial assembly and minimally invasive medical surgery.
A multimodal perception module is used to generate a fine 3D contour of the object. The optimal contact point is selected by combining differential geometry path planning and sequential quadratic programming. The joint angle and torque are optimized by reinforcement learning and force-position hybrid control. The response time is optimized by using the chaotic gray wolf optimization algorithm.
It improves the precision of path planning, ensures the safety and smoothness of the grasping process, shortens response time, and enhances the stability and efficiency of grasping operations.
Smart Images
Figure CN121374580B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot control and intelligent algorithms, specifically to a multi-sensory intelligent humanoid five-fingered dexterous hand. Background Technology
[0002] This invention belongs to the field of robot control and intelligent algorithms, and is applicable to human-machine collaboration scenarios such as industrial precision assembly. It aims to solve the technical defects of existing humanoid dexterous hands in terms of perception, path planning, control and response synchronization.
[0003] In the current path generation process of dexterous hands, the selection of accessible areas of objects relies heavily on experience or simple geometric rules. It does not use surface features such as quantified principal curvature to select low-stress candidate contact points, and tends to select sharp areas with high curvature, which can lead to local pressure overload on the object. The path planning does not fully integrate multimodal data and constraints of the object, and often uses linear or simple curve planning methods, which cannot conform to the object's surface to generate a smooth geodesic path. This may result in problems such as finger-object collision during grasping and path redundancy leading to response delay.
[0004] The sensor sampling, joint calculation, torque calculation, and motor execution of a dexterous hand all have inherent time lags. Existing technologies do not address these lags through coordinated optimization, relying only on simple delay compensation or fixed timing control. This results in asynchronous actions between modules and an excessively long overall response time, failing to meet the real-time requirements of scenarios such as industrial assembly and minimally invasive medical surgery. It also affects the stability and efficiency of grasping operations. Therefore, a multi-sensory intelligent humanoid five-fingered dexterous hand is needed to solve these problems. Summary of the Invention
[0005] In order to solve the technical problems mentioned in the background, the present invention provides a multi-sensory intelligent humanoid five-fingered dexterous hand.
[0006] The objective of this invention can be achieved through the following technical solutions:
[0007] This invention provides a multi-sensory intelligent humanoid five-fingered dexterous hand, comprising the following steps:
[0008] This invention provides a multi-sensory intelligent humanoid five-fingered dexterous hand, including a multimodal perception module, a path generation module, a control module, and a response optimization module;
[0009] The multimodal perception module collects object features and locates them through sensors, generates a detailed 3D contour of the object, and matches it with a preset contour in the database to obtain multimodal data of the object, specifically:
[0010] Once the fingers are returned to their initial position, the RGB-D camera in the palm of the hand is activated to scan the operating area, locate the center coordinates and angle of the object, acquire the object's two-dimensional image and depth information, and then generate a rough three-dimensional outline of the object through point cloud reconstruction. Geometric features, color features, and texture features are identified. Geometric features include the maximum diameter, height, and surface curvature. Color features include hue and transparency-related colors. Geometric features include structural texture and bump depth. Each feature is embedded into the rough three-dimensional outline of the object to obtain a fine three-dimensional outline of the object.
[0011] The fine 3D contour of the object is compared with the 3D contours of several objects in the database. The specific information corresponding to the object is matched and extracted. The specific information of the object includes the material, name and hardness coefficient of the object. The material, hardness coefficient, center coordinates, angle and fine 3D contour of the object are integrated into multimodal data of the object and sent to the path generation module.
[0012] The path generation module filters candidate contact points in the reachable area based on the object's fine 3D contour, obtains constraint parameters, and embeds them into a sequential quadratic programming algorithm to solve for the optimal contact point of the object. Then, it uses a differential geometric path planning algorithm to generate an initial grasping path curve by combining the constraint parameters and the object's multimodal data. The specific process is as follows:
[0013] The object's fine 3D contour is discretized to obtain sampling points. Several neighborhood points corresponding to each sampling point are obtained. The maximum principal curvature HY of the corresponding sampling point is calculated by fitting these neighborhood points. The calculation logic is as follows: ,in Let be the second partial derivative of the surface along the x-direction. Let be the second partial derivative of the surface along the y-direction. The mixed second-order partial derivative of the surface is used; sampling points with maximum principal curvature < preset principal curvature threshold are retained to obtain the candidate contact points of the object;
[0014] The resultant force equilibrium constraint, friction cone constraint, and pressure constraint are obtained separately and embedded into a sequential quadratic programming problem. The sequential quadratic programming problem randomly selects five points as initial contact points from each candidate contact point, with each of the five points corresponding to a finger. The nonlinear constraints are linearized to construct a quadratic programming subproblem. The construction logic of the quadratic programming subproblem is as follows: ,in This is the correction vector for the contact point. This is the correction vector for the contact force. This is the total correction amount. Let g be the Hessian matrix and g be the gradient vector. Solve the quadratic programming subproblem to obtain the total correction value. Embed the total correction value into the current contact point coordinates for correction to obtain the optimal contact points corresponding to the five points.
[0015] Calculate the average principal curvature in the neighborhood of the optimal contact point. The curvature weight bc is obtained through the average principal curvature, and its calculation logic is as follows: In high-curvature regions, the curvature weight is small, and the path is closer to the surface. The material correction accuracy is obtained based on the curvature weight and the hardness coefficient. The calculation logic is as follows: ,in The material's hardness coefficient is used; the Riemann metric is obtained by embedding the material correction accuracy into the local tangent plane of the optimal contact point, and its calculation logic is as follows: ,in and Let the orthogonal unit basis vectors of the local tangent plane at the contact point be... The surface normal vector at the contact point. The outer product of a vector and its transpose is the Hamiltonian function, which uses the Riemannian metric to obtain the path energy. Its calculation logic is as follows: ,in This is a position vector, representing the three-dimensional coordinates along the path. As a dynamic vector, reflecting the finger movement trend along the path, a geodesic equation based on differential geometry is established based on the path energy. The geodesic equation includes position-time equations and momentum-time equations, and their construction logic is as follows: t is a time variable. Solve the geodesic equation to obtain discrete path points. Repeat the above steps to obtain discrete path points corresponding to the five fingers. Fit the discrete path points into a continuous curve and mark it as the initial grasping path curve.
[0016] The control module adjusts the finger joints of the five-finger dexterity hand using a finger motion model and the Newton-Raphson numerical method, outputting an optimal joint angle sequence. Based on the initial grasping path curve and the optimal joint angle sequence, it generates grasping commands and adjusts the initial grasping torque using reinforcement learning and a force-position hybrid control algorithm to obtain the optimal control torque. The specific process is as follows:
[0017] The system controls the random movement of several joints in a dexterous five-fingered hand, obtains the joint angle vectors and corresponding fingertip positions during random movements, marks them as sample data, sends the sample data to the finger motion model, trains the model, and completes the model training by merging through activation and loss functions. It should be noted that after training, the model can directly output approximate joint angles when any fingertip position is input.
[0018] Read the target coordinates of each fingertip in the initial grasping path curve. It should be noted that the target coordinates are the target coordinates that the fingertip of the dexterous hand needs to reach precisely at a certain point in time in order to complete operations such as grasping and carrying. Input the target coordinates of each fingertip into the finger motion model to obtain the initial joint angle, obtain the real-time fingertip coordinates corresponding to the initial joint angle, and calculate the difference between the target coordinates and the real-time fingertip coordinates to obtain the position error er.
[0019] The Jacobian matrix is calculated using numerical differentiation. The Jacobian matrix describes the effect of small changes in joint angle on fingertip position. The calculation logic is as follows: ,in This refers to the angle of minute joint perturbation. For joint angle The joint angle is updated based on the measured value corresponding to the fingertip position and the Jacobian matrix and position error. Its update logic is as follows: ,in Let be the joint angle vector for the m-th iteration. If the position error norm is less than the preset error threshold, the iteration stops. Output the optimal joint angle for each position. Integrate the optimal joint angles to obtain the optimal joint angle sequence for all fingers. Embed the optimal joint angle sequence into the initial grasping path curve to obtain the overall finger motion curve. Generate grasping instructions based on the overall finger motion curve and send them to the main controller to grasp the object.
[0020] The measured joint angles, position errors, and measured contact forces are integrated to construct a torque state space. Preset torque correction coefficients are extracted from the database. The torque weights and torque correction coefficients output by reinforcement learning are integrated into a torque action space. The joint position tracking reward *ru* and the force tracking reward *rs* are obtained and added together to obtain the total torque reward function. This reward function quantifies the quality of the current action, guiding the overall controller to learn the optimal strategy. The joint position tracking reward primarily encourages the reduction of position errors, and its calculation logic is as follows: ,in This refers to the joint position reward coefficient. Let L2 be the joint angle error vector. The force tracking reward is to minimize the torque error, and its calculation logic is as follows: ,in To track rewards, Let L2 be the norm of the force error vector; input the real-time acquired force-torque state space into the Actor network and output the action. ,in Adaptive weights for force-potential mixing This is the torque correction factor;
[0021] The joint angles are collected in real time and subtracted from the optimal joint angles to obtain the joint position error vector. The current joint angular velocity is collected in real time and subtracted from the preset joint angular velocity to obtain the joint angular velocity error vector. The PD controller outputs the position loop torque based on the joint position error vector and the joint angular velocity error vector. Its calculation logic is as follows: ,in This is a proportionality coefficient matrix. This is the differential coefficient matrix; the current torque value is collected in real time and the difference between it and the preset torque value is calculated to obtain the force error vector. The position correction equation is constructed based on the force error vector, and its construction logic is as follows: Where MR is the inertia coefficient, BS is the damping coefficient, and K is the stiffness coefficient. The fingertip position correction is obtained by solving the position correction equation through numerical integration. The fingertip position correction is then converted into a joint angle correction using a Jacobian matrix. Its calculation logic is as follows: The force loop torque is obtained by multiplying the joint angle correction amount and the preset force loop gain coefficient; the force loop torque and the position loop torque are then integrated to obtain the mixed torque, the calculation logic of which is as follows: The optimal control torque is obtained by truncation of the mixed torque at the boundary. The calculation logic is as follows: , and These are the minimum safe torques of the joint and the minimum safe torque of the joint. This is a numerical constraint function.
[0022] The response optimization module initializes the gray wolf population using chaos, calculates initial fitness, updates the positions of different wolf packs, perturbs the positions, and iteratively outputs time compensation for each step to optimize the response time. The specific process is as follows:
[0023] An initial population is generated using a Logistic chaotic sequence. Each wolf in the initial population corresponds to a set of time compensation parameters. The time compensation parameters of each gray wolf are then substituted into the fitness function to obtain the time-compensated fitness. Its calculation logic is as follows: ,in This is a hysteresis error. To compensate for the time, and The preset fixed weights are 0.38 and 0.62, respectively. is the total number of lag elements, such as joint angle calculation delay or controller torque calculation delay, and i is the number of the lag element; sorted in ascending order of time compensation fitness, the top 3 wolves are taken as leader wolves, and the rest are non-leader wolves. Leader wolves include alpha wolf, beta wolf and delta wolf. Alpha wolf has the lowest fitness and is the global optimal solution, beta wolf has the second lowest fitness and is the local optimal solution, and delta wolf has the third lowest fitness and is the auxiliary search.
[0024] Calculate the distance between each non-leader wolf and the leader wolf to obtain the alpha wolf distance. beta wolf distance Distance from delta wolf Its calculation logic is as follows: , and ,in For the current non-leader wolf position, , and A vector of random coefficients. , and To correspond to the position of the leader wolf, the position is updated based on the distance to each wolf to obtain a new position. If the fitness value of the alpha wolf does not improve in several consecutive iterations, a perturbation is triggered, and the position of the alpha wolf is adjusted to obtain the adjusted position. The logic behind its adjustment is as follows: ,in and Upper and lower bound constraints for time compensation. The perturbation intensity coefficient is used. If the fitness of the adjusted alpha wolf is better than that of the original alpha wolf, the alpha wolf position is updated; otherwise, the original position is retained. When the fitness value of the alpha wolf is less than the preset fitness threshold, the iteration stops and the final alpha wolf position is output, which is marked as the optimal time compensation for each stage.
[0025] Compared with the prior art, the beneficial effects of the present invention are:
[0026] The path generation module calculates the principal curvature of sampling points through discrete contours to screen low-stress candidate contact points. It then embeds a sequence of quadratic programming based on resultant force balance, friction cone, and pressure constraints to solve for the optimal contact point. Finally, it combines differential geometry to construct Riemannian metrics and geodesic equations to generate a path that fits the object's surface, ensuring contact point safety and path smoothness, avoiding grabbing collisions and local overloads, and improving the precision of path planning.
[0027] The control module is based on the trained finger movement model and combines the Newton-Raphson method to accurately output the optimal joint angle sequence. At the same time, it dynamically adjusts the torque weight and correction coefficient through reinforcement learning and force-position hybrid control to balance the positional accuracy and force compliance at different grasping stages.
[0028] The response optimization module adopts the chaotic gray wolf optimization algorithm, which initializes the population in a chaotic manner, iteratively updates the wolf pack position, and perturbs to optimize the optimal solution. It outputs time compensation for each stage, solves the problem of asynchronous actions caused by time lag in multiple stages, and shortens the overall response time. Attached Figure Description
[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The following drawings are not drawn to scale according to the actual size, but are intended to illustrate the main idea of the present invention.
[0030] Figure 1 This is a schematic diagram of the principle of the present invention. Detailed Implementation
[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are also within the scope of protection of the present invention.
[0032] Please refer to Figure 1 As shown, the present invention provides a multi-sensory intelligent humanoid five-finger dexterous hand, including a five-finger dexterous hand, a multimodal perception module, a path generation module, a control module and a response optimization module;
[0033] The multimodal perception module collects object features and locates them through sensors, generates a detailed 3D contour of the object, and matches it with a preset contour in the database to obtain multimodal data of the object, specifically:
[0034] Once the fingers are returned to their initial position, the RGB-D camera in the palm of the hand is activated to scan the operating area, locate the center coordinates and angle of the object, acquire the object's two-dimensional image and depth information, and then generate a rough three-dimensional outline of the object through point cloud reconstruction. Geometric features, color features, and texture features are identified. Geometric features include the maximum diameter, height, and surface curvature. Color features include hue and transparency-related colors. Geometric features include structural texture and bump depth. Each feature is embedded into the rough three-dimensional outline of the object to obtain a fine three-dimensional outline of the object.
[0035] The fine 3D contour of the object is compared with the 3D contours of several objects in the database. The specific information corresponding to the object is matched and extracted. The specific information of the object includes the material, name and hardness coefficient of the object. The material, hardness coefficient, center coordinates, angle and fine 3D contour of the object are integrated into multimodal data of the object and sent to the path generation module.
[0036] The path generation module filters candidate contact points in the reachable area based on the object's fine 3D contour, obtains constraint parameters, and embeds them into a sequential quadratic programming algorithm to solve for the optimal contact point of the object. Then, it uses a differential geometric path planning algorithm to generate an initial grasping path curve by combining the constraint parameters and the object's multimodal data. The specific process is as follows:
[0037] The object's fine 3D contour is discretized to obtain sampling points. Several neighborhood points corresponding to each sampling point are obtained. The maximum principal curvature HY of the corresponding sampling point is calculated by fitting these neighborhood points. The calculation logic is as follows: ,in Let be the second partial derivative of the surface along the x-direction. Let be the second partial derivative of the surface along the y-direction. The mixed second-order partial derivative of the surface is used; sampling points with maximum principal curvature < preset principal curvature threshold are retained to obtain the candidate contact points of the object;
[0038] The resultant force equilibrium constraint, friction cone constraint, and pressure constraint are obtained separately and embedded into a sequential quadratic programming problem. The sequential quadratic programming problem randomly selects five points as initial contact points from each candidate contact point, with each of the five points corresponding to a finger. The nonlinear constraints are linearized to construct a quadratic programming subproblem. The construction logic of the quadratic programming subproblem is as follows: ,in This is the correction vector for the contact point. This is the correction vector for the contact force. This is the total correction amount. Let g be the Hessian matrix and g be the gradient vector. Solve the quadratic programming subproblem to obtain the total correction value. Embed the total correction value into the current contact point coordinates for correction to obtain the optimal contact points corresponding to the five points.
[0039] It should be noted that the resultant force balance constraint is that the sum of the five-finger contact force vectors equals the target total force, ensuring that the object is in overall force balance; the friction cone constraint is that the tangential force does not exceed the maximum static friction force; and the pressure constraint is that the contact pressure does not exceed the material's tolerance limit.
[0040] Calculate the average principal curvature in the neighborhood of the optimal contact point. The curvature weight bc is obtained through the average principal curvature, and its calculation logic is as follows: In high-curvature regions, the curvature weight is small, and the path is closer to the surface. The material correction accuracy is obtained based on the curvature weight and the hardness coefficient. The calculation logic is as follows: ,in The material's hardness coefficient is used; the Riemann metric is obtained by embedding the material correction accuracy into the local tangent plane of the optimal contact point, and its calculation logic is as follows: ,in and Let the orthogonal unit basis vectors of the local tangent plane at the contact point be... The surface normal vector at the contact point. The outer product of a vector and its transpose is the Hamiltonian function, which uses the Riemannian metric to obtain the path energy. Its calculation logic is as follows: ,in This is a position vector, representing the three-dimensional coordinates along the path. As a dynamic vector, reflecting the finger movement trend along the path, a geodesic equation based on differential geometry is established based on the path energy. The geodesic equation includes position-time equations and momentum-time equations, and their construction logic is as follows: t is a time variable. Solve the geodesic equation to obtain discrete path points. Repeat the above steps to obtain discrete path points corresponding to the five fingers. Fit the discrete path points into a continuous curve and mark it as the initial grasping path curve.
[0041] The control module adjusts the finger joints of the five-finger dexterity hand using a finger motion model and the Newton-Raphson numerical method, outputting an optimal joint angle sequence. Based on the initial grasping path curve and the optimal joint angle sequence, it generates grasping commands and adjusts the initial grasping torque using reinforcement learning and a force-position hybrid control algorithm to obtain the optimal control torque. The specific process is as follows:
[0042] The system controls the random movement of several joints in a dexterous five-fingered hand, obtains the joint angle vectors and corresponding fingertip positions during random movements, marks them as sample data, sends the sample data to the finger motion model, trains the model, and completes the model training by merging through activation and loss functions. It should be noted that after training, the model can directly output approximate joint angles when any fingertip position is input.
[0043] Read the target coordinates of each fingertip in the initial grasping path curve. It should be noted that the target coordinates are the target coordinates that the fingertip of the dexterous hand needs to reach precisely at a certain point in time in order to complete operations such as grasping and carrying. Input the target coordinates of each fingertip into the finger motion model to obtain the initial joint angle, obtain the real-time fingertip coordinates corresponding to the initial joint angle, and calculate the difference between the target coordinates and the real-time fingertip coordinates to obtain the position error er.
[0044] The Jacobian matrix is calculated using numerical differentiation. The Jacobian matrix describes the effect of small changes in joint angle on fingertip position. The calculation logic is as follows: ,in This refers to the angle of minute joint perturbation. For joint angle The joint angle is updated based on the measured value corresponding to the fingertip position and the Jacobian matrix and position error. Its update logic is as follows: ,in Let be the joint angle vector for the m-th iteration. If the position error norm is less than the preset error threshold, the iteration stops. Output the optimal joint angle for each position. Integrate the optimal joint angles to obtain the optimal joint angle sequence for all fingers. Embed the optimal joint angle sequence into the initial grasping path curve to obtain the overall finger motion curve. Generate grasping instructions based on the overall finger motion curve and send them to the main controller to grasp the object.
[0045] The measured joint angles, position errors, and measured contact forces are integrated to construct a torque state space. Preset torque correction coefficients are extracted from the database. The torque weights and torque correction coefficients output by reinforcement learning are integrated into a torque action space. The joint position tracking reward *ru* and the force tracking reward *rs* are obtained and added together to obtain the total torque reward function. This reward function quantifies the quality of the current action, guiding the overall controller to learn the optimal strategy. It should be noted that the joint position tracking reward primarily encourages the reduction of position errors; its calculation logic is as follows: ,in This refers to the joint position reward coefficient. Let L2 be the joint angle error vector. The force tracking reward is to minimize the torque error, and its calculation logic is as follows: ,in To track rewards, Let L2 be the norm of the force error vector; input the real-time acquired force-torque state space into the Actor network and output the action. ,in Adaptive weights for force-potential mixing This is the torque correction factor;
[0046] The joint angles are collected in real time and subtracted from the optimal joint angles to obtain the joint position error vector. The current joint angular velocity is collected in real time and subtracted from the preset joint angular velocity to obtain the joint angular velocity error vector. The PD controller outputs the position loop torque based on the joint position error vector and the joint angular velocity error vector. Its calculation logic is as follows: ,in This is a proportionality coefficient matrix. This is the differential coefficient matrix; the current torque value is collected in real time and the difference between it and the preset torque value is calculated to obtain the force error vector. The position correction equation is constructed based on the force error vector, and its construction logic is as follows: Where MR is the inertia coefficient, BS is the damping coefficient, and K is the stiffness coefficient. The fingertip position correction is obtained by solving the position correction equation through numerical integration. The fingertip position correction is then converted into a joint angle correction using a Jacobian matrix. Its calculation logic is as follows: The force loop torque is obtained by multiplying the joint angle correction amount and the preset force loop gain coefficient; the force loop torque and the position loop torque are then integrated to obtain the mixed torque, the calculation logic of which is as follows: The optimal control torque is obtained by truncation of the mixed torque at the boundary. The calculation logic is as follows: , and These are the minimum safe torques of the joint and the minimum safe torque of the joint. This is a numerical constraint function.
[0047] The response optimization module initializes the gray wolf population using chaos, calculates initial fitness, updates the positions of different wolf packs, perturbs the positions, and iteratively outputs time compensation for each step to optimize the response time. The specific process is as follows:
[0048] It should be noted that the core function of the Chaotic Gray Wolf Optimization Algorithm is to accurately optimize the compensation parameters for time lag in multiple stages, and solve the problems of slow response and asynchronous actions of various modules from the perspective of time coordination, so as to enable the response speed and synchronization of the dexterous hand to meet the requirements of precision operation.
[0049] An initial population is generated using a Logistic chaotic sequence. Each wolf in the initial population corresponds to a set of time compensation parameters. The time compensation parameters of each gray wolf are then substituted into the fitness function to obtain the time-compensated fitness. Its calculation logic is as follows: ,in This is a hysteresis error. To compensate for the time, and The preset fixed weights are 0.38 and 0.62, respectively. is the total number of lag elements, such as joint angle calculation delay or controller torque calculation delay, and i is the number of the lag element; sorted in ascending order of time compensation fitness, the top 3 wolves are taken as leader wolves, and the rest are non-leader wolves. Leader wolves include alpha wolf, beta wolf and delta wolf. Alpha wolf has the lowest fitness and is the global optimal solution, beta wolf has the second lowest fitness and is the local optimal solution, and delta wolf has the third lowest fitness and is the auxiliary search.
[0050] Calculate the distance between each non-leader wolf and the leader wolf to obtain the alpha wolf distance. beta wolf distance Distance from delta wolf Its calculation logic is as follows: , and ,in For the current non-leader wolf position, , and A vector of random coefficients. , and To correspond to the position of the leader wolf, the position is updated based on the distance to each wolf to obtain a new position. If the fitness value of the alpha wolf does not improve in several consecutive iterations, a perturbation is triggered, and the position of the alpha wolf is adjusted to obtain the adjusted position. The logic behind its adjustment is as follows: ,in and Upper and lower bound constraints for time compensation. The perturbation intensity coefficient is used. If the fitness of the adjusted alpha wolf is better than that of the original alpha wolf, the alpha wolf position is updated; otherwise, the original position is retained. When the fitness value of the alpha wolf is less than the preset fitness threshold, the iteration stops and the final alpha wolf position is output, which is marked as the optimal time compensation for each stage.
[0051] The above formulas are all dimensionless calculations. Dimensionlessness can be achieved through standardization, normalization, or other methods (such as mapping the time compensation parameter to the [0,20] ms range and normalizing the joint angle error to the [0,1] interval), which will not be elaborated here. The formulas are obtained by collecting a large amount of dexterity hand grasping experimental data (covering grasping scenarios of different materials such as ceramics and plastics, and objects of different sizes such as 0402 capacitors and micro glass beads), and fitting the data through software simulation (such as PyBullet simulation). This results in formulas that closely reflect the actual physical laws of dexterity hand operation. The preset parameters in the formulas (such as the principal curvature threshold, the PD controller proportional coefficient, the chaotic disturbance intensity coefficient, etc.) are set by those skilled in the art in combination with the hardware parameters of the dexterity hand (such as the mechanical limits of the degree of freedom joints and the accuracy of the force sensor) and the specific application scenario (such as industrial precision assembly or medical rehabilitation).
[0052] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions or computer programs. When these computer instructions or computer programs are loaded or executed on a computer, they generate, in whole or in part, the processes or functions described in the embodiments of the present invention (such as driving the dexterous hand controller to perform grasping actions, or the response optimization module outputting time compensation parameters). The computer can be a general-purpose computer or a special-purpose computer (such as one equipped with an ARM processor). The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., CAN bus, SPI bus) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more sets of available media. The available media can be magnetic media (e.g., floppy disks, portable hard drives (ATA interface), magnetic tape), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).
[0053] It should be understood that in the various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and the inherent logic of the dexterous hand grasping (e.g., first obtain object data through the multimodal perception module, then plan the path through the path generation module, and finally execute the grasping through the control module and the response optimization module), and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0054] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution (e.g., torque calculation for a PD controller is preferably implemented in hardware to reduce latency, while Actor network inference for reinforcement learning can be deployed in software on the main controller); those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.
[0055] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods (such as integrating the path generation module with the joint angle calculation function of the control module to reduce data interaction delay), or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection between the devices or units shown or discussed may be through some interfaces, and the indirect coupling or communication connection between the devices or units may be electrical, mechanical, or other forms (such as the joint encoder and the control module communicating indirectly through an SPI interface).
[0056] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units (e.g., the log storage function of the health self-check module may be distributed across a local USB and a remote server). Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0057] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit (for example, the joint angle calculation and torque optimization functions in the control module can be integrated into the same FPGA chip to improve calculation efficiency).
[0058] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a dexterous hand controller, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives (ATA interface), read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. The software product may additionally include a health self-check log storage function (e.g., stored in CSV format, supporting USB export).
[0059] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims, especially covering the collaborative implementation scheme of the four major modules of multimodal perception, path generation, control, and response optimization.
Claims
1. A multi-sensory intelligent humanoid five-fingered dexterous hand, comprising a multimodal perception module, a path generation module, a control module, and a response optimization module, characterized in that: The multimodal perception module collects object features and localizes them through sensors, generates a fine three-dimensional outline of the object, and matches it with a preset outline in the database to obtain multimodal data of the object. The path generation module filters candidate contact points in the reachable area based on the object's fine 3D contour, obtains constraint parameters, and embeds them into a sequence quadratic programming algorithm to solve for the optimal contact point of the object. The constraint parameters and the object's multimodal data are then used to generate an initial grasping path curve through a differential geometry path planning algorithm. The control module adjusts the finger joints of a five-finger dexterity hand using a finger motion model and the Newton-Raphson numerical method, outputting an optimal joint angle sequence. The specific process is as follows: The system controls the random movement of several joints of a five-fingered dexterous hand, obtains the joint angle vectors and corresponding fingertip positions during random movements, marks them as sample data, sends the sample data to the finger motion model, trains the model, and completes the model training by convergence through activation functions and loss functions. Read the target coordinates of each fingertip in the initial grasping path curve, input the target coordinates of each fingertip into the finger motion model to obtain the initial joint angle, obtain the real-time fingertip coordinates corresponding to the initial joint angle, and calculate the position error by subtracting the target coordinates from the real-time fingertip coordinates. The Jacobian matrix is calculated by numerical differentiation. The joint angles are updated based on the Jacobian matrix and the position error. The iteration stops when the position error norm is less than the preset error threshold. The optimal joint angles at each position are output. The optimal joint angles are integrated to obtain the optimal joint angle sequence of all fingers. The optimal joint angle sequence is embedded into the initial grasping path curve to obtain the overall finger motion curve. The grasping command is generated based on the overall finger motion curve and sent to the main controller to grasp the object. Grasping instructions are generated based on the initial grasping path curve and the optimal joint angle sequence. The initial grasping torque is adjusted through reinforcement learning and force-position hybrid control algorithm to obtain the optimal control torque. Specifically, the measured joint angle, position error and measured contact force are integrated to construct the torque state space. The preset torque correction coefficients in the database are extracted. The torque weights and torque correction coefficients output by reinforcement learning are integrated into the torque action space. The joint position tracking reward and force tracking reward are obtained and added to obtain the total torque reward function. The real-time collected torque state space is input into the Actor network and the action is output. The joint angle is collected in real time and subtracted from the optimal joint angle to obtain the joint position error vector. The current joint angular velocity is collected in real time and subtracted from the preset joint angular velocity to obtain the joint angular velocity error vector. The PD controller outputs the position loop torque based on the joint position error vector and the joint angular velocity error vector. The current torque value is collected in real time and subtracted from the preset torque value to obtain the force error vector. A position correction equation is constructed based on the force error vector. The position correction equation is solved by numerical integration to obtain the fingertip position correction amount. The fingertip position correction amount is converted into a joint angle correction amount through the Jacobian matrix. The joint angle correction amount is multiplied by the preset force loop gain coefficient to obtain the force loop torque. The force loop torque and the position loop torque are integrated to obtain the mixed torque, and the optimal control torque is obtained by truncation of the mixed torque at the boundary. The response optimization module initializes the gray wolf population with chaos, calculates the initial fitness and updates the positions of different wolf packs, perturbs the positions and iteratively outputs the time compensation for each step to optimize the response time.
2. The multi-sensory intelligent anthropomorphic five-fingered dexterous hand of claim 1, wherein, The path generation module filters candidate contact points in the accessible area based on the object's fine 3D contour, obtains constraint parameters, and embeds them into a sequence of quadratic programming to solve for the optimal contact point of the object. The specific process is as follows: The fine three-dimensional contour of the object is discretized to obtain each sampling point. Several neighborhood points corresponding to each sampling point are obtained. The maximum principal curvature of the corresponding sampling point is calculated by fitting the neighborhood points. Sampling points with a maximum principal curvature less than a preset principal curvature threshold are retained to obtain each candidate contact point of the object. The resultant force balance constraint, friction cone constraint, and pressure constraint are obtained respectively, and embedded into a sequential quadratic programming problem. The sequential quadratic programming problem randomly selects five points as initial contact points from each candidate contact point. The five points correspond to the five fingers respectively. The nonlinear constraints are linearized to construct a quadratic programming subproblem. Solving the quadratic programming subproblem yields the total correction amount. The total correction amount is embedded into the current contact point coordinates for correction, and the optimal contact points corresponding to the five points are obtained.
3. The multi-sensory intelligent anthropomorphic five-fingered dexterous hand of claim 2, wherein, The constraint parameters and multimodal data of the object are used to generate an initial grasping path curve through a differential geometric path planning algorithm. The specific process is as follows: The average principal curvature of the neighborhood of the optimal contact point is calculated, and the material correction accuracy is obtained based on the curvature weight and hardness coefficient. The material correction accuracy is embedded into the local tangent plane of the optimal contact point to obtain the Riemann metric. The Hamiltonian function obtains the path energy through the Riemann metric. Based on the path energy, the geodesic equation of differential geometry is established. The geodesic equation includes the position-time equation and the momentum-time equation. The discrete path points output by solving the geodesic equation are solved. The above steps are repeated to obtain the discrete path points corresponding to the five fingers. The discrete path points are fitted into a continuous curve and marked as the initial grasping path curve.
4. The multi-sensory intelligent anthropomorphic five-fingered dexterous hand of claim 1, wherein, The response optimization module initializes the gray wolf population using chaos, calculates initial fitness, updates the positions of different wolf packs, perturbs the positions, and iteratively outputs time compensation for each step to optimize the response time. The specific process is as follows: An initial population is generated using a Logistic chaotic sequence. Each wolf in the initial population corresponds to a set of time compensation parameters. The time compensation parameters of each gray wolf are substituted into the fitness function to obtain the time compensation fitness. The wolves are sorted in ascending order of time compensation fitness, and the top 3 wolves are taken as leader wolves, and the rest are non-leader wolves. Leader wolves include alpha wolves, beta wolves and delta wolves. Calculate the distance between each non-leader wolf and the leader wolf to obtain the alpha wolf distance, beta wolf distance, and delta wolf distance. Update the position based on each wolf distance to obtain the new position. If the fitness value of the alpha wolf does not improve in several consecutive iterations, a perturbation is triggered to adjust the position of the alpha wolf to obtain the adjusted position. If the fitness of the alpha wolf at the adjusted position is better than that of the original alpha wolf, then the alpha wolf position is updated; otherwise, the original position is retained. When the fitness value of the alpha wolf is less than the preset fitness threshold, the iteration stops and the final position of the alpha wolf is output, which is marked as the optimal time compensation for each stage.
5. A multi-sensory intelligent humanoid five-fingered dexterous hand according to claim 1, characterized in that, The multimodal perception module acquires object features and localizes them through sensors, generating a detailed 3D contour of the object. The specific steps are as follows: Once the fingers return to their initial position, the RGB-D camera in the palm of the hand is activated to scan the operating area, locate the center coordinates and angle of the object, acquire the object's two-dimensional image and depth information, and then generate a rough three-dimensional outline of the object through point cloud reconstruction. Geometric features, color features, and texture features are identified. Geometric features include the maximum diameter, height, and surface curvature; color features include hue and transparency-related colors; and texture features include structural texture and bump depth. Each feature is embedded into the rough three-dimensional outline of the object to obtain a fine three-dimensional outline of the object.
6. A multi-sensory intelligent humanoid five-fingered dexterous hand according to claim 5, characterized in that, The detailed 3D contour of the object is matched with the preset contour in the database to obtain the object's multimodal data. The specific steps are as follows: The fine 3D contour of the object is compared with the 3D contours of several objects in the database. The specific information corresponding to the object is matched and extracted. The specific information of the object includes the material, name and hardness coefficient of the object. The material, hardness coefficient, center coordinates, angle and fine 3D contour of the object are integrated into multimodal data of the object and sent to the path generation module.