Dexterous hand multi-object stable grabbing method and system based on vision and touch perception

Abstract

The invention provides a dexterous hand multi-object stable grabbing method and system based on vision and touch perception. The dexterous hand multi-object stable grabbing method comprises the steps that RGB images and depth maps of target objects are collected through a monocular RGB-D camera, and dense point cloud data are generated; geometric features are extracted by using a three-dimensional point cloud neural network, and composite object distribution features are analyzed in combination with visual information; an initial grasping strategy is generated based on the features, actions are executed, and the joint state and contact force data are synchronously obtained in real time through body sensing and a fingertip force sensor; in the carrying process, multi-dimensional force perception and the joint state are fused to analyze object stress distribution and center-of-gravity shift; vision and force sensing data are used as input. Vision, body sensing and force sensing data are fused, the problems of three-dimensional structure recognition and dynamic stress regulation and control in multi-object composite grabbing are solved, compared with a single sensing scheme, the grabbing planning precision and the self-adaptive capacity in a complex scene are remarkably improved, and the method is particularly suitable for stable carrying of combinations of loose objects such as dinner plates carrying tableware.

Description

Technical Field

[0001] The present invention relates to the technical field of robot grasping and control. Specifically, it relates to a method and system for stable multi-object grasping of a dexterous hand based on visual and tactile perception. Background Art

[0002] With the rapid development of robot technology, the application demand for dexterous hands in industrial, service, and household scenarios is increasing day by day. However, most of the existing research on robot grasping technology focuses on the stable grasping of a single object, such as target positioning based on a depth camera and grasping control of a two-finger gripper. For example, the "Robotic Arm Grasping Method" disclosed in Chinese Patent CN201810458201.4 can use a depth camera to achieve the grasping of a single object, but its dependence on the structure of two parallel fingers restricts its applicability in complex scenarios. Especially when multiple loose objects are stacked on the same carrier (such as bowls and chopsticks on a dinner plate), the existing technology is difficult to effectively identify the overall distribution characteristics of the composite target, and it is even more impossible to perceive the dynamic force changes in real time during the grasping process and make adaptive adjustments.

[0003] In practical applications, the grasping of composite targets needs to meet the following requirements simultaneously: accurately identify the spatial distribution and geometric features of multiple objects to plan a reasonable initial grasping posture; dynamically perceive the force changes during the handling process, such as the moment fluctuations caused by the center-of-gravity shift of the object or external disturbances; and adjust the grasping strategy in real time to ensure the stability of the overall target. However, the existing technology has significant deficiencies:

[0004] Limitations of visual perception: Traditional methods rely on single-depth information or two-dimensional images, making it difficult to extract the global three-dimensional structural features of composite targets, resulting in inaccurate initial grasping planning;

[0005] Rigid control strategies: Most algorithms adopt fixed thresholds or offline optimization strategies, lacking the online response ability to dynamic environments (such as object sliding and tipping);

[0006] Insufficient force perception and fusion: Existing systems mostly neglect the cooperation between tactile information and proprioception, resulting in the inability to effectively suppress the relative displacement of multiple objects during the handling process.

[0007] Academia and industry have explored the above problems. For example, teams such as Liu Houde from the Shenzhen Graduate School of Tsinghua University have conducted in-depth research in the field of dexterous hand grasping, but their achievements still focus on single-object scenarios and do not solve the robustness problem of multi-object composite grasping. In addition, the solutions proposed by peer companies mostly rely on high-cost multi-camera vision systems or complex mechanical structures, making it difficult to achieve lightweight deployment.

[0008] Therefore, there is an urgent need for a method that integrates multi-modal perception and intelligent decision-making, which can achieve stable grasping and dynamic adjustment of multiple objects in complex scenarios. Summary of the Invention

[0009] Aiming at the defects in the prior art, the purpose of the present invention is to provide a stable multi-object grasping method and system for a dexterous hand based on vision and tactile perception.

[0010] A stable multi-object grasping method for a dexterous hand based on vision and tactile perception provided by the present invention, the method comprises the following steps:

[0011] Step S1: Collect the RGB image and depth map of the target object through a monocular RGB-D camera to generate dense point cloud data containing the three-dimensional structure information of the target object;

[0012] Step S2: Input the point cloud data into a three-dimensional point cloud neural network to extract the geometric structure features of the object, and combine the RGB image and depth information to estimate the distribution features of the composite objects in the scene;

[0013] Step S3: Generate an initial grasping strategy based on the extracted visual features, including the clamping point position, clamping direction and grasping mode;

[0014] Step S4: Control the dexterous hand to perform the initial grasping action, and obtain the joint state, gripper posture and contact force information in real time through the body perception module and the fingertip force sensor;

[0015] Step S5: During the grasping and handling process, analyze the force distribution and center of gravity offset of the composite object based on the contact force direction and joint state sensed by the multi-dimensional force sensor;

[0016] Step S6: Taking the vision and force sensor data as input, adopt a reinforcement learning algorithm to dynamically adjust the clamping force and posture parameters of the dexterous hand to optimize the grasping stability;

[0017] Step S7: Continuously perform tactile perception and policy fine-tuning during the handling process to ensure the overall posture stability of the composite object.

[0018] Preferably, the three-dimensional point cloud neural network is PointNet or PointNet++, which is used to extract the geometric features of the point cloud data and fuse the texture information of the RGB image.

[0019] Preferably, the reinforcement learning algorithm is a policy gradient algorithm, including PPO or DDPG, and its reward function is constructed based on clamping stability, no slippage of the object and energy consumption factors.

[0020] Preferably, the dynamic adjustment in step S6 includes:

[0021] Step S6.1: Define the state space as the joint angles of the dexterous hand, the clamping force and the feedback of the tactile sensor;

[0022] Step S6.2: Realize real-time fine-tuning of the clamping force and attitude parameters through online policy iteration optimization.

[0023] Preferably, the composite object is a plurality of loose objects placed on a carrier, including a dinner plate and tableware thereon, and the dynamic adjustment ensures that each object does not tip over or move relatively during the handling process.

[0024] The present invention also provides a dexterous hand multi-object stable grasping system based on vision and tactile perception, and the system includes the following modules:

[0025] Module M1: Collect the RGB image and depth map of the target object through a monocular RGB-D camera to generate dense point cloud data containing the three-dimensional structure information of the target object;

[0026] Module M2: Input the point cloud data into a three-dimensional point cloud neural network to extract the geometric structure features of the object, and combine the RGB image and depth information to estimate the distribution features of the composite object in the scene;

[0027] Module M3: Generate an initial grasping strategy based on the extracted visual features, including the clamping point position, clamping direction, and grasping method;

[0028] Module M4: Control the dexterous hand to perform the initial grasping action, and obtain the joint state, gripper attitude, and contact force information in real time through the body perception module and fingertip force sensors;

[0029] Module M5: During the grasping and handling process, analyze the force distribution and center of gravity offset of the composite object based on the contact force direction and joint state sensed by the multi-dimensional force sensor;

[0030] Module M6: Take the vision and force sensor data as input, and use a reinforcement learning algorithm to dynamically adjust the clamping force and attitude parameters of the dexterous hand to optimize the grasping stability;

[0031] Module M7: Continuously perform tactile perception and policy fine-tuning during the handling process to ensure the overall attitude stability of the composite object.

[0032] Preferably, the three-dimensional point cloud neural network is PointNet or PointNet++, which is used to extract the geometric features of the point cloud data and fuse the texture information of the RGB image.

[0033] Preferably, the reinforcement learning algorithm is a policy gradient algorithm, including PPO or DDPG, and its reward function is constructed based on factors such as clamping stability, no slippage of the object, and energy consumption.

[0034] Preferably, the dynamic adjustment in Module M6 includes:

[0035] Module M6.1: Define the state space as the dexterous hand joint angle, clamping force, and tactile sensor feedback;

[0036] Module M6.2: Realize real-time fine-tuning of the clamping force and attitude parameters through online policy iteration optimization.

[0037] Preferably, the composite object is a plurality of loose objects placed on a carrier, including a dinner plate and tableware thereon, and the dynamic adjustment ensures that no object topples or moves relatively during the handling process.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. The present invention realizes high-precision perception of the three-dimensional structural features and dynamic force distribution of the multi-object composite grasping target by integrating monocular RGB-D visual information, dexterous hand proprioception, and fingertip force sensor data; compared with traditional single-vision or force control schemes, this method can capture the spatial layout and real-time force changes of the target object more comprehensively, significantly improving the grasping planning accuracy and environmental adaptability in complex scenarios;

[0040] 2. The dynamic adjustment mechanism of the present invention based on the reinforcement learning algorithm, combined with the design of the composite reward function of "maximizing stability", enables the system to perceive dynamic disturbances such as the center-of-gravity offset and contact force change of the object in real time during the handling process, and quickly optimize the clamping force and attitude parameters; this feature effectively suppresses the relative sliding and overturning problems between multi-objects, greatly improving the robustness and anti-interference ability of the grasping process;

[0041] 3. Aiming at the limitation of the prior art that only supports single-object grasping, the present invention generates an initial grasping strategy through a neural network and combines an online fine-tuning mechanism to achieve overall stable grasping of multi-object targets in loose stacks or non-fixed arrangements; it is particularly suitable for practical scenarios such as food service and logistics sorting that require handling multiple objects simultaneously, expanding the application range of the dexterous hand. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] By reading the following detailed description of the non-limiting embodiments with reference to the accompanying drawings, other features, objects, and advantages of the present invention will become more apparent:

[0043] Figure 1 The flow schematic diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those of ordinary skill in the art can make several changes and improvements without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0045] Example 1:

[0046] Referring to Figure 1 , a multi-object stable grasping method for a dexterous hand based on visual and tactile perception provided by the present invention, the method includes the following steps:

[0047] Step S1: Collect the RGB image and depth map of the target object through a monocular RGB-D camera to generate dense point cloud data containing the three-dimensional structure information of the target object;

[0048] Step S2: Input the point cloud data into a three-dimensional point cloud neural network to extract the geometric structure features of the object, and estimate the distribution features of the composite objects in the scene in combination with the RGB image and depth information; the three-dimensional point cloud neural network is PointNet or PointNet++, which is used to extract the geometric features of the point cloud data and fuse the texture information of the RGB image.

[0049] Step S3: Generate an initial grasping strategy based on the extracted visual features, including the clamping point position, clamping direction and grasping method;

[0050] Step S4: Control the dexterous hand to execute the initial grasping action, and obtain the joint state, gripper posture and contact force information in real time through the body perception module and the fingertip force sensor;

[0051] Step S5: During the grasping and handling process, analyze the force distribution and centroid offset of the composite object based on the contact force direction and joint state sensed by the multi-dimensional force sensor; during the grasping process, based on the body sense of the dexterous hand and the tactile sensors at its fingertips, the force and action point of each fingertip on the object can be obtained so as to obtain the resultant force and the resultant moment

[0052] so as to calculate and obtain the moment centroid

[0053] For the estimation of the object centroid, there are two modes to choose from:

[0054] If it is known that the materials of multiple objects are the same, for example, there are several wooden strips placed on a wooden board, the existing point cloud-based centroid estimation method can be directly used.

[0055] If it is known that the materials of multiple objects are different, the mass of each object needs to be known. Using the existing point cloud segmentation algorithm, calculate the centroid of the point cloud segmented from each object respectively, and then combine the mass of each object to obtain the overall centroid.

[0056] During the grasping and moving processes, if the spatial positions of the moment center of the moment and the center of mass of the object are close, the risk of tipping is considered small. Therefore, a penalty term can be added to the reward function. When the spatial distance between the moment center of the moment and the center of gravity of the object is greater than a certain threshold, a penalty is imposed to prevent tipping.

[0057] Step S6: Using the vision and force sensor data as inputs, a reinforcement learning algorithm is used to dynamically adjust the clamping force and posture parameters of the dexterous hand to optimize the grasping stability; the reinforcement learning algorithm is a policy gradient algorithm, including PPO or DDPG, and its reward function is constructed based on the clamping stability, no slippage of the object, and energy consumption factors; the dynamic adjustment includes:

[0058] Step S6.1: Define the state space as the joint angles of the dexterous hand, the clamping force, and the feedback of the tactile sensor; define the state space as the joint angles of the dexterous hand and the measured values of the tactile sensor; State space definition: Suppose the dexterous hand has n degrees of freedom, and its state space S t At time t is represented as: S t =[θ1, θ2, …, θ n , F 1x , F 1y , F 1z , …F jx , F jy , F jz . Where θ i is the i-th joint angle (unit: radian), and F k{x / y / z} is the three-dimensional contact force measured by the k-th fingertip force sensor (unit: Newton), where k = 1, 2, …j, and j is the number of tactile (force) sensors.

[0059] Step S6.2: Through online policy iteration optimization, real-time fine-tuning of the clamping force and posture parameters is achieved. Online policy iteration optimization: Use a policy gradient algorithm (such as PPO) to construct a policy network Π φ (a t |s t ), where φ is the policy network parameter, and a t is the action space, including the joint angle increment Δθ i and the clamping force adjustment amount ΔF. The optimization process is to minimize the loss function L(φ) as: Where r t (φ) is the policy update ratio. is the estimated value of the advantage function, and ∈ is the clipping threshold (default 0.2). The policy is updated every 50 ms to achieve millisecond-level real-time response.

[0060] Step S7: Continuously perform tactile perception and strategy fine-tuning during the handling process to ensure the overall posture stability of the composite object. The composite object is a plurality of loose objects placed on the carrier, including a dinner plate and tableware thereon, and the dynamic adjustment ensures that no object topples or moves relatively during the handling process.

[0061] The present invention also provides a dexterous hand multi-object stable grasping system based on vision and tactile perception. The dexterous hand multi-object stable grasping system based on vision and tactile perception can be implemented by executing the process steps of the dexterous hand multi-object stable grasping method based on vision and tactile perception. That is, those skilled in the art can understand the dexterous hand multi-object stable grasping method based on vision and tactile perception as a preferred implementation manner of the dexterous hand multi-object stable grasping system based on vision and tactile perception.

[0062] Example 2:

[0063] The present invention also provides a dexterous hand multi-object stable grasping system based on vision and tactile perception. The system includes the following modules:

[0064] Module M1: Collect the RGB image and depth map of the target object through a monocular RGB-D camera to generate dense point cloud data containing the three-dimensional structure information of the target object;

[0065] Module M2: Input the point cloud data into a three-dimensional point cloud neural network to extract the geometric structure features of the object, and estimate the distribution features of the composite object in the scene in combination with the RGB image and depth information; the three-dimensional point cloud neural network is PointNet or PointNet++, which is used to extract the geometric features of the point cloud data and fuse the texture information of the RGB image.

[0066] Module M3: Generate an initial grasping strategy based on the extracted visual features, including the position of the clamping point, the clamping direction, and the grasping method;

[0067] Module M4: Control the dexterous hand to execute the initial grasping action, and obtain the joint state, gripper posture, and contact force information in real time through the proprioception module and the fingertip force sensor;

[0068] Module M5: During the grasping and handling process, analyze the force distribution and center of gravity offset of the composite object based on the contact force direction and joint state sensed by the multi-dimensional force sensor;

[0069] Module M6: Take the vision and force sensor data as input, and use a reinforcement learning algorithm to dynamically adjust the clamping force and posture parameters of the dexterous hand to optimize the grasping stability; the reinforcement learning algorithm is a policy gradient algorithm, including PPO or DDPG, and its reward function is constructed based on the clamping stability, no slippage of the object, and energy consumption factors; the dynamic adjustment includes:

[0070] Module M6.1: Define the state space as the dexterous hand joint angles, grasping force, and tactile sensor feedback;

[0071] Module M6.2: Through online policy iteration optimization, achieve real-time fine-tuning of the grasping force and posture parameters.

[0072] Module M7: Continuously perform tactile perception and policy fine-tuning during the handling process to ensure the overall posture stability of the composite object; the composite object is multiple loose objects placed on the carrier, including a dinner plate and the tableware on it, and the dynamic adjustment ensures that no object overturns or moves relatively during the handling process.

[0073] Example 3:

[0074] The present invention proposes a stable grasping control method that integrates monocular RGB-D visual information, dexterous hand proprioception, and fingertip force sensors. It estimates the object's visual features through a neural network and generates an initial grasping posture, and dynamically adjusts the dexterous hand grasping posture and output torque based on reinforcement learning. This method is particularly suitable for grasping an overall target containing multiple loose objects (such as a dinner plate with bowls and chopsticks), and realizing stable handling and posture control, including the following steps:

[0075] Step 1: Scene perception and point cloud generation;

[0076] Use a monocular RGB-D camera to collect the RGB image and depth map of the grasping target (such as a dinner plate and the tableware on it), and convert them into dense point cloud data. This data contains the three-dimensional structure information of the target object and is the basis for subsequent grasping reasoning and policy generation.

[0077] Step 2: Feature extraction;

[0078] Input the point cloud data into a three-dimensional point cloud neural network (such as PointNet, PointNet++), and extract the geometric structure features of the object. Combine the RGB image and depth information to estimate the distribution features of the composite object (such as the tableware on the dinner plate) in the scene.

[0079] Step 3: Generation of the initial grasping strategy;

[0080] Based on the visual features, use the existing grasping strategy generation method to generate an initial grasping posture suitable for the current task, including: the position of the grasping point, the grasping direction, the grasping method (grasping, lifting, etc.).

[0081] Step 4: Execute the initial grasping action;

[0082] Control the robotic arm and the dexterous hand at its end to execute the grasping action according to the generated strategy, and at the same time activate the proprioception and fingertip force sensor module to obtain the joint state, gripper posture, and fingertip contact force information in real time for subsequent adjustment.

[0083] Step 5: Force perception and force analysis;

[0084] During the grasping and handling process, multi-dimensional force sensors arranged at the fingertips are used to sense the contact force and its direction received by each finger. Combining with the joint states of the dexterous hand body, the force distribution, moment change and center-of-gravity offset of the composite object are understood.

[0085] Step 6: Dynamic pose and force application optimization based on reinforcement learning;

[0086] The state information from the vision and force sensors is used as the input of the reinforcement learning model (such as policy gradient algorithms like PPO and DDPG). With the goal of "maximizing grasping stability", the finger clamping force and pose parameters are dynamically adjusted to optimize the current action strategy.

[0087] Step 6.1: State space definition: including joint angles, clamping force, etc.;

[0088] Step 6.2: Reward function design: A composite reward is constructed based on factors such as clamping stability (no tipping and no slipping of the object) and energy consumption;

[0089] Step 6.3: Policy iteration optimization: Continuously execute, sense, evaluate and update the policy to achieve the ability of online fine-tuning.

[0090] Step 7: Stable handling and dynamic adjustment;

[0091] During the handling process (such as moving a dinner plate from table A to table B), the system continuously performs tactile perception and policy fine-tuning to ensure the stability of the overall pose of the target object, and to avoid tilting, slipping or relative movement of the objects above.

[0092] By adopting the structure of "monocular RGB-D vision perception + point cloud feature extraction", the present invention solves the problem that existing dexterous hands are difficult to recognize the overall distribution characteristics when facing the composite grasping target of multiple stacked objects, achieving the effect of reasonably planning the grasping pose and clamping points before the initial grasping, and laying a foundation for subsequent stable handling; by adopting the structure of "force sensor perception + body perception + reinforcement learning control", the present invention solves the problem that it is difficult to adjust the grasping pose and force in real time due to the dynamic change of the force on the composite object during the handling process, achieving the stable grasping effect of continuously adapting and adjusting during the grasping process and effectively suppressing the sliding and tipping of the object.

[0093] Those skilled in the art can understand this embodiment as a more specific illustration of Embodiment 1 and Embodiment 2.

[0094] Those skilled in the art know that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in the form of pure computer-readable program code, the method steps can be logically programmed to enable the system and its various devices, modules, and units provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc., to achieve the same functions. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a kind of hardware component, and the devices, modules, and units included therein for implementing various functions can also be regarded as the structures within the hardware component; the devices, modules, and units for implementing various functions can also be regarded as either software modules for implementing the method or structures within the hardware component.

[0095] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A stable multi-object grasping method for a dexterous hand based on visual and tactile perception, characterized in that, The method includes the following steps: Step S1: Collect the RGB image and depth map of the target object through a monocular RGB-D camera to generate dense point cloud data containing the three-dimensional structure information of the target object; Step S2: Input the point cloud data into a three-dimensional point cloud neural network to extract the geometric structure features of the object, and estimate the distribution features of the composite objects in the scene by combining the RGB image and depth information; Step S3: Generate an initial grasping strategy based on the extracted visual features, including the position of the gripping point, the gripping direction, and the grasping method; Step S4: Control the dexterous hand to execute the initial grasping action, and obtain the joint state, gripper posture, and contact force information in real time through the proprioceptive module and the fingertip force sensor; Step S5: During the grasping and handling process, analyze the force distribution and center of gravity offset of the composite object based on the contact force direction and joint state sensed by the multi-dimensional force sensor; Step S6: Using the visual and force sensor data as input, adopt a reinforcement learning algorithm to dynamically adjust the gripping force and posture parameters of the dexterous hand to optimize the grasping stability; Step S7: Continuously perform tactile perception and policy fine-tuning during the handling process to ensure the overall posture stability of the composite object.

2. The stable multi-object grasping method for a dexterous hand based on visual and tactile perception according to claim 1, wherein The three-dimensional point cloud neural network is PointNet or PointNet++, which is used to extract the geometric features of the point cloud data and fuse the texture information of the RGB image.

3. The method for stable multi-object grasping of a dexterous hand based on visual and tactile perception according to claim 1, wherein, The reinforcement learning algorithm is a policy gradient algorithm, including PPO or DDPG, and its reward function is constructed based on factors such as gripping stability, no slippage of the object, and energy consumption.

4. The method for stable multi-object grasping of a dexterous hand based on visual and tactile perception according to claim 1, wherein, The dynamic adjustment in Step S6 includes: Step S6.1: Define the state space as the joint angles of the dexterous hand, the gripping force, and the tactile sensor feedback; Step S6.2: Through online policy iteration optimization, realize the real-time fine-tuning of the gripping force and posture parameters.

5. The method for stable multi-object grasping of a dexterous hand based on visual and tactile perception according to claim 1, wherein The composite object is a plurality of loose objects placed on a carrier, including a dinner plate and the tableware on it, and the dynamic adjustment ensures that there is no tipping or relative movement of each object during the handling process.

6. A dexterous hand multi-object stable grasping system based on visual and tactile perception, characterized in that, The system includes the following modules: Module M1: Collect the RGB image and depth map of the target object through a monocular RGB-D camera to generate dense point cloud data containing the three-dimensional structure information of the target object; Module M2: Input the point cloud data into a three-dimensional point cloud neural network to extract the geometric structure features of the object, and estimate the distribution features of the composite objects in the scene by combining the RGB image and depth information; Module M3: Generate an initial grasping strategy based on the extracted visual features, including the position of the gripping point, the gripping direction, and the grasping method; Module M4: Control the dexterous hand to execute the initial grasping action, and obtain the joint state, gripper posture, and contact force information in real time through the proprioceptive module and the fingertip force sensor; Module M5: During the grasping and handling process, analyze the force distribution and center of gravity offset of the composite object based on the contact force direction and joint state sensed by the multi-dimensional force sensor; Module M6: Using the visual and force sensor data as input, adopt a reinforcement learning algorithm to dynamically adjust the gripping force and posture parameters of the dexterous hand to optimize the grasping stability; Module M7: Continuously perform tactile perception and policy fine-tuning during the handling process to ensure the overall posture stability of the composite object.

7. The dexterous hand multi-object stable grasping system based on visual and tactile perception according to claim 6, characterized in that, The three-dimensional point cloud neural network is PointNet or PointNet++, which is used to extract the geometric features of point cloud data and fuse the texture information of RGB images.

8. The dexterous hand multi-object stable grasping system based on visual and tactile perception according to claim 6, characterized in that, The reinforcement learning algorithm is a policy gradient algorithm, including PPO or DDPG, and its reward function is constructed based on factors such as grasping stability, no slippage of the object, and energy consumption.

9. The dexterous hand multi-object stable grasping system based on visual and tactile perception according to claim 6, characterized in that, The dynamic adjustment in module M6 includes: Module M6.1: Define the state space as the dexterous hand joint angles, grasping force, and tactile sensor feedback; Module M6.2: Through online policy iteration optimization, achieve real-time fine-tuning of the grasping force and pose parameters.

10. The dexterous hand multi-object stable grasping system based on visual and tactile perception according to claim 6, characterized in that, The composite object is multiple loose objects placed on a carrier, including a dinner plate and the tableware on it, and the dynamic adjustment ensures that there is no tipping or relative movement of each object during the handling process.

Images