Dexterous hand multi-object stable grasping method and system based on visual and tactile perception

Abstract

The application provides a kind of based on visual and tactile perception nimble hand multi-object stable grasping method and system, comprising: the RGB image and depth map of target object are collected by monocular RGB-D camera and generate dense point cloud data;Geometric features are extracted using three-dimensional point cloud neural network, and the distribution characteristics of composite object are analyzed in combination with visual information;Initial grasping strategy is generated based on features and action is performed, joint state and contact force data are obtained in real time by synchronous through proprioceptive and fingertip force sensor;In the process of carrying, multi-dimensional force perception and joint state analysis object stress distribution and barycentric shift are fused;With visual and force sensing data as input.The application fuses vision, proprioceptive and force sensing data, solves the three-dimensional structure recognition and dynamic stress control problem in multi-object composite grasping, compared with single perception scheme, significantly improves the grasping planning accuracy and adaptive ability in complex scene, especially suitable for stable carrying of loose object combination such as dinner plate carrying cutlery.

Description

Technical Field

[0001] This invention relates to the technical field of robot grasping and control, specifically to a method and system for stable multi-object grasping using a dexterous hand based on visual and tactile perception. Background Technology

[0002] With the rapid development of robotics technology, the demand for dexterous hands in industrial, service, and home scenarios is increasing. However, most existing research on robotic grasping technology focuses on the stable grasping of single objects, such as target localization based on depth cameras and grasping control using two-finger grippers. For example, the "robotic arm grasping method" disclosed in Chinese patent CN201810458201.4, although capable of grasping single objects using a depth camera, limits its applicability in complex scenarios due to its reliance on a two-finger parallel gripper structure. Especially when faced with multiple loosely stacked objects on the same carrier (such as bowls and chopsticks on a plate), existing technology struggles to effectively identify the overall distribution characteristics of the composite target, and is unable to perceive dynamic force changes in real time and make adaptive adjustments during the grasping process.

[0003] In practical applications, composite target grasping needs to simultaneously meet the following requirements: accurately identify the spatial distribution and geometric features of multiple objects to plan a reasonable initial grasping posture; dynamically sense changes in force during the handling process, such as torque fluctuations caused by object center of gravity shifts or external disturbances; and adjust the grasping strategy in real time to ensure the overall stability of the target. However, existing technologies have significant shortcomings: Limitations of visual perception: Traditional methods rely on single depth information or two-dimensional images, making it difficult to extract the global three-dimensional structural features of composite targets, resulting in inaccurate initial grasping planning; Rigid control strategies: Most algorithms use fixed thresholds or offline optimization strategies, lacking the ability to respond online to dynamic environments (such as object sliding or overturning); Insufficient force perception and fusion: Existing systems often neglect the coordination between tactile information and proprioception, resulting in the inability to effectively suppress the relative displacement of multiple objects during handling.

[0004] Both academia and industry have explored solutions to these problems. For example, the team led by Liu Houde at Tsinghua University's Shenzhen Graduate School has conducted in-depth research in the field of dexterous hand grasping, but their results are still focused on single-object scenes and have not solved the robustness problem of multi-object composite grasping. In addition, solutions proposed by peer companies often rely on high-cost multi-view vision systems or complex mechanical structures, making it difficult to achieve lightweight deployment.

[0005] Therefore, there is an urgent need for a method that integrates multimodal perception and intelligent decision-making to achieve stable grasping and dynamic adjustment of multiple objects in complex scenarios. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for stable multi-object grasping using a dexterous hand based on visual and tactile perception.

[0007] According to the present invention, a method for stable multi-object grasping based on visual and tactile perception using a dexterous hand includes the following steps: Step S1: Acquire RGB images and depth maps of the target object using a monocular RGB-D camera to generate dense point cloud data containing the three-dimensional structural information of the target object; Step S2: Input the point cloud data into the 3D point cloud neural network to extract the geometric structure features of the complex, and combine the RGB image and depth information to estimate the distribution features of the complex in the scene; Step S3: Generate an initial gripping strategy based on the extracted visual features, including gripping point position, gripping direction and gripping method; Step S4: Control the dexterous hand to perform the initial grasping action, and obtain joint status, gripper posture and contact force information in real time through the body perception module and fingertip force sensor; Step S5: During the grasping and handling process, based on the contact force direction and joint state sensed by the multi-dimensional force sensor, analyze the force distribution and center of gravity shift of the complex body. Step S6: Using visual and force sensor data as input, a reinforcement learning algorithm is used to dynamically adjust the gripping force and posture parameters of the dexterous hand to optimize gripping stability; Step S7: During the handling process, continuous tactile sensing and strategy fine-tuning are performed to ensure the overall posture stability of the complex.

[0008] Preferably, the three-dimensional point cloud neural network is PointNet or PointNet++, used to extract the geometric features of the point cloud data and fuse the texture information of the RGB image.

[0009] Preferably, the reinforcement learning algorithm is a policy gradient algorithm, including PPO or DDPG, and its reward function is constructed based on clamping stability, object slippage, and energy consumption factors.

[0010] Preferably, the dynamic adjustment in step S6 includes: Step S6.1: Define the state space as the angle of the dexterous hand joints, the gripping force, and the feedback from the tactile sensor; Step S6.2: Real-time fine-tuning of clamping force and attitude parameters is achieved through online strategy iterative optimization.

[0011] Preferably, the composite body consists of multiple loose objects placed on a carrier, including a plate and tableware on it, and the dynamic adjustment ensures that the objects do not tip over or move relative to each other during transport.

[0012] The present invention also provides a dexterous hand multi-object stable grasping system based on visual and tactile perception, the system comprising the following modules: Module M1: Acquires RGB images and depth maps of the target object using a monocular RGB-D camera, generating dense point cloud data containing the three-dimensional structural information of the target object; Module M2: Inputs point cloud data into a 3D point cloud neural network to extract the geometric features of the complex objects, and combines RGB images and depth information to estimate the distribution features of the complex objects in the scene; Module M3: Generates an initial gripping strategy based on the extracted visual features, including gripping point position, gripping direction, and gripping method; Module M4: Controls the dexterous hand to perform the initial grasping action, and acquires joint status, gripper posture and contact force information in real time through the body perception module and fingertip force sensor; Module M5: During the grasping and handling process, based on the contact force direction and joint status sensed by the multi-dimensional force sensor, the force distribution and center of gravity shift of the complex body are analyzed. Module M6: Using visual and force sensor data as input, it employs reinforcement learning algorithms to dynamically adjust the gripping force and posture parameters of the dexterous hand, thereby optimizing gripping stability; Module M7: Continuously performs tactile sensing and strategy fine-tuning during the handling process to ensure the overall stability of the complex body's posture.

[0013] Preferably, the three-dimensional point cloud neural network is PointNet or PointNet++, used to extract the geometric features of the point cloud data and fuse the texture information of the RGB image.

[0014] Preferably, the reinforcement learning algorithm is a policy gradient algorithm, including PPO or DDPG, and its reward function is constructed based on clamping stability, object slippage, and energy consumption factors.

[0015] Preferably, the dynamic adjustment in module M6 includes: Module M6.1: Defines the state space as the angle of the dexterous hand joints, the gripping force, and the feedback from the tactile sensors; Module M6.2: Real-time fine-tuning of clamping force and attitude parameters is achieved through online strategy iterative optimization.

[0016] Preferably, the composite body consists of multiple loose objects placed on a carrier, including a plate and tableware on it, and the dynamic adjustment ensures that the objects do not tip over or move relative to each other during transport.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention achieves high-precision perception of the three-dimensional structural features and dynamic force distribution of multi-object composite grasping targets by integrating monocular RGB-D visual information, dexterous hand proprioception, and fingertip force sensor data. Compared with traditional single vision or force control solutions, this method can more comprehensively capture the spatial layout and real-time force changes of target objects, significantly improving the accuracy of grasping planning and environmental adaptability in complex scenarios. 2. This invention is based on a dynamic adjustment mechanism of reinforcement learning algorithm, combined with a composite reward function design of "maximizing stability", which enables the system to perceive dynamic disturbances such as the shift of the object's center of gravity and changes in contact force in real time during the handling process, and quickly optimize the clamping force and attitude parameters. This feature effectively suppresses the problem of relative sliding and overturning between multiple objects, and greatly improves the robustness and anti-interference ability of the grasping process. 3. To address the limitation of existing technologies that only support grasping a single object, this invention generates an initial grasping strategy through a neural network and combines it with an online fine-tuning mechanism to achieve stable overall grasping of loosely stacked or non-fixed-arranged multi-object targets; it is particularly suitable for practical scenarios such as catering services and logistics sorting that require the simultaneous handling of multiple objects, thus expanding the application scope of dexterous hands. Attached Figure Description

[0018] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 The flowchart of this invention. Detailed Implementation

[0019] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0020] Example 1: Reference Figure 1 According to the present invention, a method for stable multi-object grasping based on visual and tactile perception using a dexterous hand includes the following steps: Step S1: Acquire RGB images and depth maps of the target object using a monocular RGB-D camera to generate dense point cloud data containing the three-dimensional structural information of the target object; Step S2: Input the point cloud data into a 3D point cloud neural network to extract the geometric features of the objects, and combine the RGB image and depth information to estimate the distribution features of the complex objects in the scene; the 3D point cloud neural network is PointNet or PointNet++, which is used to extract the geometric features of the point cloud data and fuse the texture information of the RGB image.

[0021] Step S3: Generate an initial gripping strategy based on the extracted visual features, including gripping point position, gripping direction and gripping method; Step S4: Control the dexterous hand to perform the initial grasping action, and obtain joint status, gripper posture and contact force information in real time through the body perception module and fingertip force sensor; Step S5: During the grasping and handling process, based on the contact force direction and joint state sensed by the multi-dimensional force sensor, the force distribution and center of gravity shift of the complex body are analyzed; during the grasping process, based on the proprioception of the dexterous hand and the tactile sensors of its fingertips, the force and point of application of each fingertip on the object can be obtained. Thus, a combined force is achieved. sum moment ;

[0022] Thus, the moment center is calculated. .

[0023] For object centroid estimation, two modes are available: If it is known that multiple objects are made of the same material, such as several wooden strips placed on a wooden board, existing centroid estimation methods based on point clouds can be used directly.

[0024] If multiple objects are known to have different materials, then the mass of each object must be known. Using existing point cloud segmentation algorithms, the centroid of each segmented point cloud is calculated separately, and then combined with the mass of each object, the overall centroid can be obtained.

[0025] During the grasping and moving process, if the center of torque and the object's center of mass are close in space, the risk of tipping over is considered low. Therefore, a penalty term can be added to the reward function. When the spatial distance between the center of torque and the object's center of mass exceeds a certain threshold, a penalty is imposed to prevent tipping over.

[0026] Step S6: Using visual and force sensor data as input, a reinforcement learning algorithm is employed to dynamically adjust the gripping force and posture parameters of the dexterous hand, optimizing gripping stability. The reinforcement learning algorithm is a policy gradient algorithm, including PPO or DDPG, and its reward function is constructed based on gripping stability, object slippage prevention, and energy consumption factors. Dynamic adjustment includes: Step S6.1: Define the state space as the dexterous hand joint angles, gripping force, and tactile sensor feedback; define the state space as the dexterous hand joint angles and tactile sensor measurements; state space definition: Assume the dexterous hand has Each degree of freedom, its state space At any moment Represented as: .in For the first Joint angles (unit: radians). For the first Three-dimensional contact force (unit: Newton) measured by a fingertip force sensor, of which This refers to the number of tactile (force) sensors.

[0027] Step S6.2: Achieve real-time fine-tuning of clamping force and attitude parameters through online policy iterative optimization. Online policy iterative optimization: Construct a policy network using a policy gradient algorithm (such as PPO). ,in For policy network parameters, The motion space includes the increment of joint angles. and clamping force adjustment amount The optimization process minimizes the loss function. for: .in The strategy update ratio. This is the estimated value of the dominance function. Sets the pruning threshold (default 0.2). The policy is updated every 50ms, achieving millisecond-level real-time response.

[0028] Step S7: Continuously perform tactile sensing and strategy fine-tuning during the handling process to ensure the overall posture stability of the complex. The complex consists of multiple loose objects placed on a carrier, including a plate and its tableware, and the dynamic adjustment ensures that the objects do not tip over or move relative to each other during handling.

[0029] The present invention also provides a dexterous hand multi-object stable grasping system based on visual and tactile perception. The dexterous hand multi-object stable grasping system based on visual and tactile perception can be implemented by executing the process steps of the dexterous hand multi-object stable grasping method based on visual and tactile perception. That is, those skilled in the art can understand the dexterous hand multi-object stable grasping method based on visual and tactile perception as a preferred embodiment of the dexterous hand multi-object stable grasping system based on visual and tactile perception.

[0030] Example 2: The present invention also provides a dexterous hand multi-object stable grasping system based on visual and tactile perception, the system comprising the following modules: Module M1: Acquires RGB images and depth maps of the target object using a monocular RGB-D camera, generating dense point cloud data containing the three-dimensional structural information of the target object; Module M2: Inputs point cloud data into a 3D point cloud neural network to extract geometric features of objects, and combines RGB images and depth information to estimate the distribution features of complex objects in the scene; the 3D point cloud neural network is PointNet or PointNet++, used to extract geometric features of point cloud data and fuse texture information from RGB images.

[0031] Module M3: Generates an initial gripping strategy based on the extracted visual features, including gripping point position, gripping direction, and gripping method; Module M4: Controls the dexterous hand to perform the initial grasping action, and acquires joint status, gripper posture and contact force information in real time through the body perception module and fingertip force sensor; Module M5: During the grasping and handling process, based on the contact force direction and joint status sensed by the multi-dimensional force sensor, the force distribution and center of gravity shift of the complex body are analyzed. Module M6: Using visual and force sensor data as input, it employs reinforcement learning algorithms to dynamically adjust the gripping force and posture parameters of the dexterous hand, optimizing grasping stability. The reinforcement learning algorithm is a policy gradient algorithm, including PPO or DDPG, and its reward function is constructed based on gripping stability, object slippage prevention, and energy consumption factors. Dynamic adjustments include: Module M6.1: Defines the state space as the angle of the dexterous hand joints, the gripping force, and the feedback from the tactile sensors; Module M6.2: Real-time fine-tuning of clamping force and attitude parameters is achieved through online strategy iterative optimization.

[0032] Module M7: Continuously performs tactile sensing and strategy fine-tuning during the handling process to ensure the overall posture stability of the complex body; the complex body consists of multiple loose objects placed on the carrier, including a plate and the tableware on it, and the dynamic adjustment ensures that the objects do not tip over or move relative to each other during the handling process.

[0033] Example 3: This invention proposes a stable gripping control method that integrates monocular RGB-D visual information, dexterous hand proprioception, and fingertip force sensors. It estimates the visual features of the object and generates an initial gripping posture through a neural network, and dynamically adjusts the dexterous hand gripping posture and output torque based on reinforcement learning. This method is particularly suitable for grasping a whole target containing multiple loose objects (such as a plate containing bowls and chopsticks), achieving smooth handling and posture control, and includes the following steps: Step 1: Scene perception and point cloud generation; An RGB-D camera is used to acquire RGB images and depth maps of the target object (such as a plate and its utensils), which are then converted into dense point cloud data. This data contains the three-dimensional structural information of the target object and forms the basis for subsequent grasping inference and policy generation.

[0034] Step 2: Feature extraction; Point cloud data is input into a 3D point cloud neural network (such as PointNet or PointNet++) to extract the geometric features of objects. By combining RGB images and depth information, the distribution characteristics of complex objects (such as tableware on a plate) in the scene are estimated.

[0035] Step 3: Initial gripping strategy generation; Based on visual features, existing gripping strategy generation methods are used to generate an initial gripping posture suitable for the current task, including: gripping point position, gripping direction, gripping method (clamping, lifting, etc.).

[0036] Step 4: Perform the initial gripping motion; The robotic arm and its dexterous end effector execute grasping actions according to the generated strategy. At the same time, the body perception and fingertip force sensor modules are activated to acquire joint status, gripper posture and fingertip contact force information in real time, in preparation for subsequent adjustments.

[0037] Step 5: Force sensing and force analysis; During the grasping and handling process, the multi-dimensional force sensors deployed at the fingertips sense the contact force and its direction on each finger. Combined with the joint state of the dexterous hand, the force distribution, torque changes and center of gravity shift of the complex body are understood.

[0038] Step 6: Optimize dynamic posture and force application based on reinforcement learning; State information from vision and force sensors is used as input to a reinforcement learning model (such as policy gradient algorithms like PPO and DDPG). The goal is to maximize grip stability, and the finger gripping force and posture parameters are dynamically adjusted to optimize the current action strategy.

[0039] Step 6.1: State space definition: including joint angles, clamping forces, etc.; Step 6.2: Reward function design: Construct a composite reward based on factors such as clamping stability (no tilting or slippage of the object) and energy consumption; Step 6.3: Strategy Iteration and Optimization: Continuously execute, perceive, evaluate and update the strategy to achieve online fine-tuning capabilities.

[0040] Step 7: Stable handling and dynamic adjustment; During the handling process (such as moving a plate from tabletop A to tabletop B), the system continuously performs tactile sensing and strategy fine-tuning to ensure the stability of the overall posture of the target object and prevent items above from tilting, slipping, or moving relative to each other.

[0041] This invention solves the problem that existing dexterous hands struggle to identify the overall distribution characteristics when facing complex grasping targets with multiple stacked objects by employing a structure of "monocular RGB-D visual perception + point cloud feature extraction." This achieves the effect of rationally planning the grip posture and clamping points before the initial grasp, laying the foundation for stable handling. By adopting a structure of "force sensor perception + proprioception + reinforcement learning control," it solves the problem that it is difficult to adjust the grip posture and grip force in real time when the dynamic changes in the force on the complex object during handling. This achieves a stable grasping effect that continuously adapts and effectively suppresses object slippage and tipping during the grasping process.

[0042] Those skilled in the art can understand this embodiment as a more specific description of Embodiment 1 and Embodiment 2.

[0043] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0044] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for stable multi-object grasping using a dexterous hand based on visual and tactile perception, characterized in that, The method includes the following steps: Step S1: Acquire RGB images and depth maps of the target object using a monocular RGB-D camera to generate dense point cloud data containing the three-dimensional structural information of the target object; Step S2: Input the point cloud data into the 3D point cloud neural network to extract the geometric structure features of the complex, and combine the RGB image and depth information to estimate the distribution features of the complex in the scene; Step S3: Generate an initial gripping strategy based on the extracted visual features, including gripping point position, gripping direction and gripping method; Step S4: Control the dexterous hand to perform the initial grasping action, and obtain joint status, gripper posture and contact force information in real time through the body perception module and fingertip force sensor; Step S5: During the grasping and handling process, based on the contact force direction and joint state sensed by the multi-dimensional force sensor, analyze the force distribution and center of gravity shift of the complex body. Step S6: Using visual and force sensor data as input, a reinforcement learning algorithm is used to dynamically adjust the gripping force and posture parameters of the dexterous hand to optimize gripping stability; Step S7: Continuously perform tactile sensing and strategy fine-tuning during the handling process to ensure the overall posture stability of the complex body; The composite body consists of multiple loose objects placed on a carrier, and the dynamic adjustment ensures that the objects do not tip over or move relative to each other during the handling process. During the grasping process, based on the dexterous hand's proprioception and the tactile sensors at its fingertips, the force and point of application of each fingertip on the object are obtained. Thus, a combined force is achieved. sum moment ; Thus, the moment center is calculated. ; If it is known that multiple objects have the same material, the existing centroid estimation method based on point cloud can be used directly; If it is known that multiple objects have different materials, then the mass of each object must be known. Using existing point cloud segmentation algorithms, the centroid of each segmented point cloud is calculated separately, and then combined with the mass of each object, the overall centroid can be obtained. During the grasping and moving process, if the center of moment and the center of mass of the object are close in space, the risk of tipping over is considered to be small. A penalty term is added to the reward function. When the spatial distance between the center of moment and the center of mass of the object is greater than a certain threshold, a penalty is imposed to prevent tipping over. Dynamic adjustments include: The state space is defined as the angle of the dexterous hand joints, the gripping force, and the feedback from the tactile sensors. State space definition: Suppose a dexterous hand has Each degree of freedom, its state space At any moment Represented as: ;in For the first Each joint angle For the first Three-dimensional contact force measured by a fingertip force sensor, among which This refers to the number of tactile sensors. Real-time fine-tuning of clamping force and attitude parameters is achieved through online policy iterative optimization; online policy iterative optimization: a policy gradient algorithm is used to construct the policy network. ,in For policy network parameters, The motion space includes the increment of joint angles. and clamping force adjustment amount The optimization process minimizes the loss function. for: ;in For strategy update ratio; This is the estimated value of the dominance function. The pruning threshold is set; the policy is updated every 50ms to achieve millisecond-level real-time response.

2. The method for stable multi-object grasping based on visual and tactile perception using a dexterous hand according to claim 1, characterized in that, The three-dimensional point cloud neural network is PointNet or PointNet++, used to extract the geometric features of point cloud data and fuse the texture information of RGB images.

3. The method for stable multi-object grasping based on visual and tactile perception using a dexterous hand according to claim 1, characterized in that, The reinforcement learning algorithm is a policy gradient algorithm, including PPO or DDPG, and its reward function is constructed based on clamping stability, object slippage, and energy consumption factors.

4. A dexterous hand multi-object stable grasping system based on visual and tactile perception, characterized in that, The system includes the following modules: Module M1: Acquires RGB images and depth maps of the target object using a monocular RGB-D camera, generating dense point cloud data containing the three-dimensional structural information of the target object; Module M2: Inputs point cloud data into a 3D point cloud neural network to extract the geometric features of the complex objects, and combines RGB images and depth information to estimate the distribution features of the complex objects in the scene; Module M3: Generates an initial gripping strategy based on the extracted visual features, including gripping point position, gripping direction, and gripping method; Module M4: Controls the dexterous hand to perform the initial grasping action, and obtains joint status, gripper posture and contact force information in real time through the body perception module and fingertip force sensor; Module M5: During the grasping and handling process, based on the contact force direction and joint status sensed by the multi-dimensional force sensor, the force distribution and center of gravity shift of the complex body are analyzed. Module M6: Using visual and force sensor data as input, it employs reinforcement learning algorithms to dynamically adjust the gripping force and posture parameters of the dexterous hand, thereby optimizing gripping stability; Module M7: Continuously performs tactile sensing and strategy fine-tuning during the handling process to ensure the overall stability of the complex body's posture; The composite body consists of multiple loose objects placed on a carrier, and the dynamic adjustment ensures that the objects do not tip over or move relative to each other during the handling process. During the grasping process, based on the dexterous hand's proprioception and the tactile sensors at its fingertips, the force and point of application of each fingertip on the object are obtained. Thus, a combined force is achieved. sum moment ; Thus, the moment center is calculated. ; If it is known that multiple objects have the same material, the existing centroid estimation method based on point cloud can be used directly; If it is known that multiple objects have different materials, then the mass of each object must be known. Using existing point cloud segmentation algorithms, the centroid of each segmented point cloud is calculated separately, and then combined with the mass of each object, the overall centroid can be obtained. During the grasping and moving process, if the center of moment and the center of mass of the object are close in space, the risk of tipping over is considered to be small. A penalty term is added to the reward function. When the spatial distance between the center of moment and the center of mass of the object is greater than a certain threshold, a penalty is imposed to prevent tipping over. Dynamic adjustments include: The state space is defined as the angle of the dexterous hand joints, the gripping force, and the feedback from the tactile sensors. State space definition: Suppose a dexterous hand has Each degree of freedom, its state space At any moment Represented as: ;in For the first Each joint angle For the first Three-dimensional contact force measured by a fingertip force sensor, among which This refers to the number of tactile sensors. Real-time fine-tuning of clamping force and attitude parameters is achieved through online policy iterative optimization; online policy iterative optimization: a policy gradient algorithm is used to construct the policy network. ,in For policy network parameters, The motion space includes the increment of joint angles. and clamping force adjustment amount The optimization process minimizes the loss function. for: ;in For strategy update ratio; This is the estimated value of the dominance function. The pruning threshold is set; the policy is updated every 50ms to achieve millisecond-level real-time response.

5. The dexterous hand multi-object stable grasping system based on visual and tactile perception according to claim 4, characterized in that, The three-dimensional point cloud neural network is PointNet or PointNet++, used to extract the geometric features of point cloud data and fuse the texture information of RGB images.

6. The dexterous hand multi-object stable grasping system based on visual and tactile perception according to claim 4, characterized in that, The reinforcement learning algorithm is a policy gradient algorithm, including PPO or DDPG, and its reward function is constructed based on clamping stability, object slippage, and energy consumption factors.

Images