Intelligent learning method for grabbing of three-finger dexterous hand for multiple types of workpieces

Abstract

The invention discloses an intelligent learning method for grabbing of a three-finger dexterous hand for multiple types of workpieces, and belongs to the technical field of intelligent grabbing application of manipulators. A pre-defined rule and simulation expert double-layer mode is provided, a high-quality demonstration data set is automatically generated, a simulation grabbing scene of a mechanical arm and a three-fingered dexterous hand is constructed, recognition and classification of multiple types of workpieces are achieved by analyzing workpiece contour features, grabbing hand shapes are matched according to the recognition and classification, strategy pre-training is carried out through simulation learning, and the recognition and classification efficiency is improved. And optimizing the strategy based on a PPO algorithm and a composite reward function. Aiming at the problem of lack of a standardized three-fingered dexterous hand parameter model, a modeling method based on parameterized URDF and automatic script generation is used to realize efficient configuration and optimization of the three-fingered dexterous hand in simulation. Through performance evaluation, stable and high-quality grabbing of multiple types of workpieces can be achieved.

Description

Technical Field

[0001] This invention relates to the field of intelligent grasping application technology of robotic arms, specifically to an intelligent learning method for grasping various types of workpieces using a three-finger dexterous hand. Background Technology

[0002] Research on robotic dexterous hands aims to develop multi-fingered robotic hands capable of mimicking the fine motor skills of human hands through high-degree-of-freedom joint structures and precision sensor systems. For the specific needs of industrial grasping scenarios, three-fingered dexterous hands, with their biomimetic grasping mechanism and real-time perception capabilities, are widely used in tasks such as workpiece grasping and assembly. However, when faced with diverse workpieces in industrial settings that vary greatly in shape, size, and material, most grasping planning methods do not adequately consider grasping gestures, making it difficult to achieve precise object manipulation. Designing universal grasping devices for unknown objects faces even greater technical challenges when performing the same tasks.

[0003] Domestic and international scholars have conducted extensive exploratory research in the field of robot dexterous hand grasping strategy optimization. Traditional grasping strategies rely on predefined rules or limited databases, requiring manual adjustment of grasping parameters for specific workpieces. They struggle to adapt to unfamiliar objects, and industrial environments present challenges such as random workpiece postures and fluctuating physical properties. Fixed grasping strategies are prone to slippage, collisions, or grasping failures. The rapid development of reinforcement learning technology can effectively address key challenges in dexterous hand operations, such as high-dimensional state spaces and complex motion planning. Reinforcement learning models dynamically utilize closed-loop feedback control based on visual and / or tactile perception, updating strategies while accumulating new observation data. While physical simulation-based reinforcement learning can improve adaptability, direct training from scratch requires massive amounts of interactive data, resulting in low training efficiency and a tendency to get trapped in local optima. Pure imitation learning is limited by the quality of demonstration datasets and struggles to cope with environmental uncertainties. By combining imitation learning to obtain high-quality initial grasping strategies and then using reinforcement learning for fine-tuning in a physical simulation environment, the grasping ability of a three-finger dexterous hand for various types of workpieces can be significantly improved. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an intelligent learning method for three-finger dexterous hand grasping of various workpieces. This invention combines reinforcement learning and imitation learning, which is an effective solution to these challenges. By introducing physical parameter randomization and work feature perception modules, the robot's grasping skills in handling various objects are constructed and enhanced, ultimately achieving the generation of the optimal grasping posture for the workpiece.

[0005] The intelligent learning method for three-finger dexterous hand grasping of various workpiece types proposed in this invention specifically includes the following steps: S1. Build a grasping task scenario for the robotic arm and the three-finger dexterous hand, and configure the dataset acquisition module, workpiece perception module, and simulation strategy learning module; S2. The dataset acquisition module uses a two-layer mode of predefined rule strategies and simulation expert agents to acquire the dataset required for imitation learning. The predefined rule strategies provide stable and safe basic actions, while the simulation expert agents generate fine actions that adapt to complex scenarios and seamlessly switch to the predefined rule strategies when they fail. S3. Map the dexterous hand movements to the simulation, control the dexterous hand in the simulation to move and interact with objects in the simulation, and obtain a high-quality demonstration dataset. S4. The workpiece perception module acquires RGB images of the scene, extracts geometric features, functional area information and hand shape classification information of the workpiece, and realizes workpiece type recognition and gripping part labeling. S5. The simulation strategy learning module inputs a parameterized URDF three-finger dexterous hand model into the Mujoco simulation environment; the strategy network is trained through imitation learning pre-training and reinforcement learning to obtain a converged strategy network; during the imitation learning pre-training process, a composite reward function is designed to reward proximity, contact, normal alignment, friction constraints, grasping quality, slip detection, lifting and stability, and physical parameter randomization is introduced to improve the robustness and generalization ability of the strategy. The trained and converged policy network takes robot body perception information, workpiece geometry information to be grasped, fingertip-workpiece interaction information, and task environment information as input, and outputs 18-dimensional motion control commands for the robotic arm and three-finger dexterous hand to grasp the workpiece.

[0006] Compared with the prior art, the present invention has the following advantages: 1. This invention automatically generates high-quality demonstration data through a "predefined rules + simulation expert" model, overcoming the dependence on a large amount of human expert teaching data and reducing the difficulty and cost of data acquisition.

[0007] 2. This invention employs the PPO algorithm and its Clip mechanism for fine-tuning. Many current methods only set a sparse reward for successful grasping. Before receiving this reward, the agent receives no learning signals, making it unable to understand the importance of intermediate steps like approach and alignment, resulting in low learning efficiency and most random actions being ineffective. The multi-objective composite reward function designed in this invention is key to the method's superior performance. It rewards the final success, providing fine-tuning guidance for the entire grasping process from geometric, physical, and mechanical quality perspectives. This invention's method can actively learn grasping postures with good force closure, full utilization of friction, accurate normal alignment, and strong anti-interference capabilities. Its grasping success rate and grasping quality are far superior to strategies relying solely on sparse or simple rewards.

[0008] 3. This invention proposes to generate a pre-set posture for the dexterous hand before grasping, and intelligently triggers the switching of control modes based on tactile perception signals. When facing different types of objects, the grasping posture is judged based on the result of workpiece perception. When facing objects such as flexible bags whose center of gravity changes, after reliable contact occurs, the thumb and index finger, which are dynamically assigned to the main pinching finger role, are switched to force control mode to apply a precise and smooth grasping force, and to provide support for the stabilizing finger role to maintain position control. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart of the intelligent learning method for three-finger dexterous hand grasping of various types of workpieces proposed in this invention. Figure 2 This is a schematic diagram of the structure of the parameterized three-finger dexterous hand described in this invention; Figure 3 This is a flowchart illustrating the training process of the grasping strategy optimized by pre-training and reinforcement learning in this invention. Figure 4 This is a schematic diagram of the strategy network architecture and training process of the present invention; Figure 5 This is a flowchart of the three-finger dexterous hand pinching and collaborative control process of the present invention; Figure 6 This invention is based on a three-finger dexterous hand grasping decision map for perceiving small workpieces; Figure 7 This invention relates to a three-finger dexterous hand grasping decision map based on perception of flexible packaging. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0013] This invention proposes an intelligent learning method for grasping three-finger dexterous hands that include a robotic arm. The purpose of this method is to enable the three-finger dexterous hand to perform grasping tasks on various types of workpieces. The three-finger dexterous hand is located at the end of the robotic arm, which drives the three-finger dexterous hand to move in space. Through the coordinated movement of the three fingers, the three-finger dexterous hand can be used in industrial scenarios to achieve fine operations such as grasping and manipulating objects.

[0014] like Figure 1 As shown, the intelligent learning method for three-finger dexterous hand grasping of various workpieces proposed in this invention mainly includes: S1. Build a grasping task scenario for the robotic arm and the three-finger dexterous hand, and configure the dataset acquisition module, workpiece perception module, simulation strategy learning module, and performance index evaluation module. S2. The dataset acquisition module uses a two-layer mode of predefined rule strategies and simulation expert agents to acquire the dataset required for imitation learning. The predefined rule strategies provide stable and safe basic actions, while the simulation expert agents generate fine actions that adapt to complex scenarios and seamlessly switch to the predefined rule strategies when they fail. The dataset acquisition module of this invention acquires the dataset required for imitation learning through two methods: a predefined rule strategy and a simulation expert agent. For workpieces with regular geometric shapes, the predefined rule strategy directly calculates a set of predefined gripping points based on the size and orientation of their minimum bounding box. Utilizing force closure theory, it prioritizes combinations of gripping points that can form force-closed grips, and preliminarily evaluates the gripping quality by calculating the singular values ​​of the gripping matrix, filtering out obviously unstable gripping postures. The simulation expert agent plans and executes a time-optimal, energy-lowering motion trajectory for the robotic arm that satisfies complex dynamic constraints, achieving intelligent, anthropomorphic motion from the starting point to the gripping posture. It fine-tunes the gripping posture using complete simulation environment state information and workpiece features extracted by the workpiece perception module.

[0015] In a specific embodiment of this invention, predefined rules and strategies serve as a basic protection layer, providing stable and secure, but potentially suboptimal, actions. Simulated expert agents, as a high-performance layer, generate human-like, sophisticated actions adapted to complex scenarios. The two methods collaborate, with the simulated expert agent making priority decisions. When the simulated expert agent fails, a seamless switch to the predefined rules and strategies ensures the continuity of data generation.

[0016] S3. After obtaining the dataset based on step S2, the dexterous hand movements are mapped to the simulation. The dexterous hand in the simulation is controlled to move and interact with the objects in the simulation to obtain a high-quality demonstration dataset.

[0017] S4. The workpiece perception module acquires RGB images of the scene, extracts geometric features, functional area information and hand shape classification information of the workpiece, and realizes workpiece type recognition and gripping part labeling. In one specific embodiment of the present invention, the workpiece perception module first employs a center-left-lower cropping strategy on the acquired RGB image, using 2 / 3 of the original image size as the cropping area. This cropping strategy is based on the empirical knowledge that workpieces are usually located under the operating table in industrial scenarios, effectively removing background interference and focusing on key working areas. The cropped RGB image is then converted into a grayscale image, and the Otsu adaptive thresholding algorithm is used for binarization segmentation. The Otsu algorithm automatically determines the optimal segmentation threshold by maximizing the inter-class variance, adapting to workpiece recognition under different lighting conditions. Morphological closing operations are performed on the binarized image to fill holes, and morphological opening operations are used to remove noise. The contour outer boundary detection algorithm is used to extract the main contour of the workpiece, and the extracted contour is displayed as a green line superimposed on the original cropped image with a line width set to 2 pixels. This visualization result is used for debugging and effect verification to ensure the accuracy of contour extraction. The extracted contour features are combined with a classification network to identify the workpiece type (types include: circular, rectangular, and irregular shapes). The optimal grasping point of the workpiece is marked on the key points of interaction between the workpiece and the three-finger dexterous hand and the functional area of ​​the workpiece in the collected RGB image data of the workpiece, so as to grasp the corresponding parts of different types of objects with the optimal hand shape.

[0018] In a preferred embodiment of this invention, the workpiece sensing module comprises hardware and software components. The hardware component includes an RGB-D camera, fixedly mounted at the front of the operating space, to obtain RGB images, depth maps, and surface normal maps of the object. The software component is deployed in the MuJoCo simulation environment, randomly placing different types of parameterized workpieces. In each workpiece's XML model file, [the following is used to...]. <site>The system predefines keypoint names and 3D coordinates, assigns a functional label to each link in the workpiece model, and implements this by assigning different material names to different parts. Parts with the material name "GraspRegion" are classified as graspable areas during rendering. The workpiece's position and pose are randomly initialized. A standard image is acquired using the renderer, and the normal direction in the world coordinate system for each pixel is obtained. The 3D coordinates of each predefined keypoint in the current world coordinate system are directly queried via API. These 3D coordinates are projected onto the 2D image pixel coordinates using camera parameters. Based on the complete simulation state and the grasping keypoint information obtained from the projection, a grasping trajectory is planned. The simulation expert agent executes this trajectory during simulation. Predefined rule policies generate a successful grasping pose based on the workpiece's bounding box geometry and predefined keypoints, and record this pose as the grasping hand shape for the scene. A lightweight CNN classification network is used to classify the workpiece shape (circular, rectangular, or irregular), outputting its category label and converting it to one-hot encoding. This category information is input into the policy network as part of the task and environment information.

[0019] S5. The simulation strategy learning module inputs a parameterized URDF three-finger dexterous hand model into the Mujoco simulation environment. The strategy network is trained using imitation learning pre-training and reinforcement learning methods to obtain a converged strategy network. During the imitation learning pre-training process, a composite reward function is designed to reward proximity, contact, normal alignment, friction constraints, grasping quality, slip detection, lifting, and stability. Randomization of physical parameters is introduced to improve the robustness and generalization ability of the strategy. The converged strategy network takes robot body perception information, workpiece geometry information, fingertip-workpiece interaction information, and task environment information as input, and outputs eighteen-dimensional motion control commands for the robotic arm and three-finger dexterous hand to grasp the workpiece.

[0020] The simulation strategy learning module of this invention constructs a three-finger dexterous hand based on a parametric URDF model in the Mujoco simulation environment, and trains the strategy network through a combination of imitation learning and reinforcement learning. The strategy network takes robot body perception information, workpiece geometric information, fingertip-workpiece interaction information, and task and environment information as inputs, and outputs eighteen-dimensional motion control commands.

[0021] This invention takes into account that there is currently no parametric model like MANO that is unified, standardized, widely adopted, and specifically designed for "three-finger morphology," achieving the same level of precision and sophistication as MANO in the field of five-finger human hand models. Figure 2 As shown, this invention argues that rigid body models based on parametric URDF are currently the most effective and standardized solution. They provide sufficient parametric capabilities to adapt to different designs or for optimization, and directly serve the core needs of simulation and control. The rigid body model imported from parametric URDF contains accurate inertial properties, collision meshes, joint constraints, damping, friction, and other parameters.

[0022] like Figure 3 As shown, this invention uses a combination of imitation learning and reinforcement learning. First, using the high-quality demonstration dataset constructed in steps S2 and S3, the policy network is pre-trained using imitation learning to obtain a basic imitation policy. Its state space consists of robot body perception information, workpiece geometry and perception information, fingertip-workpiece interaction information, and task and environment information.

[0023] The robot's body perception information includes joint position, velocity, and torque.

[0024] The workpiece geometry and perception information includes geometric features and feature vector z extracted by the perception network. z is a feature vector extracted from the neural network backbone of the workpiece perception module. It encodes the visual and geometric features of the object and is the key to replacing the original point cloud data.

[0025] Fingertip-workpiece interaction information is a feature set used to quantify the fine physical interaction state during the gripping process, including the directed distance (SDF) between the fingertip and the workpiece, contact normal, friction tolerance, and gripping quality indicators. Directed distance d i The proximity between the fingertip and the workpiece surface is characterized. In simulation, this is obtained by querying the shortest directed distance from the geometric center of the fingertip link to the surface of the object model. This distance can be calculated using functions such as mjv_nearest provided by MuJoCo, and its return value is d. i This represents the precise distance value at the current moment. Contact normal n i Used to evaluate the geometric alignment quality of contact points, when the directed distance d i When the contact value is less than a preset contact threshold, contact is determined to have occurred. At this point, the unit normal vector of the object's surface at the contact point is obtained through the simulation interface, denoted as the contact normal n. i Friction tolerance μ_margin i The formula for real-time monitoring of slippage risk during capture is μ_margin. i =max( ),in and Let be the tangential and normal components of the force at the i-th contact point, respectively. A smaller friction coefficient indicates more sufficient frictional constraint and more stable gripping. The gripping quality index Q_wrench comprehensively evaluates the overall gripping quality from a mechanical perspective. It constructs the gripping matrix G in real time based on the spatial position and normal direction of all effective contact points and calculates its minimum singular value. As an indicator, Q_wrench, the larger the value, the better the force containment of the gripping and the stronger the anti-interference ability. Task and environmental information includes the pose of the robotic arm's end effector and the pose of the workpiece in the environment. The motion space is a combination of 12-DOF joint angle control of the three-finger dexterous hand and 6-DOF end effector pose control of the robotic arm, thereby realizing the gripping and manipulation of the workpiece within an 18-dimensional control command space.

[0026] The reward function for the pre-training phase is designed as follows: r = r approach + r contact + r align + r friction + r wrench + r slip + r lift + r stability +r pinch ; Where, r approach The function representing the reward for proximity to an object, r approach =-λ1· ;r contact This represents the contact reward function between the fingers of a three-finger dexterous hand and the contact surface of an object. Indicates the preset target distance; r contact =-λ2· , This parameter represents the width of the reward decay; r align The normal alignment reward function, r, represents the area of ​​contact between the finger and the object. align =λ3· , and Let r represent the normal vector at the i-th fingertip contact point and the normal vector at the object contact point, respectively; friction Let r represent the friction-constrained reward function for grasping. friction =-λ4· , and These represent the tangential and normal components of the force at the i-th contact point, respectively. r is the coefficient of friction; wrench This represents the crawling quality reward function, r. wrench =λ5·( ), Represents the minimum singular value of the capture matrix G; r slip This represents the reward function for detecting fingertip-object slippage, r. slip =-λ6· , r represents the relative tangential velocity at the i-th contact point; lift This represents the boosting reward function after the grab is formed, r lift =λ7·( ), and These represent the object's current height and initial height, respectively; r stability This represents the stability reward function after capture and boosting, r stability =-λ8·(| |+| |+| |), r represents the change in Euler angles of an object over a period of time. pinch This represents a composite reward function designed for the pinch operation, r. pinch =λ9·(| | | |)+λ 10 ·exp , and Let represent the unit normal vectors at the points of contact between the tips of the thumb and index finger, respectively. This indicates the actual pinching force obtained through the tactile sensor. Indicates the preset target pinching force; λ i This represents the weight assigned to each reward function.

[0027] like Figure 4 As shown, the policy network architecture and reinforcement learning training process are as follows: Policy function of policy network and value function All parameters are parameterized by a deep neural network. The policy network input layer receives a composite state space S_t, the feature extraction layer consists of three fully connected layers using the ReLU activation function to fuse and extract high-dimensional state features, and the output layer outputs the mean of a multidimensional Gaussian distribution. Its dimensions are the same as the 18-dimensional action space. The standard deviation of this Gaussian distribution... It is set as a learnable, independent parameter vector, and outputs a scalar representing an estimate of the value of the current state.

[0028] High-quality demonstration dataset generated in steps S2 and S3 Reinforcement-supervised learning is applied to the policy network, and the network parameters are initialized. The reinforcement-supervised learning loss function is: ; L bc This is the behavioral cloning loss function used in the imitation learning phase. State-action samples are taken from a demonstration dataset generated by predefined rule policies and simulation expert agents. The action output by the current policy network in the given state. The corresponding expert will demonstrate the action.

[0029] In the MuJoCo simulation environment, a three-finger dexterous hand interacts with the environment to collect experience data, which is stored in an experience replay buffer. The PPO algorithm is used to update the policy network by maximizing the clipped substitution objective function. [min( )]; in To optimize the objective function of the PPO-based shearing strategy, This represents the ratio of the probability of actions to the current policy and the probability of actions to the old policy under the given state. The value is the estimate of the dominance function calculated using the GAE method. This is a hyperparameter. Physical parameter randomization is introduced during training: initially, the object is fixed in a simple, easy-to-grasp location. As the policy performance improves, the randomness of placement is gradually increased. At the beginning of each training episode, the physical parameters in the simulation environment are randomly set, forcing the policy to learn robust grasping behavior effective under various physical and dynamic conditions, rather than overfitting to specific physical parameters of the simulation.

[0030] Once the policy network training converges, it can take the robot's own perception information, the geometric information of the workpiece to be grasped, the fingertip-workpiece interaction information, and the task environment information as inputs, and output eighteen-dimensional motion control commands for the robotic arm and the three-finger dexterous hand to grasp various types of workpieces.

[0031] To verify the effectiveness of the method of the present invention, a specific pinching operation scenario is set up, such as... Figure 5 As shown, in this scenario, the three-finger hand performs a pinching operation on the workpiece, and the specific process is as follows: The workpiece sensing module identifies the input RGB image. If the workpiece's envelope radius is less than a set threshold, it determines that a pinching operation is required, and the simulation strategy learning module then activates the pinching collaborative mode. For example... Figure 6 As shown, in this mode, the system assigns dynamic roles to the three fingers: the thumb and index finger act as the main pinching fingers, undertaking the primary gripping task; the middle finger acts as the stabilizing finger, adjusted to a slightly bent posture, ready to provide stable support under or to the side of the workpiece to prevent it from rolling. Figure 7 As shown, when the object is identified as flexible packaging, the main pinch finger moves to the high gripping point according to its original joint angle control, aligning its fingertip normal vectors relative to each other in the gripping space. The stabilizing finger actively supports the object, its joint angle no longer a fixed posture, but controlled to a position below the expected center of gravity migration path of the object. It should move in advance to the bottom or lower-middle part of the packaging bag to prepare to support the falling center of gravity. When the tactile sensor on the stabilizing finger detects a continuous increase in pressure, it elastically sinks within a certain range to buffer and absorb the impact of the downward shift of the center of gravity. At the same time, it generates an upward supporting force to actively support the downward shift of the center of gravity. Throughout the lifting process, the main pinch finger provides a stable gripping force, while the stabilizing finger acts like an active tray, finely adjusting the support force and position in real time through its tactile feedback, working together with the main pinch finger to form a dynamic balance system to resist the overturning torque caused by the change in the center of gravity.

[0032] The strategy network outputs a set of initial joint angle commands based on the position and orientation of the target workpiece, driving the dexterous hand to form a pre-position for grasping. The goal of this pre-position is to make the normal vector of the contact point between the fingertips of the thumb and index finger... and Align them as much as possible in space while keeping their directions opposite, i.e., maximize | | | The value of | is set close to -1. This geometric constraint is encoded in the normal alignment reward function r. align During the training phase, the policy network is guided to learn the correct angle output. All fingers are controlled by joint angle position, precisely executing pre-positioning. Upon contact, when the soft tactile sensors integrated into the thumb and fingertips detect pressure values ​​exceeding the contact threshold, the system triggers a control mode switch. For the dominant pinching fingers (thumb and index finger), a force control mode is switched. The target value of their joint angle is no longer fixed but dynamically generated by a force servo controller, which uses real-time force signals fed back from the sensors. With the expected pinching force Using the deviation as input, the control algorithm calculates the required joint angle increment in real time. The stabilizing mechanism controls the joint angle position of the middle finger to maintain its supporting posture. If the workpiece's posture changes during lifting, its angle can be fine-tuned via feedback from a vision or inertial measurement unit to continuously provide stable support.

[0033] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.< / site>

Claims

1. A three-finger dexterous hand grasping intelligent learning method for multiple types of workpieces, characterized in that, The method includes: S1. Build a grasping task scenario for the robotic arm and the three-finger dexterous hand, and configure the dataset acquisition module, workpiece perception module, and simulation strategy learning module; S2. The dataset acquisition module uses a two-layer mode of predefined rule strategies and simulation expert agents to acquire the dataset required for imitation learning. The predefined rule strategies provide stable and safe basic actions, while the simulation expert agents generate fine actions that adapt to complex scenarios and seamlessly switch to the predefined rule strategies when they fail. S3. Map the dexterous hand movements to the simulation, control the dexterous hand in the simulation to move and interact with objects in the simulation, and obtain a high-quality demonstration dataset. S4. The workpiece perception module acquires RGB images of the scene, extracts geometric features, functional area information and hand shape classification information of the workpiece, and realizes workpiece type recognition and gripping part labeling. S5. The simulation strategy learning module inputs a parameterized URDF three-finger dexterous hand model into the Mujoco simulation environment; the strategy network is trained through imitation learning pre-training and reinforcement learning to obtain a converged strategy network; during the imitation learning pre-training process, a composite reward function is designed to reward proximity, contact, normal alignment, friction constraints, grasping quality, slip detection, lifting and stability, and physical parameter randomization is introduced to improve the robustness and generalization ability of the strategy. The trained and converged policy network takes robot body perception information, workpiece geometry information to be grasped, fingertip-workpiece interaction information, and task environment information as input, and outputs 18-dimensional motion control commands for the robotic arm and three-finger dexterous hand to grasp the workpiece.

2. The intelligent learning method for three-finger dexterous hand grasping of multiple types of workpieces according to claim 1, characterized in that, In S2, the predefined rule strategy calculates a set of predefined gripping points for workpieces with regular geometric shapes based on the size and orientation of their minimum bounding box. Using the force closure theory, it prioritizes the selection of gripping point combinations that can form force closure gripping, and preliminarily evaluates the gripping quality by calculating the singular values ​​of the gripping matrix, filtering out obviously unstable gripping postures. The simulation expert agent plans and executes a motion trajectory for the robotic arm that is time-optimal, energy-lower, and meets complex dynamic constraints, realizing intelligent and human-like motion from the starting point to the grasping posture. It uses complete simulation environment state information and combines workpiece features extracted by the workpiece perception module to fine-tune the grasping posture.

3. The intelligent learning method for three-finger dexterous hand grasping of multiple types of workpieces according to claim 1, characterized in that, The workpiece sensing module in S4 includes hardware and software components. The hardware component includes an RGB-D camera, which is used to obtain RGB images, depth maps, and object surface normal maps. The software is deployed in the MuJoCo simulation environment. Different types of parametric workpieces are randomly placed. In the XML model file of each workpiece, the names and 3D coordinates of key points are predefined. A function tag is assigned to each link of the workpiece model. This is achieved by assigning different material names to different parts. Parts with the material name GraspRegion are classified as graspable areas during rendering. The position and pose of the workpiece are randomly initialized. A standard image is obtained using the renderer. The normal direction in the world coordinate system corresponding to each pixel is obtained. The 3D coordinates of each predefined key point in the current world coordinate system are directly queried through the API. These 3D coordinates are projected onto the pixel coordinates of the 2D image using camera parameters. Based on the complete simulation state and the grasping key point information obtained from the above projection, the grasping trajectory is planned. The simulation expert agent executes the trajectory in the simulation. The predefined rule strategy will generate a successful grasping pose based on the bounding box geometry of the workpiece and the predefined key points, and record this pose as the grasping hand shape of the scene.

4. The intelligent learning method for three-finger dexterous hand grasping of multiple types of workpieces according to claim 1, characterized in that, In S4, the workpiece perception module first employs a center-left-lower-center cropping strategy on the acquired RGB images, using 2 / 3 of the original image size as the cropping area. The cropped RGB image is then converted to a grayscale image, and the Otsu adaptive thresholding algorithm is used for binarization segmentation. The Otsu algorithm automatically determines the optimal segmentation threshold by maximizing the inter-class variance, adapting to workpiece recognition under different lighting conditions. Morphological closing operations are performed on the binarized image to fill holes, and morphological opening operations are used to remove noise, extracting the main contours of the workpiece. The extracted contours are then overlaid on the original cropped image as green lines with a line width of 2 pixels. This visualization result is used for debugging and effect verification to ensure the accuracy of contour extraction. The extracted contour features are combined with a classification network to identify the workpiece type, which includes circular, rectangular, or irregular shapes. The optimal grasping points of the workpiece are marked on the key points of interaction between the workpiece and the three-finger dexterous hand in the acquired workpiece RGB image data, and the corresponding parts of different types of objects are grasped with the optimal hand shape.

5. The intelligent learning method for three-finger dexterous hand grasping of multiple types of workpieces according to claim 1, characterized in that, In S5, the simulation strategy learning module first uses imitation learning pre-training to pre-train the strategy network to obtain the basic imitation strategy. The state space of the strategy network consists of robot body perception information, workpiece geometry and perception information, fingertip-workpiece interaction information, and task and environment information. The robot's body perception information includes joint position, velocity, and torque; The workpiece geometry and perception information includes geometric features and feature vector z extracted by the perception network. z is a feature vector extracted from the neural network backbone of the workpiece perception module, which encodes the visual and geometric features of the object. Fingertip-workpiece interaction information is a set of features used to quantify the fine physical interaction state during the gripping process, including the directional distance (SDF) between the fingertip and the workpiece, contact normal, friction tolerance, and gripping quality indicators. Task and environmental information includes the pose of the robotic arm's end effector and the pose of the workpiece in the environment; The motion space is a combination of 12-DOF joint angle control of a three-finger dexterous hand and 6-DOF end-effector pose control of a robotic arm, thereby enabling the grasping and manipulation of workpieces within an 18-dimensional control command space.

6. The intelligent learning method for three-finger dexterous hand grasping of multiple types of workpieces according to claim 1, characterized in that, In S5, the composite reward function is as follows: r = r approach + r contact + r align + r friction + r wrench + r slip + r lift + r stability +r pinch ； Where, r approach This represents the reward function for proximity to an object; r contact The contact reward function r represents the contact surface between the fingers of a three-finger dexterous hand and the object. align The normal alignment reward function, r, represents the area of ​​contact between the finger and the object. friction This represents the frictional constraint reward function for grasping; r wrench This represents the crawling quality reward function; r slip This represents the reward function for detecting fingertip-object slippage; r lift This represents the boosting reward function after the grab is formed; r stability This represents the stability reward function after capture and boosting; r pinch This represents a composite reward function designed for the pinch operation.

7. The intelligent learning method for three-finger dexterous hand grasping of multiple types of workpieces according to claim 1, characterized in that, In S5, reinforcement learning is used to train the policy network, specifically including: Using the high-quality demonstration dataset generated in step S3 Supervised learning is performed on the policy network to initialize the network parameters. In the MuJoCo simulation environment, the three-finger dexterous hand interacts with the environment to collect experience data and stores it in the experience playback buffer. The PPO algorithm is used to update the policy network by maximizing the clipped substitution objective function. During training, physical parameter randomization is introduced: In the early stage of training, the object is fixed in a simple and easy-to-grasp position; as the policy performance improves, the randomness of the placement is gradually increased. At the beginning of each training episode, the physical parameters in the simulation environment are randomly set, forcing the policy to learn robust grasping behavior that is effective under various physical dynamic conditions, rather than overfitting to the specific physical parameters of the simulation.

Images