Dexterous hand general grasping method and device based on single-view vision and demonstration optimization

Abstract

The application discloses a kind of based on single visual angle vision and demonstration optimization nimble hand general grabbing method and device, it is related to nimble hand grabbing and artificial intelligence field.The method first builds real scene and teleoperation equipment, collects the expert data of multiple sizes, shape object.Combined with the expert data collected, based on the algorithm after the improvement of DAPG algorithm, learn the general grasping strategy of cross object shape primitive in simulation environment in combination with domain randomization operation.The geometric characteristics of unknown object are analyzed in real time by first visual angle vision information, and initial grasping trajectory is generated, and grasping force and posture are adjusted by fusing dynamic feedback of force touch sensor, and high generalization and high reliability are grasped.The application solves the problems that traditional method relies on a large amount of real expert data, needs a large number of object models to train, strategy generalization is poor, and simulation migration stability is insufficient.

Description

Technical Field

[0001] This invention belongs to the field of dexterous hand grasping technology, specifically relating to a dexterous hand universal grasping method and device based on single-view vision and demonstration optimization. Background Technology

[0002] Stable grasping is a prerequisite for a robot's hand to complete all operational tasks. However, in practical applications, robots mostly operate in open environments, encountering many complex and unfamiliar objects. In the field of robotic grasping, although traditional analytical methods, learning strategies based on large amounts of pre-collected grasping data, and reinforcement learning through environmental interaction each have their strengths, they still face significant challenges and limitations in practical applications. Specifically, analytical methods rely on precise modeling of the geometric relationship between the object and the robotic arm, which is not flexible enough in the face of dynamic and changing factors encountered in open environments, making it difficult to handle grasping tasks involving unknown or novel objects.

[0003] While big data-based learning strategies can extract experience from past grasping examples, their generalization ability and adaptability are limited when encountering unfamiliar objects or scenarios, and the cost of acquiring, processing, and labeling large-scale datasets is extremely high. Reinforcement learning explores the optimal grasping strategy through continuous trial and error. Although it can theoretically learn and adapt autonomously, its learning efficiency is low, and trial and error in the physical world may bring safety risks and equipment damage.

[0004] In this context, imitation learning methods, with their ability to learn directly from human grasping behavior, offer an efficient and intuitive solution for robotic grasping technology. They effectively utilize human experience and intuition to quickly learn effective grasping actions, significantly accelerating learning speed and improving grasping success rates. However, even imitation learning faces challenges when dealing with a wide variety of unknown objects, especially in complex scenarios. The need for large amounts of labeled example data has become a bottleneck, limiting its widespread application. Summary of the Invention

[0005] To address these shortcomings, this invention proposes a dexterous hand universal grasping method and device based on single-view vision and demonstration optimization. It combines imitation learning methods with atomic actions of a domain-randomized grasping basic model, data dimensionality reduction techniques for visual information, and the introduction of a force feedback mechanism. This transforms the grasping task of complex, unknown new objects into a simplified task that the robot can handle with its already learned basic grasping actions, thus achieving data dimensionality reduction. This reduces reliance on novel and large amounts of data and accelerates the learning process. Simultaneously, by learning and applying meta-operations—a set of basic but widely applicable grasping skills—the robot can flexibly cope with various unknown grasping scenarios. Furthermore, through real-time force feedback, the robot can perceive contact forces and object characteristics during grasping actions, enabling immediate adjustments to the grasping strategy, compensating for information loss that may result from dimensionality reduction, and ensuring the accuracy and adaptability of the grasping actions. The technical solution of this disclosure is as follows:

[0006] According to the first aspect of this disclosure: a general grasping method for dexterous hands based on single-view vision and presentation optimization is provided, comprising:

[0007] S110, Construct a general shape primitive grasping strategy learning method, whose input is shape primitive information and output is the grasping trajectory of its shape primitive;

[0008] S120, construct a single-view unknown object visual information processing module, acquire its RGB-D information for unknown complex objects, process the RGB-D information by the single-view unknown object visual information processing module to obtain the combination of shape primitives and the information of each shape primitive, and input it into the general shape primitive grasping strategy set constructed in S110.

[0009] S130: The grasping part is determined, and the grasping trajectory of each shape element obtained in S120 is used to grasp the corresponding part of the object. The optimal grasping part is selected by comparing the grasping success rate.

[0010] S140: Perform real device grabbing. Based on the optimal grabbing position and grabbing trajectory evaluated in S130, and combined with the force feedback of the finger joints, perform real device grabbing.

[0011] According to a second aspect of this disclosure: a control method for a demonstration-optimized general-purpose grasping robot is provided, the control method being executed based on a general-purpose grasping robot control model trained on a single-view vision and demonstration-optimized dexterous hand general-purpose grasping method, the control method comprising:

[0012] S310, Obtain target task information, wherein the target task information includes information about the target object of the operation task to be performed;

[0013] S320, The target task information is input into the general grasping robot control model, and joint control signals are obtained from the general grasping robot control model;

[0014] S330, based on the joint control signal, control the general grasping robot to perform operation tasks on the target object.

[0015] According to a third aspect of this disclosure: a dexterous hand universal grasping device based on single-view vision and demonstration optimization is provided. The device is applied to a joint control model of a universal grasping robot used to demonstrate the robot performing operational tasks. The device includes:

[0016] The single-view unknown object visual information processing module is used to process visual data from a single-view image sensor, reconstruct the geometric information of the target object, perform point cloud extraction and geometric fitting on the depth data in the RGB-D image, identify the shape category, size parameters and pose parameters of the unknown object, and output the task state representation to provide a perceptual basis for subsequent strategy learning and grasping.

[0017] A general shape primitive grasping strategy learning module is used to learn grasping strategies that adapt to various geometric shapes. Combining expert demonstration data and domain randomized task instances, the initial strategy is first obtained through behavior cloning pre-training. Then, the interaction between the strategy and the task is executed in a simulation environment, experiential trajectories are collected, and a reinforcement learning objective function is constructed. The strategy parameters are optimized through the natural policy gradient and conjugate gradient methods, and finally a grasping strategy with generalization ability is obtained.

[0018] The grasping part discrimination module infers the suitable grasping parts for the dexterous hand based on the known geometric parameters of the object. By calculating the set of contact points on the object surface, combined with force closure constraints, grasping stability and hand structure limitations, it filters out the grasping point combinations that meet the conditions and outputs the corresponding finger configuration to drive the control system to grasp.

[0019] The real-machine grasping module deploys the learned grasping strategy on the actual robot platform, receives the output results of the strategy learning module and the grasping discrimination module, controls the robotic arm and dexterous hand to complete the approach, grasping and placement, monitors the contact state and error during the grasping process through sensor feedback, supports dynamic adjustment, and achieves stable and reliable object grasping.

[0020] According to a fourth aspect of this disclosure: a control device for a demonstration-optimized general-purpose grasping robot is provided, the control device being executed based on a general-purpose grasping robot control model, the model being trained based on the aforementioned dexterous hand general-purpose grasping method based on single-view vision and demonstration optimization, the control device comprising:

[0021] The acquisition unit is configured to acquire target task information, wherein the target task information includes information about the target object of the operation task to be performed;

[0022] The signal determination unit is configured to input the target task information into the general grasping robot control model and obtain joint control signals from the general grasping robot control model;

[0023] The control unit is configured to control the general-purpose grasping robot to perform an operation task on the target object based on the joint control signals.

[0024] According to a fifth aspect of this disclosure: an electronic device is provided, the electronic device comprising:

[0025] processor;

[0026] Memory used to store the processor's executable instructions.

[0027] When the processor executes the executable instructions, it causes the processor to execute a dexterous hand general grasping method based on single-view vision and demonstration optimization or a control method for a demonstration-optimized general grasping robot.

[0028] According to a sixth aspect of this disclosure: a demonstration-optimized general-purpose grasping robot is provided, the demonstration-optimized general-purpose grasping robot including an electronic device, or the demonstration-optimized general-purpose grasping robot being communicatively connected to the electronic device.

[0029] According to the seventh aspect of this disclosure: a computer program product is provided, including computer-executable instructions, which, when executed by at least one processor, implement a dexterous hand general grasping method based on single-view vision and demonstration optimization or a control method for a demonstration-optimized general grasping robot.

[0030] Beneficial effects:

[0031] This invention establishes a novel workflow that combines visual information and optimizations from simulation learning to grasp complex and unknown objects using only a small amount of expert data. The entire grasping scheme exhibits strong task generalization and data efficiency. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating a general grasping method for dexterous hands based on single-view vision and demonstration optimization according to exemplary embodiments of the present disclosure.

[0033] Figure 2 This is a schematic block diagram of a dexterous hand general grasping method based on single-view vision and demonstration optimization according to exemplary embodiments of the present disclosure;

[0034] Figure 3 This is a flowchart of a general shape primitive grasping strategy learning according to an exemplary embodiment of the present disclosure;

[0035] Figure 4 This is a flowchart illustrating an optimized control method for a general-purpose grasping robot according to exemplary embodiments of the present disclosure;

[0036] Figure 5 This is a schematic block diagram of a dexterous hand universal grasping device based on single-view vision and demonstration optimization according to exemplary embodiments of the present disclosure;

[0037] Figure 6 This is a schematic block diagram illustrating an optimized control device for a general-purpose grasping robot according to exemplary embodiments of the present disclosure;

[0038] Figure 7 This is a block diagram of an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0039] The present invention will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present invention and should not be construed as limiting the scope of protection of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention based on the above-described contents.

[0040] As mentioned earlier, the related technologies suffer from problems such as reliance on a large amount of real expert data, the need for extensive object model training, poor policy generalization, and insufficient simulation transfer stability, making it difficult to meet the requirements for autonomous grasping by robots.

[0041] Specifically, the analysis method relies on precise modeling of the geometric relationship between the object and the robotic arm, which is not flexible enough in the face of dynamic and variable factors encountered in open environments and is difficult to handle grasping tasks of unknown or novel objects.

[0042] While big data-based learning strategies can extract experience from past grasping examples, their generalization ability and adaptability are limited when encountering unfamiliar objects or scenarios, and the cost of acquiring, processing, and labeling large-scale datasets is extremely high. Reinforcement learning explores the optimal grasping strategy through continuous trial and error. Although it can theoretically learn and adapt autonomously, its learning efficiency is low, and trial and error in the physical world may bring safety risks and equipment damage.

[0043] In this context, imitation learning methods, with their ability to learn directly from human grasping behavior, offer an efficient and intuitive solution for robotic grasping technology. They effectively utilize human experience and intuition to quickly learn effective grasping actions, significantly accelerating learning speed and improving grasping success rates. However, even imitation learning faces challenges when dealing with a wide variety of unknown objects, especially in complex scenarios. The need for large amounts of labeled example data has become a bottleneck, limiting its widespread application.

[0044] In view of the foregoing, exemplary embodiments of this disclosure provide a dexterous hand universal grasping method based on single-view vision and demonstration optimization, a control method for a demonstration-optimized universal grasping robot, a dexterous hand universal grasping device based on single-view vision and demonstration optimization, a control device for a demonstration-optimized universal grasping robot, an electronic device, a computer-readable storage medium, and a computer program product, which can solve or at least alleviate the aforementioned problems.

[0045] In a first aspect of an exemplary embodiment of this disclosure, a general grasping method for dexterous hands based on single-view vision and demonstration optimization is provided.

[0046] The dexterous hand general grasping method based on single-view vision and demonstration optimization according to the exemplary embodiments of the present disclosure can be applied to the joint control model or joint controller of a general grasping robot. For example, training software can be loaded on the user terminal, and the user can input training instructions for the joint control model on the user terminal. The user terminal can obtain the trained joint control model of the general grasping robot by executing the dexterous hand general grasping method based on single-view vision and demonstration optimization according to the exemplary embodiments of the present disclosure.

[0047] Specifically, the joint control model of a general-purpose grasping robot can be used to control the robot to perform operational tasks. As an example, the joint control model can be set in the robot's processor, which can also be referred to as a learning device for a dexterous hand general-purpose grasping method based on single-view vision and demonstration optimization. This processor is connected to the robot's actuator, which may include a robotic arm and a dexterous hand. The processor can send joint control signals to the actuator, thereby controlling its motion state and enabling the robot's autonomous movement.

[0048] The aforementioned user terminal can be such as a tablet computer, a laptop computer, a digital assistant, etc. However, the implementation scenario of the above method is only an example scenario. The dexterous hand general grasping method based on single-view vision and demonstration optimization according to the exemplary embodiment of this disclosure can also be applied to other application scenarios. For example, it can also be that the user requests a training model from the server via the network on the user terminal (e.g., mobile phone, desktop computer, tablet computer, etc.). The server can complete the request by executing the dexterous hand general grasping method based on single-view vision and demonstration optimization according to the exemplary embodiment of this disclosure. Here, the server can be an independent server, a server cluster, a cloud computing platform, or a virtualization center.

[0049] According to the exemplary embodiments of this disclosure, the dexterous hand general grasping method based on single-view vision and demonstration optimization can perform dimensionality reduction processing on the object image in the single-view unknown object visual information processing module. The dimensionality-reduced shape primitive combination is input into the general shape primitive grasping strategy learning module to generate and acquire the grasping trajectory of each shape primitive. Then, the optimal grasping part is selected by the grasping part discrimination. Finally, the force feedback is combined in the real machine grasping module to reduce the error caused by image dimensionality reduction and real machine transfer. This can improve the success rate of performing operation tasks on the one hand, and improve the model's learning ability to perform operation tasks on novel objects that have not been learned before. The joint control model of the general grasping robot with strong robustness, high success rate and strong generalization can be obtained, so as to control the general grasping robot to meet the operation task requirements of various objects.

[0050] The following will refer to Figure 1 and Figure 2 This description describes a general grasping method for dexterous hands based on single-view vision and demonstration optimization according to embodiments of the present disclosure. Figure 1 This is a flowchart illustrating a general grasping method for dexterous hands based on single-view vision and demonstration optimization according to exemplary embodiments of the present disclosure. Figure 2 This is a schematic block diagram of a dexterous hand general grasping method based on single-view vision and demonstration optimization according to exemplary embodiments of the present disclosure. Figure 1 and Figure 2 As shown, the method includes the following steps:

[0051] S110 constructs a general shape primitive grasping strategy learning method, whose input is shape primitive information and output is the grasping trajectory of its shape primitive.

[0052] Figure 3 This is a flowchart illustrating the learning of a general shape primitive grasping strategy according to exemplary embodiments of the present disclosure. Figure 3 As shown, this step may include the following procedures:

[0053] S110-1, Constructing the Task Parameter Space The parameter space is used to uniformly represent the geometric properties and spatial states of various objects, providing standardized task input for subsequent grasping strategy training. Modeling each shape primitive to be grasped is taken as a task, with task parameters being triples. ,in, Represents an object task instance:

[0054] The The category label represents the shape primitive, indicating which geometric category the shape primitive belongs to, such as cuboid, cylinder, sphere, etc., belonging to discrete sets. ;

[0055] The Represents a vector of size parameters, each Indicates the first The object in the first Dimensional information in each dimension;

[0056] The Represents the six-dimensional spatial pose of an object, where Represents the spatial position vector of a shape primitive. Its spatial coordinate axes are , , , Rotation angle of shape primitive , respectively around , , Rotation of the axis, measured in radians or degrees.

[0057] All task instances constitute the task parameter space:

[0058] ,

[0059] The task parameter space is used for unified modeling of the input structure of the driving policy network and provides conditional support for the training and generalization of the grasping policy.

[0060] S110-2, Collect expert datasets: Select multiple task instances in the task space, generate corresponding state-action pairs through the teleoperation system, and construct an expert demonstration dataset. Step S110-2 includes the following:

[0061] S110-2-1, Initialize the environment, treating the crawling problem as a Markov decision process, defined by tuples:

[0062] ,

[0063] in, and Representing the state and action spaces respectively. It is a reward function used to measure task progress. It is a dynamic transition, possessing randomness. It is a discount factor.

[0064] S110-2-2: Perform geometric analysis on the dexterous hand used to calculate the maximum grasping size of the dexterous hand:

[0065] ,

[0066] in, and These represent the spatial coordinates of the two contact points when any two fingers of a dexterous hand form a closed force. This indicates the crawling process. Indicates the maximum size that can be grabbed. and It refers to any two different fingers out of the five fingers.

[0067] S110-2-3: Sample the task parameters in S110-1 to obtain task data, which serves as the shape primitives to be captured in the expert data, where the shape category is... The shapes involve basic shapes such as cuboids, cylinders, and spheres. It is a shape category parameter. It represents the set of general shape primitives, that is, the collection of all categories of basic geometric shapes that are considered. It is the first in the set Each shape category Represents a specific geometric shape, using random sampling; dimensional parameters Different shapes have different dimensions; for example, a sphere has one dimension, a cylinder has two dimensions, and a cuboid has three dimensions. Representing the first The first object The parameters in each dimension, including the size parameter, are also sampled randomly. , Indicates 0- Uniform sampling of the range The maximum grasping size obtained in S110-2-2; the pose parameters are... , Include and There are 6 parameters in total. Indicates the position of the target object. , , ,in Within the workspace axis, axis, The maximum and minimum values ​​of the axis. This indicates the orientation of the target object, that is, its rotational attitude in three-dimensional space. , , The sampling method is uniform sampling.

[0068] S110-2-4: Based on the task data sampled in S110-2-3, use the teleoperation system to operate the robot to collect the grasping trajectory of the object to be grasped in the task, including: operating the robotic arm and dexterous hand to perform the grasping task, collecting the grasping trajectory, and defining the grasping trajectory as:

[0069] ,

[0070] in, It is the joint angle of the robotic arm at time t. It is the joint angle of a dexterous hand in time t. It refers to the trajectory length; multiple collected trajectories form the captured trajectory dataset.

[0071] ,

[0072] The dataset contains objects of various shapes and sizes, among which These are N trajectories collected by the remote operating system. For the collected first There are trajectories, where N is the number of trajectories.

[0073] S110-2-5: Reproduce the collected grasping trajectory in a simulation environment and record the dataset. :

[0074] ,

[0075] in, For expert datasets, Let be the state at time t. This represents the action of the agent at time t. Indicates that an action is being performed. The state at time t+1. The reward at time t.

[0076] S110-3, Pre-policy learning: Training a policy network based on the expert demonstration dataset. Imitation learning is performed to minimize the behavioral cloning loss between the predicted action and the expert action, thus obtaining the initial policy parameters. S110-3 includes the following steps:

[0077] S110-3-1, Proposed Training Policy Network , indicating the state Next task action , It is a set of learnable parameters.

[0078] S110-3-2, Define the supervised learning loss function, for... Perform supervised learning to minimize the behavior cloning loss:

[0079] ,

[0080] in, It is the behavioral cloning loss function. These are policy network parameters. It is an expert demonstration dataset. Is the policy network in state and tasks Making predictions It is a task Chinese experts are in a state of readiness The actual actions below, The mean squared error (MSE) between two actions is used as a loss metric.

[0081] S110-3-2, Minimize the behavioral cloning loss function Continuously update network parameters , so that:

[0082] ,

[0083] Conditions met: ,

[0084] in The behavior cloning loss for the Kth iteration number, The behavior cloning loss is for the (K-1)th iteration. A small positive value, meaning when the loss between two iterations is less than When the optimal strategy is obtained, the optimal strategy can be obtained.

[0085] The final iteration result is used as the initialization strategy parameter. ,use Policy network Initialize as a pre-policy .

[0086] S110-4, Domain Randomization, Task Parameter Space Constructed from S110-1 Domain randomization sampling is performed on the task parameters. The parameters involved, shape categories Dimensions , and pose parameters Perform perturbation sampling to generate task parameter instances. .

[0087] S110-5, Reward Reshaping: A reward reshaping mechanism is introduced during the reinforcement learning phase. The reshaping rewards include, but are not limited to, auxiliary reward items such as grasping stability and target position error penalty. S110-5 includes the following:

[0088] S110-5-1, a hand approaching an object is defined as:

[0089] ,

[0090] in, It's the reward for the stage where the hand touches the object. It is a correction value. It is the three-dimensional coordinate of the center of the hand. These are the three-dimensional coordinates of the object's center. This indicates the calculation of the Euclidean distance between two locations. This represents the Euclidean distance between the hand and the object.

[0091] S110-5-2, a finger approaching an object is defined as:

[0092] ,

[0093] in, It's a reward for the stage where the finger approaches the object. It is a correction value. It refers to the position of the fingers in a dexterous hand. It can be any one of the five fingers. This represents the sum of the distances between all fingers and the object. This indicates the calculation of the Euclidean distance between two locations. This represents the distance between an object and a particular finger. This represents a constant value, indicating that the distance between the hand and the object is within a certain range. Not zero.

[0094] S110-5-3, the position of an object near a target is defined as:

[0095] ,

[0096] in, It is the reward for the phase when the object approaches the target position. This is a correction value. Represents the Euclidean distance between the object and the target position. The reward direction factor is as follows:

[0097] ,

[0098] in, Let be the Euclidean distance between the object and the target positions at step t. Let be the Euclidean distance between the object and the target positions at step t-1. This means that when the distance between the object and the target position at step t is closer than the distance at step t-1, =1, A positive value is defined as the distance between the object and the target position at step t being the same as the distance at step t-1. =0, The value is 0 when the distance between the object and the target position at step t is greater than the distance at step t-1. =-1, It is a negative value;

[0099] S110-5-4, the punishment is defined as:

[0100] ,

[0101] in, This represents a heavy penalty in the design of rewards. It is a very large negative value;

[0102] In the first punishment, Represents the height of the palm. The height of the hand is indicated by the height of the palm, which is higher than the height of the back of the hand. This is to ensure that the palm is facing down during the grasping process, so as to prevent the object from being placed in the target position by holding it or other means.

[0103] In the second punishment, This indicates the center of the object; a penalty is imposed when the hand's height is lower than the object's height to ensure the hand grasps the object and prevent the hand from imparting speed to the object from below.

[0104] In the third punishment, This indicates the height of the tabletop; a penalty is imposed when the hand's height is lower than the tabletop height to ensure that the object does not fall below the tabletop.

[0105] In the fourth punishment, This indicates a height slightly above the desktop. Indicates a smaller constant value. This represents the Euclidean distance between the hand and the object. It is a penalty imposed when the hand and the object are far apart when the target is slightly above the table, in order to prevent the hand from separating from the object when it tries to grab the object off the table.

[0106] No punishment will be given in other circumstances.

[0107] S110-5-5, based on the reward design of S110-5-1-S110-5-4, the reward is defined as follows:

[0108] ,

[0109] in, To reinforce rewards in learning, The reward is for bringing the hand close to the object. Rewards are given when the finger approaches the object. This indicates the reward for the phase where the object approaches the target position. This is a penalty item.

[0110] S110-6, Reinforcement learning, utilizes the policy network to interact with the environment on task instances after domain randomization, and introduces a reward reshaping mechanism. The policy parameters are optimized using a reinforcement learning algorithm, thereby improving the policy's generalization ability and training stability on unseen tasks. The steps of S110-6 are as follows:

[0111] S110-6-1, Load the pre-policy obtained in the imitation learning stage of S110-3. As an initial strategy;

[0112] S110-6-2, Task instance generated based on domain randomization in S110-4 This allows it to interact with the environment and be recorded. ;

[0113] S110-6-3, based on the initial policy of S110-2, introduces a hybrid policy gradient update policy. The overall objective function is:

[0114] ,

[0115] in: It is the total learning loss. It is a reinforcement learning loss, using the natural policy gradient approximation. The initial behavior cloning loss coefficient, As the attenuation factor, k is the number of iterations;

[0116] ,

[0117] in, It is the reinforcement learning loss function. It is a strategy In state Task The following actions The probability, The advantage function is a weighted term in the policy gradient calculation, used to measure the extent to which the current action is superior to the policy's average behavior; for the advantage function , Indicates the state Next action The difference between the expected return that can be obtained and the expected return of the strategy in this state is defined as:

[0118] ,

[0119] Indicates in the task medium state Next, according to the strategy The average return that can be obtained Indicates in The subsequent expected return is defined as:

[0120] ,

[0121] in, Indicates in the task medium state Next action The reward obtained is calculated according to the reward function designed in S110-5, and the data used in the calculation is recorded in S110-6-2. Provided; The discount factor has a value range of [0, 1]. Indicates the execution of an action After that, the environment transitions to the next state; It is a functional distribution, in the task medium state Next action The transition probabilities of all possible states; express Obedience in the task medium state Next action The subsequent state transition probability distribution; Indicates in the task medium state Next action After all possible probability Perform expectation calculations to estimate the long-term value of the current action in the future; Indicates the state Next, according to the strategy Average return that can be obtained:

[0122] ,

[0123] in, Representation Strategy In the mission medium state All actions below; Representation Strategy In the mission medium state Sampling in the action distribution below ; Indicates the strategy In the mission medium state The mathematical expectation of the generated action probability distribution;

[0124] S110-6-4, Construct an approximate natural gradient update equation:

[0125] With the current strategy parameters as Under the given conditions, let the gradient of the objective function be:

[0126] ,

[0127] in, The loss function for the hybrid learning policy constructed in S110-6-3, for Regarding strategy parameters The gradient;

[0128] Construct the following constrained optimization problem, where the optimization objective is to maximize the inner product of the gradient direction and the update direction, and the constraint is that the quadratic form of this direction under the Fisher information matrix does not exceed a preset threshold:

[0129] ,

[0130] ,

[0131] in, ,

[0132] in, It is the gradient function of the target. represents the update direction of the policy parameters; F is the Fisher information matrix, which measures the local geometric information in the policy parameter space. It is a strategy In state Task Take action below The probability, The log probability of the policy is expressed with respect to the parameters. gradient, Represents all possible states and corresponding actions Calculate the weighted average. It is a preset KL divergence limit threshold used to control the magnitude of single-step updates and prevent excessive changes in policy parameters.

[0133] From the above, we can conclude that , Step size;

[0134] S110-6-5, Update strategy parameters:

[0135] The natural gradient direction is solved using the conjugate gradient (CG) method to ensure the stability and convergence of the training process. The update expression for the policy parameters is as follows:

[0136] ,

[0137] in, These are the policy parameters for the (k+1)th iteration. The strategy parameters are for the k-th iteration. The gradient of the objective function constructed in S110-6-4, It is the natural gradient direction, a direction vector that takes into account the geometry of the policy space. The step size is expressed as follows:

[0138] ,

[0139] in, >0 indicates the step size for parameter updates, which depends on the update direction and the quadratic form of the Fisher information matrix, used to ensure that the parameter updates satisfy the KL divergence constraint. Denominator Measure the curvature or change cost of the update direction in the parameter space. It is the preset KL divergence limit threshold.

[0140] S110-6-6, iterative execution until convergence:

[0141] Repeat steps 110-6-2 to 110-6-5 until the average return converges, thus obtaining the final policy parameters. To form a general shape primitive grasping strategy .

[0142] S120: Construct a single-view unknown object visual information processing module to obtain the RGB-D information of the unknown complex object. The RGB-D information is processed by the single-view unknown object visual information processing module to obtain the combination of shape primitives and the information of each shape primitive, which is then input into the general shape primitive grasping strategy set constructed in S110.

[0143] S120 single-view unknown object visual information processing includes:

[0144] S120-1: Acquire RGB-D information of unfamiliar, unknown objects. Using camera intrinsics obtained beforehand through Halcon calibration, prepare for the transformation from image coordinates to camera coordinates. Acquire RGB-D information of objects on the table using a depth camera. ;

[0145] S120-2: Acquire point cloud information and use an image-depth fusion processing method to map each depth pixel to a 3D point, along with corresponding color information, to reconstruct the dense point cloud of objects in the scene. ;

[0146] S120-3, Segment the object: Segment the point cloud of the unfamiliar, unknown object obtained in S120-2. The image was processed using RANSAC and SAM image processing methods, and segmented into m relatively simple parts, where the first... The point cloud information of each part is .

[0147] The segmented objects in S120-3 include:

[0148] S120-3-1, by identifying the desktop (plane) using RANSAK and removing the background, the single-view point cloud information of the object to be grasped is obtained. Let this be... ;

[0149] S120-3-2 uses SAM-6D to segment the point cloud information of objects, thus segmenting complex objects. Divide into simpler parts , ,in, The number of segments for the object.

[0150] S120-4, fitted into shape primitives, using point cloud information of the segmented simple parts. The RANSAC method is used for fitting to obtain the fitted shape primitives. Status information in, It is the fitted first State information of each shape primitive. It is the first Shape categories of shape primitives For the first Size information of each shape primitive No. Pose information of each shape primitive;

[0151] S120-5: Obtain the combination of shape primitives, and then fit the first... Shape primitives According to its status information In a simulation environment, the fitted m shape primitives are placed and combined sequentially to obtain a shape primitive combination. .

[0152] S130: The grasping part is determined, and the grasping trajectory of each shape element obtained in S120 is used to grasp the corresponding part of the object. The optimal grasping part is selected by comparing the grasping success rate.

[0153] The S130's grasping part determination includes:

[0154] S130-1, Obtain the grasping strategy for each shape primitive and combine the shape primitives. Various shape primitives Status information As input, the general shape primitive grasping strategy obtained in S110-6 is used. The strategy for grasping the shape primitives of each part. ;

[0155] S130-2, Grasping under Simulation Environment: General Shape Primitive Grasping Strategy For each shape primitive Perform n grabbing operations in a simulation environment to generate n grabbing trajectories. , For shape primitives The generated first There are 10 capture trajectories, assuming a total of 1000 trajectories. Each section was successfully crawled, with a success rate of [percentage missing]. :

[0156] ,

[0157] in, To determine the number of times each shape primitive is grasped. For shape primitives Scraping The number of successes in the subshape primitive.

[0158] S130-3, Compare the success rate of grabbing different parts; the success rate of grabbing each part is... Compare the success rates of each capture, and select the part with the highest success rate as the capture part. The formula is:

[0159] ,

[0160] Combination of shape primitives The shape primitive with the highest success rate in grabbing.

[0161] S130-4, Select the optimal grasping part, and select the k-th shape primitive with the highest success rate. The corresponding local area of ​​the real object is determined as the actual grasping part.

[0162] S140; Perform real-device grabbing, using the optimal grabbing position and grabbing trajectory evaluated in S130, combined with the force feedback of the finger joints to perform real-device grabbing.

[0163] Real device capture in S140 includes:

[0164] S140-1, Select the grab trajectory for shape primitives. Select the successfully captured trajectory generated in S130-2 The trajectory of the kth part of the grasped object;

[0165] S140-2, The grab trajectory is transferred to the real device, based on the grab trajectory generated in S140-1. Capture data on a real device.

[0166] S140-3, force feedback data acquisition: a force feedback sensor for the joints or fingertips of a dexterous hand, acquiring force feedback data in real time. , Corresponding to the five fingers;

[0167] S140-4, the action stops when the force threshold is reached, and the action stops when the following conditions are met:

[0168] and,

[0169] any ,

[0170] Finger force threshold Force feedback from the thumb, This provides force feedback for the remaining four fingers.

[0171] S140-5 completes the grasping process. Through a single-view depth vision system integrated into the end of the robotic arm, the position of the target object is observed and analyzed in real time, and the success of the grasp is determined from a visual perspective.

[0172] In a second aspect of exemplary embodiments of this disclosure, a control method for demonstrating and optimizing a general-purpose grasping robot is provided. This control method is executed based on a general-purpose grasping robot control model trained using a single-view unknown object general policy learning method. Figure 4 As shown, the control method may include the following steps:

[0173] In step S310, target task information is obtained, wherein the target task information includes information about the target object of the operation task to be performed;

[0174] In step S320, the target task information is input into the general grasping robot control model, and joint control signals are obtained from the general grasping robot model;

[0175] In step S330, based on the joint control signal, the general-purpose grasping robot is controlled to perform an operation task on the target object.

[0176] This control method can be applied to the processor of a general-purpose grasping robot. The processor can be connected to the actuator of the general-purpose grasping robot. The actuator can include a dexterous hand and a robotic arm. The processor can send joint control signals to the actuator to control the motion state of the actuator and realize the autonomous movement of the general-purpose grasping robot.

[0177] The specific steps and details of this control method can be implemented and understood in a similar manner to the dexterous hand general grasping method based on single-view vision and demonstration optimization described in the first aspect above. Accordingly, this control method has the same or similar beneficial effects as the dexterous hand general grasping method based on single-view vision and demonstration optimization described above, and will not be repeated here.

[0178] In a third aspect of the exemplary embodiments of this disclosure, a learning device for a dexterous hand universal grasping method based on single-view vision and demonstration optimization is provided. The device is applied to a joint control model of a universal grasping robot for demonstration optimization. The joint control model is used to demonstrate the universal grasping robot performing operational tasks, such as... Figure 5 As shown, the device may include a single-view unknown object visual information processing module 410, a general shape primitive grasping strategy learning module 420, a grasping part discrimination module 430, and a real machine grasping module 440.

[0179] The single-view unknown object visual information processing module 410 processes visual data from a single-view image sensor to reconstruct the geometric information of the target object. By performing point cloud extraction and geometric fitting on the depth data in the RGB-D image, it identifies the shape category (such as sphere, cylinder, or cuboid), size parameters, and pose parameters of the unknown object, and outputs a task state representation, providing a perceptual foundation for subsequent policy learning and grasping.

[0180] The general shape primitive grasping strategy learning module 420 is used to learn grasping strategies that adapt to various geometric shapes. Combining expert demonstration data and domain randomized task instances, it first obtains an initial strategy through behavior cloning pre-training, then executes the interaction between the strategy and the task in a simulation environment, collects experience trajectories and constructs a reinforcement learning objective function, optimizes the strategy parameters through the natural policy gradient and conjugate gradient methods, and finally obtains a grasping strategy with generalization ability.

[0181] The grasping part determination module 430 infers the suitable grasping parts for a dexterous hand based on the known geometric parameters of the object. By calculating the set of contact points on the object surface, combined with force closure constraints, grasping stability, and hand structure limitations, it selects grasping point combinations that meet the conditions and outputs the corresponding finger configuration to drive the control system for grasping.

[0182] The actual robot grasping module 440 deploys the learned grasping strategy onto the actual robot platform. It receives the output results from the strategy learning module and the grasping discrimination module, and controls the robotic arm and dexterous hand to complete operations such as approaching, grasping, and placing. Sensor feedback monitors the contact state and error during the grasping process, supports dynamic adjustments, and achieves stable and reliable object grasping.

[0183] As an example, the single-view unknown object visual information processing module 410 is configured to: extract image data from a monocular RGB-D camera, obtain the object point cloud contour through image segmentation and depth filtering, fit the point cloud using the RANSAC (Random Sample Consensus) method, determine whether the target object belongs to a sphere, cylinder, or cuboid, and further extract the corresponding set of size parameters. The pose parameters are , Include and There are 6 parameters in total. , , ,in Within the workspace axis, axis, The maximum and minimum values ​​of the axes. The fitted information is output as the task status. This provides a perceptual foundation for subsequent strategy learning and grasping.

[0184] As an example, the general shape primitive grasping strategy learning module 420 is configured to: upon receiving task state parameters output by the visual information processing module... Next, an expert dataset is constructed from the expert demonstration data. For policy networks Perform behavioral clone training by minimizing the mean squared error loss. The initial policy parameters are obtained. Subsequently, the task parameters are sampled using domain randomization in the simulation environment to generate various variant task instances, which interact with the environment to collect policy trajectories. A hybrid loss function is then constructed. , which indicates Based on the reinforcement learning loss using the advantage function, an optimization problem is constructed with the objective of maximizing the inner product of the gradient direction and the update direction, constrained by the condition that this direction lies within the Fisher information matrix. The quadratic form below does not exceed the preset threshold. , It is a preset KL divergence limit threshold, optimizing the strategy parameters. Ultimately, it learns a grasping strategy that can generalize to multiple tasks. .

[0185] As an example, the grasping part discrimination module 430 is configured to combine the shape primitives output by the single-view unknown object visual information processing module 410. Various shape primitives Status information As input, the general shape primitive grasping strategy obtained by the general shape primitive grasping strategy learning module 420 The strategy for grasping the shape primitives of each part. In a simulated environment, the crawling success rate of each part is obtained. Compare the success rates of grabbing, select the part with the highest success rate as the grabbing part, and obtain... Combination of shape primitives The shape primitive with the highest success rate in grabbing Combination of shape primitives The k-th shape primitive is selected as the k-th part of the object to be grasped.

[0186] As an example, the real device grasping module 440 is configured to: grasp the intended grasping part output by the grasping part discrimination module 430 and the general shape primitive grasping strategy obtained by the general shape primitive grasping strategy learning module 420. The robot generates a grasping action sequence, and the real-machine grasping module 440 deploys the grasping strategy into the robot control system. This module controls the robotic arm to guide the dexterous hand to approach the target object and execute the grasping operation according to the planned grasping trajectory. At the same time, the module integrates real-time feedback from force sensors and position sensors to determine whether the grasping contact is successful. This ensures the stability and success rate of the entire grasping process.

[0187] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs the operation has been described in detail in the method embodiments. Each unit in the dexterous hand general grasping method learning device based on single-view vision and demonstration optimization can perform the corresponding steps in the method according to the dexterous hand general grasping method learning method based on single-view vision and demonstration optimization in the method embodiments of the first aspect above, which will not be described in detail here.

[0188] In a fourth aspect of exemplary embodiments of this disclosure, a control device for demonstrating and optimizing a general-purpose grasping robot is provided, such as... Figure 6 As shown, the control device may include an acquisition unit 510, a signal determination unit 520, and a control unit 530.

[0189] The acquisition unit 510 is configured to acquire target task information, wherein the target task information includes information about the target object to be operated; the signal determination unit 520 is configured to input the target task information into a general grasping robot control model and obtain joint control signals from the general grasping robot control model; the control unit 530 is configured to control the general grasping robot to perform an operation task on the target object based on the joint control signals. The general grasping robot control model is trained according to the dexterous hand general grasping method learning method based on single-view vision and demonstration optimization according to embodiments of this disclosure.

[0190] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0191] In a fifth aspect of exemplary embodiments of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing processor-executable instructions, wherein, when executed by the processor, the processor-executable instructions cause the processor to perform a single-view unknown object-based general policy learning method or a demonstration and optimization control method for a general grasping robot according to embodiments of the present disclosure.

[0192] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include, for example, a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the general policy learning method based on a single-view unknown object according to embodiments of the present disclosure or to demonstrate and optimize the control method for a general grasping robot.

[0193] As an example, an electronic device does not necessarily have to be a single device; it can also be a collection of any means or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. An electronic device can also be part of an integrated control system or system manager, or can be configured to interface with a server, either locally or remotely (e.g., via wireless transmission).

[0194] In an electronic device, processor 610 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor 610 may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0195] The processor 610 can execute instructions or code stored in the memory 630, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0196] The memory 630 may be integrated with the processor 610, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 630 may include a separate device, such as an external disk drive, a storage array, or other storage device that can be used by any database system. The memory 630 and the processor 610 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 610 to read files stored in the memory 630.

[0197] In addition, electronic devices may include video displays (such as liquid crystal displays) and user interaction interfaces (such as keyboards, mice, touch input devices, etc.). All components of the electronic device can be interconnected via buses and / or networks.

[0198] In a sixth aspect of exemplary embodiments of the present disclosure, a demonstration-optimized general-purpose grasping robot is provided, which may include the electronic device described in the present disclosure, or the demonstration-optimized general-purpose grasping robot may be communicatively connected to the electronic device described in the present disclosure.

[0199] For example, the demonstration optimized general-purpose grasping robot itself is equipped with electronic devices to execute the control method of the demonstration optimized general-purpose grasping robot according to the embodiments of this disclosure; as another example, the demonstration optimized general-purpose grasping robot can be communicatively connected to electronic devices such as servers to receive control commands from electronic devices and / or send real-time poses to electronic devices, etc.

[0200] In a seventh aspect of exemplary embodiments of the present disclosure, a computer-readable storage medium is provided, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform a learning method for a dexterous hand general grasping method based on single-view vision and demonstration optimization, or a control method for a demonstration-optimized general grasping robot according to embodiments of the present disclosure.

[0201] For example, when the logical instructions in the aforementioned memory 630 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this disclosure, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0202] Computer-readable storage media may be, for example, a memory that includes instructions. Optionally, computer-readable storage media may be: read-only memory (ROM), random access memory (RAM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0203] In addition, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, enables the processor to perform a learning method for a dexterous hand general grasping method based on single-view vision and demonstration optimization, or a control method for a demonstration-optimized general grasping robot according to embodiments of the present disclosure.

[0204] In an eighth aspect of exemplary embodiments of the present disclosure, a computer program product is provided, including computer-executable instructions that, when executed by at least one processor, implement a learning method for a dexterous hand general grasping method based on single-view vision and demonstration optimization, or a control method for a demonstration-optimized general grasping robot according to embodiments of the present disclosure.

[0205] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0206] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0207] This invention establishes a novel workflow that combines single-view visual information with optimizations from simulation learning, enabling the grasping of complex and unknown objects using only a small amount of expert data. The entire grasping scheme exhibits strong task generalization and data efficiency.

[0208] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0209] Although the invention has been described with respect to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes, and not for the purpose of explaining or limiting the subject matter of the invention.

Claims

1. A general grasping method for dexterous hands based on single-view vision and demonstration optimization, characterized in that, include: S110, Construct a general shape primitive grasping strategy learning method, whose input is shape primitive information and output is the grasping trajectory of its shape primitive; S120, construct a single-view unknown object visual information processing module, acquire its RGB-D information for unknown complex objects, process the RGB-D information by the single-view unknown object visual information processing module to obtain the combination of shape primitives and the information of each shape primitive, and input it into the general shape primitive grasping strategy set constructed in S110. S130: The grasping part is determined, and the grasping trajectory of each shape element obtained in S120 is used to grasp the corresponding part of the object. The optimal grasping part is selected by comparing the grasping success rate. S140, execute real device grabbing, using the optimal grabbing part and grabbing trajectory obtained from the evaluation in S130, combined with the force feedback of the finger joints to execute real device grabbing; S110 includes: S110-1, Constructing the Task Parameter Space Modeling each shape primitive to be grasped is taken as a task, and the task parameters are triples. ,in, Represents an object task instance: The The category label of a shape primitive indicates which geometric category the shape primitive belongs to, and whether it belongs to a discrete set. ; The Represents a vector of size parameters, each Indicates the first The object in the first Dimensional information in each dimension; The Represents the six-dimensional spatial pose of an object, where Represents the spatial position vector of a shape primitive. Its spatial coordinate axes are , , , Rotation angle of shape primitive , respectively around , , Rotation of the axis, measured in radians or degrees; All task instances constitute the task parameter space: ； S110-2, Collect expert datasets: Select multiple task instances in the task parameter space, generate corresponding state-action pairs through the teleoperation system, and construct an expert demonstration dataset. ; S110-3, Pre-policy learning: Training a policy network based on the expert demonstration dataset. Imitation learning is performed to minimize the behavioral cloning loss between the predicted action and the expert action, thus obtaining the initial policy parameters. ; S110-4, Domain Randomization: Task parameter space constructed from S110-1 Domain randomization sampling is performed on the task parameters. The parameters involved include shape category c and size parameters. and pose parameters Perform perturbation sampling to generate task parameter instances. ; S110-5, Reward Reshaping: A reward reshaping mechanism is introduced during the reinforcement learning phase, where the rewards include grasping stability and target position error penalty; S110-6, Reinforcement Learning: The policy network is used to interact with the environment on a task instance after domain randomization, and a reward reshaping mechanism is introduced to optimize the policy parameters through a reinforcement learning algorithm. S120 single-view unknown object visual information processing includes: S120-1: Acquire RGB-D information of unfamiliar, unknown objects. Using camera intrinsics obtained beforehand through Halcon calibration, prepare for the transformation from image coordinates to camera coordinates. Acquire RGB-D information of objects on the table using a depth camera. ; S120-2: Acquire point cloud information and use an image-depth fusion processing method to map each depth pixel to a 3D point, along with corresponding color information, to reconstruct the dense point cloud of objects in the scene. ; S120-3, Segment the object: Segment the point cloud of the unfamiliar, unknown object obtained in S120-2. The image was processed using RANSAC and SAM image processing methods, and segmented into m relatively simple parts, where the first... The point cloud information of each part is ; S120-4, fitted into shape primitives, using point cloud information of the segmented simple parts. The RANSAC method is used for fitting to obtain the fitted shape primitives. Status information ,in, It is the fitted first State information of each shape primitive. It is the first Shape categories of shape primitives For the first Size information of each shape primitive No. Pose information of each shape primitive; S120-5: Obtain the combination of shape primitives, and then fit the first... Shape primitives According to its status information In a simulation environment, the fitted m shape primitives are placed and combined sequentially to obtain a shape primitive combination. ; The S130's grasping part determination includes: S130-1, Obtain the grasping strategy for each shape primitive and combine the shape primitives. Various shape primitives Status information As input, the general shape primitive grasping strategy obtained in S110-6 is used. The shape primitives of each part are formed by the grasping strategy. ; S130-2, Grasping Strategy for Shape Primitives in Simulation Environment For each shape primitive Perform n grabbing operations in a simulation environment to generate n grabbing trajectories. , For shape primitives The generated first There are 10 capture trajectories, assuming a total of 1000 trajectories. Each section was successfully crawled, with a success rate of [percentage missing]. : ， in, To determine the number of times each shape primitive is grasped. For shape primitives Scraping The number of successes in the sub-shape primitive; S130-3, Compare the success rate of grabbing different parts; the success rate of grabbing each part is... Compare the success rates of each capture, and select the part with the highest success rate as the capture part. The formula is: ， Combination of shape primitives The shape primitive with the highest success rate in grabbing; S130-4, Select the optimal gripping area, set Combination of shape primitives The k-th shape primitive is used to select the k-th part of the object as the part to be grasped; Real device capture in S140 includes: S140-1, Select the grab trajectory for shape primitives. We selected the successfully captured trajectory generated in S130-2. The trajectory of the kth part of the grasped object; S140-2, The grab trajectory is transferred to the real device, based on the grab trajectory generated in S140-1. Capture data on a real device; S140-3, force feedback data acquisition: a force feedback sensor for the joints or fingertips of a dexterous hand, acquiring force feedback data in real time. , Corresponding to the five fingers; S140-4, the action stops when the force threshold is reached, and the action stops when the following conditions are met: and, any , Finger force threshold Force feedback from the thumb, For the force feedback of the remaining four fingers; S140-5 completes the grasping process. Through a single-view depth vision system integrated into the end of the robotic arm, the position of the target object is observed and analyzed in real time, and the success of the grasp is determined from a visual perspective.

2. The dexterous hand general grasping method based on single-view vision and demonstration optimization according to claim 1, characterized in that, S110-2 collects expert datasets, including: S110-2-1, Initialize the environment, treating the crawling problem as a Markov decision process, defined by tuples: ， in, and Representing the state and action spaces respectively. It is a reward function used to measure task progress. It is a dynamic transition, possessing randomness. It is a discount factor; S110-2-2: Perform geometric analysis on the dexterous hand used to calculate the maximum grasping size of the dexterous hand: ， in, and These represent the spatial coordinates of the two contact points when any two fingers of a dexterous hand form a closed force. This indicates the crawling process. Indicates the maximum size that can be grabbed. and It represents any two different fingers out of the five fingers; S110-2-3: Sample the task parameters in S110-1 to obtain task data, which serves as the shape primitives to be captured in the expert data, where the shape category is c. The shapes involve cuboids, cylinders, and spheres. 'c' is a shape category parameter, and C represents the set of general shape primitives, that is, the collection of all categories of the basic geometric shapes considered. It is the first in the set Each shape category Represents a specific geometric shape, using random sampling; dimensional parameters Different shapes have different dimensions. Representing the first The first object The parameters in each dimension, including the size parameter, are also sampled randomly. , Indicates 0- Uniform sampling of the range The maximum grasping size of the dexterous hand obtained in S110-2-2; the pose parameters are... , Include and There are 6 parameters in total. Indicates the position of the target object. , , ,in Within the workspace axis, axis, The maximum and minimum values ​​of the axis. This indicates the orientation of the target object, that is, its rotational attitude in three-dimensional space. , , The sampling method is uniform sampling; S110-2-4: Based on the task data sampled in S110-2-3, use the teleoperation system to operate the robot to collect the grasping trajectory of the object to be grasped in the task, including: operating the robotic arm and dexterous hand to perform the grasping task, collecting the grasping trajectory, and defining the grasping trajectory as: ， in, It is the joint angle of the robotic arm at time t. It is the joint angle of a dexterous hand in time t. It refers to the trajectory length; multiple collected trajectories form the captured trajectory dataset. ， The dataset contains objects of various shapes and sizes, among which These are N trajectories collected by the remote operating system. For the collected first There are trajectories, where N is the number of trajectories; S110-2-5: Reproduce the collected grasping trajectory in a simulation environment and record the dataset. : ， in, For expert datasets, Let be the state at time t. This represents the action of the agent at time t. Indicates that an action is being performed. The state at time t+1. The reward at time t.

3. The dexterous hand universal grasping method based on single-view vision and demonstration optimization according to claim 2, characterized in that, Pre-policy learning in S110-3 includes: S110-3-1, Proposed Training Policy Network , indicating the state Next task action , It is a set of learnable parameters; S110-3-2, Define the supervised learning loss function, for... Perform supervised learning to minimize the behavior cloning loss: ， in, It is the behavioral cloning loss function. These are policy network parameters. It is an expert demonstration dataset. Is the policy network in state and tasks Making predictions It is a task Chinese experts are in a state of readiness The actual actions below, The mean squared error between two actions is used as a loss metric. S110-3-2, Minimize the behavioral cloning loss function Continuously update network parameters , making ， Conditions met: , in, The behavior cloning loss for the Kth iteration number, The behavior cloning loss is for the (K-1)th iteration. A positive value means that when the loss between two iterations is less than 100%. When the optimal strategy is obtained; The final iteration result is used as the initialization strategy parameter. ,use Policy network Initialize as a pre-policy .

4. The dexterous hand universal grasping method based on single-view vision and demonstration optimization according to claim 3, characterized in that, The reward refactoring mechanism in S110-5 includes: S110-5-1, a hand approaching an object is defined as: ， in, It's the reward for the stage where the hand touches the object. It is a correction value. It is the three-dimensional coordinate of the center of the hand. These are the three-dimensional coordinates of the object's center. This indicates the calculation of the Euclidean distance between two locations. Represents the Euclidean distance between the hand and the object; S110-5-2, a finger approaching an object is defined as: ， in, It's a reward for the stage where the finger approaches the object. It is a correction value. It refers to the position of the fingers in a dexterous hand. It can be any one of the five fingers. This represents the sum of the distances between all fingers and the object. This indicates the calculation of the Euclidean distance between two locations. This represents the distance between an object and a particular finger. This represents a constant value, indicating that the distance between the hand and the object is within a certain range. Not equal to 0; S110-5-3, the position of an object near a target is defined as: ， in, It is the reward for the phase when the object approaches the target position. This is a correction value. Represents the Euclidean distance between the object and the target position. The reward direction factor is as follows: ， in, Let be the Euclidean distance between the object and the target positions at step t. Let be the Euclidean distance between the object and the target positions at step t-1. This means that when the distance between the object and the target position at step t is closer than the distance at step t-1, =1, A positive value is defined as the distance between the object and the target position at step t being the same as the distance at step t-1. =0, The value is 0 when the distance between the object and the target position at step t is greater than the distance at step t-1. =-1, It is a negative value; S110-5-4, the punishment is defined as follows: ， in, This represents a heavy penalty in the design of rewards. It is a negative value; In the first punishment, Represents the height of the palm. The punishment is given when the height of the palm is higher than the height of the back of the hand. In the second punishment, The punishment is given when the center of the object is indicated, i.e., when the height of the hand is lower than the height of the object. In the third punishment, This indicates the height of the desktop; a penalty is imposed when the height of an object is lower than the height of the desktop. In the fourth punishment, This indicates a height above the desktop. Represents a constant value. It represents the Euclidean distance from the hand to the object, indicating a penalty when the hand and the object are a fixed distance apart, and the target is higher than the table. No punishment will be given in other circumstances; S110-5-5, based on the reward design of S110-5-1-S110-5-4, the reward is defined as follows: ， in, To reinforce rewards in learning, The reward is for bringing the hand close to the object. Rewards are given when the finger approaches the object. This indicates the reward for the phase where the object approaches the target position. This is a penalty item.

5. The dexterous hand general grasping method based on single-view vision and demonstration optimization according to claim 4, characterized in that, The steps in S110-6 are as follows: S110-6-1, Load the pre-policy obtained in the imitation learning stage of S110-3. As an initial strategy; S110-6-2, Task instance generated based on domain randomization in S110-4 This allows it to interact with the environment and be recorded. ; S110-6-3, based on the initial policy of S110-2, introduces a hybrid policy gradient update policy. The overall objective function is: ， in, It is the total learning loss. It is a reinforcement learning loss, using the natural policy gradient approximation. The initial behavior cloning loss coefficient, As the attenuation factor, k is the number of iterations; ， in, It is the reinforcement learning loss function. It is a strategy In state Task The following actions The probability, The advantage function is a weighted term in the policy gradient calculation, used to measure the extent to which the current action is superior to the policy's average behavior; for the advantage function , Indicates the state Next action The difference between the expected return that can be obtained and the expected return of the strategy in this state is defined as: ， Indicates the state Next, according to the strategy The average return that can be obtained Indicates in The subsequent expected return is defined as follows: ， in, Indicates in the task medium state Next action The reward obtained is calculated according to the reward function designed in S110-5, and the data used in the calculation is recorded in S110-6-2. supply; This is the discount factor, with a value range of [0,1]. Indicates the execution of an action After that, the environment transitions to the next state; It is a functional distribution, in the task medium state Next action The transition probabilities of all possible states; express Obedience in the task medium state Next action The subsequent state transition probability distribution; Indicates the task medium state Next action After all possible probability Perform expectation calculations to estimate the long-term value of the current action in the future; Indicates the state Next, according to the strategy Average return that can be obtained: ， in, Representation strategy In the mission medium state All actions below; Representation strategy In the mission medium state Sampling in the action distribution below ; Indicates the strategy In the mission medium state The mathematical expectation of the generated action probability distribution; S110-6-4, Construct an approximate natural gradient update equation: With the current strategy parameters as Under the given conditions, let the gradient of the objective function be: ， in, The loss function for the hybrid learning policy constructed in S110-6-3, for Regarding strategy parameters The gradient; Construct the following constrained optimization problem, where the optimization objective is to maximize the inner product of the gradient direction and the update direction, and the constraint is that the quadratic form of this direction under the Fisher information matrix does not exceed a preset threshold: ， ， in, , in, It is the gradient function of the target. The Fisher information matrix measures the local geometric information in the policy parameter space. It is a strategy In state Task Take action below The probability, The log probability of the policy is expressed with respect to the parameters. gradient, Represents all possible states and corresponding actions Calculate the weighted average. It is a preset KL divergence limit threshold, used to control the magnitude of single-step updates and avoid excessive changes in policy parameters; From the above, we can conclude that: , It is the step size; S110-6-5, Update strategy parameters: The natural gradient direction is solved using the conjugate gradient method to ensure the stability and convergence of the training process. The update expression for the policy parameters is as follows: ， in, These are the policy parameters for the (k+1)th iteration. The strategy parameters are for the k-th iteration. The gradient of the objective function constructed in S110-6-4, It is the direction of the natural gradient, a direction vector considering the geometry of the policy space. The step size is expressed as follows: ， in, >0 indicates the step size for parameter updates, which depends on the update direction and the quadratic form of the Fisher information matrix. This ensures that the parameter updates satisfy the KL divergence constraint. The denominator... Measure the curvature or change cost of the update direction in the parameter space; S110-6-6, iterative execution until convergence: Repeat steps 110-6-2 to 110-6-5 until the average return converges, thus obtaining the final policy parameters. To form a general shape primitive grasping strategy .

6. The dexterous hand universal grasping method based on single-view vision and demonstration optimization according to claim 5, characterized in that, The segmented objects in S120-3 include: S120-3-1, by identifying the desktop (plane) using RANSAK and removing the background, the single-view point cloud information of the object to be grasped is obtained. Let this be... ; S120-3-2 uses SAM-6D to segment the point cloud information of objects, thus segmenting complex objects. Divide into simpler parts , ,in, The number of segments for the object.

7. A control method for demonstrating and optimizing a general-purpose grasping robot, characterized in that, The control method is executed based on a general grasping robot control model, which is trained based on the dexterous hand general grasping method based on single-view vision and demonstration optimization as described in any one of claims 1-6. The control method includes: S310, Obtain target task information, wherein the target task information includes information about the target object of the operation task to be performed; S320, The target task information is input into the general grasping robot control model, and joint control signals are obtained from the general grasping robot control model; S330, based on the joint control signal, control the general grasping robot to perform operation tasks on the target object.

8. A learning device based on the dexterous hand universal grasping method based on single-view vision and demonstration optimization as described in any one of claims 1-6, characterized in that, The device is used to demonstrate and optimize the joint control model of a general-purpose grasping robot. The joint control model is used to demonstrate and optimize the general-purpose grasping robot's execution of operational tasks. The device includes: A single-view unknown object visual information processing module (410) is used to process visual data from a single-view image sensor, reconstruct the geometric information of the target object, perform point cloud extraction and geometric fitting on the depth data in the RGB-D image, identify the shape category, size parameters and pose parameters of the unknown object, and output the task state representation to provide a perceptual basis for subsequent strategy learning and grasping. A general shape primitive grasping strategy learning module (420) is used to learn grasping strategies that adapt to various geometric shapes. Combining expert demonstration data and domain randomized task instances, the initial strategy is first obtained through behavior cloning pre-training, and then the interaction between the strategy and the task is executed in the simulation environment. Experience trajectories are collected and reinforcement learning objective functions are constructed. The strategy parameters are optimized through natural policy gradient and conjugate gradient methods, and finally a grasping strategy with generalization ability is obtained. The grasping part discrimination module (430) infers the suitable grasping part for the dexterous hand based on the known geometric parameters of the object. By calculating the set of contact points on the object surface, combined with force closure constraints, grasping stability and hand structure limitations, it selects the grasping point combination that meets the conditions and outputs the corresponding finger configuration to drive the control system to grasp. The real-machine grasping module (440) deploys the learned grasping strategy on the actual robot platform, receives the output results of the strategy learning module and the grasping discrimination module, controls the robotic arm and dexterous hand to complete the approach, grasping and placement, monitors the contact state and error during the grasping process through sensor feedback, supports dynamic adjustment, and realizes stable and reliable object grasping.

9. A control device for demonstrating and optimizing a general-purpose grasping robot, characterized in that, The control device is based on a general grasping robot control model, which is trained based on the dexterous hand general grasping method based on single-view vision and demonstration optimization as described in any one of claims 1-6. The control device includes: The acquisition unit (510) is configured to acquire target task information, wherein the target task information includes information about the target object of the operation task to be performed; The signal determination unit (520) is configured to input the target task information into the general grasping robot control model and obtain joint control signals from the general grasping robot control model; The control unit (530) is configured to control the general-purpose grasping robot to perform an operation task on the target object based on the joint control signal.

10. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; When the processor executes the executable instructions, it causes the processor to execute the dexterous hand general grasping method based on single-view vision and demonstration optimization according to any one of claims 1 to 6, or the control method of a demonstration-optimized general grasping robot according to claim 7.

11. A demonstration-optimized general-purpose grasping robot, characterized in that, The demonstration optimized general-purpose grasping robot includes the electronic device of claim 10, or the demonstration optimized general-purpose grasping robot is communicatively connected to the electronic device of claim 10.

12. A computer program product comprising computer-executable instructions, characterized in that, When the computer-executable instructions are executed by at least one processor, they implement the dexterous hand general grasping method based on single-view vision and demonstration optimization according to any one of claims 1 to 6, or the control method of the demonstration-optimized general grasping robot according to claim 7.

Images