Task-oriented robotic grasping method based on intuitive physics

Abstract

The task-oriented robotic arm grasping method based on intuitive physics belongs to the field of robotics / artificial intelligence. It solves the problem of poor versatility when using existing robotic grasping methods to grasp objects. The present invention inputs the color image of the target object into the instance segmentation network to obtain the mask of the target object; uses the mask of the target object to extract the target object point cloud P; inputs the target object point cloud into the trained task-independent grasping network and the trained task intuitive physics network respectively to obtain the reliability score Stag of the grasping posture Gtag that satisfies the geometric relationship and the task operation score Stog of the grasping posture Gtag that satisfies the geometric relationship; selects the grasping posture Gtag corresponding to the N values ​​with the highest reliability score Stag, uses the farthest point sampling method to extract m sample grasping postures from the grasping posture Gtag corresponding to the N values, and selects the grasping posture with the smallest task operation score Stog from the m grasping postures as the optimal grasping posture. The present invention is suitable for robotic arm control.

Description

Technical Field

[0001] The present invention belongs to the field of robotics / artificial intelligence. Background Art

[0002] The robotic arm is the primary component of a robot's interaction with its environment, and grasping is a crucial step in a robot's manipulation tasks. Existing research on intelligent grasping methods primarily focuses on generating algorithms capable of picking up and placing objects from one location to another, solely considering the effects of gravity. However, in manipulation tasks, the positional relationship between the robotic arm and the target remains essentially unchanged after grasping the object, maintaining a specific physical relationship. Therefore, the choice of grasping position significantly impacts the efficiency of the task. For example, a rotary valve requires grasping the edge rather than near the axis to reduce operating force, while a single person picking up a long profile requires grasping near the center of gravity to reduce torque. Due to the constraints of the task, an inappropriate grasping position can increase the burden of the manipulation task. Therefore, grasping that ignores both task and motion constraints often fails to meet other tasks beyond grasping.

[0003] Existing algorithms for task-oriented grasping often combine tasks into a complete sequence through reinforcement learning or self-supervised learning, undergoing extensive simulation exploration. While these algorithms can accomplish specific tasks, they require setting a reasonable reward function, conducting numerous simulations, and ensuring that the camera view and environment during the actual grasping process remain consistent or similar to those in the simulation. This leads to complex and inefficient grasping training, poor object generalization, and strict camera view requirements, significantly limiting the widespread adoption of robotics in practical applications. Most importantly, existing methods focus solely on task completion, without considering the underlying logic of the robotic arm's manipulation tasks from a fundamental perspective of physical forces. This makes it impossible to use a common architecture for training different manipulation tasks, forcing the implementation of each task to be redesigned, significantly increasing the difficulty of training. Furthermore, compared to simply teaching a neural network the underlying logic, the sheer scope of task completion training exponentially increases the difficulty of converging the neural network to the target result. Summary of the Invention

[0004] The present invention aims to solve the problem of poor versatility when using existing robot grasping methods to grasp objects, and now provides a task-oriented robotic arm grasping method based on intuitive physics.

[0005] The task-oriented robotic arm grasping method based on intuitive physics of the present invention includes:

[0006] Step 1: Collect a color image and a depth image of the target object, input the color image of the target object into the instance segmentation network, and obtain a mask of the target object;

[0007] Step 2: Using the mask of the target object, extract the target object point cloud P from the depth image of the target object;

[0008] Step 3: Input the target object point cloud into the trained task-independent grasping network to obtain the reliability score Stag of the grasping posture Gtag that satisfies the geometric relationship;

[0009] Input the target object point cloud into the trained task intuitive physical network to obtain the task operation score Stog of the grasping posture Gtag that satisfies the geometric relationship;

[0010] Step 4. Select the grasping postures Gtag corresponding to the N values ​​with the highest reliability scores Stag, use the farthest point sampling method to extract m sampling grasping postures from the grasping postures Gtag corresponding to the N values, and select the grasping posture with the smallest task operation score Stog from the m grasping postures as the optimal grasping posture. N and m are both positive integers, and N>m.

[0011] Furthermore, in the present invention, the task-independent grasping network is a generalized grasping network with convolutional networks (GG-CNN), a point cloud-based grasping posture detection network (PointGPD), or a contact point grasping network (Contact Graspnet).

[0012] Furthermore, in the present invention, in step 1, the acquisition of the color image and depth image of the target object is achieved using an RGBD camera.

[0013] Furthermore, in the present invention, in step three, the task-independent grasping network is trained based on a six-degree-of-freedom grasping dataset using a contact point grasping method.

[0014] In this implementation, after obtaining a dataset, a task-independent grasping pose generation neural network model is retrained. This method trains the grasping network model based on the contact point grasping method. The retrained task-independent grasping network model is configured to take as input a point cloud P sampled from a single depth camera's perspective and output a grasping pose Gtag corresponding to the number of point clouds and its corresponding confidence score Stag.

[0015] Furthermore, in the present invention, in step three, the task-intuitive physics network is trained by placing the Franka robotic arm model into the NVIDIA Isaacgym simulation environment, and adding different objects in Isaacgym to train the task-intuitive physics model for different types of operation tasks.

[0016] Furthermore, in the present invention, in step 3, the specific training process of the task-intuitive physical network is:

[0017] Step 31: Obtain a training data set, wherein the training data set includes sampling points Ps of the target object point cloud and the true value Sgt of the grasping force score corresponding to the sampling point Ps;

[0018] Step 32: Input the sampling points Ps of the target object point cloud into the task intuitive physical neural network model to predict the task operation score Stog;

[0019] Step 33: Use the average loss function to calculate the loss value from the predicted task operation score Stog to the true value Sgt of the grasping force score. Use the loss value to train the task intuitive physical network. Repeat steps 32 and 33 until the number of training times reaches the iteration threshold, completing the network training.

[0020] Furthermore, in the present invention, in step 31, the method for obtaining the training data set is:

[0021] A robotic arm is added to the Isaacgym simulation environment to simulate the grasping of different objects. First, the depth image of the target object in the Isaacgym simulation environment is converted into a point cloud. The farthest point sampling is used to obtain n sampling points Ps. According to the position of the sampling points, the grasping posture is selected to perform the corresponding operation task action M. The maximum grasping force Fmax of each sampling point when each grasping posture is executed is obtained. The n maximum values ​​Fmax are normalized to [0,1], and the true value Sgt of the grasping force score of the n sampling points is obtained. The training data set is obtained, where n is a positive integer.

[0022] Furthermore, in the present invention, the method for acquiring the depth image of the target object in the Isaacgym simulation environment is as follows:

[0023] X camera perspective images of the target object are added to the Isaacgym simulation environment, and k depth images are collected from each perspective image, where X is greater than or equal to 4 and k is greater than or equal to 200.

[0024] The task-oriented robotic arm grasping method based on intuitive physics described in the present invention designs a task-oriented grasping neural network model based on a single camera perspective by imitating human intuitive physical experience. This method improves the success rate of the robotic arm in operating tasks by selecting grasping postures with lower operating forces. At the same time, the imitation of intuitive physical experience makes the training of multiple tasks easier, and only one object needs to be trained to achieve direct generalization of all objects under the same physical experience. By training grasping postures that are truly geometrically satisfied, a larger solution space for task-related grasping postures is achieved, which improves the simplicity of the method in practical application. Quantitative and qualitative experimental results show that the results generated by this method have better task completion effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flow chart of the method of the present invention;

[0026] Figure 2 A flowchart comparing the design framework of the human-simulated method of the present invention and the manual grasping process;

[0027] Figure 3 To capture the comparison chart of the training data set;

[0028] Figure 4 A comparison of the solution space for task grasping predicted by the task-independent grasping network of this method and the task-independent grasping network of the traditional method;

[0029] Figure 5 An example diagram of a human performing manipulation tasks using intuitive physics;

[0030] Figure 6 This is a comparison chart between the simulation environment and the actual operation experiment;

[0031] Figure 7 This is a diagram of the generalization experiment for the manipulation task;

[0032] Figure 8 A three-dimensional diagram of the Franka manipulator performing the generalized experimental manipulation task. In the figure, AXisY and AXisZ are the three coordinate axes at the end of the Franka manipulator, and Joint1, Joint2, ..., Joint7 are the joint numbers of the Franka manipulator.

[0033] Figure 9 This is a comparison chart of predictions between the method of the present invention and the existing Contact Graspnet method;

[0034] Figure 10 The force diagram of the end portion of the grasping posture predicted by the present invention when performing the operation;

[0035] Figure 11 The end force diagram of the operation performed for the grasping posture predicted by the existing method Contact Graspnet;

[0036] Figure 12 Franka manipulator joint torque diagram for performing the operation of the grasping posture predicted by the present invention;

[0037] Figure 13 Graph of the Franka manipulator joint torques for executing a grasp pose predicted by the existing method Contact Graspnet. DETAILED DESCRIPTION

[0038] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other in the absence of conflict.

[0039] Specific implementation method 1: refer to Figure 1 and Figure 2 Specifically describing this embodiment, the task-oriented robotic arm grasping method based on intuitive physics described in this embodiment includes:

[0040] Step 1: Collect the color image and depth image of the target object, input the color image of the target object into the instance segmentation network to obtain the mask of the target object; use the mask of the target object to extract the target object point cloud P from the depth image of the target object;

[0041] Step 2: Input the target object point cloud into the trained task-independent grasping network to obtain the grasping posture Gtag that satisfies the geometric relationship and the reliability score Stag of the grasping posture Gtag;

[0042] Input the target object point cloud into the trained task-intuitive physical network to obtain the task operation score Stog of the grasping posture Gtag;

[0043] Step 3. Select the grasping postures Gtag corresponding to the N values ​​with the highest reliability scores Stag, use the farthest point sampling method to extract m sampling grasping postures from the grasping postures Gtag corresponding to the N values, and select the grasping posture with the smallest task operation score Stog from the m grasping postures as the optimal grasping posture, where N and m are both positive integers, and N>m.

[0044] Furthermore, in this embodiment, the task-independent grasping network is generalized grasping with convolutional networks (GG-CNN), point cloud-based grasping posture detection (PointGPD), or contact point grasping (Contact Graspnet).

[0045] Furthermore, in this embodiment, in step 1, an RGBD camera is used to collect the color image and depth image of the target object.

[0046] Furthermore, in this embodiment, in step three, the task-independent grasping network is trained based on a six-degree-of-freedom grasping dataset using a contact point grasping method.

[0047] In this implementation, after obtaining a dataset, a task-independent grasping pose generation neural network model is retrained. This method trains the grasping network model based on the contact point grasping method. The retrained task-independent grasping network model is configured to take as input a point cloud P sampled from a single depth camera's perspective and output a grasping pose Gtag corresponding to the number of point clouds and its corresponding confidence score Stag.

[0048] Furthermore, in this embodiment, in step three, the task-independent grasping network is trained by placing the Franka robotic arm model into the NVIDIA Isaacgym simulation environment, and adding different objects in Isaacgym to implement the training of task-intuitive physics models for different types of operation tasks.

[0049] Furthermore, in this embodiment, in step 3, the specific training process of the task-independent crawling network is as follows:

[0050] Step 31: Obtain a training data set, wherein the training data set includes sampling points Ps of the target object point cloud and the true value Sgt of the grasping force score corresponding to the sampling point Ps;

[0051] Step 32: Input the sampling points Ps of the target object point cloud into the task intuitive physical neural network model to predict the task operation score Stog;

[0052] Step 3.3: Use the average loss function to calculate the loss value from the predicted task operation score Stog to the true value Sgt of the grasping force score, and determine whether the iteration reaches the iteration number threshold. If so, the task intuitive physical network is completed. Otherwise, return to step 3.2 until the iteration number threshold is reached and the task-independent grasping network completes training.

[0053] In this implementation, network training typically involves setting a specific number of steps to ensure the network converges to its minimum possible value. This is because the network's convergence capacity is finite. For different tasks or networks, after a certain number of training steps, the network will converge to its limit and then fluctuate around that limit. The number of iterations typically set is often greater than the number required for optimal convergence.

[0054] Furthermore, in this embodiment, in step 31, the method for obtaining the training data set is:

[0055] A robotic arm is added to the Isaacgym simulation environment to simulate the grasping of different objects. First, the depth image of the target object in the Isaacgym simulation environment is converted into a point cloud. The farthest point sampling is used to obtain n sampling points Ps. According to the position of the sampling points, the grasping posture is selected to perform the corresponding operation task action M. The maximum grasping force Fmax of each sampling point when each grasping posture is executed is obtained. The n maximum values ​​Fmax are normalized to between [0,1], and the true value Sgt of the grasping force score of the n sampling points is obtained. The training data set is obtained, where n is a positive integer.

[0056] Furthermore, in this embodiment, the method for acquiring the depth image of the target object in the Isaacgym simulation environment is as follows:

[0057] X camera perspective images of the target object are added to the Isaacgym simulation environment, and k depth images are collected from each perspective image, where X is greater than or equal to 4 and k is greater than or equal to 200. Specific embodiment:

[0059] By retaining all the grasping postures in the ACRONYM dataset as a task-independent grasping dataset, it is compared with the existing Contact Graspnet datasets such as Figure 3 As shown. At the same time, Figure 4 A comparison of the solution space obtained by this method with that obtained by existing direct pick-up / drop grasping methods is shown. A geometry-based task-independent grasping neural network model is trained using the contact point grasping training method. The input point cloud P is 1024 points, and the output is 1024 6-DoF grasp poses Gtag and their corresponding scores Stag.

[0060] according to Figure 5Based on the example of humans using intuitive physics experience to perform manipulation tasks, the logic behind this is summarized into three points, and the conclusions are used to guide the design of a neural network based on intuitive physics experience. The conclusions are: 1. The object is directly related to the magnitude of the manipulation force; 2. The same intuitive physics experience can be quickly transferred between objects with the same physical logic; 3. Different physical experiences should be used for the same task with different physical logics. The Franka robot arm model is placed in the Isaacgym simulation environment, and different objects are added to Isaacgym to train the intuitive physics model for different types of manipulation tasks. The image is converted into a point cloud. Farthest point sampling is then used to obtain 1024 sampling points Ps. At each sampling point, the corresponding manipulation task action M is performed with a specific grasping posture. The contact force F between the Franka gripper and the target object is captured (1024 groups) during the execution of the manipulation task action. The maximum contact force Fmax corresponding to each sampling point is then extracted (1024 groups), corresponding to the maximum manipulation force at each point. The maximum contact force Fmax was then normalized to a value between [0, 1], yielding the true value of the manipulation force score Sgt (1024 points). This, along with the 1024 sample points Ps of the target object, formed a dataset. Furthermore, to prevent the trained model from falling into a local optimum due to a dataset containing only data from a single camera perspective, five camera perspectives were evenly added to the simulation environment, distributed across the top, bottom, left, right, and center of the target object to obtain more generalizable data. Ultimately, 300 sets of data were collected from each perspective, resulting in a dataset of 1500 sets of data for model training.

[0061] To allow the trained model to ignore the shape of the target object, we avoid using network layers with point cloud encoding and decoding. We use a 256x256x128 convolutional layer with a 1x1 fully connected layer as the task-intuitive physics model for training. We input 1024 sample points Ps and output 1024 manipulation force scores, Stog. We average the loss function between Stog and its corresponding ground truth Stg to calculate the loss, and continue training the model. Finally, we obtain a point cloud P with 1024 input points and output the corresponding manipulation force score Stog.

[0062] Finally, we first select the top 50 grasping postures in Stag, and use the farthest point sampling method to extract the grasping postures corresponding to 50 sampling points from the 200 with the highest Stag scores. We then obtain the Stog scores corresponding to these 50 grasping postures, and the one with the smallest Stog score among the 50 is taken as the final grasping posture.

[0063] The statistics are spread over the entire target object as much as possible, and then the corresponding Stog scores of the 50 postures are obtained. The grasping posture with the lowest score (i.e., the smallest operating force) is taken as the execution posture of the robot arm for the operation task.

[0064] Experimental results:

[0065] In the virtual manipulation task, we selected Where2Act (grasp where to grasp the network) that uses supervised learning to implement grasping position and grasping trajectory selection and RLAfford (reinforcement learning affordance grasping) method that uses reinforcement learning to implement grasping position selection for comparison. Since only the contact force between the end gripper gripper and the target object can be collected in the simulation environment, we compared the difference in the size of the contact force between the gripper and the target object (force ratio, no unit) and the success rate (%). The closer the force ratio is to 1, the better, and the higher the success rate, the better. The experimental results of the comparison are shown in Table 1. For different manipulation task experiments, turning the door handle requires turning the door handle 30° to complete the task, and turning the valve also requires turning the valve 30° to complete the task. A single person needs to lift the object completely off the ground, and two people need to lift the center of gravity of the object 10cm off the ground. (Note: Although the objects for single-person and double-person grasping are the same shape, the single-person grasping block is lighter and can be fully grasped by grasping it in the middle. The object for double-person grasping is heavier. Because it is difficult to ensure the same efficiency of the robotic arm in the simulation, a single robotic arm is used for simulation. However, a single robotic arm cannot lift the object completely, and grasping the middle will not lift it off the ground. Grasping at both ends, the other side is supported by the ground, which allows the center of gravity to be lifted off the ground.)

[0066] As can be seen from Table 1, the force required to obtain the grasping posture obtained by this method is the minimum among all four tasks. In addition, the proposed method can significantly improve the success rate of task execution.

[0067] Table 1 Simulation grasping task experiment

[0068]

[0069] For a real-world robotic arm manipulation task, we used a Realsense D435 RGBD camera to acquire a point cloud of the target object, MaskRCNN to segment the target object mask, and a Franka robotic arm. Five camera views were selected around the target object, and three manipulation tasks were performed for each viewpoint, totaling 15 experiments per object. Because there is currently limited research on real-world manipulation tasks, the transition from simulated tasks to real-world scenarios often encounters Sim2Real discrepancies, leading to higher requirements for real-world applications. For example, RLAfford requires the use of a digital twin to implement its real-world application, significantly increasing the difficulty. However, this method, due to its simple implementation, easily achieves good results in real-world scenarios. Since this method primarily improves the success rate of manipulation tasks by reducing the operating force, we use end-joint sensors on the robotic arm to detect end-joint forces and torque sensors on each joint to detect joint forces to determine changes in operating force. For the convenience of comparison, we tested the difference in gripping force (force ratio, no unit) and success rate (%) between this method and the task-independent method through real-object experiments. The experiments with objects close to the real objects and the simulated environment are shown in Table 2. The experimental prediction results are shown in Figure 6 At the same time, since our method has the generalization ability of the same physical and logical tasks, we tested the effect of the method on different objects. The generalization experiment is shown in Table 3 and the generalization experiment results are shown in Figure 3. Figure 7 As shown in the figure. Since the force changes of each joint of the robot arm are not fixed, we Figure 9 The comparison of the grasping results predicted by the proposed method and the traditional method in a task is shown in Figure 10-13 The figure shows the changes in the forces on the end of the manipulator and each joint due to different grasping positions, where the positions of the end of the manipulator and the joints are as follows: Figure 8 Finally, because Franka's gripping force is fixed, it cannot be dynamically adjusted according to the specific object like a human. Therefore, we set different gripping forces to test the reduction of the force on the gripper. The success rate under different gripping forces will reflect its demand for gripping force and thus the effectiveness of this method. The experiment is shown in Table 4.

[0070] Table 2 Object grasping task experiments

[0071]

[0072] Table 3. Object generalization experiment of physical grasping task

[0073]

[0074] Table 4 Experiments on different gripping force operation tasks

[0075]

[0076] As can be seen from Table 2, compared with grasping without task orientation, this method can significantly reduce the grasping operation force, thereby maintaining a high task execution success rate. As can be seen from Table 3, this method has good generalization ability for different objects. Figure 6 It can be seen that through this method, the operating force of each joint of the robotic arm in the operation task can be reduced, and then the task with high load requirements can be completed as much as possible under the condition that the load of the robotic arm itself is certain.

[0077] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in conjunction with other described embodiments.

Claims

1. A task-oriented robotic arm grasping method based on intuitive physics, characterized by: include: Step 1: Collect a color image and a depth image of the target object, input the color image of the target object into the instance segmentation network, and obtain a mask of the target object; Using the mask of the target object, extract the target object point cloud P from the depth image of the target object; Step 2: Input the target object point cloud P into the trained task-independent grasping network to obtain the reliability score Stag of the grasping posture Gtag that satisfies the geometric relationship; Input the target object point cloud P into the trained task intuitive physical network to obtain the task operation score Stog of the grasping posture Gtag that satisfies the geometric relationship; Step 3: Select the grasping postures Gtag corresponding to the N values ​​with the highest reliability scores Stag, extract m sample grasping postures from the grasping postures Gtag corresponding to the N values ​​using the farthest point sampling method, and select the grasping posture with the smallest task operation score Stog from the m sample grasping postures as the optimal grasping posture; In step 2, the specific training process of the task-intuitive physical network is as follows: Step 2.1, obtaining a training data set, the training data set includes a sampling point Ps of the target object point cloud P, and the true value of the grasping force score corresponding to the sampling point Ps Sgt; Step 22: Input the sampling points Ps of the target object point cloud P into the task intuitive physical network to predict the task operation score Stog; Step 23: Use the average loss function to calculate the loss from the predicted task operation score Stog to the true value Sgt of the grasping force score. Use the loss value to train the task intuitive physics network. Repeat steps 22 and 23 until the number of training times reaches the iteration threshold, completing the network training. In step 21, the method for obtaining the training data set is: A robotic arm is added to the Isaacgym simulation environment to simulate the grasping of different objects. First, the depth image of the target object in the Isaacgym simulation environment is converted into a point cloud. The farthest point sampling method is used to obtain n sampling points Ps. According to the position of the sampling points, the grasping posture is selected to perform the corresponding operation task action M. The maximum grasping force Fmax of each sampling point when each grasping posture is executed is obtained. The n maximum values ​​Fmax are normalized to between [0,1], and the true value Sgt of the grasping force score of the n sampling points Ps is obtained. The training data set is obtained, where n is a positive integer.

2. The task-oriented robotic arm grasping method based on intuitive physics according to claim 1, characterized in that: In step 1, the color image and depth image of the target object are collected using an RGBD camera.

3. The task-oriented robotic arm grasping method based on intuitive physics according to claim 1, characterized in that: In step 2, the task-independent grasping network is a convolutional network generalized grasping network, a point cloud-based grasping posture detection network, or a contact point grasping network.

4. The task-oriented robotic arm grasping method based on intuitive physics according to claim 1 or 2, characterized in that: In step 2, the task-independent grasping network is trained based on a six-degree-of-freedom grasping dataset using a contact point grasping method.

5. The task-oriented robotic arm grasping method based on intuitive physics according to claim 1, characterized in that: The method for acquiring the depth image of the target object in the Isaacgym simulation environment is: X camera perspective images of the target object are added to the Isaacgym simulation environment, and k depth images are collected from each perspective image, where X is greater than or equal to 4 and k is greater than or equal to 200.

Images