Method, device and equipment for generating training data of robot control model

Abstract

This application discloses a method, apparatus, computer device, and storage medium for generating training data for a robotic arm control model, belonging to the field of computer technology. The method includes: acquiring a set of gripping points on an object surface, the set including at least three gripping points, which are contact points for the robotic arm when gripping the object; determining a gripping discrimination result based on the contact geometric parameters of the gripping points in the set, where the contact geometric parameters represent the position of the gripping point, the normal direction of the object surface at the gripping point, and the tangential direction; if the gripping discrimination result indicates that the object can be gripped based on the set of gripping points, then generating a gripping label for the object based on the contact geometric parameters of the gripping points in the set; adding the gripping label to a training dataset, which is used to train the robotic arm control model. This application achieves intelligent generation of training datasets, improving the efficiency of training data generation.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data generation method, apparatus, computer device, and storage medium. Background Technology

[0002] With the widespread application of artificial intelligence and robotics, robotic arms are becoming increasingly popular and diversified in various fields such as industrial automation, medical assistance, and home services.

[0003] In related technologies, data-driven control methods are used to control robotic arms to grasp objects. Data-driven control methods refer to using large-scale training datasets to train robotic arm control models, enabling the robotic arm control models to predict grasping strategies based on relevant information about the object, thereby achieving object grasping.

[0004] However, training datasets are usually collected and labeled manually, which requires a lot of manpower and time, resulting in low efficiency in generating training data for robotic arm control models. Summary of the Invention

[0005] This application provides a method, apparatus, computer device, and storage medium for generating training data for a robotic arm control model. It achieves intelligent generation of training datasets capable of grasping objects, thus improving the efficiency of generating training data for the robotic arm control model. The technical solution is as follows:

[0006] On the one hand, a method for generating training data for a robotic arm control model is provided, the method comprising:

[0007] Obtain a set of gripping points on the surface of an object, the set of gripping points including at least three gripping points, the at least three gripping points being contact points when the robotic arm grips the object;

[0008] Based on the contact geometry parameters of the gripping points in the gripping point set, the gripping discrimination result of the gripping point set is determined. The contact geometry parameters of the gripping points are used to represent the position of the gripping points, the normal direction and the tangential direction of the object surface at the gripping points;

[0009] If the grasping determination result indicates that the object can be grasped based on the grasping point set, then a grasping tag for the object is generated based on the contact geometry parameters of the grasping points in the grasping point set. The grasping tag for the object is used to indicate the pose of the robot arm when grasping the object.

[0010] The object's grasping label is added to the training dataset, which is used to train the robotic arm control model.

[0011] On the other hand, a training data generation device for a robotic arm control model is provided, the device comprising:

[0012] The acquisition module is used to acquire a set of gripping points on the surface of an object, the set of gripping points including at least three gripping points, the at least three gripping points being the contact points when the robotic arm grips the object;

[0013] The discrimination module is used to determine the discrimination result of the gripping point set based on the contact geometry parameters of the gripping points in the gripping point set. The contact geometry parameters of the gripping points are used to represent the position of the gripping point, the normal direction and the tangential direction of the object surface at the gripping point;

[0014] The tag generation module is used to generate a gripping tag for the object based on the contact geometry parameters of the gripping points in the gripping point set if the gripping discrimination result indicates that the object can be gripped based on the gripping point set. The gripping tag of the object is used to indicate the pose of the robot arm when gripping the object.

[0015] The label adding module is used to add the object's grasping labels to the training dataset, which is used to train the robotic arm control model.

[0016] Optionally, the discrimination module is used to:

[0017] The conversion unit is used to convert the contact forces satisfying the constraint conditions on the gripping points in the set of gripping points into multiple gripping spins based on the contact geometry parameters of the gripping points in the set of gripping points.

[0018] The conversion unit is used to determine a grasping spinner space based on the plurality of grasping spinners, wherein the grasping spinner space is the smallest space containing the plurality of grasping spinners;

[0019] The discrimination unit is configured to determine, if the grasping spinor space satisfies the stable grasping condition, that the grasping discrimination result indicates that the object can be grasped based on the grasping point set; and if the grasping spinor space does not satisfy the stable grasping condition, determine that the grasping discrimination result indicates that the object cannot be grasped based on the grasping point set.

[0020] Optionally, the contact geometry parameters of the gripping point include the position vector, normal vector, and tangent vector of the gripping point; the transformation unit is used for:

[0021] For each crawl point in the set of crawl points, perform the following operations:

[0022] Based on the position vector, normal vector, and tangent vector of the grab point, determine the grab matrix of the grab point;

[0023] Multiply the gripping matrix of the gripping point by all the contact forces at the gripping point that satisfy the constraint conditions to obtain multiple gripping spinors corresponding to the gripping point.

[0024] Optionally, the conversion unit is used for:

[0025] Determine the position vector, normal vector, first tangent vector, and second tangent vector of the grab point, wherein the normal vector, the first tangent vector, and the second tangent vector are perpendicular to each other;

[0026] The position vector is cross-multiplied with the normal vector to obtain a first mapping vector; the position vector is cross-multiplied with the first tangent vector to obtain a second mapping vector; the position vector is cross-multiplied with the second tangent vector to obtain a third mapping vector.

[0027] The grasping matrix is ​​constructed based on the normal vector, the first tangent vector, the second tangent vector, the first mapping vector, the second mapping vector, and the third mapping vector.

[0028] Optionally, the contact force at the gripping point includes normal force and tangential force; the constraint conditions include:

[0029] The normal force is non-negative;

[0030] The tangential force is not greater than the product of the normal force and the coefficient of friction of the object.

[0031] Optionally, the stable grasping conditions include:

[0032] The origin of the screw coordinate system is located in the grasping screw space. The screw coordinate system refers to the coordinate system in which the grasping screw space is located. The vectors in the screw coordinate system are used to describe forces.

[0033] The object's gravity lies within the grasp spin space.

[0034] Optionally, the spinor coordinate system is a three-dimensional coordinate system, and the grasping spinor space constitutes a three-dimensional convex polyhedron. The three-dimensional coordinate system includes a first coordinate axis, a second coordinate axis, and a third coordinate axis that are mutually perpendicular. The first and second coordinate axes are perpendicular to the direction of gravity, and the third coordinate axis is parallel to the direction of gravity. The discrimination module is further used for:

[0035] The convex polyhedron formed by the grasping spinor space is mapped onto the plane containing the first coordinate axis and the second coordinate axis to obtain a polygon;

[0036] Determine the minimum distance from the origin of the spinor coordinate system to the edge of the polygon;

[0037] If the value of the minimum distance is greater than the value of the object's gravity, then the gravity is determined to be located in the grasping spin space.

[0038] Optionally, the apparatus further includes a fraction generation module for:

[0039] If the minimum distance is greater than the weight of the object, a grasping score is determined based on the minimum distance. The grasping score is positively correlated with the minimum distance and is used to reflect the grasping stability of grasping the object based on the grasping point set.

[0040] Optionally, the acquisition module is used to:

[0041] The surface of the object's three-dimensional mesh model is divided into multiple candidate patches, and the angle between adjacent meshes in each candidate patch is less than an angle threshold.

[0042] Among the plurality of candidate facets, at least one target facet is determined, wherein the area of ​​the target facet is greater than an area threshold;

[0043] The set of grab points is sampled on at least one target patch.

[0044] Optionally, the acquisition module is used to:

[0045] At least three gripping points are randomly sampled from the surface of the object's three-dimensional mesh model;

[0046] If the distance between the at least three gripping points satisfies the finger distance condition of the robotic hand, and the angle between the normal vectors of the at least three gripping points satisfies the finger angle condition of the robotic hand, then the at least three gripping points constitute the gripping point set.

[0047] Optionally, the label generation module is used to:

[0048] The simulation unit is used to simulate the set of grasping points and obtain simulation results if the grasping judgment result indicates that the object can be grasped based on the set of grasping points.

[0049] The tag generation unit is used to generate a gripping tag for the object based on the contact geometry parameters of the gripping points in the gripping point set if the simulation result indicates that the robot successfully grips the object using the gripping point set.

[0050] Optionally, the simulation unit is used for:

[0051] Based on the contact geometry parameters of the gripping points in the set of gripping points, the fingertip target pose of the robotic arm is determined;

[0052] In the simulation environment, the robotic arm is controlled to grasp the object according to the target pose of the fingertip, and the simulation result is obtained.

[0053] Optionally, the simulation unit is used for:

[0054] The following iterative process is performed: The joint pose of the robotic hand is converted into the fingertip pose using the pose conversion parameters of the robotic hand; the pose conversion parameters are updated based on the fingertip pose error between the fingertip pose and the target fingertip pose; the fingertip pose error is converted into joint pose error using the updated pose conversion parameters; and the joint pose of the robotic hand is updated based on the joint pose error.

[0055] The iteration process is stopped when the iteration process meets the iteration termination condition.

[0056] In the simulation environment, the robotic arm is controlled to grasp the object according to the updated joint pose, and the simulation result is obtained.

[0057] Optionally, the simulation unit is used for:

[0058] During the iteration process, the updated joint pose is converted into an updated fingertip pose using the updated pose conversion parameters;

[0059] If the fingertip pose error between the updated fingertip pose and the target fingertip pose is less than the error threshold, then the iteration process is stopped.

[0060] Optionally, the robotic hand is a three-finger robotic hand, comprising a first fingertip, a second fingertip, and a third fingertip; the set of grasping points includes a first grasping point, a second grasping point, and a third grasping point; the object grasping tag includes:

[0061] The position vector and normal vector of the first fingertip;

[0062] The center position vector and roll angle of the grasping circle, wherein the grasping circle refers to the circumcircle of the first grasping point, the second grasping point and the third grasping point;

[0063] The angle between the first fingertip and the second fingertip along the grasping circle;

[0064] The angle between the first fingertip and the third fingertip along the grasping circle;

[0065] The normal vector of the second fingertip and the normal vector of the third fingertip.

[0066] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to perform the operations performed by the training data generation method for the robotic arm control model as described above.

[0067] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to perform the operations performed by the training data generation method for the robotic arm control model as described above.

[0068] On the other hand, a computer program product is provided, including a computer program loaded and executed by a processor to perform the operations performed by the training data generation method for the robotic arm control model as described above.

[0069] The solution provided in this application obtains a set of gripping points on the surface of an object. Whether the object can be stably gripped based on this set of gripping points depends on the position and orientation of each gripping point in the set. The contact geometry parameters of the gripping points represent their position and orientation. Therefore, based on the contact geometry parameters of each gripping point in the set, it can be determined whether the object can be gripped. If the object can be gripped, a gripping label for the object is generated based on the contact geometry parameters of the gripping points in the set to represent the pose of the robotic arm when gripping the object. This gripping label can then be added to the training dataset used to train the robotic arm control model. Therefore, this method achieves intelligent generation of training datasets capable of gripping objects, ensuring the quality of the training dataset, eliminating the need for manual collection and annotation, and improving the efficiency of generating training data for the robotic arm control model. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a schematic diagram of a computer system provided in an embodiment of this application;

[0072] Figure 2 This is a flowchart illustrating a method for generating training data for a robotic arm control model, as provided in an embodiment of this application.

[0073] Figure 3This is a flowchart of another method for generating training data for a robotic arm control model provided in an embodiment of this application;

[0074] Figure 4 This is a schematic diagram of a patch division result provided in an embodiment of this application;

[0075] Figure 5 This is a flowchart of another method for generating training data for a robotic arm control model provided in an embodiment of this application;

[0076] Figure 6 This is a simulation result of a robotic arm grasping an object, provided in an embodiment of this application;

[0077] Figure 7 This is a schematic diagram of a joint pose provided in an embodiment of this application;

[0078] Figure 8 This is a schematic diagram of a tag-grabbing method provided in an embodiment of this application;

[0079] Figure 9 This is a flowchart of another method for generating training data for a robotic arm control model provided in an embodiment of this application;

[0080] Figure 10 This is a flowchart of another method for generating training data for a robotic arm control model provided in an embodiment of this application;

[0081] Figure 11 This is a schematic diagram of the structure of a training data generation device for a robotic arm control model provided in an embodiment of this application;

[0082] Figure 12 This is a schematic diagram of the structure of a training data generation device for another robotic arm control model provided in an embodiment of this application;

[0083] Figure 13 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0084] Figure 14 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0085] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0086] It is understood that the terms "first," "second," etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, a first mapping vector may be referred to as a second mapping vector, and similarly, a second mapping vector may be referred to as a first mapping vector.

[0087] "At least one" refers to one or more. For example, at least one crawl point can be one crawl point, two crawl points, three crawl points, or any integer number of crawl points greater than or equal to one. "Multiple" refers to two or more. For example, multiple crawl points can be two crawl points, three crawl points, or any integer number of crawl points greater than or equal to two. "Each" refers to each of the at least one crawl point. For example, each crawl point refers to each of the multiple crawl points. If the multiple crawl points are three crawl points, then each crawl point refers to each of the three crawl points.

[0088] It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this application have all been fully authorized by the user or relevant parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0089] For example, the data such as the set of grab points, contact geometry parameters, grab discrimination results, and grab tags involved in this application are all fully authorized by the user or relevant parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0090] For ease of explanation, the concepts involved in the embodiments of this application are introduced below.

[0091] (1) Circular representation: A method for describing the grasping posture of a three-finger robot using the geometric properties of a circle. The grasping configuration is parameterized by data such as the center of the circle, the normal vector and the angle, which simplifies the computational complexity.

[0092] (2) Overlapping surface division method: an algorithm that divides the surface of an object into multiple overlapping surfaces, used to filter and select large surface areas suitable for sampling.

[0093] (3) Grip point set: The set of contact points of the robot arm selected through sampling and geometric constraints is a set of potential effective grip points that meet the distance and angle conditions of the robot arm.

[0094] (4) Grasp Wrench Space (GWS): A multidimensional convex hull that describes the possible forces and torques applied during the gripping process of the robot arm and is used to determine the gripping stability.

[0095] (5) Friction cone: The contact force constraint conditions defined at the gripping point ensure that the normal force is non-negative and the tangential force satisfies the friction coefficient constraint.

[0096] (6) Grasping matrix: used to establish a six-dimensional structure of the local coordinate system, including the normal vector and tangent vector of the grasping point, providing a basis for analyzing the contact force applied to the grasping point.

[0097] (7) Inverse Kinematics Solution: Based on the fingertip pose of the manipulator, the joint pose of the manipulator is solved using the pseudo-inverse iteration method of the Jacobian matrix. This allows the manipulator's joint movements to be controlled so that the fingertip reaches the desired pose. The pseudo-inverse iteration method of the Jacobian matrix is ​​a numerical method for solving inverse kinematics problems. The joint pose of the manipulator is updated by solving the pseudo-inverse of the Jacobian matrix.

[0098] (8) Pybullet simulation environment: a physics engine for simulating robotic arm grasping tasks. In this application embodiment, it is used to verify the feasibility of the grasping point set in order to filter the grasping point set that can be successfully grasped.

[0099] (9) Robotic Arm: A robotic arm is the actuator of a robot. It is a programmable mechanical device with functions similar to those of a human hand. It is a high-precision, multi-input multi-output, highly nonlinear, and strongly coupled system capable of performing various physical operations such as grasping and handling. With the widespread application of artificial intelligence, robotic arms play an important role in production and daily life, and can be widely used in fields such as industrial automation, medical assistance, and home services, becoming an indispensable piece of equipment.

[0100] A robotic hand typically consists of joints and an end effector. The robotic hand forms a kinematic chain through the connection of multiple joints, with the end of the kinematic chain called the end effector. The end effector mimics the human hand; the robotic hand controls the posture of the joints to drive the end effector, enabling it to perform rotational or translational movements to complete various grasping tasks. The joints of the robotic hand mimic the joints of a human hand and are typically driven by motors or hydraulic systems, allowing the robotic hand to move in multiple directions. The end effector of a robotic hand is usually the "fingertip" part, mimicking the fingertip of a human finger, but it can also be a gripper or other component.

[0101] A three-finger robotic hand is a type of robotic hand. It's a mechanical device with three moving parts, similar to three human fingers, such as the thumb, index finger, and middle finger. Each of the three "fingers" can be controlled independently. By controlling the joint position of each finger, it can bend and extend, thereby driving the fingertip position and performing grasping and other actions to achieve object grasping tasks. Through its relatively simple yet flexible design, the three-finger robotic hand can provide excellent object manipulation capabilities in various situations, and is particularly suitable for tasks requiring high dexterity and grasping precision.

[0102] With the rapid development of robotics technology, the application of robotic arms in various fields such as industrial automation, medical assistance, and home services has become increasingly widespread and diversified. As a highly efficient tool, the grasping ability of a robotic arm directly determines its performance in various application scenarios. Whether it's material handling on a production line or delicate operations in surgery, the grasping accuracy and stability of a robotic arm are crucial. Traditional robotic arm grasping methods typically rely on manually designed predefined grasping strategies or control methods based on control rules. However, these methods have limitations because they usually assume that the geometric characteristics and grasping conditions of the object are known. Therefore, while the pre-set grasping strategies or control rules can be effectively applied, they ignore the diversity and uncertainty of object changes in the actual environment. For example, factors such as the coefficient of friction, weight distribution, and shape changes of different objects can all affect the success of grasping. Therefore, when faced with complex and diverse objects, traditional robotic arm grasping methods often struggle to handle various changing object materials, shapes, and weights, resulting in difficulty in guaranteeing the success rate and stability of grasping.

[0103] Based on this, this application proposes a data-driven grasping method. Unlike traditional control methods based on grasping strategies or control rules, this data-driven grasping method generates a large-scale training dataset and combines machine learning and deep learning algorithms to train a robotic arm control model. This allows the robotic arm control model to learn and extract features applicable to various object grasping tasks. By training the robotic arm control model, it can autonomously identify the shape, position, and material of objects, and predict grasping strategies based on this information, thereby improving the success rate and stability of grasping.

[0104] While data-driven crawling methods hold great potential, generating high-quality training datasets remains a technical challenge. Manually collecting and labeling training datasets is extremely time-consuming and labor-intensive. To improve the efficiency of training dataset generation, this application provides a method for generating training data for a robotic arm control model, capable of intelligently generating training datasets. The following embodiments provide a detailed description of this method.

[0105] The method for generating training data for a robotic arm control model provided in this application is executed by a computer device. When the method is executed by the computer device, the computer device generates object grasping labels and adds the grasping labels to the training dataset. The training dataset is used to train the robotic arm control model, which is used to control the robotic arm, and the robotic arm is used to grasp objects.

[0106] Optionally, the computer device may be a terminal or a server. Optionally, the server may be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, or in-vehicle terminal, but is not limited to these.

[0107] Figure 1 This is a schematic diagram of a computer system provided in an embodiment of this application. See also... Figure 1 The computer system includes a computer device 101 and a robotic arm 102. The computer device 101 and the robotic arm 102 are connected via a wireless or wired network.

[0108] The computer device 101 generates object grasping labels, adds the grasping labels to the training dataset, and uses the training dataset to train the robot control model.

[0109] In one possible implementation, after the computer device 101 has trained the robotic arm control model, the model is deployed in a control device. The control device controls the robotic arm 102, and the control device and the robotic arm 102 are connected via a wireless or wired network. During the robotic arm 102's grasping task, the control device predicts the grasping pose of the robotic arm 102 using the control model and sends control commands to the robotic arm 102 so that it can achieve the grasping pose indicated by the control commands, thereby grasping the object. Optionally, the control device and the computer device 101 can be the same device or different devices.

[0110] In one possible implementation, after the computer device 101 has trained the robotic arm control model, it deploys the robotic arm control model in the robotic arm 102. During the process of the robotic arm 102 performing a grasping task, the robotic arm 102 predicts the grasping pose through the robotic arm control model, and then grasps the object.

[0111] under Figure 2 The example is a brief description of the method for generating training data for a robotic arm control model. Figure 2 This is a flowchart illustrating a method for generating training data for a robotic arm control model, as provided in this application embodiment. This application embodiment is executed by a computer device, for example, the computer device is... Figure 1 Computer device 101 in the computer system. See also Figure 2 The method includes:

[0112] 201. A computer device acquires a set of gripping points on the surface of an object, the set of gripping points including at least three gripping points, which are contact points of the robotic arm when gripping the object.

[0113] The object surface refers to the surface of the object to be grasped. The purpose of this application embodiment is to determine a set of grasping points on the object surface that can grasp the object. The set of grasping points obtained in step 201 is a set of candidate grasping points sampled from the object surface. Therefore, it is necessary to determine whether the set of grasping points can grasp the object through the operation in step 202 below.

[0114] The set of gripping points includes at least three gripping points. The number of gripping points in the set is equal to the number of fingers of the robotic hand. Each finger of the robotic hand corresponds to a gripping point. The gripping point corresponding to a finger of the robotic hand is the contact point between the finger and the surface of the object when the robotic hand grips the object.

[0115] For example, if the robotic hand is a three-finger robotic hand, then the set of gripping points includes three gripping points.

[0116] 202. The computer device determines the grasping judgment result of the grasping point set based on the contact geometric parameters of the grasping points in the grasping point set. The contact geometric parameters of the grasping points are used to represent the position of the grasping points and the normal and tangential directions of the object surface at the grasping points.

[0117] Each gripping point in the set of gripping points has its own contact geometry parameters. The contact geometry parameters of the gripping point can reflect the contact geometry characteristics of the gripping point on the object surface. The contact geometry parameters of the gripping point can represent the position of the gripping point on the object surface and the normal and tangential directions of the object surface at the gripping point. Therefore, the contact geometry parameters cover the spatial position and geometric characteristics of the gripping point.

[0118] Wherein, the normal direction of the object's surface at the gripping point refers to the direction of the normal line of the object's surface at the gripping point, and the tangential direction of the object's surface at the gripping point refers to the direction of the tangential plane of the object's surface at the gripping point. The normal direction of the object at the gripping point and the tangential direction of the object's surface at the gripping point are perpendicular to each other.

[0119] The contact geometry parameters of the gripping point are determined when the gripping point is sampled on the object surface.

[0120] In this embodiment, the computer device determines the grasping judgment result of the grasping point set based on the contact geometric parameters of each grasping point in the grasping point set. This grasping judgment result indicates whether an object can be grasped based on the grasping point set, that is, whether the robot arm can grasp an object by applying contact force at each grasping point in the grasping point set. Since the contact geometric parameters of the grasping points can reflect the spatial position and geometric characteristics of the grasping points, the contact geometric parameters of each grasping point determine the combination of force and torque applied at the grasping point. The mode of action of the force and torque applied at the grasping point determines whether the object can be stably grasped. Therefore, the computer device can determine the grasping judgment result of the grasping point set based on the contact geometric parameters of each grasping point in the grasping point set.

[0121] 203. If the grasping judgment result indicates that the object can be grasped based on the grasping point set, the computer device generates a grasping label for the object based on the contact geometry parameters of the grasping points in the grasping point set. The grasping label of the object is used to indicate the pose of the robot arm when grasping the object.

[0122] If the grasping judgment result indicates that the object can be grasped based on the grasping point set, then the grasping point set is a valid grasping point set. The computer device then generates a grasping label for the object based on the contact geometry parameters of each grasping point in the grasping point set. The grasping label of the object indicates the pose of each finger of the robot arm when grasping the object.

[0123] Since the gripping point is the contact point between the robotic arm's fingers and the object when the robotic arm grasps it, the position of the gripping point determines the position of the robotic arm's fingers on the object's surface, and the direction of the gripping point determines how the robotic arm's fingers should apply force on the object's surface (that is, the robotic arm's posture). Therefore, the position and posture (pose) of the robotic arm are aligned one-to-one with the position and direction of the gripping point. Thus, the computer equipment can determine the robotic arm's posture when grasping an object based on the contact geometry parameters of the gripping point.

[0124] Additionally, if the grabbing result indicates that an object cannot be grabbed based on the grabbing point set, then the grabbing point set is invalid, and the computer device discards the grabbing point set.

[0125] 204. The computer device adds the object's grasping label to the training dataset, which is used to train the robotic arm control model.

[0126] The object's grasping label indicates the robot's pose when the object is effectively grasped. Therefore, the computer device adds the object's grasping label to the training dataset. Subsequently, the training dataset can be used to train the robot's control model, which is used to predict the robot's pose when grasping an object.

[0127] Optionally, the training dataset also includes object data representing the object's pose. For example, the object data might include a 3D mesh model of the object or a photograph of the object. The robotic arm control model is used to predict the pose of the robotic arm when grasping the object based on the object data.

[0128] Optionally, the training dataset includes multiple grab labels for the object, each grab label being different. The multiple grab labels include grab labels determined based on a set of grab points, or may also include grab labels determined in other ways.

[0129] Optionally, the training dataset includes multiple sets of training data, each set of training data including object data of an object and at least one grasping label of the object.

[0130] The method provided in this application obtains a set of gripping points on the surface of an object. Whether the object can be stably gripped based on this set of gripping points depends on the position and orientation of each gripping point in the set. The contact geometry parameters of the gripping points represent their position and orientation. Therefore, based on the contact geometry parameters of each gripping point in the set, it can be determined whether the object can be gripped. If the object can be gripped, a gripping label for the object is generated based on the contact geometry parameters of the gripping points in the set to represent the pose of the robotic arm when gripping the object. This gripping label can then be added to the training dataset used to train the robotic arm control model. Therefore, this method achieves intelligent generation of training datasets capable of gripping objects, ensuring the quality of the training dataset, eliminating the need for manual collection and annotation, and improving the efficiency of generating training data for the robotic arm control model.

[0131] under Figure 3 The embodiments are detailed descriptions of the training data generation method for the robot control model. Figure 3 This is a flowchart illustrating another method for generating training data for a robotic arm control model provided in this application embodiment. This application embodiment is executed by a computer device. See also... Figure 3 The method includes:

[0132] 301. A computer device acquires a set of gripping points on the surface of an object, the set of gripping points including at least three gripping points, which are contact points of the robotic arm when gripping the object.

[0133] In one possible implementation, the robotic hand is a three-finger robotic hand, comprising a first finger, a second finger, and a third finger. The set of grasping points includes a first grasping point, a second grasping point, and a third grasping point. The first grasping point is the contact point between the first finger and the object when the robotic hand grasps the object; the second grasping point is the contact point between the second finger and the object when the robotic hand grasps the object; and the third grasping point is the contact point between the third finger and the object when the robotic hand grasps the object. For example, the first, second, and third fingers can be referred to as the middle finger, left finger, and right finger of the robotic hand, respectively.

[0134] In one possible implementation, the computer device divides the surface of a 3D mesh model of an object into multiple candidate patches, where the angle between adjacent meshes in each candidate patch is less than an angle threshold; determines at least one target patch among the multiple candidate patches, where the area of ​​the target patch is greater than an area threshold; and samples a set of grab points in the at least one target patch.

[0135] The surface of an object's 3D mesh model is composed of interconnected meshes, such as triangles. For any two adjacent meshes, the computer determines the angle between them. If the angle is less than a threshold, the two adjacent meshes are assigned to the same candidate facet; otherwise, they are assigned to different candidate facets. This ensures that the resulting candidate facets are relatively flat, resulting in good surface continuity across the multiple candidate facets.

[0136] After dividing the target surface into multiple candidate patches, the target patches with an area greater than a threshold are selected, while smaller candidate patches are filtered out. Grasping point sets are then sampled from the larger target patches. Since larger target patches provide more contact areas, they typically have more potentially valid grasping points. Therefore, sampling grasping points from larger target patches improves the feasibility of grasping points. Smaller candidate patches usually represent detailed parts of the object's surface, such as sharp corners, edges, or local curved surfaces. Grasping points from these candidate patches may not be suitable for grasping, as they are likely to experience slippage or instability. Therefore, filtering out smaller candidate patches reduces unnecessary computation and saves computational resources.

[0137] For example, the aforementioned angle threshold can be a pre-set threshold.

[0138] For example, the area threshold mentioned above can be a pre-set threshold.

[0139] Figure 4 This is a schematic diagram of a patch division result provided in an embodiment of this application, as shown below. Figure 4 As shown, the surface of the object is divided into multiple candidate patches. Among them, the light-colored candidate patch 401 has a larger area, so the sampling point is sampled on the light-colored candidate patch 401, and the dark-colored candidate patch 402 has a smaller area, so the dark-colored candidate patch 402 is filtered out.

[0140] In this implementation, the surface of the object's 3D mesh model is divided into multiple candidate patches. A set of grasping points is sampled from target patches with an area greater than a threshold. This ensures that the sampled grasping points are located in relatively flat, suitable grasping areas, increasing the likelihood of successful grasping. This initial sampling of a more suitable set of grasping points improves the feasibility of the sample, reduces the probability of invalid computation, and saves computational resources. Therefore, sampling grasping points by dividing and selecting patches improves the efficiency of sampling the grasping point set, ensures its richness and diversity, and provides a solid foundation for the robot's adaptability in complex tasks.

[0141] In one possible implementation, the computer device randomly samples at least three gripping points on the surface of the object's three-dimensional mesh model; if the distance between the at least three gripping points satisfies the finger distance condition of the robotic hand, and the angle between the normal vectors of the at least three gripping points satisfies the finger angle condition of the robotic hand, then the at least three gripping points constitute a gripping point set.

[0142] The finger distance condition refers to the distance requirements that each finger of the robotic arm must meet. For example, the distance between each finger cannot exceed a distance threshold, therefore the distance between at least three grasping points must also not exceed the distance threshold. The finger angle condition refers to the angle requirements that each finger of the robotic arm must meet. For example, the angle between each finger cannot exceed an angle threshold, therefore the angle between the normal vectors of at least three grasping points must also not exceed the angle threshold. Therefore, at least three grasping points corresponding to at least three fingers of the robotic arm must satisfy the above finger distance and finger angle conditions to ensure the feasibility of the sampled at least three grasping points and to avoid situations where the physical structure of the robotic arm prevents the use of these at least three grasping points for grasping.

[0143] Optionally, the robotic hand is a three-finger robotic hand, comprising a first finger, a second finger, and a third finger. The set of grasping points includes a first grasping point, a second grasping point, and a third grasping point. The first grasping point is the contact point between the first finger and the object, the second grasping point is the contact point between the second finger and the object, and the third grasping point is the contact point between the third finger and the object. The following description uses an example of a set of three grasping points to illustrate the process of sampling three grasping points.

[0144] The computer device randomly samples multiple points on the surface of a 3D mesh model, forming a point set. For each point in the set, the computer device designates it as a first grasping point, and then randomly samples a second and third grasping point. The computer device determines whether the distances between the first, second, and third grasping points satisfy the finger distance condition, and whether the angles between the normal vectors of the first, second, and third grasping points satisfy the finger angle condition. If both the finger distance and finger angle conditions are satisfied, then the first, second, and third grasping points are combined into a grasping point set. This process is repeated for each point in the set, resulting in multiple grasping point sets. The number of points sampled can be a pre-set number.

[0145] In this implementation, a suitable set of gripping points is selected based on the distance and angle conditions of the robotic arm's fingers to ensure the feasibility of the gripping point set. This avoids situations where the robotic arm cannot use the gripping point set for gripping, optimizes the selection process of the gripping point set, and thus reduces the probability of invalid calculations and saves computing resources.

[0146] 302. The computer device converts the contact forces satisfying the constraints on the gripping points in the gripping point set into multiple gripping spindles based on the contact geometry parameters of the gripping points in the gripping point set.

[0147] Each gripping point in the set of gripping points has its own contact geometry parameters. The contact geometry parameters of the gripping point can reflect the contact geometry characteristics of the gripping point on the object surface. The contact geometry parameters of the gripping point indicate the position of the gripping point on the object surface and the normal and tangential directions of the object surface at the gripping point.

[0148] In this embodiment of the application, for each gripping point in the set of gripping points, the computer device converts the contact force satisfying the constraint conditions on the gripping point into multiple gripping spins based on the contact geometry parameters of the gripping point, thereby obtaining multiple gripping spins corresponding to each gripping point.

[0149] Here, the multiple gripping spins corresponding to the gripping point refer to the combination of forces and torques that can be applied at that gripping point. The constraint condition refers to the conditions that the contact force must satisfy to ensure the physical feasibility of the contact force; this constraint condition can be a pre-set condition.

[0150] In one possible implementation, the contact force at the gripping point includes a normal force and a tangential force. For example, for ease of calculation, the contact force is represented as a four-dimensional vector comprising a first tangential force, a second tangential force, a normal force, and a normal moment. The first and second tangential forces are two forces perpendicular to each other in the tangential plane direction, and the first, second, and normal forces are mutually perpendicular. This four-dimensional vector can describe the state of all forces applied to the gripping point during the robotic arm's gripping process.

[0151] For example, the contact force can be represented by the following formula (1).

[0152] f i =[f i1 ,f i2 ,f i3 ,f i4 ] T ; Formula (1)

[0153] Among them, f i f represents the contact force applied at the i-th gripping point in the set of gripping points. i1 f represents the first tangential force at the i-th grasping point. i2 f represents the second tangential force at the i-th grasping point. i3 f represents the normal force at the i-th grasping point. i4 Let T represent the normal torque at the i-th grasping point, and let T represent the transpose.

[0154] In one possible implementation, the contact force at the gripping point includes both normal and tangential forces. The aforementioned constraints include the following.

[0155] (1) The normal force is non-negative.

[0156] In this context, a non-negative normal force indicates that the force applied to the gripping point is a pressure (or thrust) towards the object's surface, rather than a tension force. In other words, the normal force is a force perpendicular to the object's surface and pointing inwards at the gripping point, not outwards. This constraint ensures stability and effectiveness during the gripping process.

[0157] (2) The tangential force is not greater than the product of the normal force and the friction coefficient of the object.

[0158] Tangential force is a parallel force acting on the contact surface and is related to the relative motion or potential sliding of the objects. The product of the normal force and the coefficient of friction of the object is the maximum value that the tangential force can reach. When friction exists at the contact surface, the magnitude of the tangential force cannot exceed this maximum value; otherwise, the contact surface will slide or slip. Therefore, it is necessary to ensure that the tangential force is not greater than the product of the normal force and the coefficient of friction of the object to ensure that the object does not slide due to excessive tangential force during the grasping process, thus maintaining a stable grasp.

[0159] For example, if the tangential force is divided into a first tangential force and a second tangential force, then the constraint condition is that the resultant force of the first tangential force and the second tangential force is not greater than the product of the normal force and the friction coefficient of the object.

[0160] For example, the forces that satisfy the above constraints can be called a friction cone. A friction cone is a set, where a cone represents all forces applied at the gripping point that satisfy the constraints. Then, the friction cone can be represented by the following formula (2).

[0161] F={f|f tangent ≤μ s f normal ,f normal ≥0}; formula (2)

[0162] Where F represents the friction cone, f represents the contact force, and f tangent f represents the tangential force in the contact force. normal μ represents the normal force in the contact force. s This represents the coefficient of friction.

[0163] In this implementation, the contact force satisfying the constraints is mapped to spinor space. These constraints include ensuring the normal force is non-negative to guarantee inward pressure on the object's surface, and that the tangential force is no greater than the product of the normal force and the object's coefficient of friction to prevent excessive tangential force from causing the object to slip. By appropriately setting these constraints, the grasping process can be guaranteed to conform to grasping mechanics, and the applied contact force can satisfy physical feasibility, thus ensuring the reliability of the grasping judgment result.

[0164] In one possible implementation, the contact geometry parameters of the gripping point include the position vector, normal vector, and tangent vector of the gripping point. For each gripping point in the set of gripping points, steps 3021-3022 are performed to determine multiple gripping spinors corresponding to each gripping point.

[0165] 3021. Determine the grabbing matrix of the grabbing point based on the position vector, normal vector, and tangent vector of the grabbing point.

[0166] Among them, the position vector of the gripping point represents the position of the gripping point on the object surface, the normal vector of the gripping point represents the normal direction of the object surface at the gripping point, and the tangent vector of the gripping point represents the tangent direction of the object surface at the gripping point.

[0167] The computer device determines the gripping matrix of the gripping point based on the position vector, normal vector, and tangent vector of the gripping point. This gripping matrix is ​​a mapping tool used to map the contact forces that satisfy the constraints at the gripping point to the spinor space.

[0168] Optionally, the process by which the computer device determines the crawling matrix includes: determining the position vector, normal vector, first tangent vector, and second tangent vector of the crawling point, wherein the normal vector, first tangent vector, and second tangent vector are mutually perpendicular. The position vector and normal vector are cross-producted to obtain a first mapping vector; the position vector and first tangent vector are cross-producted to obtain a second mapping vector; the position vector and second tangent vector are cross-producted to obtain a third mapping vector; and the crawling matrix is ​​constructed based on the normal vector, first tangent vector, second tangent vector, first mapping vector, second mapping vector, and third mapping vector.

[0169] Wherein, the first tangent vector represents the direction of the first tangential force, the second tangent vector represents the direction of the second tangential force, and the normal vector represents the direction of the normal force. For example, the normal vector, the first tangent vector, and the second tangent vector are all unit vectors.

[0170] For example, the normal vector, the first tangent vector, the second tangent vector, and 0 are used as the first row elements of the capture matrix, and the first mapping vector, the second mapping vector, the third mapping vector, and the normal vector are used as the second row elements of the capture matrix. Then, the capture matrix can be represented by the following formula (3).

[0171]

[0172] Among them, G i Let n represent the crawling matrix of the i-th crawling point in the crawling point set. i Let o represent the normal vector of the i-th capture point. i Let e ​​represent the first tangent vector of the i-th capture point. i Let p represent the second tangent vector of the i-th grasping point. i This represents the position vector of the i-th capture point.

[0173] It should be noted that the position vector, normal vector, first tangent vector, and second tangent vector mentioned above are all three-dimensional vectors in three-dimensional space. Therefore, the grasping matrix is ​​a six-dimensional structure, that is, a 6×4 matrix.

[0174] In this implementation, multiple mapping vectors are determined based on the position vector, normal vector, and tangent vector of the grasping point. Finally, the grasping matrix is ​​constructed by the normal vector, tangent vector, and mapping vector, which ensures a comprehensive consideration of the mechanical state of the grasping point, improves the accuracy and reliability of the grasping matrix, and makes the calculation of the grasping spinor more accurate.

[0175] 3022. Multiply the gripping matrix of the gripping point by all the contact forces that satisfy the constraints at the gripping point to obtain multiple gripping spinors corresponding to the gripping point.

[0176] For example, if the grasping matrix is ​​a 6×4 matrix and the contact force is a 4×1 matrix (i.e., a four-dimensional vector), then the grasping spinor obtained by multiplying the grasping matrix and the contact force is a 6×1 matrix (i.e., a six-dimensional vector).

[0177] In this implementation, the grasping matrix is ​​determined based on the position vector, normal vector, and tangent vector of each grasping point. The grasping matrix is ​​then multiplied by the contact force that satisfies the constraint conditions to obtain multiple grasping spinors. Therefore, the grasping spinors can accurately describe the mechanical behavior of the grasping point, which is beneficial for describing the force in complex grasping situations, thereby ensuring the accuracy of grasping judgment.

[0178] 303. The computer device determines the gripping screw space based on multiple gripping screws. The gripping screw space is the smallest space that contains multiple gripping screws.

[0179] After determining the multiple gripping screws corresponding to each gripping point in the set of gripping points, the computer device determines the gripping screw space based on these gripping screws. In the gripping mechanics of the robotic arm, the focus is not only on the forces and torques applied independently at each gripping point, but also on all possible resultant forces and torques that can be formed when the forces and torques applied at all gripping points act together. Therefore, the gripping screw space, determined based on the multiple gripping screws corresponding to each gripping point, can describe the possibilities of forces and torques applied at all gripping points in the set of gripping points.

[0180] In one possible implementation, multiple grasping spinors are all six-dimensional vectors. The computer device determines the smallest space containing these multiple grasping spinors as the grasping spinor space, which is then a six-dimensional convex hull.

[0181] In one possible implementation, multiple grasping spinors are all six-dimensional vectors. For each grasping spinor, the computer device determines the first, second, and last elements of the grasping spinor, and these elements constitute a three-dimensional target grasping spinor. The computer device defines the minimum space containing the multiple target grasping spinors as the grasping spinor space, which is then a three-dimensional convex hull.

[0182] In one possible implementation, the grasping spinor space can be represented by the following formula (4).

[0183]

[0184] in, G represents the grasping of the spinor space. i Let f represent the capture matrix of the i-th capture point in the capture point set. i F represents the contact force applied at the i-th gripping point. i Let m represent the friction cone formed by the contact forces satisfying the constraints at the i-th gripping point, and m represent the number of at least three gripping points in the set of gripping points.

[0185] 304. If the gripping screw space satisfies the stable gripping condition, the computer device determines that the gripping judgment result of the gripping point set indicates that the object can be gripped based on the gripping point set. If the gripping screw space does not satisfy the stable gripping condition, the computer device determines that the gripping judgment result of the gripping point set indicates that the object cannot be gripped based on the gripping point set.

[0186] After determining the gripping screw space, the computer device checks whether the gripping screw space satisfies the stable gripping conditions. If the gripping screw space satisfies the stable gripping conditions, it is determined that the robot arm can grasp the object by applying force at each gripping point in the gripping point set. If the gripping screw space does not satisfy the stable gripping conditions, it is determined that the robot arm cannot grasp the object by applying force at each gripping point in the gripping point set.

[0187] The stable capture condition is a pre-set condition.

[0188] In this implementation, the contact geometry parameters of the grasping points are converted into multiple grasping screws. The smallest space containing these grasping screws is taken as the grasping screw space. Then, by checking whether the grasping screw space satisfies the stable grasping condition, it is determined whether the set of grasping points can grasp an object. Therefore, for robotic hands with at least three fingers, a quantitative analysis method based on grasping screw space is provided to determine whether a set of grasping points can grasp an object, ensuring the reliability of the grasping task and improving the success rate and stability of grasping.

[0189] In one possible implementation, the stable fetching conditions include the following.

[0190] (1) The origin of the screw coordinate system is located in the grasping screw space. The screw coordinate system refers to the coordinate system in which the grasping screw space is located. The vectors in the screw coordinate system are used to describe forces.

[0191] The origin of this screwonic coordinate system lies in the grasping screw space, indicating that this grasping screw space is force-closed. This means that when the robot grips at each grasping point in the grasping point set, it can balance certain forces and torques applied externally, preventing the object from sliding or flipping during the grasping process. If the grasping screw space does not contain the origin of the screwonic coordinate system, the robot gripping process lacks force closure, and the object may slide or rotate, leading to grasping failure. Therefore, the grasping screw space corresponding to the grasping point set needs to satisfy this condition to ensure that the robot can grasp objects based on the grasping point set.

[0192] In this system, the origin of the screw coordinate system is the zero vector, which has the same dimension as the grasping screw. For example, the grasping screw is a three-dimensional vector. The grasping screw space is a three-dimensional space, and the screw coordinate system is a three-dimensional coordinate system, the origin of which is the three-dimensional zero vector.

[0193] (2) The object’s gravity is located in the grasp spinor space.

[0194] The fact that an object's gravity lies within the gripping spinor space means that when a robotic arm grasps an object at various gripping points within the gripping point set, it can balance or counteract the object's gravity by adjusting the applied forces and torques to ensure a stable grasp. If the object's gravity does not lie within the gripping spinor space, the robotic arm may be unable to balance or counteract the object's gravity by adjusting the applied forces and torques, and the object may slide or rotate, leading to grasping failure. Therefore, the gripping spinor space corresponding to the gripping point set needs to satisfy this condition to ensure that the robotic arm can grasp objects based on the gripping point set.

[0195] Optionally, the screw coordinate system is a three-dimensional coordinate system, which captures the screw space to form a three-dimensional convex polyhedron. The three-dimensional coordinate system includes a first coordinate axis, a second coordinate axis, and a third coordinate axis that are perpendicular to each other. The first and second coordinate axes are perpendicular to the direction of gravity, and the third coordinate axis is parallel to the direction of gravity.

[0196] The process of determining whether the gravity of an object is located in the grasping screw space includes: mapping the convex polyhedron formed by the grasping screw space onto the plane containing the first and second coordinate axes to obtain a polygon; determining the minimum distance from the origin of the screw coordinate system to the edge of the polygon; if the value of the minimum distance is greater than the value of the object's gravity, then the gravity is determined to be located in the grasping screw space.

[0197] Since the grasping screw space is a three-dimensional convex polyhedron, directly analyzing whether the object's gravity lies within it is complex. Therefore, the three-dimensional convex polyhedron is mapped onto the plane containing the first and second coordinate axes, resulting in a two-dimensional polygon. After obtaining the two-dimensional polygon, the minimum distance from the origin to its edge is determined. If this minimum distance is greater than the value of gravity, it indicates that gravity lies within the grasping screw space, and the grasping screw space can balance gravity. If the minimum distance is not greater than the value of gravity, it indicates that gravity does not lie within the grasping screw space, and the grasping screw space cannot balance gravity.

[0198] In this implementation, the grasping screw space is mapped as a polygon on a plane perpendicular to the direction of gravity. By comparing the minimum distance from the origin to the edge of the polygon with the magnitude of gravity, it is determined whether gravity is located in the grasping screw space. Therefore, an effective way to determine whether gravity is located in the grasping screw space is provided. This effectively simplifies the high-dimensional geometry of the grasping screw space, simplifies complex calculations in a two-dimensional plane, facilitates rapid analysis and processing, and helps improve processing efficiency.

[0199] Optionally, if the minimum distance is greater than the object's weight, then a grasping score is determined based on the minimum distance. The grasping score is positively correlated with the minimum distance and is used to reflect the grasping stability of the object based on the grasping point set.

[0200] If the minimum distance is greater than the object's weight, it indicates that the gripping spinor space of the gripping point set can balance gravity, and the gripping point set is likely a valid gripping point set. Therefore, the computer device determines a gripping score based on the minimum distance to represent the gripping stability of the object based on the gripping point set. A larger minimum distance results in a larger gripping score, indicating higher gripping stability. Conversely, a smaller minimum distance results in a smaller gripping score, indicating lower gripping stability.

[0201] For example, the computer device normalizes the minimum distance to obtain a crawl score, which can then be used as one of the data items in the crawl tag corresponding to the crawl point set.

[0202] In this implementation, when gravity is located in the grasping spinor space, the grasping score of the grasping point set is determined based on the minimum distance to reflect the grasping stability of grasping objects based on the grasping point set. This provides a quantitative indicator for the grasping stability, which helps to select a more suitable grasping point set based on the grasping score, thereby improving the grasping success rate.

[0203] In this embodiment, the ability to grasp an object is determined by judging whether the origin of the screw coordinate system and the object's gravity are located in the grasping screw space. Therefore, after mapping the contact force to the grasping screw space, the discrimination method is very simple and no complex judgment is required, which improves the convenience of determining the grasping discrimination result.

[0204] Furthermore, the introduction of gripping spinor space to determine the gripping stability of the gripping point set enhances the reliability of determining the gripping point set, ensuring that the applied forces and torques during the gripping process meet physical feasibility requirements, and significantly improving the success rate and stability of the gripping point set. Especially when handling fragile or precision objects, it can effectively prevent damage caused by excessive contact force.

[0205] 305. If the grasping judgment result of the grasping point set indicates that the object can be grasped based on the grasping point set, the computer device generates a grasping label for the object based on the contact geometry parameters of the grasping points in the grasping point set. The grasping label of the object is used to indicate the pose of the robot arm when grasping the object.

[0206] If the grasping judgment result indicates that the object can be grasped based on the grasping point set, then the grasping point set is a valid grasping point set. The computer device then generates a grasping label for the object based on the contact geometry parameters of each grasping point in the grasping point set. The grasping label of the object indicates the pose of each finger of the robot arm when grasping the object.

[0207] In one possible implementation, if the grasping determination result of the grasping point set indicates that an object can be grasped based on the grasping point set, then the computer device simulates the grasping point set to obtain a simulation result. If the simulation result indicates that the robot arm successfully grasps the object using the grasping point set, then a grasping tag for the object is generated based on the contact geometry parameters of the grasping points in the grasping point set. The simulation process is described below. Figure 5 The embodiments shown are not described here.

[0208] In one possible implementation, the object's grab tag is represented by a circular notation, as detailed in the following embodiments, which will not be elaborated here.

[0209] 306. The computer device adds the object's grasping label to the training dataset, which is used to train the robotic arm control model.

[0210] The object's grasping label indicates the robot's pose when the object is effectively grasped. Therefore, the computer device adds the object's grasping label to the training dataset. Subsequently, the training dataset can be used to train the robot's control model, which is used to predict the robot's pose when grasping an object.

[0211] Optionally, the training dataset includes object data and a grasping label, whereby the object data represents the object's pose. The training process of the robotic arm control model then includes: inputting the object data into the robotic arm control model to obtain the predicted pose; and training the robotic arm control model based on the predicted pose and the grasping label. The training objective is to reduce the discrepancy between the predicted pose and the grasping label.

[0212] The method provided in this application obtains a set of gripping points on the surface of an object. Whether the object can be stably gripped based on this set of gripping points depends on the position and orientation of each gripping point in the set. The contact geometry parameters of the gripping points represent their position and orientation. Therefore, based on the contact geometry parameters of each gripping point in the set, it can be determined whether the object can be gripped. If the object can be gripped, a gripping label for the object is generated based on the contact geometry parameters of the gripping points in the set to represent the pose of the robotic arm when gripping the object. This gripping label can then be added to the training dataset used to train the robotic arm control model. Therefore, this method achieves intelligent generation of training datasets capable of gripping objects, ensuring the quality of the training dataset, eliminating the need for manual collection and annotation, and improving the efficiency of generating training data for the robotic arm control model.

[0213] Based on the above embodiments, after determining that an object can be grasped based on the set of grasping points, it is also necessary to simulate the set of grasping points. See below for details. Figure 5 Examples of implementations. Figure 5 This is a flowchart illustrating another method for generating training data for a robotic arm control model provided in this application embodiment. This application embodiment is executed by a computer device. See also... Figure 5 The method includes:

[0214] 501. A computer device acquires a set of gripping points on the surface of an object, the set of gripping points including at least three gripping points, which are contact points of the robotic arm when gripping the object.

[0215] The process of step 501 is the same as that of step 301 above, and will not be repeated here.

[0216] 502. The computer device determines the grasping judgment result of the grasping point set based on the contact geometric parameters of the grasping points in the grasping point set. The contact geometric parameters of the grasping points are used to represent the position of the grasping points and the normal and tangential directions of the object surface at the grasping points.

[0217] The process of step 502 is the same as that of steps 302-304 above, and will not be repeated here.

[0218] 503. If the grasping judgment result indicates that an object can be grasped based on the grasping point set, then the computer device determines the fingertip target pose of the robot arm based on the contact geometry parameters of the grasping points in the grasping point set.

[0219] In this embodiment, after determining the grasping discrimination result based on the contact geometry parameters of each grasping point in the grasping point set, if the grasping discrimination result indicates that the object can be grasped, it is necessary to simulate the grasping point set to obtain the simulation result. If the simulation result also indicates that the object can be grasped, then a grasping label is generated based on the contact geometry parameters of each grasping point in the grasping point set. Steps 503 and 504 are the process of simulating the grasping point set.

[0220] Since the gripping point is the contact point between the fingers and the object when the robotic arm grasps it, the position of the gripping point determines the position of the robotic arm's fingertips on the object's surface, and the direction of the gripping point determines how the robotic arm's fingertips should apply force on the object's surface (that is, the posture of the robotic arm's fingertips). Therefore, the position and direction (posture) of the robotic arm's fingertips are aligned one-to-one with the position and direction of the gripping points. Thus, the computer equipment can determine the target pose of the robotic arm's fingertips based on the contact geometry parameters of the gripping points. This target pose represents the position and posture of the robotic fingertips when the robotic arm applies contact force at each gripping point in the set of gripping points.

[0221] In other words, when the position and posture of the robotic fingertip reach the position and posture indicated by the target fingertip posture, the robotic fingertip contacts each gripping point in the gripping point set, and then grips the object by applying contact force.

[0222] 504. In a simulation environment, computer equipment controls a robotic arm to grasp an object according to the target pose of the fingertip, and obtains simulation results.

[0223] After determining the target pose of the robotic arm's fingertip, the computer device verifies it in the Pybullet simulation environment, controls the robotic arm to move to the position and posture indicated by the target pose of the fingertip, performs object grasping, and obtains the grasping result, which is also the simulation result. The simulation result is used to indicate whether the robotic arm has successfully grasped the object using the set of grasping points.

[0224] Figure 6 This is a simulation result of a robotic arm grasping an object, as provided in the embodiments of this application. Figure 6 As shown, in the simulation environment, the robotic arm 601 is controlled to grasp the object 602 according to the target pose of the fingertip.

[0225] In this implementation, the fingertip target pose of the robot is determined based on the contact geometry parameters of the gripping point set, and the gripping is simulated in a simulation environment to verify the actual gripping effect of the gripping point set, thereby improving the reliability of the gripping point set and helping to select the most suitable gripping point set more efficiently in complex environments.

[0226] In one possible implementation, the robotic hand includes finger joints and fingertips. By controlling the joint pose of each finger, each finger can bend and extend, thereby driving the fingertip pose. Therefore, based on the target pose of the fingertip, an inverse kinematics algorithm is needed to determine the target joint pose of the robotic hand. By controlling the joints of the robotic hand to move to that target joint pose, the fingertip of the robotic hand can be moved to that target fingertip pose.

[0227] The computer device performs inverse kinematics solving by executing the following iterative process: It converts the joint pose of the manipulator into the fingertip pose using the manipulator's pose transformation parameters; it updates the pose transformation parameters based on the fingertip pose error between the fingertip pose and the target fingertip pose; it converts the fingertip pose error into joint pose error using the updated pose transformation parameters; and it updates the joint pose of the manipulator based on the joint pose error. This is an iterative process. During iteration, the computer device stops the iteration process when the iteration termination condition is met. The currently updated joint pose (i.e., the latest joint pose) is then the target joint pose. Therefore, in the simulation environment, the computer device controls the manipulator to grasp the object according to the updated joint pose, obtaining the simulation results.

[0228] The pose transformation parameters of the robotic arm are used to convert between the joint pose and the fingertip pose. The joint pose represents the position and orientation of the finger joints, while the fingertip pose represents the position and orientation of the fingertip. It should be noted that the pose transformation parameters of the robotic arm are iteratively updated during the iteration process to make them increasingly accurate.

[0229] First, the computer determines the current joint pose of the robotic arm. Using the current pose transformation parameters, it converts the current joint pose into the fingertip pose of the robotic arm, determining the fingertip pose error between this pose and the target fingertip pose. If the current pose transformation parameters and the current joint pose are accurate enough, the current fingertip pose will be sufficiently close to the target fingertip pose, meaning the fingertip pose error is small enough. Therefore, based on the fingertip pose error, the pose transformation parameters are updated, with the goal of reducing the fingertip pose error. The updated pose transformation parameters then convert the fingertip pose error into joint pose error, which represents the difference between the current joint pose and the target joint pose. Therefore, based on the joint pose error, the joint pose of the robotic arm is updated, completing one iteration. In the next iteration, the updated joint pose and the updated pose transformation parameters are used in the calculation.

[0230] For example, the pose transformation parameters of a robotic arm are represented by its Jacobian matrix, which describes the transformation relationship between joint poses and fingertip poses. For instance, a computer device uses the Jacobian matrix to convert joint poses into fingertip poses. Conversely, the computer device uses the pseudo-inverse of the Jacobian matrix to convert fingertip poses into joint poses.

[0231] For example, the computer device adds the joint pose error to the joint pose to obtain the updated joint pose. For example, the joint pose includes the rotation angles of each finger joint of the robotic arm. Figure 7 This is a schematic diagram of a joint pose provided in an embodiment of this application, such as... Figure 7 As shown, the robotic hand includes fingers F1, F2, and F3. The joint poses include the rotation angles θ of each joint on finger F1. 11 θ 12 and θ 13 The rotation angle θ of each joint on finger F2 21 θ 22 and θ 23 The rotation angle θ of each joint on finger F3 31 and θ 32 .

[0232] Optionally, the computer device stops the iteration process in response to the iteration process meeting the iteration termination condition, including: during the iteration process, if the fingertip pose error between the updated fingertip pose and the fingertip target pose is less than the error threshold after converting the updated joint pose to the updated fingertip pose using the updated pose conversion parameters, then the iteration process is stopped.

[0233] If the fingertip pose error between the updated fingertip pose and the target fingertip pose is less than the error threshold, it means that the updated pose transformation parameters are accurate enough, and the updated joint pose is accurate enough. The updated joint pose matches the target fingertip pose, so the iteration process stops. In the simulation environment, the computer controls the robot to grasp the object according to the updated joint pose, and the simulation results are obtained.

[0234] For example, the computer device implements the simulation process through the following steps 1-4.

[0235] 1. Input the target pose of the fingertip and the current joint pose.

[0236] 2. Begin iteration. Convert joint pose to fingertip pose using the Jacobian matrix.

[0237] 3. Determine the fingertip pose error between the fingertip pose and the target fingertip pose. If the fingertip pose error is greater than the error threshold, update the Jacobian matrix based on the fingertip pose error and execute step 4 below. If the fingertip pose error is not greater than the error threshold, stop the iteration process and use the current joint pose as the joint pose for controlling the robot.

[0238] For example, the fingertip pose error can be represented by the following formula (5).

[0239] e = [x * -x,log(R * R T )] T ; Formula (5)

[0240] Where e represents the fingertip pose error, x * Let R represent the position parameters (e.g., position vector) in the fingertip target pose, where x represents the position parameters in the fingertip pose. * R represents the attitude parameters (e.g., rotation matrix) in the fingertip target pose. T represents the transpose.

[0241] 4. Convert the fingertip pose error into joint pose error by using the pseudo-inverse of the Jacobian matrix. Update the joint pose based on the joint pose error to obtain the updated joint pose, and return to step 2.

[0242] For example, the joint pose error can be represented by the following formula (6), and the updated joint pose can be represented by the following formula (7).

[0243]

[0244] q ′ =q + Δq; Formula (7)

[0245] Where Δq represents the joint pose error, and J represents the Jacobian matrix. Let e ​​represent the pseudo-inverse of the Jacobian matrix, e represent the fingertip pose error, α represent the scaling factor, and q represent the current joint pose. ′ This indicates the updated joint pose.

[0246] In this implementation, the pose transformation parameters of the manipulator are used to convert the joint pose into the fingertip pose. Based on the error between the fingertip pose and the target fingertip pose, the pose transformation parameters and the joint pose are updated synchronously. During the iteration process, the accuracy of the pose transformation parameters becomes higher and higher, and consequently the accuracy of the joint pose also becomes higher and higher. When the iterative contact condition is reached, the manipulator is controlled to grasp according to the updated joint pose, which ensures that the fingertip pose of the manipulator is the target fingertip pose during grasping. This improves the accuracy and efficiency of controlling the manipulator and ensures the reliability and efficiency of obtaining simulation results.

[0247] Furthermore, if the error between the fingertip pose and the target fingertip pose is small enough, it indicates that the joint pose used to determine the fingertip pose is accurate enough, and the iteration process stops. Therefore, the introduction of an error threshold judgment method can avoid redundant adjustments caused by excessive iteration, thereby improving the real-time performance and accuracy of grasping control.

[0248] Furthermore, the inverse kinematics solution method based on the Jacobian matrix not only improves the solution speed but also enhances the accuracy of the results. This means that the robotic arm can adjust its pose more quickly to adapt to different grasping tasks, thereby exhibiting greater flexibility and responsiveness in dynamic environments.

[0249] 505. If the simulation results indicate that the robotic arm successfully grasps the object using the set of gripping points, the computer device generates a gripping label for the object based on the contact geometry parameters of the gripping points in the set of gripping points.

[0250] In this implementation, after calculating and determining that the set of gripping points can grasp the object, the feasibility of the selected set of gripping points is first verified through simulation to determine whether the robotic arm can successfully grasp the object. Simulation ensures that the set of gripping points is not only theoretically feasible but also successfully performs the grasping action in practice. Therefore, it provides more reliable gripping tags, thereby helping to optimize the gripping point selection process. Furthermore, filtering the gripping point set first through calculation and then through simulation effectively reduces the number of simulations, saving costs and time, and increasing the accuracy of the grasping strategy.

[0251] In this embodiment, when both the grasping judgment result and the simulation result indicate that the robotic arm can grasp the object based on the grasping point set, a grasping label for the object is generated based on the contact geometry parameters of the grasping points in the grasping point set. That is, the computer device determines the effective grasping point set by executing steps 1-6.

[0252] 1. The computer device randomly samples multiple points on the surface of the object's three-dimensional mesh model to obtain a point set.

[0253] 2. The computer device creates a first candidate list and a second candidate list. The first candidate list is used to store the set of candidate crawl points, and the second candidate list is used to store the set of valid crawl points that have been filtered out.

[0254] 3. The computer device creates a third candidate list for each point in the point set. The second candidate list is used to store other points that match the current point, that is, to store other points that satisfy the finger distance condition and the finger angle condition with the current point.

[0255] 4. For point i in the point set, the computer device traverses the other points in the point set. For the currently traversed point j, if point i and point j satisfy the above finger angle condition and finger distance condition, then point j is added to the third candidate list of point i.

[0256] 5. For point i in the point set, the computer device traverses the third candidate list of point i. If point i and points j and k in the second candidate list satisfy the above finger angle condition and finger distance condition, then point i, point j and point k are added to the first candidate list as a set of grasping points.

[0257] 6. For each set of grab points in the first candidate list, call checkstable to determine the grab judgment result of the grab point set, and call checkIK to determine the simulation result of the grab point set. If both the grab judgment result and the simulation result indicate that the object can be grabbed based on the grab point set, then add the grab point set to the second candidate list. Subsequently, grab tags for the object are generated based on the grab point sets in the second candidate list.

[0258] 506. The computer device adds the object's grasping label to the training dataset, which is used to train the robotic arm control model.

[0259] The method provided in this application obtains a set of gripping points on the surface of an object. Whether the object can be stably gripped based on this set of gripping points depends on the position and orientation of each gripping point in the set. The contact geometry parameters of the gripping points represent their position and orientation. Therefore, based on the contact geometry parameters of each gripping point in the set, it can be determined whether the object can be gripped. If the object can be gripped, a gripping label for the object is generated based on the contact geometry parameters of the gripping points in the set to represent the pose of the robotic arm when gripping the object. This gripping label can then be added to the training dataset used to train the robotic arm control model. Therefore, this method achieves intelligent generation of training datasets capable of gripping objects, ensuring the quality of the training dataset, eliminating the need for manual collection and annotation, and improving the efficiency of generating training data for the robotic arm control model.

[0260] In some embodiments, the robotic hand is a three-finger robotic hand, comprising a first finger, a second finger, and a third finger. The fingertip of the first finger is designated as the first fingertip, the fingertip of the second finger as the second fingertip, and the fingertip of the third finger as the third fingertip. The set of grasping points includes a first grasping point, a second grasping point, and a third grasping point. The first grasping point is the contact point between the first fingertip and the object when the robotic hand grasps the object; the second grasping point is the contact point between the second fingertip and the object when the robotic hand grasps the object; and the third grasping point is the contact point between the third fingertip and the object when the robotic hand grasps the object. For example, the first, second, and third fingers can be referred to as the middle finger, left finger, and right finger of the robotic hand, respectively.

[0261] In the above embodiments, the computer device generates a gripping tag for an object based on the contact geometry parameters of the gripping points in the gripping point set. The gripping tag for the object includes the following:

[0262] (1) The position vector and normal vector of the first fingertip.

[0263] The contact geometry parameters of the first gripping point include the position vector and normal vector of the first gripping point. The computer device uses the position vector of the first gripping point as the position vector of the first fingertip and the normal vector of the first gripping point as the normal vector of the first fingertip.

[0264] The position vector and normal vector are vectors in a three-dimensional coordinate system, which can be either the object coordinate system or the world coordinate system.

[0265] Figure 8 This is a schematic diagram of a tag-grabbing method provided in an embodiment of this application, such as... Figure 8As shown, the position vector of the first fingertip C is p3, and the normal vector is n3.

[0266] (2) The center position vector and roll angle of the grab circle, which refers to the circumcircle of the first grab point, the second grab point and the third grab point.

[0267] The first, second, and third gripping points are not on a straight line. Therefore, based on the first, second, and third gripping points, a unique circumcircle containing these three points can be determined. The computer device uses this circumcircle as the gripping circle of the three-finger robotic hand, which represents the circumcircle containing the three fingertips of the three-finger robotic hand.

[0268] Among them, the center position vector of the grab circle represents the location of the center of the grab circle, and the roll angle of the grab circle refers to the angle between the plane on which the grab circle is located and the horizontal plane. Figure 8 This is a schematic diagram of a tag-grabbing method provided in an embodiment of this application, such as... Figure 8 As shown, the first fingertip C, the second fingertip A, and the third fingertip B are located on the boundary line of the grasping circle, and the center position vector of the grasping circle is O. The center position vector O = {O x O y O z The roll angle of the grasping circle is}

[0269] For example, the contact geometry parameters of the first gripping point include the position vector of the first gripping point, the contact geometry parameters of the second gripping point include the position vector of the second gripping point, and the contact geometry parameters of the third gripping point include the position vector of the third gripping point. Therefore, based on the position vectors of the first gripping point, the second gripping point, and the third gripping point, the computer device can determine the circumcircle where the first gripping point, the second gripping point, and the third gripping point are located, that is, determine the gripping circle.

[0270] (3) The angle between the first fingertip along the grasping circle and the second fingertip.

[0271] The grasping label includes the position vector of the first fingertip and the angle between the first fingertip and the second fingertip along the grasping circle. Therefore, the position vector of the second fingertip can be determined based on the position vector of the first fingertip and this angle. In this embodiment, the grasping label does not include the position vector of the second fingertip, but includes the angle, in order to represent the grasping label with a circular representation, which is more intuitive and easier for the mechanical control model to understand and learn.

[0272] Figure 8 This is a schematic diagram of a tag-grabbing method provided in an embodiment of this application, such as... Figure 8 As shown, the angle between the first fingertip C along the grasping circle and the second fingertip A is...

[0273] For example, the contact geometry parameters of the first gripping point include the position vector of the first gripping point, and the contact geometry parameters of the second gripping point include the position vector of the second gripping point. The position vector of the first gripping point can be used as the position vector of the first fingertip, and the position vector of the second gripping point can be used as the position vector of the second fingertip. Therefore, given the gripping circle, the computer device can determine the angle between the first fingertip and the second fingertip along the gripping circle based on these two position vectors.

[0274] (4) The angle between the first fingertip along the grasping circle and the third fingertip.

[0275] The grasping label includes the position vector of the first fingertip and the angle between the first fingertip and the third fingertip along the grasping circle. Therefore, the position vector of the third fingertip can be determined based on the position vector of the first fingertip and this angle. In this embodiment, the grasping label does not include the position vector of the third fingertip, but includes the angle. This is to represent the grasping label using a circular representation, which is more intuitive and facilitates understanding and learning by the mechanical control model.

[0276] Figure 8 This is a schematic diagram of a tag-grabbing method provided in an embodiment of this application, such as... Figure 8 As shown, the angle between the first fingertip C and the third fingertip B along the grasping circle is...

[0277] For example, the contact geometry parameters of the first gripping point include the position vector of the first gripping point, and the contact geometry parameters of the third gripping point include the position vector of the third gripping point. The position vector of the first gripping point can be used as the position vector of the first fingertip, and the position vector of the third gripping point can be used as the position vector of the third fingertip. Therefore, given the gripping circle, the computer device can determine the angle between the first fingertip and the third fingertip along the gripping circle based on these two position vectors.

[0278] (5) The normal vector of the second fingertip and the normal vector of the third fingertip.

[0279] The contact geometry parameters of the second gripping point include the normal vector of the second gripping point, which the computer device uses as the normal vector of the second fingertip. The contact geometry parameters of the third gripping point include the normal vector of the third gripping point, which the computer device uses as the normal vector of the third fingertip.

[0280] Figure 8 This is a schematic diagram of a tag-grabbing method provided in an embodiment of this application, such as... Figure 8 As shown, the normal vector of the second fingertip A is n1, and the normal vector of the third fingertip A is n2.

[0281] In one possible implementation, the object grasping label also includes a grasping score, which reflects the grasping stability of the robotic arm when grasping the object using the pose indicated by the grasping label. The method for determining this grasping score is described in the explanation of step 304 above. Subsequently, when training the robotic arm control model, the loss parameter corresponding to the grasping label can be weighted based on the grasping score.

[0282] For example, the grab tag of an object can be represented by the following formula (8).

[0283]

[0284] Where g represents the tag to be captured, and O represents the center position vector of the captured circle. p3 represents the roll angle of the grasping circle, p3 represents the position vector of the first fingertip, and n3 represents the normal vector of the first fingertip. This indicates the angle between the first fingertip and the second fingertip along the grasping circle. The angle between the first fingertip and the third fingertip along the grasping circle is represented by n1, the normal vector of the second fingertip is represented by n2, the normal vector of the third fingertip is represented by q, and the grasping score is represented by q, which has a range of 0 ≤ q ≤ 1.

[0285] In this implementation, due to the structural characteristics of the three-finger manipulator, a circular representation is used to represent the object's grasping label, which makes the description of the grasping pose of the three-finger manipulator more intuitive. This helps reduce computational complexity, improves the efficiency of subsequent training of the manipulator model, and enhances the efficiency of grasping through the manipulator control model.

[0286] Figure 9 This is a flowchart of another method for generating training data for a robotic arm control model provided in this application embodiment, such as... Figure 9 As shown, the method includes the following steps:

[0287] 901. Based on the superimposed surface partitioning method, the surface of an object is divided into multiple candidate patches.

[0288] 902. Based on the sampling strategy, sample and capture a set of points from multiple target patches.

[0289] Among them, the target patch is the candidate patch whose area is greater than the area threshold.

[0290] 903. Determine the grasping discrimination result based on the grasping spinor space.

[0291] The computer device determines the gripping screw space corresponding to the gripping point set based on the contact geometry parameters of each gripping point in the gripping point set, and determines the gripping discrimination result based on whether the gripping screw space satisfies the stable gripping condition.

[0292] 904. Obtain the inverse solution and joint target pose by using the pseudo-inverse method based on the Jacobian matrix.

[0293] The computer device determines the fingertip target pose of the robot arm based on the contact geometry parameters of each gripping point in the gripping point set, and obtains the inverse solution based on the pseudo-inverse method of the Jacobian matrix, converting the fingertip target pose into the joint target pose.

[0294] 905. In the simulation environment, control the robot arm to move according to the target joint pose and obtain the simulation results.

[0295] 906. Generate object grabbing labels and add them to the training dataset.

[0296] If both the grasping judgment result and the simulation result of the grasping point set indicate that the object can be grasped, then the grasping label of the object is generated based on the contact geometry parameters of each grasping point in the grasping point set.

[0297] Figure 10 This is a flowchart of another method for generating training data for a robotic arm control model provided in this application embodiment, such as... Figure 10 As shown, the method includes the following steps:

[0298] 1001. Divide the surface of an object into multiple facets.

[0299] 1002. Determine if the area of ​​the patch is greater than the area threshold. If it is greater than the area threshold, proceed to step 1003 below; if it is not greater than the area threshold, discard the patch.

[0300] 1003. Sample a set of grab points on multiple facets with an area greater than the area threshold.

[0301] 1004. Determine whether the set of grasping points satisfies the finger distance and finger angle conditions. If satisfied, proceed to step 1005 below; otherwise, discard the set of grasping points.

[0302] 1005. For a set of grasp points that meet the conditions, determine the grasp spinor space of the grasp point set.

[0303] 1006. Determine whether the gripping spinor space of the gripping point set satisfies the stable gripping condition. If it does, proceed to step 1007 below; otherwise, discard the gripping point set.

[0304] 1007. For a set of grab points that meet the conditions, perform a simulation on the set of grab points and obtain the simulation results.

[0305] 1008. Determine whether the simulation results indicate that the object can be successfully grasped. If yes, proceed to step 1009 below; otherwise, discard the set of grasping points.

[0306] 1009. For a set of crawl points that meet the conditions, generate a crawl tag based on the set of crawl points.

[0307] 1010. Add the captured labels to the training dataset.

[0308] Robotic hands, including three-fingered robotic hands, offer significant advantages in fine manipulation tasks due to their dexterity and grasping stability. However, generating training datasets for these three-fingered robotic hands still faces several challenges. First, the grasping labels for three-fingered robotic hands are complex. Traditional methods for representing grasping labels struggle to intuitively describe the contact relationship between the three fingertips and the object's surface, and they also involve high computational complexity. Second, when generating grasping labels, it's necessary to determine whether the labels possess grasping stability, i.e., whether they can stably grasp objects. Traditional methods for determining grasping stability are based on the mechanical models of two-fingered robotic hands and are not applicable to three-fingered robotic hands.

[0309] The training data generation method for the robotic arm control model provided in this application can be used to generate training data for the control model of a three-fingered robotic arm. On one hand, using circular representation to represent grasping labels provides an intuitive description of the grasping pose, effectively reducing computational complexity. On the other hand, sampling the grasping point set on the object surface using a superimposed surface partitioning method improves the sampling efficiency of the grasping point set, ensuring the richness and diversity of the dataset and providing a solid foundation for the robotic arm's adaptability in complex tasks. Furthermore, by using the grasping spinor space to determine whether the contact force applied by the three-fingered robotic arm when grasping an object satisfies the stable grasping condition, the reliability of the grasping task is enhanced, ensuring that the applied force and torque during the grasping process meet physical feasibility, thereby significantly improving the success rate and stability of the grasping. Finally, based on the inverse kinematics solution method, the joint target pose of the robotic arm is solved using the pseudo-inverse method of the Jacobian matrix according to the fingertip target pose, which not only improves the processing speed but also enhances the accuracy of the results. On the other hand, the set of grasping points is verified in a simulation environment to ensure that the selected set of grasping points not only has high quality in theory, but also performs well in actual simulation, thereby improving the reliability of grasping labels and generating a high-quality training dataset for the three-finger manipulator, providing reliable data support for the manipulator's grasping task.

[0310] Therefore, the training data generation method for the robotic arm control model provided in this application provides strong support for the robotic arm's grasping ability through large-scale and diverse data acquisition and processing. This method not only has strong versatility but can also be customized for specific needs in different fields, thereby achieving excellent application results in industrial automation, medical assistance, home services, and other complex tasks.

[0311] For example, in the field of industrial automation, robotic arms are used in precision assembly scenarios. The method provided in this application can be used to generate a training dataset for such scenarios to train the robotic arm's control model. In precision assembly scenarios, the robotic arm needs to accurately grasp electronic components or complex parts. These objects are typically small in size, irregular in shape, and extremely sensitive to assembly errors, requiring extremely high accuracy in grasping. The robotic arm control model trained on the training dataset generated by this method ensures that the robotic arm can grasp different objects based on their shape, weight, surface friction, and other characteristics, thereby improving the success rate of grasping and assembly efficiency.

[0312] For example, in the field of medical assistance, robotic arms are used to perform high-precision, high-stability grasping tasks, such as the precise operation of surgical instruments and the dispensing and delivery of medications. The method provided in this application can generate a training dataset in this scenario to train the robotic arm's control model. In medical surgery, the robotic arm control model trained on the training dataset generated by this method ensures that the robotic arm can automatically adjust its grasping strategy according to the shape, material, size, and other characteristics of different instruments, achieving more precise grasping and flexible operation, reducing human error, and improving the success rate of surgery.

[0313] For example, in the field of home services, robotic arms are used to perform tasks such as grasping everyday objects. The method provided in this application provides a training dataset for this scenario to train the robotic arm's control model. Everyday household items are diverse, including tableware, water cups, books, remote controls, and clothing. These objects not only vary in shape but also in weight distribution and grasping posture requirements. The robotic arm control model trained on the training dataset generated by this method ensures that the robotic arm can effectively identify the grasping difficulties and operation methods for each type of object.

[0314] For example, in the field of logistics sorting, robotic arms are used to sort packages. The method provided in this application provides a training dataset for this scenario to train the robotic arm's control model. In logistics sorting, packages vary greatly in shape, size, and weight. The robotic arm control model trained on the training dataset generated by this method ensures that the robotic arm adaptively adjusts its gripping posture. Regardless of changes in the size, shape, or surface characteristics of the package, the robotic arm can perform sorting operations efficiently and stably, greatly improving logistics efficiency.

[0315] In summary, the training dataset generation method provided in this application not only overcomes the limitations of traditional methods in grasping diverse objects, but also greatly enhances the application potential of robotic arms in various industries. Through systematic and large-scale accumulation of training datasets, robotic arms can perform efficient and precise grasping operations in more complex and dynamic environments, thereby promoting the development of industrial automation, medical assistance, home services, and other fields.

[0316] Figure 11 This is a schematic diagram of a training data generation device for a robotic arm control model provided in an embodiment of this application. See also... Figure 11 The device includes:

[0317] The acquisition module 1101 is used to acquire a set of gripping points on the surface of an object. The set of gripping points includes at least three gripping points, which are the contact points when the robotic arm grips the object.

[0318] The discrimination module 1102 is used to determine the discrimination result of the gripping point set based on the contact geometric parameters of the gripping points in the gripping point set. The contact geometric parameters of the gripping points are used to represent the position of the gripping point, the normal direction of the object surface at the gripping point, and the tangential direction.

[0319] The tag generation module 1103 is used to generate a gripping tag for an object based on the contact geometry parameters of the gripping points in the gripping point set if the gripping judgment result indicates that the object can be gripped based on the gripping point set. The gripping tag of the object is used to indicate the pose of the robot arm when gripping the object.

[0320] The label adding module 1104 is used to add the object grasping labels to the training dataset, which is used to train the robot control model.

[0321] The robotic arm control model training data generation device provided in this application acquires a set of gripping points on the surface of an object. Whether the gripping point set can stably grip the object depends on the position and orientation of each gripping point in the set. The contact geometry parameters of the gripping points represent their position and orientation. Therefore, based on the contact geometry parameters of each gripping point in the set, it can be determined whether the object can be gripped. If the object can be gripped, a gripping label for the object is generated based on the contact geometry parameters of the gripping points in the set to represent the pose of the robotic arm when gripping the object. This gripping label can then be added to the training dataset used to train the robotic arm control model. Therefore, this method achieves intelligent generation of training datasets capable of gripping objects, ensuring the quality of the training dataset, eliminating the need for manual collection and annotation, and improving the efficiency of generating training data for the robotic arm control model.

[0322] Optionally, see Figure 12 The discrimination module 1102 is used for:

[0323] The conversion unit 1112 is used to convert the contact forces satisfying the constraint conditions on the gripping points in the gripping point set into multiple gripping spins based on the contact geometry parameters of the gripping points in the gripping point set.

[0324] The conversion unit 1112 is used to determine the gripping screw space based on multiple gripping screws, wherein the gripping screw space is the smallest space containing multiple gripping screws.

[0325] The discrimination unit 1122 is used to determine that if the gripping screw space satisfies the stable gripping condition, the gripping discrimination result indicates that an object can be gripped based on the set of gripping points; if the gripping screw space does not satisfy the stable gripping condition, the gripping discrimination result indicates that an object cannot be gripped based on the set of gripping points.

[0326] Optionally, see Figure 12 The contact geometry parameters of the gripping point include the position vector, normal vector, and tangent vector of the gripping point; the transformation unit 1112 is used for:

[0327] For each crawl point in the crawl point set, perform the following operations:

[0328] The grabbing matrix of the grabbing point is determined based on the position vector, normal vector, and tangent vector of the grabbing point;

[0329] Multiply the gripping matrix of the gripping point by all the contact forces that satisfy the constraints at the gripping point to obtain multiple gripping spinors corresponding to the gripping point.

[0330] Optionally, see Figure 12 The conversion unit 1112 is used for:

[0331] Determine the position vector, normal vector, first tangent vector, and second tangent vector of the grab point, ensuring that the normal vector, first tangent vector, and second tangent vector are mutually perpendicular;

[0332] The first mapping vector is obtained by cross-product of the position vector and the normal vector; the second mapping vector is obtained by cross-product of the position vector and the first tangent vector; and the third mapping vector is obtained by cross-product of the position vector and the second tangent vector.

[0333] A capture matrix is ​​constructed based on the normal vector, the first tangent vector, the second tangent vector, the first mapping vector, the second mapping vector, and the third mapping vector.

[0334] Optionally, see Figure 12 The contact forces at the gripping point include normal and tangential forces; constraints include:

[0335] The normal force is non-negative;

[0336] The tangential force is no greater than the product of the normal force and the coefficient of friction of the object.

[0337] Optionally, see Figure 12 Stable crawling conditions include:

[0338] The origin of the screw coordinate system is located in the grasping screw space. The screw coordinate system refers to the coordinate system in which the grasping screw space is located. The vectors in the screw coordinate system are used to describe forces.

[0339] The object's gravity lies in the grasp spinor space.

[0340] Optionally, see Figure 12 The screw coordinate system is a three-dimensional coordinate system. The screw space is captured to form a three-dimensional convex polyhedron. The three-dimensional coordinate system includes a first coordinate axis, a second coordinate axis, and a third coordinate axis that are mutually perpendicular. The first and second axes are perpendicular to the direction of gravity, and the third axis is parallel to the direction of gravity. The discrimination module 1102 is also used for:

[0341] Mapping the convex polyhedron formed by the grasping spinor space onto the plane containing the first and second coordinate axes yields a polygon;

[0342] Determine the minimum distance from the origin of the spinor coordinate system to the edge of the polygon;

[0343] If the minimum distance is greater than the object's weight, then the gravity is determined to be located in the grasp spinor space.

[0344] Optionally, see Figure 12 The device also includes a fraction generation module 1105, used for:

[0345] If the minimum distance is greater than the object's weight, then the grasp score of the grasp point set is determined based on the minimum distance. The grasp score is positively correlated with the minimum distance and is used to reflect the grasping stability of the object based on the grasp point set.

[0346] Optionally, see Figure 12 Module 1101 is used for:

[0347] The surface of the object's 3D mesh model is divided into multiple candidate patches, where the angle between adjacent meshes in each candidate patch is less than an angle threshold.

[0348] Among multiple candidate patches, at least one target patch is identified, wherein the area of ​​the target patch is greater than an area threshold.

[0349] Sample a set of grab points for at least one target patch.

[0350] Optionally, see Figure 12 Module 1101 is used for:

[0351] Randomly sample at least three gripping points on the surface of the object's 3D mesh model;

[0352] If the distance between at least three gripping points satisfies the finger distance condition of the robotic arm, and the angle between the normal vectors of at least three gripping points satisfies the finger angle condition of the robotic arm, then at least three gripping points constitute a gripping point set.

[0353] Optionally, see Figure 12 Tag generation module 1103 is used for:

[0354] The simulation unit 1113 is used to simulate the set of grasping points and obtain the simulation result if the grasping judgment result indicates that the object can be grasped based on the set of grasping points.

[0355] The tag generation unit 1123 is used to generate a gripping tag for the object based on the contact geometry parameters of the gripping points in the gripping point set if the simulation result indicates that the robot successfully grips the object using the gripping point set.

[0356] Optionally, see Figure 12 Simulation unit 1113 is used for:

[0357] Based on the contact geometry parameters of the gripping points in the gripping point set, the fingertip target pose of the robot is determined;

[0358] In the simulation environment, the robotic arm is controlled to grasp the object according to the target pose of the fingertip, and the simulation results are obtained.

[0359] Optionally, see Figure 12 Simulation unit 1113 is used for:

[0360] The following iterative process is performed: the joint pose of the robot hand is converted into the fingertip pose using the pose conversion parameters of the robot hand; the pose conversion parameters are updated based on the fingertip pose error between the fingertip pose and the target fingertip pose; the fingertip pose error is converted into joint pose error using the updated pose conversion parameters; and the joint pose of the robot hand is updated based on the joint pose error.

[0361] The iteration process stops when the iteration termination condition is met.

[0362] In the simulation environment, the robotic arm is controlled to grasp the object according to the updated joint pose, and the simulation results are obtained.

[0363] Optionally, see Figure 12 Simulation unit 1113 is used for:

[0364] During the iteration process, the updated joint pose is converted into the updated fingertip pose using the updated pose transformation parameters;

[0365] If the fingertip pose error between the updated fingertip pose and the target fingertip pose is less than the error threshold, then the iteration process stops.

[0366] Optionally, see Figure 12 The robotic arm is a three-fingered robotic arm, comprising a first fingertip, a second fingertip, and a third fingertip. The set of grasping points includes a first grasping point, a second grasping point, and a third grasping point. The object's grasping label includes:

[0367] The position vector and normal vector of the first fingertip;

[0368] The center position vector and roll angle of the grab circle; the grab circle refers to the circumcircle of the first grab point, the second grab point, and the third grab point.

[0369] The angle between the first fingertip and the second fingertip along the grasping circle;

[0370] The angle between the first fingertip and the third fingertip;

[0371] The normal vector of the second fingertip and the normal vector of the third fingertip.

[0372] It should be noted that the training data generation device for the robotic arm control model provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the training data generation device for the robotic arm control model provided in the above embodiments and the training data generation method embodiments for the robotic arm control model belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0373] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one computer program, which is loaded and executed by the processor to perform the operations performed in the training data generation method for the robotic arm control model described above.

[0374] Optionally, the computer device is provided as a terminal. Figure 13 A schematic diagram of the structure of a terminal 1300 provided in an exemplary embodiment of this application is shown.

[0375] Terminal 1300 includes a processor 1301 and a memory 1302.

[0376] Processor 1301 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1301 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1301 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1301 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1301 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0377] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory. The memory 1302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1302 are used to store at least one computer program, which is used by the processor 1301 to implement the training data generation method for the robot control model provided in the method embodiments of this application.

[0378] In some embodiments, the terminal 1300 may also optionally include: a peripheral device interface 1303 and at least one peripheral device. The processor 1301, memory 1302, and peripheral device interface 1303 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1303 via a bus, signal line, or circuit board. Optionally, the peripheral device includes at least one of: a radio frequency circuit 1304, a display screen 1305, a camera assembly 1306, an audio circuit 1307, and a power supply 1308.

[0379] Peripheral device interface 1303 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1301 and memory 1302. In some embodiments, processor 1301, memory 1302 and peripheral device interface 1303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1301, memory 1302 and peripheral device interface 1303 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0380] The radio frequency (RF) circuit 1304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1304 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1304 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1304 can communicate with other devices through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1304 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0381] Display screen 1305 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1305 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1301 for processing. In this case, display screen 1305 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1305, disposed on the front panel of terminal 1300; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1300 or in a folded design; in still other embodiments, display screen 1305 may be a flexible display screen, disposed on a curved or folded surface of terminal 1300. Furthermore, display screen 1305 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1305 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0382] The camera assembly 1306 is used to acquire images or videos. Optionally, the camera assembly 1306 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the terminal 1300, and the rear-facing camera is disposed on the back of the terminal 1300. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1306 may also include a flash. The flash may be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0383] The audio circuit 1307 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1301 for processing, or input to the radio frequency circuit 1304 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 1300. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1301 or the radio frequency circuit 1304 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1307 may also include a headphone jack.

[0384] Power supply 1308 is used to power the various components in terminal 1300. Power supply 1308 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1308 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0385] Those skilled in the art will understand that Figure 13 The structure shown does not constitute a limitation on terminal 1300 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0386] Optionally, the computer device is provided as a server. Figure 14This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1400 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1401 and one or more memories 1402. The memories 1402 store at least one computer program, which is loaded and executed by the processor 1401 to implement the methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0387] This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the operations performed by the training data generation method for the robotic arm control model described above.

[0388] This application also provides a computer program product, including a computer program loaded and executed by a processor to perform the operations performed by the training data generation method for the robotic arm control model as described in the above embodiments.

[0389] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0390] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. A method for generating training data for a robotic arm control model, characterized in that, The method includes: Obtain a set of gripping points on the surface of an object, the set of gripping points including at least three gripping points, the at least three gripping points being contact points when the robotic arm grips the object; Based on the contact geometry parameters of the gripping points in the gripping point set, the gripping discrimination result of the gripping point set is determined. The contact geometry parameters of the gripping points are used to represent the position of the gripping points, the normal direction and the tangential direction of the object surface at the gripping points; If the grasping determination result indicates that the object can be grasped based on the grasping point set, then a grasping tag for the object is generated based on the contact geometry parameters of the grasping points in the grasping point set. The grasping tag for the object is used to indicate the pose of the robot arm when grasping the object. The object's grasping label is added to the training dataset, which is used to train the robotic arm control model.

2. The method according to claim 1, characterized in that, The step of determining the grasping discrimination result of the grasping point set based on the contact geometry parameters of the grasping points in the grasping point set includes: Based on the contact geometry parameters of the gripping points in the set of gripping points, the contact forces satisfying the constraints on the gripping points in the set of gripping points are converted into multiple gripping spins. A grasping spinner space is determined based on the plurality of grasping spinners, wherein the grasping spinner space is the smallest space containing the plurality of grasping spinners; If the gripping spinor space satisfies the stable gripping condition, then the gripping discrimination result indicates that the object can be gripped based on the set of gripping points; if the gripping spinor space does not satisfy the stable gripping condition, then the gripping discrimination result indicates that the object cannot be gripped based on the set of gripping points.

3. The method according to claim 2, characterized in that, The contact geometry parameters of the gripping point include the position vector, normal vector, and tangent vector of the gripping point; the step of converting the contact forces satisfying the constraint conditions on the gripping points in the set of gripping points into multiple gripping spinors based on the contact geometry parameters of the gripping points in the set of gripping points includes: For each crawl point in the set of crawl points, perform the following operations: Based on the position vector, normal vector, and tangent vector of the grab point, determine the grab matrix of the grab point; Multiply the gripping matrix of the gripping point by all the contact forces at the gripping point that satisfy the constraint conditions to obtain multiple gripping spinors corresponding to the gripping point.

4. The method according to claim 3, characterized in that, The step of determining the grabbing matrix of the grabbing point based on the position vector, normal vector, and tangent vector of the grabbing point includes: Determine the position vector, normal vector, first tangent vector, and second tangent vector of the grab point, wherein the normal vector, the first tangent vector, and the second tangent vector are perpendicular to each other; The position vector is cross-multiplied with the normal vector to obtain a first mapping vector; the position vector is cross-multiplied with the first tangent vector to obtain a second mapping vector; the position vector is cross-multiplied with the second tangent vector to obtain a third mapping vector. The grasping matrix is ​​constructed based on the normal vector, the first tangent vector, the second tangent vector, the first mapping vector, the second mapping vector, and the third mapping vector.

5. The method according to claim 2, characterized in that, The contact force at the gripping point includes normal force and tangential force; the constraint conditions include: The normal force is non-negative; The tangential force is not greater than the product of the normal force and the coefficient of friction of the object.

6. The method according to claim 2, characterized in that, The stable crawling conditions include: The origin of the screw coordinate system is located in the grasping screw space. The screw coordinate system refers to the coordinate system in which the grasping screw space is located. The vectors in the screw coordinate system are used to describe forces. The object's gravity lies within the grasp spin space.

7. The method according to claim 6, characterized in that, The spinor coordinate system is a three-dimensional coordinate system, and the grasping spinor space constitutes a three-dimensional convex polyhedron. The three-dimensional coordinate system includes a first coordinate axis, a second coordinate axis, and a third coordinate axis that are mutually perpendicular. The first and second coordinate axes are perpendicular to the direction of gravity, and the third coordinate axis is parallel to the direction of gravity. The method further includes: The convex polyhedron formed by the grasping spinor space is mapped onto the plane containing the first coordinate axis and the second coordinate axis to obtain a polygon; Determine the minimum distance from the origin of the spinor coordinate system to the edge of the polygon; If the value of the minimum distance is greater than the value of the object's gravity, then the gravity is determined to be located in the grasping spin space.

8. The method according to claim 7, characterized in that, The method further includes: If the minimum distance is greater than the weight of the object, a grasping score is determined based on the minimum distance. The grasping score is positively correlated with the minimum distance and is used to reflect the grasping stability of grasping the object based on the grasping point set.

9. The method according to any one of claims 1 to 8, characterized in that, The acquisition of the set of gripping points on the object surface includes: The surface of the object's three-dimensional mesh model is divided into multiple candidate patches, and the angle between adjacent meshes in each candidate patch is less than an angle threshold. Among the plurality of candidate facets, at least one target facet is determined, wherein the area of ​​the target facet is greater than an area threshold; The set of grab points is sampled on at least one target patch.

10. The method according to any one of claims 1 to 8, characterized in that, The acquisition of the set of gripping points on the object surface includes: At least three gripping points are randomly sampled from the surface of the object's three-dimensional mesh model; If the distance between the at least three gripping points satisfies the finger distance condition of the robotic hand, and the angle between the normal vectors of the at least three gripping points satisfies the finger angle condition of the robotic hand, then the at least three gripping points constitute the gripping point set.

11. The method according to any one of claims 1 to 8, characterized in that, If the grasping determination result indicates that the object can be grasped based on the grasping point set, then based on the contact geometry parameters of the grasping points in the grasping point set, a grasping tag for the object is generated, including: If the grasping judgment result indicates that the object can be grasped based on the grasping point set, then the grasping point set is simulated to obtain the simulation result; If the simulation results indicate that the robotic arm successfully grasps the object using the set of gripping points, then a gripping label for the object is generated based on the contact geometry parameters of the gripping points in the set of gripping points.

12. The method according to claim 11, characterized in that, The simulation of the set of grasped points, and the resulting simulation results, include: Based on the contact geometry parameters of the gripping points in the set of gripping points, the fingertip target pose of the robotic arm is determined; In the simulation environment, the robotic arm is controlled to grasp the object according to the target pose of the fingertip, and the simulation result is obtained.

13. The method according to claim 12, characterized in that, In the simulation environment, controlling the robotic arm to grasp the object according to the target pose of the fingertip, and obtaining the simulation results, includes: The following iterative process is performed: The joint pose of the robotic hand is converted into the fingertip pose using the pose conversion parameters of the robotic hand; the pose conversion parameters are updated based on the fingertip pose error between the fingertip pose and the target fingertip pose; the fingertip pose error is converted into joint pose error using the updated pose conversion parameters; and the joint pose of the robotic hand is updated based on the joint pose error. The iteration process is stopped when the iteration process meets the iteration termination condition. In the simulation environment, the robotic arm is controlled to grasp the object according to the updated joint pose, and the simulation result is obtained.

14. The method according to claim 13, characterized in that, The step of stopping the iteration process in response to the iteration process satisfying the iteration termination condition includes: During the iteration process, the updated joint pose is converted into an updated fingertip pose using the updated pose conversion parameters; If the fingertip pose error between the updated fingertip pose and the target fingertip pose is less than the error threshold, then the iteration process is stopped.

15. The method according to any one of claims 1 to 8, characterized in that, The robotic hand is a three-fingered robotic hand, comprising a first fingertip, a second fingertip, and a third fingertip; the set of grasping points includes a first grasping point, a second grasping point, and a third grasping point; the object's grasping tag includes: The position vector and normal vector of the first fingertip; The center position vector and roll angle of the grasping circle, wherein the grasping circle refers to the circumcircle of the first grasping point, the second grasping point and the third grasping point; The angle between the first fingertip and the second fingertip along the grasping circle; The angle between the first fingertip and the third fingertip along the grasping circle; The normal vector of the second fingertip and the normal vector of the third fingertip.

16. A training data generation device for a robotic arm control model, characterized in that, The device includes: The acquisition module is used to acquire a set of gripping points on the surface of an object, the set of gripping points including at least three gripping points, the at least three gripping points being the contact points when the robotic arm grips the object; The discrimination module is used to determine the discrimination result of the gripping point set based on the contact geometry parameters of the gripping points in the gripping point set. The contact geometry parameters of the gripping points are used to represent the position of the gripping point, the normal direction and the tangential direction of the object surface at the gripping point; The tag generation module is used to generate a gripping tag for the object based on the contact geometry parameters of the gripping points in the gripping point set if the gripping discrimination result indicates that the object can be gripped based on the gripping point set. The gripping tag of the object is used to indicate the pose of the robot arm when gripping the object. The label adding module is used to add the object's grasping labels to the training dataset, which is used to train the robotic arm control model.

17. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to perform the operations performed by the training data generation method for the robotic arm control model as described in any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to perform the operations performed by the training data generation method for the robotic arm control model as described in any one of claims 1 to 15.

19. A computer program product, comprising a computer program, characterized in that, The computer program is loaded and executed by a processor to perform the operations performed by the training data generation method for the robotic arm control model as described in any one of claims 1 to 15.

Images