Method and system for determining grabbing posture of manipulator

Abstract

The invention provides a method and a system for determining a grabbing posture of a manipulator. The method comprises the following steps: acquiring an approximate guide instruction, wherein the approximate guide instruction indicates a target grabbing position of a three-dimensional model of a to-be-grabbed object; based on the target grabbing position, a four-dimensional rotation matrix of a three-dimensional model of the manipulator is determined, and the four-dimensional rotation matrix is used for moving a grabbing point of the three-dimensional model of the manipulator to the target grabbing position of the three-dimensional model of the to-be-grabbed object; and determining a target grabbing posture of the three-dimensional model of the manipulator based on the four-dimensional rotation matrix. According to the method, any machine learning model training is not needed, object collision is avoided, the grabbing precision is improved, and the method is suitable for fine control over a real physical robot.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method and system for determining a grasping posture of a robot arm. Background Art

[0002] To enable robotic systems to perform a wider range of industrial and domestic tasks, such as picking and placing, painting, polishing and cutting randomly placed objects, it is important for intelligent robotic manipulation to go beyond the hard-coded robot trajectory programming commonly used in high-tech, high-precision (accurate to the micron level) industries such as semiconductor and automotive manufacturing.

[0003] Existing intelligent robot control algorithms cannot truly understand the scene, resulting in the inability to avoid collisions in real time, and are unable to handle dynamic changes in the scene. They cannot be used to perform delicate and dexterous operations on objects, and are not suitable for future combat alongside humans. They require a large amount of end-to-end training data, but also cannot ensure that the intelligent robot does not collide with objects along the way. When the test set distribution is significantly different from the training data environment, the intelligent robot may not work. Summary of the invention

[0004] The technical problem to be solved by the present disclosure is to provide a method and system for determining the grasping posture of a robot arm in order to overcome the defects of the prior art that a large amount of training data is required and collision cannot be avoided in real time and grasping accuracy cannot be guaranteed.

[0005] The present invention solves the above technical problems through the following technical solutions:

[0006] According to a first aspect of the present disclosure, a method for determining a gripping posture of a manipulator is provided, the method comprising:

[0007] Acquire an approximate guidance instruction, wherein the approximate guidance instruction indicates a target grasping position of a three-dimensional model of an object to be grasped;

[0008] Based on the target grasping position, determining a four-dimensional rotation matrix of the three-dimensional model of the manipulator, wherein the four-dimensional rotation matrix is ​​used to move the grasping point of the three-dimensional model of the manipulator to the target grasping position of the three-dimensional model of the object to be grasped;

[0009] A target grasping posture of the three-dimensional model of the manipulator is determined based on the four-dimensional rotation matrix.

[0010] Preferably, the step of determining the four-dimensional rotation matrix of the three-dimensional model of the manipulator based on the target grasping position comprises:

[0011] Based on a first coordinate system where the three-dimensional model of the object to be grasped is located, acquiring anchor coordinates corresponding to the target grasping position;

[0012] Based on a second coordinate system where the three-dimensional model of the manipulator is located, obtaining the grasping point coordinates corresponding to the grasping point;

[0013] The four-dimensional rotation matrix is ​​determined according to the anchor coordinates and the grasping point coordinates to transform the grasping point coordinates to the anchor coordinate position in the first coordinate system.

[0014] Preferably, the step of determining the four-dimensional rotation matrix according to the anchor coordinates and the grab point coordinates comprises:

[0015] Acquire a grasping axis of the three-dimensional model of the object to be grasped in the first coordinate system, wherein the grasping axis is obtained based on the center point coordinates of the three-dimensional model of the object to be grasped and the anchor coordinates;

[0016] Based on the grasping point coordinates and the grasping axis, determining a first rotation matrix, wherein the first rotation matrix is ​​used to move the grasping point coordinates to the anchor coordinate position in the first coordinate system, and the grasping axis of the manipulator is aligned with the grasping axis;

[0017] Determining a first translation matrix, wherein the first translation matrix is ​​used to move the grab point coordinates from the anchor coordinate position in the first coordinate system to the origin of the first coordinate system;

[0018] Determine a second rotation matrix, wherein the second rotation matrix is ​​used to rotate the manipulator around the grasping axis so that the grasping angle of the manipulator is suitable for grasping the object to be grasped;

[0019] Determine a second translation matrix, wherein the second translation matrix is ​​used to move the grab point coordinates from the origin in the first coordinate system to the anchor coordinate position in the first coordinate system;

[0020] The four-dimensional rotation matrix is ​​determined based on the first rotation matrix, the first translation matrix, the second rotation matrix, the second translation matrix and a joint angle parameter set of the manipulator.

[0021] Preferably, the step of determining the second rotation matrix comprises:

[0022] Acquire a plurality of point cloud vertices corresponding to the anchor coordinates and a target area formed by the point cloud vertices;

[0023] Calculating the relative position of each vertex of the point cloud and the center of the target area;

[0024] Calculate the diffusion of the relative position along an axis perpendicular to the grasping plane of the manipulator based on the maximum diffusion of the point cloud;

[0025] The second rotation matrix is ​​determined according to the axial diffusion.

[0026] Preferably, the step of determining the four-dimensional rotation matrix based on the first rotation matrix, the first translation matrix, the second rotation matrix, the second translation matrix and the joint angle parameter set of the manipulator comprises:

[0027] Based on the joint angle parameter set, obtaining the forward kinematics matrix of each component of the manipulator;

[0028] Calculate a grasping matrix corresponding to each component of the manipulator according to the first rotation matrix, the first translation matrix, the second rotation matrix, the second translation matrix, and the forward kinematics matrix;

[0029] Based on the grasping matrix of each component of the manipulator, the four-dimensional rotation matrix is ​​obtained.

[0030] Preferably, the step of obtaining the four-dimensional rotation matrix based on the grasping matrix of each component of the manipulator comprises:

[0031] Calculating the collision cost between the manipulator and the object to be grasped based on the relative position vector of the grasping matrix of each component of the manipulator and the anchor coordinates;

[0032] updating the grasping matrix of each component of the manipulator according to the collision cost so that the collision cost is minimized;

[0033] The four-dimensional rotation matrix is ​​obtained according to the updated grasping matrix.

[0034] Preferably, the step of obtaining the approximate guidance instruction includes:

[0035] Based on the three-dimensional model of the object to be grasped, displaying a two-dimensional image of the object to be grasped;

[0036] The approximate guidance instruction is generated by receiving a user input in the two-dimensional image, wherein the input is used to indicate the target grasping position of the manipulator for the object to be grasped.

[0037] Preferably, the step of receiving the user's input in the two-dimensional image and generating the approximate guidance instruction comprises:

[0038] receiving a click operation of the user on a target image area in the two-dimensional image;

[0039] The target image area is mapped onto the three-dimensional model of the object to be grasped by using ray projection to obtain the target grasping position, so as to generate the approximate guidance instruction.

[0040] Preferably, the method further comprises:

[0041] Based on the target grasping posture of the three-dimensional model of the manipulator, transform the structure and posture of the three-dimensional model of the manipulator;

[0042] Based on the transformed three-dimensional model of the manipulator, a synthetic two-dimensional image of the object to be grasped and the manipulator is displayed.

[0043] Preferably, the method further comprises:

[0044] receiving a user's adjustment instruction for a target adjustment axis of the manipulator in the synthetic two-dimensional image; wherein at least one adjustment axis of the manipulator is displayed in the synthetic two-dimensional image;

[0045] Updating the four-dimensional rotation matrix based on the adjustment instruction;

[0046] The structure and posture transformation of the manipulator are updated according to the updated four-dimensional rotation matrix, and the synthetic two-dimensional image of the object to be grasped and the manipulator after the transformation and update is displayed.

[0047] Preferably, the adjustment instruction includes adjusting the rotation angle and translation distance of the target adjustment axis.

[0048] Preferably, the method comprises:

[0049] Based on the four-dimensional rotation matrix, the target grasping posture of the manipulator and the object posture of the object to be grasped, the ideal grasping posture of the manipulator in the world coordinate system is determined, wherein the target grasping posture of the manipulator is defined by three-dimensional grasping point coordinates, a grasping direction vector and a grasping direction normal vector.

[0050] According to a second aspect of the present disclosure, a system for determining a gripping posture of a robot arm is provided, the system comprising: one or more processors, the one or more processors being used to execute a computer program to implement the method for determining a gripping posture of a robot arm described in the first aspect of the present disclosure.

[0051] Preferably, the system further comprises a display device, and the display device is communicatively connected to the processor.

[0052] According to a third aspect of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and used to run on the processor, wherein when the processor executes the computer program, the method for determining the grasping posture of the robot arm described in the first aspect of the present disclosure is implemented.

[0053] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for determining the gripping posture of a robot arm described in the first aspect of the present disclosure is implemented.

[0054] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method for determining a gripping posture of a manipulator according to the first aspect of the present disclosure is implemented.

[0055] On the basis of being in accordance with the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present disclosure.

[0056] The positive progress effect of the present disclosure is that the gripper grasping posture is quickly generated according to the approximate position of the grasped object determined by human interactive guidance, no machine learning model training is required, object collision is avoided, and grasping accuracy is improved. It is suitable for real physical robots to grasp physical objects in the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flowchart of the existing object grasping algorithm;

[0058] Figure 2 A first flow chart of the method for determining the grasping posture of a manipulator disclosed in the present invention;

[0059] Figure 3 A second flow chart of the method for determining the grasping posture of the manipulator disclosed in the present invention;

[0060] Figure 4 is a schematic diagram of a two-dimensional image of an object to be grasped according to the present disclosure;

[0061] Figure 5 is a schematic diagram of a composite two-dimensional image of an object to be grasped and a manipulator of the present disclosure;

[0062] Figure 6 is a schematic diagram of a user-adjustable composite two-dimensional image of the present disclosure;

[0063] Figure 7 Rotate for users of this disclosure A schematic diagram of a synthetic two-dimensional image;

[0064] Figure 8 Mobile for users of this disclosure A schematic diagram of a synthetic two-dimensional image;

[0065] Fig. 9 A schematic diagram of a synthetic two-dimensional image after the robot is transformed and updated according to a user adjustment instruction of the present disclosure;

[0066] Fig.10It is a structural schematic diagram of the system for determining the grasping posture of the manipulator disclosed in the present invention;

[0067] Fig.11 The figure is a schematic diagram of the structure of an electronic device disclosed in the present invention. DETAILED DESCRIPTION

[0068] The present disclosure is further described below by way of examples, but the present disclosure is not limited to the scope of the examples.

[0069] Prefixes such as "first" and "second" are used in the embodiments of the present disclosure only to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of prefixes such as ordinal numbers to distinguish description objects in the embodiments of the present disclosure does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and no unnecessary limitation should be constituted due to the use of such prefixes. In addition, in the description of the present embodiment, unless otherwise specified, the meaning of "plurality" is two or more.

[0070] In the embodiments of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0071] The existing GraspNet-1Billion (a large-scale benchmark dataset dedicated to object grasping tasks) allows the robotic arm to observe the desktop scene in the form of point cloud data (RGB-D (color image + depth information) data from a depth camera is required) and automatically determine which point in the three-dimensional space looks suitable for grasping after training with 1.1 billion grasping poses (for details, see: H.-S. Fang, C. Wang, M. Gou, and C. Lu. "GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping". In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. DOI: 10.1109 / CVPR42600.2020.01146.). Despite the impressive achievements, this approach is relatively slow. It needs to process the initial scene (which takes 10 seconds on a laptop GPU (a hardware device used for graphics processing and parallel computing)) and determine the final pose of the robotic arm gripper after a period of processing, and then use a separate classical control algorithm to achieve the desired pose, but this algorithm is not real-time, cannot avoid collisions in real time, and cannot handle dynamic changes in the scene, and is not suitable for future combat alongside humans.

[0072] Since this approach has no real understanding of the scene (e.g. it doesn't know what it's grasping, just that according to the neural network this is a good spot to grasp physically), it can't be used to perform fine-grained manipulation of objects. Also, while this approach is good at grasping random objects in any way, it can lead to undesirable behavior, e.g. it might choose to grasp the blade of a knife instead of the handle. Figure 1 As shown in the figure, GraspNet (an open source project for general object grasping) does not even realize that the banana is an object that can be grasped. Only after the table is cleared of other nearby objects does the banana become a valid candidate for grasping.

[0073] Another OpenVLA (open source vision-language-action model for dedicated general robot manipulation tasks) uses a large language model (LLM), such as ChatGPT (natural language processing tool driven by artificial intelligence technology) which has attracted much attention in recent years (for details, please refer to: MJKim, K.Pertsch, S.Karamcheti, et al.OpenVLA:An Open-Source Vision-Language-Action Model.arXiv:2406.09246(2024).DOI:10.48550 / arXiv.2406.09246.). The model takes two-dimensional RGB camera images and language instructions as input, which are converted into "input chunks" of the LLM model. The LLM outputs the actions that the robot needs to perform based on pre-training (for example, how much rotation / translation / grasping action the gripper should make). After the robot performs the action, it needs to update the captured image and repeat multiple iterations.

[0074] Since the core of the model is LLM, the entire algorithm is based on neural networks, so it can handle more general, human-like instructions, but the model needs to be trained on various scenarios that may be encountered during robot operation, and the robot does not perform well on scenarios that are not included in the training set. The model requires at least 1 million demonstration data to train the network, which makes it expensive to obtain training data, and it needs to be trained on 64 Nvidia A100 GPUs (a high-performance computing GPU) for 14 days, which makes training expensive. And because the model uses LLM, it is more suitable to run this method on a remote GPU server with a large memory footprint. In order to use this model to control the robot, it can also be run on a system equipped with two Nvidia RTX 4090GPUs (a flagship graphics card), but the inference accuracy of the model needs to be reduced so that the model runs at an inference speed of 6Hz, which is too slow for real-time applications. In addition, dynamic obstacles are not included in the training data of this model.

[0075] However, the model can accept human verbal language. For example, after receiving the instruction "Put the eggplant in the bowl", the model can view the two-dimensional RGB image and automatically move the gripper to the position of the eggplant, grab the eggplant and put it in the bowl. Since a large number of human habits and preferences are recorded in the training data, there may also be deviations in grasping the handle of the teapot.

[0076] It can be seen that to complete the training task of this OpenVLA model, a large amount of end-to-end robot training data is required. Moreover, since the model is a black box neural network system, it cannot ensure that the robot does not collide with objects on the way. When the test set scene is significantly different from the training data scene, the model may not work.

[0077] In view of this, the present disclosure provides a method and system for determining a grasping posture of a robot arm, so as to solve the problems in the prior art that a large amount of training data is required and collision cannot be avoided in real time and grasping accuracy cannot be guaranteed.

[0078] Example 1

[0079] In a specific embodiment of the present disclosure, a method for determining a gripping posture of a manipulator is provided, such as Figure 2 As shown, the method includes:

[0080] S11 obtains an approximate guidance instruction, where the approximate guidance instruction indicates a target grasping position of a three-dimensional model of an object to be grasped;

[0081] S12, based on the target grasping position, determining a four-dimensional rotation matrix of the three-dimensional model of the manipulator, wherein the four-dimensional rotation matrix is ​​used to move the grasping point of the three-dimensional model of the manipulator to the target grasping position of the three-dimensional model of the object to be grasped;

[0082] Among them, step S12 includes: based on the first coordinate system where the three-dimensional model of the object to be grasped is located, obtaining the anchor coordinates corresponding to the target grasping position; based on the second coordinate system where the three-dimensional model of the manipulator is located, obtaining the grasping point coordinates corresponding to the grasping point; determining the four-dimensional rotation matrix according to the anchor coordinates and the grasping point coordinates to transform the grasping point coordinates to the anchor coordinate position in the first coordinate system.

[0083] S13 determines the target grasping posture of the three-dimensional model of the manipulator based on the four-dimensional rotation matrix.

[0084] Specifically, the approximate guidance instruction of the user for the target grasping position of the object to be grasped is obtained through step S11, wherein, Figure 3 As shown, the two-dimensional image corresponding to the object to be grasped can be displayed in advance based on the three-dimensional model of the object to be grasped in step S0, and the three-dimensional CAD (computer-aided design) model of the object to be grasped (such as a teapot) can be loaded. For example, an input file in the format of *.obj, *.stl, *.dae or other three-dimensional object CAD model can be used. Figure 4As shown, the object to be grasped is displayed in the user interface window in the form of a two-dimensional image, for example, using a Python Qt (for graphical user interface development) software package, interacting with the user in real time through the user interface, receiving the user's input in the two-dimensional image, for example, receiving the user's click operation on the target image area in the two-dimensional image to instruct the manipulator to select the grasping area of ​​the object to be grasped in the two-dimensional image, and then using ray casting to map the target image area to the three-dimensional model of the object to be grasped, for example, drawing a virtual line (ray casting) between the camera center of the manipulator and the target image area determined by the user, finding the first intersection vertex of the virtual line and the target image area, and determining it as the target grasping position of the object to be grasped to generate an approximate guidance instruction.

[0085] Among them, in order to facilitate the transformation calculation of the manipulator's grasping posture, the first coordinate system corresponding to the three-dimensional model of the object to be grasped and the second coordinate system corresponding to the three-dimensional model of the manipulator can be constructed respectively, thereby obtaining the anchor coordinates of the target grasping position in the first coordinate system (three-dimensional coordinates), and then generate corresponding approximate guidance instructions.

[0086] Of course, the anchor coordinates of the object to be grasped in the three-dimensional model of the object may also be stored in advance, and the approximate guidance instructions may be directly generated according to the pre-stored anchor coordinates.

[0087] The grasping point of the manipulator in the second coordinate system is moved to the anchor coordinates using the four-dimensional rotation matrix pair in step S12. For example, define a point 15 cm from the origin of the manipulator base to the +z axis as the grasping point of the manipulator 3D model in the second coordinate system. By calculating a four-dimensional rotation matrix, the robot grasps the point Move to anchor coordinates Place.

[0088] Then, in step S13, the structure and posture of the manipulator are changed according to the calculated four-dimensional rotation matrix to determine the target grasping posture of the three-dimensional model of the manipulator. The target grasping posture can be visualized in the user interface by rendering for the user to view.

[0089] This specific embodiment will quickly receive interactive guidance from the user to determine the approximate location where the object should be grasped. This interactive manipulator grasping posture generation algorithm can be executed quickly, does not require any machine learning model training, and is suitable for real physical robots to grasp physical objects in the environment.

[0090] In a specific implementation, step S12 includes:

[0091] S121, obtaining a grasping axis of the three-dimensional model of the object to be grasped in the first coordinate system, where the grasping axis is obtained based on the center point coordinates of the three-dimensional model of the object to be grasped and the anchor coordinates;

[0092] S122, based on the grasping point coordinates and the grasping axis, determining a first rotation matrix, the first rotation matrix is ​​used to move the grasping point coordinates to the anchor coordinate position in the first coordinate system, and the grasping axis of the manipulator is aligned with the grasping axis;

[0093] S123, determining a first translation matrix, where the first translation matrix is ​​used to move the coordinates of the grab point from the anchor coordinate position in the first coordinate system to the origin of the first coordinate system;

[0094] S124. Determine a second rotation matrix, which is used to rotate the manipulator around the grasping axis so that the grasping angle of the manipulator is suitable for grasping the object to be grasped; wherein, step S124 includes: S1241. Obtain a number of point cloud vertices corresponding to the anchor coordinates and a target area formed by the point cloud vertices; S1242. Calculate the relative position of each point cloud vertex and the center of the target area; S1243. Calculate the diffusion of the relative position along the axis perpendicular to the grasping plane of the manipulator based on maximizing the point cloud diffusion; S1244. Determine the second rotation matrix based on the axis diffusion.

[0095] S125, determining a second translation matrix, where the second translation matrix is ​​used to move the grab point coordinates from the origin in the first coordinate system to the anchor coordinate position in the first coordinate system;

[0096] S126. Determine a four-dimensional rotation matrix based on the first rotation matrix, the second translation matrix, the second rotation matrix, the second translation moment and the joint angle parameter set of the manipulator.

[0097] Specifically, since the robot hand points to the center of the object to be grasped, the robot hand can lift the object more easily, so the center point coordinates of the three-dimensional model of the object to be grasped can be obtained. Connection center point coordinates and anchor coordinates Forming the grabbing axis Then the gripping point of the robot Translate to the anchor coordinates of the object's 3D model At the same time, ensure that the grasping axis of the manipulator is aligned with the grasping axis of the object to be grasped. Alignment, the first rotation matrix R1 needs to satisfy formulas (1)-(3):

[0098]

[0099] Among them, α, β, and γ represent the three rotation angles of the robot gripping point on the X, Y, and Z axes respectively, and t x,t y ,t z They represent the three coordinates of the robot grasping point on the X, Y, and Z axes respectively.

[0100] In the process of determining the first rotation matrix R1, the cost function C1 can be defined as the sum of the square errors of all elements of the above matrix to calculate its relative to the gradient (α, β, γ, t x , t y , t z ) and use PyTorch (an open source deep learning framework for machine learning and deep learning) Adam optimizer to iteratively update the parameters. The cost function C1 is shown in formula (4):

[0101]

[0102] Among them, since the value C1 of the cost function is on a typical GPU laptop currently on the market, the first rotation matrix can be calculated at the end of the optimization step (the number of iterations is less than 1000 times).

[0103] In order to ensure the stability of the robot's grasping, it is also necessary to ensure that the grasping direction of the robot is perpendicular to the long side of the object to be grasped. For example, taking a teapot as an example, the two parallel grasping fingers on the robot should be placed on the left and right sides of the teapot handle, and keep the grasping direction perpendicular to the long side of the teapot handle to ensure stable grasping of the teapot. Therefore, the robot can be rotated around the grasping axis Rotation is performed to maximize the diffusion of the object point cloud vertices near the image area coordinates along the axis perpendicular to the grasping plane, and the grasping angle of the manipulator is corrected to ensure that it is suitable for grasping the object to be grasped, thereby ensuring the grasping stability of the object to be grasped.

[0104] Before rotating the robot, you need to first move the gripping points of the robot's 3D model to the Move to the origin of the first coordinate system where the object to be grasped is located. This movement can be achieved by the first translation matrix T1. The first translation matrix T1 is shown in formula (5):

[0105]

[0106] In the three-dimensional model of the manipulator, assuming that the grasping plane of the manipulator is perpendicular to the X-axis, the second rotation matrix R2 can be defined as the rotation angle θ around the grasping axis First, the anchor coordinates of the object to be grasped are Separate the point cloud vertex data of the object to be grasped within a very short distance (for example, within 3 cm), and name this set of point cloud data V. The shape of V is 3×n V , where n VRepresents the number of vertices in V, and then calculates the relative position of the vertex data relative to the center of V rel , as shown in formula (6):

[0107]

[0108] Then calculate V using formulas (7)-(9) rel Along the gripping axis Rotate to move S about an axis perpendicular to the gripping plane of the manipulator:

[0109]

[0110] where · represents the dot product, is a 4×1 matrix, the last element of which is always 0. Therefore, only the first 3 elements of the matrix are considered in this specific implementation. The pointing direction of the manipulator after rotation and translation through the first rotation matrix, the first translation matrix, and the second rotation matrix can be obtained.

[0111] Since we only need to maximize S to get θ, we only need to perform the calculation with multiple θ values ​​in parallel batch processing. For example, we can calculate all possible rotation angles θ of S between 0-360°, in intervals of 10°, and select θ that maximizes S. This process can be completed instantly on a normal GPU laptop. At this point, the gripping point of the robot can be made by R2T1R1. The robot is then translated into the origin of the first coordinate system where the three-dimensional model of the object to be grasped is located, while ensuring that the angle of the robot is suitable for the actual grasping of the object. Then, the robot can be translated using the second translation matrix T2 so that the grasping point of the three-dimensional model of the robot is Reposition the anchor coordinates of the object to be grasped The second translation matrix T2 is shown in formula (10):

[0112]

[0113] From this we can get the intermediate rotation matrix R auto The calculation formula is shown in formula (11):

[0114] R auto =T2R2T1R1 (11)

[0115] Combined with the intermediate rotation matrix R auto And the joint angle parameter set of the manipulator, the four-dimensional rotation matrix of the manipulator can be determined.

[0116] In a specific implementation, step S126 includes:

[0117] S1261. Based on the joint angle parameter set of the manipulator, obtain the forward kinematics matrix of each component of the manipulator;

[0118] S1262, calculating a grasping matrix corresponding to each component of the manipulator according to the first rotation matrix, the first translation matrix, the second rotation matrix, the second translation matrix, and the forward kinematics matrix;

[0119] S1263. Based on the grasping matrix of each component of the manipulator, a four-dimensional rotation matrix is ​​obtained.

[0120] Specifically, due to R auto is an intermediate rotation matrix including the first rotation matrix, the first translation matrix, the second rotation matrix and the second translation matrix, through R auto The structure and position of the three-dimensional model of the manipulator can be converted to the correct grasping posture suitable for grasping the object to be grasped. However, in the process of controlling the manipulator, it is also necessary to control the grasping point of the manipulator Move to anchor coordinates Considering that the manipulator is a dynamic mesh, each manipulator component has its own mesh and initial pose, and is interconnected through the manipulator forward kinematics defined by the manipulator URDF (Uniform Robot Description Format) file, so for a given set of manipulator joint angle parameters, the manipulator forward kinematics parameters FK can be obtained. GT , and then the forward kinematics matrix corresponding to each manipulator component i under the manipulator forward kinematics parameters is obtained as FK i , if you want to use the intermediate rotation matrix R auto Make sure Move to the anchor coordinates of the object's 3D model The correct posture R of each robot component i is auto，i The grab matrix is ​​shown in formula (12):

[0121]

[0122] Based on the grasping matrix of each component of the manipulator, the four-dimensional rotation matrix of the manipulator grasping the object to be grasped can be obtained.

[0123] In a specific implementation, step S1263 includes:

[0124] S12631. Calculate the collision cost between the manipulator and the object to be grasped based on the grasping matrix of each component of the manipulator and the relative position vector of the anchor coordinates;

[0125] S12632, updating the grasping matrix of each component of the manipulator according to the collision cost so as to minimize the collision cost;

[0126] S12633. Obtain a four-dimensional rotation matrix based on the updated grasping matrix.

[0127] Specifically, in order to ensure that the manipulator is as close as possible to the anchor coordinates specified by the user and to avoid collision between the manipulator and the object to be grasped, the position and direction of the manipulator can also be fine-tuned. By introducing the Z-axis translation (R tr (t z ))'s limit value and the rotation around the Z axis (R ro The limiting value α of (α) limit To minimize collisions and the distance between the manipulator and the anchor coordinates, the manipulator is adjusted to keep the manipulator within the allowed translation and rotation range:

[0128]

[0129] Therefore, we can find the parameters (α, t z ) minimizes the cost C2, which can be obtained by formulas (15)-(16):

[0130]

[0131] Among them, λ is the regularization parameter used to balance the relationship between the collision cost and the overall cost and the collision cost, C collision is the collision cost, which is obtained based on the distance between the manipulator and the object to be grasped and the relative surface normal, and A is is the vertex set of the object to be grasped, B is the vertex set of the object to be grasped, P is the penetration threshold, is an indicator function, which is equal to 1 when the condition is true and 0 otherwise. A and N B are the number of vertices in sets A and B, respectively. i and b j are the vertices of sets A and B respectively, and They are respectively a i and b i The normal line at .

[0132] In a specific embodiment, the method further comprises:

[0133] S14, based on the target grasping posture of the three-dimensional model of the manipulator, transforming the structure and posture of the three-dimensional model of the manipulator;

[0134] S15. Based on the transformed three-dimensional model of the manipulator, a synthetic two-dimensional image of the object to be grasped and the manipulator is displayed.

[0135] Specifically, after the four-dimensional rotation matrix of the manipulator is obtained by calculation, the structure and posture of the manipulator are transformed according to the four-dimensional matrix and then displayed in the image user interface window together with the object to be grasped, such as Figure 5 As shown, the grasping posture of the manipulator to finally grasp the object to be grasped is visually displayed.

[0136] In one embodiment, the method further comprises:

[0137] S16, receiving an adjustment instruction from a user for a target adjustment axis of the manipulator in the synthesized two-dimensional image; wherein at least one adjustment axis of the manipulator is displayed in the synthesized two-dimensional image;

[0138] S17, updating the four-dimensional rotation matrix based on the adjustment instruction;

[0139] S18. Update the structure and posture transformation of the manipulator according to the updated four-dimensional rotation matrix, and display a synthetic two-dimensional image of the object to be grasped and the updated manipulator.

[0140] The adjustment instruction includes adjusting the rotation angle and translation distance of the target adjustment axis.

[0141] Specifically, there are many ways to grab an object at a specified anchor coordinate, and the method of any of the aforementioned embodiments can obtain an optimal posture example (assuming that the object to be grabbed is floating in the air and there are no obstacles nearby). Considering that users have different preferences for the grasping posture of the manipulator grasping the object to be grasped, for example, if there are usually other obstacles on the table where the teapot is placed, or for visual purposes, the user wants the manipulator to grasp at an angle that is tilted to make it look more beautiful, therefore, the user can customize the displayed structure and the manipulator after the posture change through the graphical user interface window.

[0142] In order to enable the user to adjust the generated manipulator grasping posture, it is necessary to display at least one adjustment axis of the manipulator in the synthetic two-dimensional image, for example, to derive three adjustment axes corresponding to the Cartesian coordinate system at the position of the manipulator grasping point. like Figure 6 As shown, the user can select any one of the adjustment axes and click or drag the manipulator to rotate and translate along the selected adjustment axis.

[0143] Assuming that the vectors of the three adjustment axes in the synthetic two-dimensional image are [1,0,0], [0,1,0], and [0,0,1], respectively, the four-dimensional rotation matrix of the manipulator relative to the vectors of the three adjustment axes are respectively as shown in formulas (17)-(19):

[0144]

[0145] When the user selects When the adjustment axis rotates by angle θ, Figure 7 As shown, the adjustment value corresponding to the four-dimensional rotation matrix of the manipulator can be calculated by formula (20):

[0146]

[0147] Similarly, when the user selects Adjust the axis or When adjusting the axis, just replace Modified to or Parameters can be used.

[0148] Then adjust the rotation matrix R adjust (ω) and the adjusted rotation matrix R of each manipulator component adjust，i (ω) is shown in formulas (21)-(22):

[0149] R adjust (ω) = T2R u (ω)R2T1R1 (21)

[0150]

[0151] When the user selects When the adjustment axis is translated by a distance of Δl, Figure 8 As shown, it can be expressed by formula (23):

[0152]

[0153] Similarly, when the user selects Adjust the axis or When adjusting the axis, just replace Modified to or Parameters can be used.

[0154] Then adjust the translation matrix T adjust (Δl) and the adjustment translation matrix T of each manipulator component adjust，i (Δl) is shown in formula (24)-(25):

[0155] T adjust (Δl) = T2T u (Δl)R2T1R1 (24)

[0156]

[0157] The four-dimensional rotation matrix of each manipulator component is updated according to the adjustment rotation matrix and adjustment translation matrix of each manipulator component, and the structure and posture transformation of the manipulator three-dimensional model are updated, such as Fig. 9 As shown, a synthetic two-dimensional image of the object to be grasped and the manipulator after the transformation update is displayed.

[0158] In one embodiment, the method further comprises:

[0159] Based on the four-dimensional rotation matrix, the target grasping posture of the manipulator and the object posture of the object to be grasped, the ideal grasping posture of the manipulator in the world coordinate system is determined, wherein the target grasping posture of the manipulator is defined by three-dimensional grasping point coordinates, a grasping direction vector and a grasping direction normal vector.

[0160] Specifically, given the user's anchor coordinate point and the user's adjustment of the robot's grasping posture, the target four-dimensional rotation matrix of the robot relative to the object to be grasped can be obtained. The target four-dimensional rotation can be represented by a 4x4 transformation matrix Among them, R G2O is the rotation matrix, T G2O is the translation vector. G2O The manipulator can be transformed from the second coordinate system to the first coordinate system to obtain a target grasping posture for the manipulator to grasp an object to be grasped.

[0161] For example, for a two-finger gripper, the target four-dimensional rotation matrix can be defined with three elements, for example, the grip point is The grab direction is Grab the normal direction in is a three-dimensional point, is a direction vector in three-dimensional space. By R G2O The matrix is ​​converted into the corresponding vector in the first coordinate system, as shown in formulas (26)-(28):

[0162]

[0163] Assume that the base of the robot is located at the origin of the world coordinate system, and give the 6D posture of the object to be grasped, represents the transformation from the first coordinate system to the world coordinate system, so the target grasping posture of the manipulator relative to the object to be grasped can be transformed into the world coordinate system. Among them, P is represented by a 4×4 rigid transformation matrix P=[R, T; 0, 1], where R is a 3×3 three-dimensional rotation matrix and T is a 3×1 three-dimensional transformation vector. The final transformation matrix in the world coordinate system can be obtained by formulas (29)-(31):

[0164]

[0165] With the final transformation matrix in the world coordinate system, the grasping posture of the manipulator can be controlled according to the final transformation matrix so that it is transformed into an ideal grasping posture in the world coordinate system to grasp the object to be grasped.

[0166] This specific embodiment quickly generates the gripping posture of the gripper based on the approximate position of the grasped object determined by human interactive guidance, does not require any machine learning model training, avoids object collision, improves grasping accuracy, and is suitable for real physical robots to grasp physical objects in the environment.

[0167] Example 2

[0168] In a specific embodiment of the present disclosure, a system for determining a gripping posture of a manipulator is provided. Fig.10 As shown, the system includes: one or more processors 100, a display device 110, the display device 110 is communicatively connected to the processor 100, and the one or more processors 100 are used to execute a computer program to implement the method for determining the gripping posture of the robot arm described in any of the above embodiments.

[0169] The processor 100 specifically includes the following modules:

[0170] The first acquisition module 101 is used to acquire an approximate guidance instruction, where the approximate guidance instruction indicates a target grasping position of a three-dimensional model of an object to be grasped;

[0171] The first determination module 102 is used to determine the four-dimensional rotation matrix of the three-dimensional model of the manipulator based on the target grasping position, wherein the four-dimensional rotation matrix is ​​used to move the grasping point of the three-dimensional model of the manipulator to the target grasping position of the three-dimensional model of the object to be grasped;

[0172] The generation module 103 is used to determine the target grasping posture of the three-dimensional model of the manipulator based on the four-dimensional rotation matrix.

[0173] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution.

[0174] This specific embodiment quickly generates the gripping posture of the gripper based on the approximate position of the grasped object determined by human interactive guidance, does not require any machine learning model training, avoids object collision, improves grasping accuracy, and is suitable for real physical robots to grasp physical objects in the environment.

[0175] Example 3

[0176] Fig.11 This is a schematic diagram of the structure of an electronic device shown in an example embodiment of the present disclosure, the electronic device includes a memory, a processor, and a computer program stored in the memory and used to run on the processor, and when the processor executes the computer program, the method for determining the gripping posture of the robot arm described in any of the above embodiments is implemented. Fig.11 The electronic device 30 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0177] like Fig.11 As shown, the electronic device 30 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including the memory 32 and the processor 31).

[0178] The bus 33 includes a data bus, an address bus, and a control bus.

[0179] The memory 32 may include a volatile memory, such as a random access memory (RAM) 321 and / or a cache memory 322 , and may further include a read-only memory (ROM) 323 .

[0180] The memory 32 may also include a program tool 325 (or utility) having a set (at least one) of program modules 324, such program modules 324 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0181] The processor 31 executes various functional applications and data processing by running the computer program stored in the memory 32, such as the method for determining the grasping posture of the manipulator provided in any of the above embodiments.

[0182] The electronic device 30 may also communicate with one or more external devices 34 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 35. Furthermore, the electronic device 30 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 36. As shown, the network adapter 36 communicates with other modules of the electronic device 30 via a bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

[0183] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0184] Example 4

[0185] The embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the method for determining the gripping posture of a robot arm provided in any of the above embodiments is implemented.

[0186] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0187] Example 5

[0188] The embodiment of the present disclosure further provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for determining a gripping posture of a robot arm.

[0189] Among them, the program code for executing the computer program product of the present disclosure can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or completely on the remote device.

[0190] Although the specific embodiments of the present disclosure are described above, those skilled in the art should understand that this is only an example, and the protection scope of the present disclosure is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present disclosure, but these changes and modifications all fall within the protection scope of the present disclosure.

Claims

1. A method for determining a gripping posture of a manipulator, characterized in that: The method comprises: Acquire an approximate guidance instruction, wherein the approximate guidance instruction indicates a target grasping position of a three-dimensional model of an object to be grasped; Based on the target grasping position, determining a four-dimensional rotation matrix of the three-dimensional model of the manipulator, wherein the four-dimensional rotation matrix is ​​used to move the grasping point of the three-dimensional model of the manipulator to the target grasping position of the three-dimensional model of the object to be grasped; A target grasping posture of the three-dimensional model of the manipulator is determined based on the four-dimensional rotation matrix.

2. The method according to claim 1, characterized in that The step of determining the four-dimensional rotation matrix of the three-dimensional model of the manipulator based on the target grasping position comprises: Based on a first coordinate system where the three-dimensional model of the object to be grasped is located, acquiring anchor coordinates corresponding to the target grasping position; Based on a second coordinate system where the three-dimensional model of the manipulator is located, obtaining the grasping point coordinates corresponding to the grasping point; The four-dimensional rotation matrix is ​​determined according to the anchor coordinates and the grasping point coordinates to transform the grasping point coordinates to the anchor coordinate position in the first coordinate system.

3. The method according to claim 2, characterized in that The step of determining the four-dimensional rotation matrix according to the anchor coordinates and the grab point coordinates comprises: Acquire a grasping axis of the three-dimensional model of the object to be grasped in the first coordinate system, wherein the grasping axis is obtained based on the center point coordinates of the three-dimensional model of the object to be grasped and the anchor coordinates; Based on the grasping point coordinates and the grasping axis, determining a first rotation matrix, wherein the first rotation matrix is ​​used to move the grasping point coordinates to the anchor coordinate position in the first coordinate system, and the grasping axis of the manipulator is aligned with the grasping axis; Determining a first translation matrix, wherein the first translation matrix is ​​used to move the grab point coordinates from the anchor coordinate position in the first coordinate system to the origin of the first coordinate system; Determine a second rotation matrix, wherein the second rotation matrix is ​​used to rotate the manipulator around the grasping axis so that the grasping angle of the manipulator is suitable for grasping the object to be grasped; Determine a second translation matrix, wherein the second translation matrix is ​​used to move the grab point coordinates from the origin in the first coordinate system to the anchor coordinate position in the first coordinate system; The four-dimensional rotation matrix is ​​determined based on the first rotation matrix, the first translation matrix, the second rotation matrix, the second translation matrix and a joint angle parameter set of the manipulator.

4. The method according to claim 3, characterized in that The step of determining the second rotation matrix comprises: Acquire a plurality of point cloud vertices corresponding to the anchor coordinates and a target area formed by the point cloud vertices; Calculating the relative position of each vertex of the point cloud and the center of the target area; Calculate the diffusion of the relative position along an axis perpendicular to the grasping plane of the manipulator based on the maximum diffusion of the point cloud; The second rotation matrix is ​​determined according to the axial diffusion.

5. The method according to claim 3, characterized in that: The step of determining the four-dimensional rotation matrix based on the first rotation matrix, the first translation matrix, the second rotation matrix, the second translation matrix and the joint angle parameter set of the manipulator comprises: Based on the joint angle parameter set, obtaining the forward kinematics matrix of each component of the manipulator; Calculate a grasping matrix corresponding to each component of the manipulator according to the first rotation matrix, the first translation matrix, the second rotation matrix, the second translation matrix, and the forward kinematics matrix; Based on the grasping matrix of each component of the manipulator, the four-dimensional rotation matrix is ​​obtained.

6. The method according to claim 5, characterized in that The step of obtaining the four-dimensional rotation matrix based on the grasping matrix of each component of the manipulator comprises: Calculating the collision cost between the manipulator and the object to be grasped based on the relative position vector of the grasping matrix of each component of the manipulator and the anchor coordinates; updating the grasping matrix of each component of the manipulator according to the collision cost so that the collision cost is minimized; The four-dimensional rotation matrix is ​​obtained according to the updated grasping matrix.

7. The method according to any one of claims 1 to 6, characterized in that The step of obtaining the approximate guidance instruction comprises: Based on the three-dimensional model of the object to be grasped, displaying a two-dimensional image of the object to be grasped; The approximate guidance instruction is generated by receiving a user input in the two-dimensional image, wherein the input is used to indicate the target grasping position of the manipulator for the object to be grasped.

8. The method according to claim 7, characterized in that The step of receiving the user's input in the two-dimensional image and generating the approximate guidance instruction comprises: receiving a click operation of the user on a target image area in the two-dimensional image; The target image area is mapped onto the three-dimensional model of the object to be grasped by using ray projection to obtain the target grasping position, so as to generate the approximate guidance instruction.

9. The method according to claim 7, characterized in that: The method further comprises: Based on the target grasping posture of the three-dimensional model of the manipulator, transform the structure and posture of the three-dimensional model of the manipulator; Based on the transformed three-dimensional model of the manipulator, a synthetic two-dimensional image of the object to be grasped and the manipulator is displayed.

10. The method according to claim 9, characterized in that The method further comprises: receiving a user's adjustment instruction for a target adjustment axis of the manipulator in the synthetic two-dimensional image; wherein at least one adjustment axis of the manipulator is displayed in the synthetic two-dimensional image; Updating the four-dimensional rotation matrix based on the adjustment instruction; The structure and posture transformation of the manipulator are updated according to the updated four-dimensional rotation matrix, and the synthetic two-dimensional image of the object to be grasped and the manipulator after the transformation and update is displayed.

11. The method according to claim 10, characterized in that The adjustment instruction includes adjusting the rotation angle and translation distance of the target adjustment axis.

12. The method according to any one of claims 1 to 11, characterized in that The method further comprises: Based on the four-dimensional rotation matrix, the target grasping posture of the manipulator and the object posture of the object to be grasped, the ideal grasping posture of the manipulator in the world coordinate system is determined, wherein the target grasping posture of the manipulator is defined by three-dimensional grasping point coordinates, a grasping direction vector and a grasping direction normal vector.

13. A system for determining a gripping posture of a robot, characterized in that: The system comprises: one or more processors, and the one or more processors are used to execute a computer program to implement the method for determining a grasping posture of a robot arm according to any one of claims 1-6.

14. The system according to claim 13, characterized in that The system also includes a display device, which is communicatively connected to the processor to implement the method for determining the grasping posture of the robot arm as described in any one of claims 7-12.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and used to run on the processor, characterized in that: When the processor executes the computer program, the method for determining the grasping posture of the manipulator according to any one of claims 1 to 12 is implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for determining the grasping posture of a robot arm according to any one of claims 1 to 12 is implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for determining the grasping posture of a robot arm according to any one of claims 1 to 12 is implemented.

Images