Mechanical arm grabbing control method, system, equipment, medium and product

Abstract

The invention discloses a mechanical arm grabbing control method, system and device, a medium and a product, and relates to the field of mechanical arm control, the method comprises the steps that a first target image is obtained, and the first target image comprises an image of a first target object; determining pixel coordinates of a first target object in the first target image; converting the pixel coordinate of the first target object into a three-dimensional coordinate of the first target object in a camera coordinate system; according to the three-dimensional coordinates of the first target object in the camera coordinate system, through a coordinate transformation matrix, three-dimensional coordinates of the first target object in a base coordinate system of the target mechanical arm are obtained through transformation; the three-dimensional coordinate of the first target object under the base coordinate system of the target mechanical arm serves as a target position, and the target mechanical arm is controlled to grab the first target object; the problem that an existing mechanical arm is poor in object grabbing flexibility is solved.

Description

Technical Field

[0001] This application relates to the field of robotic arm control, and in particular to a robotic arm grasping control method, system, device, medium and product. Background Technology

[0002] In recent years, the demand for contactless operation has increased significantly, which has driven the widespread application of intelligent robotic arms in various industries, such as manufacturing, logistics, healthcare, and scientific research. As an important component of automation equipment, intelligent robotic arms have gradually become a key technology for improving production efficiency, optimizing operational processes, reducing costs, and enhancing safety.

[0003] However, existing robotic arms require a certain position and orientation of the object when performing grasping operations. They cannot accurately grasp objects in any position within the reach of the robotic arm, making it unable to cope with grasping tasks in different scenarios and resulting in poor flexibility in grasping objects. Summary of the Invention

[0004] The purpose of this application is to provide a robotic arm grasping control method, system, device, medium, and product to solve the problem of poor flexibility in the grasping of objects by existing robotic arms.

[0005] To achieve the above objectives, this application provides the following solution:

[0006] In a first aspect, this application provides a robotic arm grasping control method, including:

[0007] Acquire a first target image, wherein the first target image includes an image of a first target object;

[0008] Determine the pixel coordinates of the first target object in the first target image;

[0009] Convert the pixel coordinates of the first target object into its three-dimensional coordinates in the camera coordinate system;

[0010] Based on the three-dimensional coordinates of the first target object in the camera coordinate system, the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm are obtained by transforming the coordinates using a coordinate transformation matrix.

[0011] The three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm are used as the target position, and the target robotic arm is controlled to grasp the first target object.

[0012] Optionally, the robotic arm grasping control method further includes:

[0013] Before transforming the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm using a coordinate transformation matrix based on the three-dimensional coordinates of the first target object in the camera coordinate system, the coordinate transformation matrix is ​​calibrated.

[0014] The calibration coordinate transformation matrix includes:

[0015] Determine the coordinates of the first target; wherein, the coordinates of the first target are the three-dimensional coordinates of the second target object located at the target position in the camera coordinate system;

[0016] After controlling the end of the target robotic arm to be perpendicular to and in close contact with the second target object located at the target position, the coordinates of the second target are acquired; wherein, the coordinates of the second target are the three-dimensional coordinates when the end of the target robotic arm is perpendicular to and in close contact with the second target object located at the target position;

[0017] After controlling the end effector of the target robotic arm to be in the target grasping posture, target pose information is acquired; wherein, the target grasping posture is the posture of the target robotic arm when grasping the second target object located at the target position, and the target pose information is the pose information of the end effector of the target robotic arm when it is in the target grasping posture;

[0018] Based on the first target coordinates, the second target coordinates, and the target pose information, determine the coordinate transformation matrix between the camera coordinate system and the base coordinate system of the target robotic arm.

[0019] Optionally, the step of converting the pixel coordinates of the target object into its three-dimensional coordinates in the camera coordinate system, wherein the target object includes a first target object or a second target object, includes:

[0020] Substitute the pixel coordinates of the target object into the following formula to calculate the 3D camera coordinates of the target object:

[0021]

[0022] Among them, (x p ,y p Let z be the pixel coordinates of the target object. c The depth value of the depth camera used to capture the target image, wherein the target image includes a first target image or a second target image, and K is the intrinsic parameter matrix of the depth camera. -1 Let K be the inverse matrix, (x c ,y c ,z c f represents the three-dimensional coordinates of the target object in the camera coordinate system of the depth camera. x f is the focal length value in the X direction of the depth camera. yLet (c) be the focal length value in the Y direction of the depth camera. x ,c y The coordinates of the principal point are two-dimensional coordinates, where the principal point refers to the point where the optical axis intersects the image plane of the target image.

[0023] Optionally, the step of transforming the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm based on the three-dimensional camera coordinates of the first target object through coordinate transformation relationships includes:

[0024] Substitute the three-dimensional camera coordinates of the first target object into the following formula to calculate the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm:

[0025]

[0026] Among them, (x c ,y c ,z c T represents the three-dimensional coordinates of the first target object in the camera coordinate system. Base-Camera The coordinate transformation matrix to be calibrated, (x Base ,y Base ,z Base ) represents the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm.

[0027] Alternatively, T can be calculated according to the following formula. Base-Camera :

[0028]

[0029] R = R z (r z )·R y (r y )·R x (r x );

[0030]

[0031] t = [x, y, z];

[0032] Where R is the rotation matrix of the camera coordinate system relative to the base coordinate system of the target robotic arm, and t is the translation vector of the camera coordinate system relative to the base coordinate system of the target robotic arm. x (r x R is the rotation matrix about the X-axis. y (r y R is the rotation matrix about the Y-axis. z (r zLet ) be the rotation matrix around the Z-axis, rx be the rotation angle of the end effector of the target robotic arm around the X-axis, ry be the rotation angle of the end effector of the target robotic arm around the Y-axis, rz be the rotation angle of the end effector of the target robotic wall around the Z-axis, x be the distance the end effector of the target robotic wall moves along the X-axis, y be the distance the end effector of the target robotic wall moves along the Y-axis, and z be the distance the end effector of the target robotic wall moves along the Z-axis.

[0033] Optionally, the robotic arm grasping control method further includes:

[0034] After the calibration coordinate transformation matrix is ​​established, and before the transformation of the first target object into its three-dimensional coordinates in the base coordinate system of the target robotic arm using the coordinate transformation matrix based on the three-dimensional coordinates of the first target object in the camera coordinate system, the pose information of the end effector of the target robotic arm [x,y,z,rx,ry,rz] is obtained.

[0035] Substitute [x,y,z,rx,ry,rz] into T Base-Camera From the formulas for calculating R and t, we obtain T. Base-Camera .

[0036] Secondly, this application provides a robotic arm grasping control system, comprising:

[0037] An image acquisition module is used to acquire a first target image, wherein the first target image includes an image of a first target object;

[0038] The target detection module is used to determine the pixel coordinates of the first target object in the first target image;

[0039] The target localization module is used to convert the pixel coordinates of the first target object into its three-dimensional coordinates in the camera coordinate system;

[0040] The coordinate transformation module is used to transform the three-dimensional coordinates of the first target object in the camera coordinate system into the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm through a coordinate transformation matrix.

[0041] The grasping control module is used to take the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm as the target position, and control the target robotic arm to grasp the first target object.

[0042] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the robotic arm grasping control method described in any one of the above.

[0043] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the robotic arm grasping control method described above.

[0044] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the robotic arm grasping control method described above.

[0045] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0046] This application provides a robotic arm grasping control method, apparatus, device, medium, and product. By acquiring a first target image, which includes an image of a first target object, determining the pixel coordinates of the first target object in the first target image, and converting the pixel coordinates of the first target object into its three-dimensional coordinates in the camera coordinate system, and then using a coordinate transformation matrix to obtain the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm based on the acquired first target image, the method achieves the positioning of the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm. It is only necessary to ensure that the acquired first target image includes the image of the first target object; no further specialization is required. The specific pose of the first target object is defined, making its placement more flexible. By using the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm as the target position, the target robotic arm is controlled to grasp the first target object, thus achieving grasping control. When the relative position between the depth camera used to acquire the first target image and the end effector of the target robotic arm remains unchanged, the target robotic arm can accurately grasp the first target object in any pose within its reach, without limiting the specific pose of the first target object. This makes the operation of the target robotic arm more flexible and can cope with grasping tasks in different scenarios, solving the problem of poor flexibility in grasping objects in existing robotic arms. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A flowchart illustrating a robotic arm grasping control method according to an embodiment of this application;

[0049] Figure 2This is a schematic diagram of the functional modules of a robotic arm grasping control system provided in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] In one exemplary embodiment, such as Figure 1 As shown, a robotic arm grasping control method is provided, including the following steps 101 to 105. Wherein:

[0053] Step 101: Obtain a first target image, wherein the first target image includes an image of a first target object.

[0054] In this embodiment, the first target object refers to the object that the robotic arm intends to grasp during the grasping operation in the application scenario. Since there is a certain distance between the first target object and the camera that acquires the first target image, a depth camera is used to acquire the first target image.

[0055] Step 102: Determine the pixel coordinates of the first target object in the first target image.

[0056] Step 103: Convert the pixel coordinates of the first target object into its three-dimensional coordinates in the camera coordinate system.

[0057] Step 104: Based on the three-dimensional coordinates of the first target object in the camera coordinate system, the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm are obtained by transforming the coordinates using a coordinate transformation matrix.

[0058] Step 105: Use the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm as the target position, and control the target robotic arm to grasp the first target object.

[0059] In this embodiment of the application, the opening and closing degree of the gripper of the target robotic arm is much greater than the length and width of the first target object being gripped.

[0060] By implementing steps 101 to 105 above, a first target image is acquired, which includes an image of a first target object. The pixel coordinates of the first target object in the first target image are determined, and these pixel coordinates are converted into its three-dimensional coordinates in the camera coordinate system. Based on the three-dimensional coordinates of the first target object in the camera coordinate system, a coordinate transformation matrix is ​​used to obtain the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm. This achieves the positioning of the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm based on the acquired first target image, providing precise position information for the control system of the target robotic arm. This is achieved simply by ensuring that the acquired first target image includes an image of the first target object. This method eliminates the need to limit the specific pose of the first target object, making its placement more flexible. By using the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm as the target position, the target robotic arm is controlled to grasp the first target object, achieving grasping control. When the relative position between the depth camera used to acquire the first target image and the end effector of the target robotic arm remains unchanged, the target robotic arm can accurately grasp the first target object in any pose within its reach, without limiting the specific pose of the first target object. This makes the operation of the target robotic arm more flexible, enabling it to cope with grasping tasks in different scenarios and solving the problem of poor flexibility in grasping objects in existing robotic arms.

[0061] Optionally, in other embodiments of this application, the above-described robotic arm grasping control method further includes, before step 104:

[0062] Step 201: Calibrate the coordinate transformation matrix.

[0063] Optionally, in other embodiments of this application, calibrating the coordinate transformation matrix includes steps 301 to 304. Wherein:

[0064] Step 301: Determine the coordinates of the first target. The coordinates of the first target are the three-dimensional coordinates of the second target object located at the target position in the camera coordinate system.

[0065] Step 302: After controlling the end of the target robotic arm to be perpendicular to and in close contact with the second target object located at the target position, the coordinates of the second target are acquired, wherein the coordinates of the second target are the three-dimensional coordinates when the end of the target robotic arm is perpendicular to and in close contact with the second target object located at the target position.

[0066] Step 303: After controlling the end effector of the target robotic arm to be in the target grasping posture, collect the target pose information; wherein, the target grasping posture is the posture of the target robotic arm when grasping the second target object located at the target position, and the target pose information is the pose information of the end effector of the target robotic arm when it is in the target grasping posture.

[0067] Step 304: Determine the coordinate transformation matrix between the camera coordinate system and the base coordinate system of the target robotic arm based on the coordinates of the first target, the coordinates of the second target, and the target pose information.

[0068] In this embodiment, the order of steps 301 to 303 can be adjusted according to actual needs, and it is not required that steps 301 to 303 be executed sequentially. For example, steps 301 to 303 can be executed simultaneously. Another example is that steps 301 to 302 can be executed simultaneously, and then step 3203 can be executed.

[0069] Optionally, in other embodiments of this application, step 301 described above includes steps 401 to 403. Wherein:

[0070] Step 401: Obtain the second target image, which includes an image of the second target object located at the target location.

[0071] In this embodiment of the application, the second target object may be the object that the target robotic arm wants to grasp in the application scenario, or it may be a model of the object that the target robotic arm wants to grasp in the application scenario.

[0072] Step 402: Determine the pixel coordinates of the second target object in the second target image.

[0073] Step 403: Convert the pixel coordinates of the second target object into its three-dimensional coordinates in the camera coordinate system.

[0074] Optionally, in other embodiments of this application, the above-described determination of the pixel coordinates of the target object in the target image, wherein the target image includes a first target image or a second target image, and the target object includes a first target object or a second target object, includes steps 501 to 503. Wherein:

[0075] Step 501: Target detection is performed using computer vision algorithms to identify target objects in the target image.

[0076] In this embodiment, the computer vision algorithm used is not specifically limited and can be set according to actual needs. For example, feature matching methods such as SIFT, SURF, and ORB can be used to identify and match feature points to perform target detection on the target image. For example, if the target object has a unique color, a color segmentation method using color thresholding can be used. For example, if the shape of the target object is fixed, a template matching method using template matching technology can be used to find the region most similar to the template. Another example is the use of a target detection model using a deep learning model such as a convolutional neural network (CNN) for classification or object detection tasks, such as YOLO, SSD, or Faster R-CNN frameworks.

[0077] Step 502: Obtain the bounding box or key points of the target object.

[0078] In this embodiment, once a target object is detected, you can obtain its bounding box, or more precisely, its keypoints. The bounding box is typically defined by the coordinates of its top-left and bottom-right corners, while keypoints are specific locations on the target object.

[0079] Step 503: Extract the pixel coordinates of the target object from the bounding box or key points of the target object.

[0080] In this embodiment, the pixel coordinates of the target object refer to its position information on the image plane. For bounding boxes, the pixel coordinates of the target object may be the coordinates of the center point of the bounding box, or the coordinates of the top-left and bottom-right corners of the bounding box. For keypoints, the pixel coordinates of the target object are directly the pixel coordinates of each keypoint.

[0081] Optionally, in other embodiments of this application, the above-described determination of the pixel coordinates of the target object in the target image further includes:

[0082] Before step 501, the target image is preprocessed.

[0083] In this embodiment, the purpose of preprocessing the target image is to improve the accuracy of subsequent target object recognition and acquisition of its bounding box or key points. This application does not limit the specific preprocessing operations; they can be set according to requirements. For example, the target image can be grayscaled, denoised, and binarized. The purpose of binarization is to distinguish the target object from the background in the target image.

[0084] Optionally, in other embodiments of this application, target detection is performed using YOLOv5 to identify target objects in the target image.

[0085] In this embodiment, YOLOv5 has high accuracy and excellent real-time performance, making it very suitable for tasks that require fast response and accurate detection, and can achieve ideal detection results in target detection tasks.

[0086] Optionally, in other embodiments of this application, the pixel coordinates of the target object are converted into its three-dimensional coordinates in the camera coordinate system. The target object includes a first target object or a second target object, including:

[0087] Substitute the pixel coordinates of the target object into the following formula to calculate the 3D camera coordinates of the target object:

[0088]

[0089] Among them, (x p ,y p (x) represents the pixel coordinates of the target object. p The x-axis pixel coordinates of the target object, y p Let z be the pixel coordinate of the target object along the Y-axis. c The depth values ​​of the depth camera used to capture the target image, which includes either a first target image or a second target image, and K is the intrinsic parameter matrix of the depth camera. -1 Let K be the inverse matrix, (x c ,y c ,z c x represents the three-dimensional coordinates of the target object in the camera coordinate system of the depth camera. c Let X be the X-axis coordinate of the target object in the camera coordinate system, and Y be the Y-axis coordinate. c Let z be the Y-axis coordinate of the target object in the camera coordinate system. c f is the Z-axis coordinate of the target object in the camera coordinate system. x f is the focal length value in the X direction of the depth camera. y Let (c) be the focal length value in the Y direction of the depth camera. x ,c y ) are the coordinates of the principal point, c x The x-axis coordinate of the principal point, c y The Y-axis coordinate of the principal point, which refers to the point where the optical axis intersects the image plane of the target image.

[0090] In this embodiment, the principal point is usually close to the center of the target image, and the center point of the target image can be selected as the principal point. The robotic arm is controlled to a fixed acquisition pose, and the Python file for acquiring camera data (referring to RGBD data, i.e., two-dimensional color images and depth images) is started. A window will open on the screen, and the center point of the target object is manually clicked with the mouse in the window. The two-dimensional coordinates of the mouse click and the three-dimensional camera coordinates will then be displayed on the terminal.

[0091] Optionally, in other embodiments of this application, step 104 described above includes:

[0092] (x) c ,y c ,z c Substitute the values ​​into the following formula to calculate the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm:

[0093]

[0094] Among them, T Base-Camera Let x be the coordinate transformation matrix between the camera coordinate system of the depth camera and the base coordinate system of the target robotic arm. Base ,y Base ,zBase Let x be the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm. Base Let X be the X-axis coordinate of the first target object in the base coordinate system of the target robotic arm, Y be the Y-axis coordinate of the first target object in the base coordinate system of the target robotic arm, and Z be the Z-axis coordinate of the first target object in the base coordinate system of the target robotic arm.

[0095] Optionally, in other embodiments of this application, the coordinate transformation matrix T described above is calculated according to the following formula. Base-Camera :

[0096]

[0097] R = R z (r z )·R y (r y )·R x (r x );

[0098]

[0099]

[0100] t = [x, y, z];

[0101] Where R is the rotation matrix of the camera coordinate system relative to the base coordinate system of the target robotic arm, and t is the translation vector of the camera coordinate system relative to the base coordinate system of the target robotic arm. x (r x R is the rotation matrix about the X-axis. y (r y R is the rotation matrix about the Y-axis. z (r z Let y be the rotation matrix around the Z-axis, rx be the rotation angle of the end effector of the target robotic arm around the X-axis, ry be the rotation angle of the end effector of the target robotic wall around the Y-axis, rz be the rotation angle of the end effector of the target robotic wall around the Z-axis, x be the distance the end effector of the target robotic wall moves along the X-axis, y be the distance the end effector of the target robotic arm moves along the Y-axis, z be the distance the end effector of the target robotic wall moves along the Z-axis, sin() be the sine function, and cos() be the cosine function.

[0102] Optionally, in other embodiments of this application, the above-described robotic arm grasping control method further includes steps 601 to 602. Wherein:

[0103] Step 601, after step 201 and before step 104, obtain the pose information of the end effector of the target robotic arm [x,y,z,rx,ry,rz].

[0104] In this embodiment, the target robotic arm can be powered off, the end of the target robotic arm can be manually positioned vertically and close to the second target object, and then the target robotic arm can be powered on to start the Python file that collects data from the target robotic arm and obtain the current pose information of the end of the target robotic arm.

[0105] Step 602, substitute the obtained [x,y,z,rx,ry,rz] into T Base-Camera From the formulas for calculating R and t, we obtain T. Base-Camera .

[0106] In this embodiment of the application, after obtaining the pose information [x,y,z,rx,ry,rz] of the end effector of the target robotic arm, it is substituted into the calibrated coordinate transformation matrix T. Base-Camera From the formulas for calculating R and t, we obtain T. Base-Camera Then use the obtained T Base-Camera The three-dimensional coordinates of the first target object in the camera coordinate system are transformed to the base coordinate system of the target robotic arm to obtain the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm.

[0107] Optionally, in other embodiments of this application, the above-described robotic arm grasping control method further includes:

[0108] Before acquiring the second target image, install the depth camera and its related drivers and dependencies. These drivers and dependencies provide the necessary hardware support for the depth camera, ensuring the proper functioning of the data acquisition capabilities. Obtain the depth camera's intrinsic parameters (including f-parameters) using visualization tools. x f y and (c x ,c y A depth camera is mounted on the target robotic arm, and the depth camera moves with the target robotic arm.

[0109] Based on the same inventive concept, this application also provides a robotic arm grasping control system for implementing the robotic arm grasping control method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the robotic arm grasping control system provided below can be found in the limitations of the robotic arm grasping control method described above, and will not be repeated here.

[0110] In one exemplary embodiment, such as Figure 2 As shown, a robotic arm grasping control system 70 is provided, including:

[0111] Image acquisition module 701 is used to acquire a first target image, wherein the first target image includes an image of a first target object;

[0112] The target detection module 702 is used to determine the pixel coordinates of the first target object in the first target image;

[0113] The target localization module 703 is used to convert the pixel coordinates of the first target object into its three-dimensional coordinates in the camera coordinate system;

[0114] The coordinate transformation module 704 is used to transform the three-dimensional coordinates of the first target object in the camera coordinate system into the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm through a coordinate transformation matrix.

[0115] The grasping control module 705 is used to take the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm as the target position, and control the target robotic arm to grasp the first target object.

[0116] In this embodiment, the grasping control module 705 drives the target robotic arm via a serial port to grasp the first target object.

[0117] Optionally, in other embodiments of this application, the robotic arm grasping control system 70 described above further includes:

[0118] The calibration module 706 is used to calibrate the coordinate transformation matrix before transforming the three-dimensional coordinates of the first target object in the base coordinate system of the target robot arm by using the coordinate transformation matrix based on the three-dimensional coordinates of the first target object in the camera coordinate system.

[0119] Optionally, in other embodiments of this application, the image acquisition module 701 described above is further used for:

[0120] Acquire a second target image, which includes an image of the second target object located at the target location.

[0121] Accordingly, the target detection module 702 described above is also used for:

[0122] Determine the pixel coordinates of the second target object in the second target image;

[0123] The aforementioned target positioning module 703 is also used for:

[0124] The pixel coordinates of the second target object are converted into its three-dimensional coordinates in the camera coordinate system.

[0125] The aforementioned robotic arm grasping control system 70 also includes:

[0126] Information collection module 707 is used for:

[0127] After controlling the end of the target robotic arm to be perpendicular to and in close contact with the second target object located at the target position, the coordinates of the second target are acquired, wherein the coordinates of the second target are the three-dimensional coordinates when the end of the target robotic arm is perpendicular to and in close contact with the second target object located at the target position;

[0128] After the end effector of the target robotic arm is in the target grasping posture, the target pose information is collected; wherein, the target grasping posture is the posture of the target robotic arm when it grasps the second target object located at the target position, and the target pose information is the pose information of the end effector of the target robotic arm when it is in the target grasping posture.

[0129] The calibration module 706 described above is also used for:

[0130] Based on the coordinates of the first target, the coordinates of the second target, and the target pose information, determine the coordinate transformation matrix between the camera coordinate system and the base coordinate system of the target robotic arm.

[0131] Optionally, in other embodiments of this application, the target detection module 702 described above is further used for:

[0132] Target detection is performed using computer vision algorithms to identify target objects in target images, including either a first target image or a second target image, and target objects including either a first target object or a second target object.

[0133] Obtain the bounding box or key points of the target object;

[0134] Extract the pixel coordinates of the target object from its bounding box or key points.

[0135] Optionally, in other embodiments of this application, the target detection module 702 described above is further used for:

[0136] Before using computer vision algorithms to detect and identify target objects in a target image, the target image is preprocessed.

[0137] Optionally, in other embodiments of this application, the target detection module 702 described above is further used for:

[0138] YOLOv5 is used for object detection to identify target objects in target images.

[0139] Optionally, in other embodiments of this application, the target positioning module 703 described above is further used for:

[0140] Substitute the pixel coordinates of the target object into the following formula to calculate the 3D camera coordinates of the target object:

[0141]

[0142] Among them, (xp ,y p (x) represents the pixel coordinates of the target object. p The x-axis pixel coordinates of the target object, y p Let z be the pixel coordinate of the target object along the Y-axis. c The depth values ​​of the depth camera used to obtain the target image, which includes either a first target image or a second target image, are used. K is the intrinsic parameter matrix of the depth camera. -1 Let K be the inverse matrix, (x c ,y c ,z c Let x be the three-dimensional coordinates of the target object in the camera coordinate system. c Let X be the X-axis coordinate of the target object in the camera coordinate system, and Y be the Y-axis coordinate. c Let z be the Y-axis coordinate of the target object in the camera coordinate system. c f is the Z-axis coordinate of the target object in the camera coordinate system. x f is the focal length value in the X direction of the depth camera. y Let (c) be the focal length value in the Y direction of the depth camera. x ,c y ) are the coordinates of the principal point, c x The x-axis coordinate of the principal point, c y The Y-axis coordinate of the principal point, which refers to the point where the optical axis intersects the image plane of the target image.

[0143] In this embodiment, the principal point is usually close to the center of the target image, and the center point of the target image can be selected as the principal point.

[0144] Optionally, in other embodiments of this application, the coordinate transformation module 704 described above is further used for:

[0145] (x) c ,y c ,z c Substitute the values ​​into the following formula to calculate the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm:

[0146]

[0147] Among them, T Base-Camera Let x be the coordinate transformation matrix between the camera coordinate system of the depth camera and the base coordinate system of the target robotic arm. Base ,y Base ,z Base Let x be the three-dimensional coordinates of the first target object in the base coordinate system of the target robotic arm. Base Let X be the X-axis coordinate of the first target object in the base coordinate system of the target robotic arm, Y be the Y-axis coordinate of the first target object in the base coordinate system of the target robotic arm, and Z be the Z-axis coordinate of the first target object in the base coordinate system of the target robotic arm.

[0148] Optionally, in other embodiments of this application, the coordinate transformation module 704 described above is further used for:

[0149] Calculate the coordinate transformation matrix T as described above using the following formula. Base-Camera :

[0150]

[0151] R = R z (r z )·R y (r y )·R x (r x );

[0152]

[0153] t = [x, y, z];

[0154] Where R is the rotation matrix of the camera coordinate system relative to the base coordinate system of the target robotic arm, and t is the translation vector of the camera coordinate system relative to the base coordinate system of the target robotic arm. x (r x R is the rotation matrix about the X-axis. y (r y R is the rotation matrix about the Y-axis. z (r z Let y be the rotation matrix around the Z-axis, rx be the rotation angle of the end effector of the target robotic arm around the X-axis, ry be the rotation angle of the end effector of the target robotic wall around the Y-axis, rz be the rotation angle of the end effector of the target robotic wall around the Z-axis, x be the distance the end effector of the target robotic wall moves along the X-axis, y be the distance the end effector of the target robotic wall moves along the Y-axis, z be the distance the end effector of the target robotic wall moves along the Z-axis, sin() be the sine function, and cos() be the cosine function.

[0155] Optionally, in other embodiments of this application, the information acquisition module 707 described above is further used for:

[0156] After calibrating the coordinate transformation matrix, and before transforming the first target object into its three-dimensional coordinates in the base coordinate system of the target robotic arm using the coordinate transformation matrix, the pose information of the end effector of the target robotic arm [x,y,z,rx,ry,rz] is obtained.

[0157] Accordingly, the coordinate transformation module 704 described above is also used for:

[0158] Substitute the obtained [x,y,z,rx,ry,rz] into T Base-CameraFrom the formulas for calculating R and t, we obtain T. Base-Camera .

[0159] Optionally, in other embodiments of this application, the aforementioned depth camera uses the Orbbec Gemini 2, a binocular structured light 3D camera based on the Orbbec MX6600 depth engine chip. It supports multiple depth working modes, providing high-precision, zero-blind-zone depth measurement, with an applicable range of 0.15 meters to 10 meters. It features a built-in six-axis IMU and hardware D2C functionality, achieving pixel-level alignment between the depth map and the RGB image, reducing computational power requirements, supporting multi-machine synchronization, and adapting to various sensing and measurement scenarios.

[0160] Optionally, in other embodiments of this application, the image acquisition module 701, target detection module 702, target localization module 703, and coordinate transformation module 704 described above are all implemented using the Atlas 200IDKA2 development board. The Atlas 200IDKA2 is equipped with a Huawei Ascend AI processor (Ascend 310) and adopts a CANN computing architecture, supporting efficient execution of deep learning inference tasks. This development board possesses powerful computing capabilities and is suitable for various AI application scenarios such as image processing, target detection, and speech recognition. It supports multiple peripheral interfaces to meet various hardware expansion needs, while also offering advantages in low power consumption and high efficiency, making it suitable for deploying and running AI inference tasks in various environments. By supporting multiple login methods (such as Type-C, Ethernet port, serial port, etc.), users can flexibly choose the operation mode according to their needs, enhancing development flexibility and debugging convenience.

[0161] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 3 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores robotic arm grasping control data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a robotic arm grasping control method.

[0162] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0163] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0164] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0165] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0166] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0167] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0168] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0169] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0170] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A robot arm grasping control method, characterized by, The mechanical arm positioning comprises: acquiring a first target image, wherein the first target image comprises an image of a first target object; determining pixel coordinates of the first target object in the first target image; converting the pixel coordinates of the first target object into three-dimensional coordinates of the first target object in a camera coordinate system; transforming the three-dimensional coordinates of the first target object in the camera coordinate system into three-dimensional coordinates of the first target object in a base coordinate system of a target mechanical arm through a coordinate transformation matrix; taking the three-dimensional coordinates of the first target object in the base coordinate system of the target mechanical arm as a target position, and controlling the target mechanical arm to grasp the first target object.

2. The method of claim 1, wherein, Further comprising: before the transformation of the three-dimensional coordinates of the first target object in the camera coordinate system into the three-dimensional coordinates of the first target object in the base coordinate system of the target mechanical arm through the coordinate transformation matrix, calibrating the coordinate transformation matrix, specifically comprising: determining a first target coordinate; wherein the first target coordinate is a three-dimensional coordinate of a second target object located at the target position in the camera coordinate system; after controlling the end of the target mechanical arm to be perpendicular to and close to the second target object located at the target position, acquiring a second target coordinate; wherein the second target coordinate is a three-dimensional coordinate of the end of the target mechanical arm when it is perpendicular to and close to the second target object located at the target position; after controlling the end of the target mechanical arm to be in a target grasping posture, acquiring target pose information; wherein the target grasping posture is a posture of the target mechanical arm when it grasps the second target object located at the target position, and the target pose information is pose information of the end of the target mechanical arm when it is in the target grasping posture; determining the coordinate transformation matrix of the camera coordinate system and the base coordinate system of the target mechanical arm according to the first target coordinate, the second target coordinate, and the target pose information.

3. The method of claim 2, wherein, The conversion of the pixel coordinates of the target object into three-dimensional coordinates of the target object in the camera coordinate system, the target object comprising a first target object or a second target object, comprises: substituting the pixel coordinates of the target object into the following formula to calculate the three-dimensional camera coordinates of the target object: wherein (x p ,y p ) is a pixel coordinate of the target object, z c is a depth value of a depth camera used for shooting a target image, the target image comprising a first target image or a second target image, K is an intrinsic matrix of the depth camera, K -1 is an inverse matrix of K, (x c ,y c ,z c ) is a three-dimensional coordinate of the target object in a camera coordinate system of the depth camera, f x is a focal length value in an X direction of the depth camera, f y is a focal length value in a Y direction of the depth camera, and (c x ,c y ) is a two-dimensional coordinate of a principal point, the principal point being a point where an optical axis intersects an image plane of the target image.

4. The method of claim 2, wherein, The transformation of the three-dimensional camera coordinates of the first target object into three-dimensional coordinates of the first target object in the base coordinate system of the target mechanical arm through a coordinate transformation relationship comprises: substituting the three-dimensional camera coordinates of the first target object into the following formula to calculate the three-dimensional coordinates of the first target object in the base coordinate system of the target mechanical arm: Wherein, (x c ,y c ,z c ) are three-dimensional coordinates of the first target object in the camera coordinate system, T Base-Camera is the coordinate transformation matrix of calibration, (x Base ,y Base ,z Base ) are three-dimensional coordinates of the first target object in the base coordinate system of the target robot arm.

5. The method of claim 4, wherein, T is calculated according to the following formula Base-Camera : R = R z (r z ) · R y (r y ) · R x (r x ) ; t=[x,y,z]; wherein R is a rotation matrix of the camera coordinate system relative to a base coordinate system of the target robot arm, t is a translation vector of the camera coordinate system relative to the base coordinate system of the target robot arm, R x (r x ) is a rotation matrix of rotation around the X-axis, R y (r y ) is a rotation matrix of rotation around the Y-axis, R z (r z ) is a rotation matrix of rotation around the Z-axis, rx is an angle of rotation of the end of the target robot arm around the X-axis, ry is an angle of rotation of the end of the target robot arm around the Y-axis, rz is an angle of rotation of the end of the target robot arm around the Z-axis, x is a movement distance of the end of the target robot arm in the X-axis, y is a movement distance of the end of the target robot arm in the Y-axis, and z is a movement distance of the end of the target robot arm in the Z-axis.

6. The method of claim 5, wherein, Further comprising: after the calibration of the coordinate transformation matrix and before the transformation of the three-dimensional coordinates of the first target object in the camera coordinate system into the three-dimensional coordinates of the first target object in the base coordinate system of the target mechanical arm through the coordinate transformation matrix, acquiring pose information [x, y, z, rx, ry, rz] of the end of the target mechanical arm; Substitute [x, y, z, rx, ry, rz] into the formula for R and t of T Base-Camera , to obtain T Base-Camera .

7. A robotic arm grasp control system, characterized by, The mechanical arm grasping control system comprises: an image acquisition module, configured to acquire a first target image, wherein the first target image comprises an image of a first target object; The target detection module is configured to determine pixel coordinates of a first target object in the first target image. The target positioning module is configured to convert the pixel coordinates of the first target object into three-dimensional coordinates of the first target object in a camera coordinate system. The coordinate conversion module is configured to convert the three-dimensional coordinates of the first target object in the camera coordinate system into three-dimensional coordinates of the first target object in a base coordinate system of a target robot arm through a coordinate transformation matrix. The grabbing control module is configured to control the target robot arm to grab the first target object by taking the three-dimensional coordinates of the first target object in the base coordinate system of the target robot arm as a target position.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the robot arm grabbing control method in any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the robot arm grabbing control method in any one of claims 1-6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the robot arm grabbing control method in any one of claims 1-6.

Images