A mechanical arm grabbing control method, system, device and electronic equipment

Abstract

This application provides a robotic arm grasping control method, system, device, and electronic device. In this embodiment, the grasping motion of the gripper is controlled by utilizing the position of the target object and the gripper in the camera coordinate system, and the grasping direction angle of the target object and the direction angle of the gripper in the image coordinate system, thereby achieving grasping control in the camera-dependent coordinate system. This eliminates the need for coordinate transformation between the camera's extrinsic parameter matrix and the robotic arm's base coordinate system, thus reducing the requirements for camera accuracy and the absolute positioning accuracy of the robotic arm. Furthermore, by performing grasping control only in the camera-dependent coordinate system, without needing coordinate transformation between the camera's extrinsic parameter matrix and the robotic arm's base coordinate system, the problem of grasping failure due to insufficient absolute positioning accuracy of the camera and robotic arm is avoided.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a robotic arm grasping control method, system, device, and electronic device. Background Technology

[0002] When a robot with a robotic arm grasps an object, it identifies the centroid coordinates and principal axis direction of the object based on images captured by a camera. Using the camera's extrinsic parameter matrix, these coordinates are transformed into the robot's base coordinate system. Based on the transformed coordinates and principal axis direction, the robot's gripper's grasping pose (grasping position and direction) is calculated. Finally, the robot controls the movement of its gripper and gripper to grasp the object based on this grasping pose.

[0003] However, during the grasping process described above, it is necessary to use the camera extrinsic parameter matrix to transform the centroid coordinates and principal axis direction to the robot's base coordinate system. The camera extrinsic parameter matrix is ​​calibrated based on the camera's coordinate system and the robot's base coordinate system. Therefore, the camera's accuracy and the robot's absolute positioning accuracy will affect the calibration accuracy of the camera extrinsic parameter matrix. Here, absolute positioning accuracy refers to the positional accuracy of the robot relative to the base coordinate system, which requires high accuracy from both the camera and the robot. Summary of the Invention

[0004] In view of this, embodiments of this application provide a robotic arm grasping control method, device, system, and electronic device, which performs grasping control in a camera-related coordinate system, eliminating the need for coordinate transformation using the camera's extrinsic parameter matrix, thereby reducing the requirements for camera accuracy and the absolute positioning accuracy of the robotic arm.

[0005] According to a first aspect of the embodiments of this application, a robotic arm grasping control method is provided, the method being applied to a robot's processor, wherein a tag is installed at a designated position on the robot's robotic arm, the tag being used to determine the position and orientation of the gripper at the end of the robotic arm; the method includes:

[0006] Obtain the coordinate information of the label pixels in the label image corresponding to the label captured by the camera;

[0007] Based on the label pixel coordinate information, determine the position of the gripper in the camera coordinate system. and the orientation angle of the gripper in the image coordinate system The direction angle The angle between the first direction pre-specified for the gripper and a specified coordinate axis in the image coordinate system;

[0008] Based on the position of the gripper in the camera coordinate system and the orientation angle in the image coordinate system And the position of the target object to be captured in the camera coordinate system has been obtained. and the orientation angle in the image coordinate system The motion control information of the gripper when grasping the target object is determined, including the direction angle. The angle between the second direction pre-specified for the target object and the specified coordinate axis;

[0009] The motion control information is used to control the movement of the gripper so that it can grasp the target object.

[0010] According to a second aspect of the embodiments of this application, a robotic arm grasping control system is provided, the system comprising:

[0011] A robotic arm, wherein a tag is installed at a designated location on the robotic arm, the tag being used to determine the position and orientation of the gripper at the end of the robotic arm;

[0012] A camera is used to capture the image of the label corresponding to the label;

[0013] A processor for use as described in the first aspect.

[0014] According to a third aspect of the embodiments of this application, a robotic arm grasping control device is provided. The device is applied to a robot, wherein a tag is installed at a designated position on the robotic arm of the robot, the tag being used to determine the position and orientation of the gripper at the end of the robotic arm; the device includes:

[0015] The acquisition module is used to obtain the coordinate information of the label pixels in the label image corresponding to the label captured by the camera;

[0016] The gripper pose information determination module is used to determine the position of the gripper in the camera coordinate system based on the label pixel coordinate information. and the orientation angle of the gripper in the image coordinate system The direction angle The angle between the first direction pre-specified for the gripper and a specified coordinate axis in the image coordinate system;

[0017] The motion control information determination module is used to determine the position of the gripper in the camera coordinate system. and the orientation angle in the image coordinate system And the position of the target object to be captured in the camera coordinate system has been obtained. and the orientation angle in the image coordinate system The motion control information of the gripper when grasping the target object is determined, including the direction angle. The angle between the second direction pre-specified for the target object and the specified coordinate axis;

[0018] The gripping control module is used to control the movement of the gripper based on the motion control information so that the gripper can grip the target object.

[0019] According to a fourth aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a processor and a memory; wherein, the memory is configured to store machine-executable instructions; and the processor is configured to read and execute the machine-executable instructions stored in the memory to implement the method as described in the first aspect.

[0020] The technical solutions provided in this application embodiment may include the following beneficial effects:

[0021] As can be seen from the above technical solutions, in this embodiment, the gripper movement is controlled by using the position of the target object to be grasped and the position of the gripper in the camera coordinate system, as well as the gripping direction angle of the target object to be grasped and the direction angle of the gripper in the image coordinate system, so as to achieve the grasping of the target object. This realizes the grasping control in the camera-related coordinate system, without the need for coordinate transformation between the camera extrinsic matrix and the base coordinate system of the robotic arm. There are no requirements for the accuracy of the camera extrinsic matrix, which reduces the requirements for camera accuracy and the absolute positioning accuracy of the robotic arm.

[0022] Furthermore, the grasping control is performed only in the camera-related coordinate system, without the need for coordinate transformation between the camera's extrinsic matrix and the robot arm's base coordinate system. This avoids the problem of grasping failure caused by insufficient absolute positioning accuracy of the robot arm. Attached Figure Description

[0023] Figure 1 This is a network example diagram of a robotic arm grasping application provided in the embodiments of this application.

[0024] Figure 2 This is a flowchart of a robotic arm grasping control method provided in an embodiment of this application.

[0025] Figure 3 This is an example diagram illustrating the determination of the pose information of the tag provided in the embodiments of this application.

[0026] Figure 4 This is an example diagram of the orientation angle of the target object to be grasped, provided in an embodiment of this application.

[0027] Figure 5 This is an example block diagram of a robotic arm grasping control system provided in an embodiment of this application.

[0028] Figure 6 This is a block diagram of a robotic arm gripping control device provided in an embodiment of this application.

[0029] Figure 7This is a hardware structure diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0031] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0032] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0033] The embodiments described in this specification will now be described in detail.

[0034] like Figure 1 As shown, Figure 1 This diagram illustrates a network configuration example for a robotic arm grasping application according to an embodiment of this application. In this embodiment, the robot includes at least: a robotic arm, a gripper (hand) disposed at the end of the robotic arm, and a processor (…). Figure 1 (Not shown in the image), wherein a tag is placed at a designated location on the robotic arm to determine the position of the gripper.

[0035] In this embodiment, the designated position can be any position where the distance between it and the position of the gripper is less than a designated distance threshold (e.g., 1 cm). This embodiment of the application does not specifically limit the location.

[0036] For example, in this embodiment, the above-mentioned label can be of many kinds, such as QR code label, animal image label, etc. This application embodiment is not specifically limited, as long as it is easy to identify.

[0037] exist Figure 1 The network shown also includes, for example, Figure 1 The camera shown here can be a depth camera, a combination of a regular camera and an inertial measurement unit (IMU), or other devices that can acquire depth information and two-dimensional images. This application embodiment is not specifically limited to these.

[0038] The depth camera can be implemented based on technologies such as binoculars, structured light, and Time-of-Flight (TOF). Examples include binocular cameras and line laser profilometers; this embodiment is not specifically limited to these.

[0039] In this embodiment, the camera can be mounted on the robot, such as on the robot's head, or at other locations other than the robot. This application embodiment does not specifically limit the mounting location of the camera, which can be determined according to the actual situation.

[0040] As one embodiment of this application, when the robot is a mobile robot, the camera is mounted on the mobile robot.

[0041] As another embodiment of this application, when the robot is a stationary robot, the camera can be mounted on the robot or at other locations that can capture images of the robot's robotic arm and the target object to be grasped, such as the roof of the space where the robot is located.

[0042] based on Figure 1 The network topology shown below will be used in conjunction with the network topology described below. Figure 2 The method provided in the embodiments of this application is described as follows:

[0043] See Figure 2 , Figure 2 This is a flowchart illustrating a method provided in an embodiment of this application. The method is applied to the processor of the aforementioned robot, such as... Figure 2 As shown, the process may include the following steps:

[0044] S210: Obtain the coordinate information of the label pixels in the label image corresponding to the label captured by the camera.

[0045] For example, in this embodiment, before performing this step, an RGB image of the label taken by the camera is obtained. Based on the RGB image of the label, the coordinate information of each label pixel in the label image can be obtained. Here, the coordinate information of each label pixel is the coordinate information in the image coordinate system.

[0046] S220: Determine the position of the gripper in the camera coordinate system based on the label pixel coordinate information. and the orientation angle of the gripper in the image coordinate system

[0047] For example, in this embodiment, the above-mentioned orientation angle The angle between a pre-specified first direction of the gripper and a specified coordinate axis in the image coordinate system is defined. Here, the first direction can be the main axis direction of the gripper. Of course, the first direction can also be other axis directions of the gripper or a custom direction. This application embodiment does not specifically limit the direction.

[0048] For example, in this embodiment, the camera coordinate system refers to the camera coordinate system used by the camera. The image coordinate system refers to the specified coordinate system corresponding to the image captured by the camera. Here, the specified coordinate system can be a coordinate system established with a specified vertex of the image as the origin and the two sides of the specified vertex as the X-axis and Y-axis.

[0049] In this embodiment, based on the above description of the image coordinate system, the specified coordinate axis can be either the Y-axis or the X-axis, and this embodiment of the application does not specifically limit it.

[0050] In this embodiment, the position of the gripper in the camera coordinate system is determined based on the label pixel coordinate information. and the orientation angle of the gripper in the image coordinate system Specifically, it can be:

[0051] Step A: Map the coordinates of the center pixel in the label image from the image coordinate system to the camera coordinate system to obtain the label's position in the camera coordinate system.

[0052] For example, in this embodiment, mapping the coordinate information of the center pixel in the label image from the image coordinate system to the camera coordinate system is a conventional coordinate system transformation process, which will not be described in detail here.

[0053] Step B: Based on the coordinates of at least two pixels in the label image along a pre-specified third direction, determine the angle between the third direction and the specified coordinate axis.

[0054] For example, in this embodiment, the third direction can be the main axis direction of the label. Of course, the third direction can also be other axis directions of the label, or a custom direction. This embodiment of the application is not specifically limited.

[0055] Figure 3 Example diagrams for determining the pose information of the label are shown below. Figure 3 For example as well as The determination process is described, such as Figure 3As shown, for a tag image, the center pixel is P0, and the coordinates of the center pixel are the coordinates of P0 in the image coordinate system. By finding the corresponding depth value in the camera depth map using the coordinates of P0, the position in the camera coordinate system can be calculated using the camera intrinsic parameter model. In this embodiment, if the third direction pre-specified for the label is one of the edges of the label image, then at least two pixels are... Figure 4 p1 and p2 are determined based on the coordinates of p1 and p2. Specifically, it can be:

[0056]

[0057] Where p2(x) is the X-axis coordinate of P2, p2(y) is the Y-axis coordinate of P2, p1(x) is the X-axis coordinate of P1, and p1(y) is the Y-axis coordinate of P1.

[0058] Step C: Based on the position of the label in the camera coordinate system Using a specified transformation matrix, determine the position of the gripper in the camera coordinate system. The specified transformation matrix is ​​used to convert the position of the label in the camera coordinate system to the position of the gripper in the camera coordinate system.

[0059] For example, in this embodiment, in step C, the position of the label in the camera coordinate system is used as a reference. Using a specified transformation matrix, determine the position of the gripper in the camera coordinate system. include:

[0060] The position of the gripper in the camera coordinate system is determined using the following formula.

[0061]

[0062] in, This indicates the position of the gripper in the camera coordinate system; Indicates the specified transformation matrix; This indicates the position of the gripper in the label coordinate system; This represents the camera extrinsic parameter matrix, which is pre-calibrated. The rotation matrix representing the label coordinate system and the base coordinate system of the robotic arm can be obtained through conventional forward kinematics. This indicates the position of the label in the camera coordinate system.

[0063] It should be noted that the specific form of the camera extrinsic parameter matrix is ​​as follows: It consists of the camera extrinsic rotation matrix. and the position of the camera in the base coordinate system Composition, camera extrinsic rotation matrix With camera extrinsic matrix They are inverse matrices;

[0064] Using the position of the label in the camera coordinate system Determine the position of the gripper in the camera coordinate system At that time, although the camera extrinsic matrix was used However, because the label is very close to the gripper, The value is very small, in the formula above It can be ignored, therefore, Also related to the camera extrinsic matrix It has little to do with the accuracy; its precision mainly depends on The calculation.

[0065] Step D: Based on Determine the orientation angle of the gripper in the image coordinate system

[0066] For example, in this embodiment, before performing step D, the positional relationship between the third direction and the first direction is first determined. Based on the positional relationship between the third direction and the first direction, according to... Determine the orientation angle of the gripper in the image coordinate system

[0067] As an example, when the positional relationship between the third direction and the first direction is that the third direction is parallel to the first direction, then directly... Determined as the orientation angle of the gripper in the image coordinate system

[0068] As another embodiment, if the positional relationship between the third direction and the first direction is such that there is an angular deviation between the third direction and the first direction, then according to... Determine the orientation angle of the gripper in the image coordinate system Specifically, it can be based on: Determine the orientation angle of the gripper in the image coordinate system based on the angle deviation. Specifically, using the aforementioned angular deviation to... Make corrections, The sum or difference between the angle deviation and the angle is used as the orientation angle of the gripper in the image coordinate system.

[0069] And specifically The sum of the angle deviation or the sum of the angle deviations The difference between the angle and the angle deviation is used as the orientation angle of the gripper in the image coordinate system. This can be determined based on the angle between the first direction and the specified coordinate axis and the angle between the third direction and the specified coordinate axis. For example, when the angle between the first direction and the specified coordinate axis is less than the angle between the third direction and the specified coordinate axis, [the following will be determined]. The sum of the angle deviations is used as the orientation angle of the gripper in the image coordinate system. For example, when the angle between the first direction and the specified coordinate axis is greater than the angle between the third direction and the specified coordinate axis, The difference between the angle and the angle deviation is used as the orientation angle of the gripper in the image coordinate system.

[0070] Of course, in this embodiment, the position of the gripper in the camera coordinate system can also be determined directly from the gripper image captured by the camera. and the orientation angle of the gripper in the image coordinate system Alternatively, the pose information of the aforementioned labels can be used to determine the position of the gripper in the camera coordinate system. and the orientation angle of the gripper in the image coordinate system The embodiments described in this application are not specifically limited.

[0071] As an example, the position of the gripper in the camera coordinate system is determined directly from the gripper image captured by the camera. and the orientation angle of the gripper in the image coordinate system Specifically, it can be:

[0072] Obtain the RGB image of the gripper corresponding to the gripper captured by the camera;

[0073] The position of the gripper in the camera coordinate system is obtained by mapping the coordinates of the center pixel in the RGB image of the gripper from the image coordinate system to the camera coordinate system.

[0074] Based on the coordinates of at least two pixels in the RGB image of the gripper along a pre-specified first direction, determine the angle between the first direction and the specified coordinate axis.

[0075] S230: Based on the position of the gripper in the camera coordinate system and the orientation angle in the image coordinate system And the position of the target object to be captured in the camera coordinate system has been obtained. and the orientation angle in the image coordinate system Determine the motion control information of the gripper when grasping the target object.

[0076] For example, in this embodiment, the position of the target object to be captured in the camera coordinate system and the orientation angle in the image coordinate system Point cloud data captured by a camera can be obtained by processing the point cloud data using conventional point cloud processing methods, or by inputting the point cloud data captured by the camera into a pre-trained neural network model. This application does not specifically limit the specific methods.

[0077] For example, in this embodiment, the above-mentioned orientation angle The angle between a pre-specified second direction and a specified coordinate axis for the target object is defined. Here, the second direction can be the main axis direction of the target object. Of course, the second direction can also be other axis directions of the target object, or a custom direction. This application embodiment does not specifically limit the definition.

[0078] Figure 4 Here is an example diagram showing the orientation angle of the target object. The following is an example diagram. Figure 4 For example, the direction angle of the target object Describe, such as Figure 4 As shown, in the image coordinate system (RGB coordinate system), for the target object, the pre-specified second direction for the target object is... Figure 4 In the `dir` parameter, the coordinate axis is specified as the y-axis of the image coordinate system. Therefore, the aforementioned direction angles... That is Figure 4 yaw in the middle.

[0079] In this embodiment, the target object to be grasped can be any object. This embodiment of the application is not specifically limited and can be determined according to the application scenario.

[0080] For example, when the application scenario is a parcel picking scenario, the target object can be a parcel box; when the application scenario is a food delivery scenario, the target object can be a food delivery box.

[0081] For example, in this embodiment, the motion control information mentioned above includes at least linear velocity and angular velocity, but this embodiment is not specifically limited.

[0082] Based on the above description of motion control information, in this embodiment, in step S230, the position of the gripper in the camera coordinate system is determined... and the orientation angle in the image coordinate system And the position of the target object to be captured in the camera coordinate system has been obtained. and the orientation angle in the image coordinate system Determine the motion control information of the gripper when grasping the target object, specifically: based on the position of the target object in the camera coordinate system. and the position of the gripper in the camera coordinate system Determine the linear velocity; based on the orientation angle of the target object in the image coordinate system. and the orientation angle of the gripper in the image coordinate system The angular velocity is determined. As for how to calculate the linear velocity and angular velocity, the following examples describe the distance, so they will not be repeated here.

[0083] S240: Controls the movement of the gripper based on motion control information to grasp the target object.

[0084] For example, in this embodiment, in step S240, the movement of the gripper is controlled based on the linear velocity and angular velocity obtained in step S230.

[0085] In this embodiment, the gripper may grasp the target object once every specified time interval (e.g., 5 seconds), or it may grasp the target object only when the specified grasping conditions are met. This embodiment of the application does not specifically limit the time interval.

[0086] Here, the specified crawling conditions can be: Less than the set angle value (e.g., 10 degrees) and / or The specified grabbing conditions are only examples and are not intended to limit the embodiments of this application. The distance is less than the set distance value (e.g., 1cm).

[0087] This concludes the process. Figure 2 The process is shown below.

[0088] pass Figure 2 As can be seen from the process, in this embodiment of the application, the gripper movement is controlled by using the position of the target object to be grasped and the position of the gripper in the camera coordinate system, as well as the grasping direction angle of the target object to be grasped and the direction angle of the gripper in the image coordinate system, so as to achieve the grasping of the target object. This realizes grasping control in the camera-related coordinate system, without the need for coordinate transformation between the camera extrinsic matrix and the base coordinate system of the robotic arm. There are no requirements for the accuracy of the camera extrinsic matrix, which reduces the requirements for camera accuracy and the absolute positioning accuracy of the robotic arm.

[0089] Furthermore, the grasping control is performed only in the camera-related coordinate system, without the need for coordinate transformation between the camera extrinsic matrix and the robot arm's base coordinate system, thus avoiding the problem of grasping failure caused by insufficient absolute positioning accuracy of the robot arm.

[0090] Furthermore, by determining the pose information of the target object and the gripper in real time, and then calculating the velocity and angular velocity in the coordinate system corresponding to the camera, a velocity control method is used to achieve real-time closed-loop control, effectively improving the accuracy of the network system.

[0091] As an optional implementation of this application, the method is based on the position of the target object to be captured in the camera coordinate system. and the position of the gripper in the camera coordinate system Determining linear velocity includes:

[0092] The linear velocity is determined using the following formula:

[0093]

[0094] Where v represents linear velocity; K v Here, is the linear velocity gain coefficient, a predefined constant; err_pos represents the position error. Let be the camera extrinsic rotation matrix, and They are inverse matrices; although the camera extrinsic rotation matrix is ​​used here. However, it only changes the direction, and only needs to be roughly accurate. The final motion accuracy is still guaranteed by err_pos.

[0095] As an optional implementation of this application, the orientation angle of the target object to be captured in the image coordinate system is used. and the orientation angle of the gripper in the image coordinate system Determining angular velocity includes:

[0096] The angular velocity is determined using the following formula:

[0097] w=K w *err_angle

[0098] w represents angular velocity; K w The angular velocity gain coefficient is a predefined constant; err_angle represents the angular error.

[0099] Taking a grasping accuracy of 1cm as an example, if a traditional method is used, it is necessary to ensure that the camera's extrinsic parameter matrix is ​​accurate. The converted positional accuracy is higher than 1cm. The camera extrinsic matrix is ​​determined by the absolute positioning accuracy of the robotic arm and the accuracy of the camera. This means that only by using a higher-precision camera and robotic arm (with absolute positioning accuracy much higher than 1cm) can a camera extrinsic matrix with an accuracy of 1cm be calibrated. If this solution is used, since closed-loop grasping is performed in the camera-related coordinate system, it is only necessary to ensure that the repeatability of the camera is 1cm, which effectively reduces the requirements for the absolute positioning accuracy of the robotic arm and the accuracy of the camera.

[0100] Corresponding to the embodiments of the foregoing methods, this specification also provides embodiments of systems, devices, and terminals to which they are applied.

[0101] like Figure 5 As shown, Figure 5 This is an example block diagram illustrating a robotic arm grasping control system according to an embodiment of this application. The robotic arm grasping control system includes:

[0102] The robotic arm has tags installed at designated locations to determine the position and orientation of the gripper at its end.

[0103] A camera is used to capture the image of the label corresponding to the label.

[0104] A processor for executing the methods corresponding to the above method embodiments.

[0105] This concludes the process. Figure 5 Description of the block diagram shown.

[0106] For specific implementation details, please refer to the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0107] like Figure 6 As shown, Figure 6 This is a block diagram illustrating a robotic arm gripping control device according to an embodiment of this application. The device is applied to a robot, and a tag is installed at a designated location on the robot's robotic arm. The tag is used to determine the position and orientation of the gripper at the end of the robotic arm. The robotic arm gripping control device includes:

[0108] The acquisition module is used to obtain the coordinate information of the label pixels in the label image corresponding to the label captured by the camera;

[0109] The gripper pose information determination module is used to determine the position of the gripper in the camera coordinate system based on the label pixel coordinate information. and the orientation angle of the gripper in the image coordinate system Direction angle This refers to the angle between a pre-specified first direction of the gripper and a specified coordinate axis in the image coordinate system;

[0110] The motion control information determination module is used to determine the position of the gripper in the camera coordinate system. and the orientation angle in the image coordinate system And the position of the target object to be captured in the camera coordinate system has been obtained. and the orientation angle in the image coordinate system Determine the motion control information of the gripper when grasping the target object, including the orientation angle. The angle between a pre-specified second direction and a specified coordinate axis for the target object;

[0111] The gripping control module is used to control the movement of the gripper based on motion control information so that the gripper can grasp the target object.

[0112] As an optional implementation of this application, the gripper pose information determination module is specifically used for:

[0113] Mapping the coordinates of the center pixel in the label image from the image coordinate system to the camera coordinate system yields the label's position in the camera coordinate system.

[0114] Based on the coordinates of at least two pixels in the label image along a pre-specified third direction, determine the angle between the third direction and the specified coordinate axis.

[0115] Based on the position of the label in the camera coordinate system Using a specified transformation matrix, determine the position of the gripper in the camera coordinate system. The specified transformation matrix is ​​used to convert the position of the label in the camera coordinate system to the position of the gripper in the camera coordinate system;

[0116] in accordance with Determine the orientation angle of the gripper in the image coordinate system

[0117] As an optional implementation method of this application, the above-mentioned basis Determine the orientation angle of the gripper in the image coordinate system include:

[0118] If the third direction is parallel to the first direction, then Determined as the orientation angle of the gripper in the image coordinate system

[0119] If there is an angular deviation between the third direction and the first direction, then according to Determine the orientation angle of the gripper in the image coordinate system based on the angle deviation.

[0120] As an optional implementation of this application, the label is a QR code; and / or, the position of the label in the camera coordinate system. Position of the gripper in the camera coordinate system The distance between them is less than a specified distance threshold.

[0121] As an optional implementation of this application, the above-mentioned method is based on the position of the label in the camera coordinate system. Using a specified transformation matrix, determine the position of the gripper in the camera coordinate system. include:

[0122] The position of the gripper in the camera coordinate system is determined using the following formula.

[0123]

[0124] in, This indicates the position of the gripper in the camera coordinate system; Indicates the specified transformation matrix; This indicates the position of the gripper in the label coordinate system; This represents the camera extrinsic parameter matrix, which is pre-calibrated. The rotation matrix represents the coordinate system of the label and the base coordinate system of the robotic arm; This indicates the position of the label in the camera coordinate system.

[0125] As an optional implementation of this application, the motion control information determination module is specifically used for:

[0126] Based on the position of the target object in the camera coordinate system and the position of the gripper in the camera coordinate system Determine the linear velocity;

[0127] Based on the orientation angle of the target object in the image coordinate system and the orientation angle of the gripper in the image coordinate system Determine the angular velocity;

[0128] Motion control information is determined based on linear velocity and angular velocity.

[0129] As an optional implementation of this application, the above-mentioned method is based on the position of the target object to be captured in the camera coordinate system. and the position of the gripper in the camera coordinate system Determining linear velocity includes:

[0130] The linear velocity is determined using the following formula:

[0131]

[0132] Where v represents linear velocity; K v The linear velocity gain coefficient is a predefined constant. Let be the camera extrinsic rotation matrix, and They are inverse matrices; err_pos represents the position error.

[0133] As an optional implementation of this application, the above-mentioned method is based on the orientation angle of the target object to be captured in the image coordinate system. and the orientation angle of the gripper in the image coordinate system Determining angular velocity includes:

[0134] The angular velocity is determined using the following formula:

[0135] w=K w *err_angle

[0136] w represents angular velocity; kw The angular velocity gain coefficient is a predefined constant; err_angle represents the angular error.

[0137] This concludes the process. Figure 6 Description of the block diagram shown.

[0138] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0139] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0140] Correspondingly, embodiments of this application also provide Figure 6 The hardware structure diagram of the device shown is as follows: Figure 7 As shown, the electronic device can be a device implementing the above-described method. Figure 7 As shown, the hardware architecture includes a processor and memory.

[0141] The memory is used to store machine-executable instructions;

[0142] The processor is used to read and execute the machine-executable instructions stored in the memory to implement the corresponding robotic arm grasping control method embodiment shown above.

[0143] As one embodiment, the memory can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, the memory can be volatile memory, non-volatile memory, or similar storage media. Specifically, the memory can be RAM (Random Access Memory), flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0144] This concludes the process. Figure 7 Description of the electronic device shown.

[0145] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0146] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this specification are indicated by the following claims.

[0147] It should be understood that this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is limited only by the appended claims.

[0148] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A robotic arm grasping control method, characterized in that, The method is applied to a robot's processor, wherein a tag is installed at a designated location on the robot's robotic arm, the tag being used to determine the position and orientation of the gripper at the end of the robotic arm; the method includes: Obtain the coordinate information of the label pixels in the label image corresponding to the label captured by the camera; Based on the label pixel coordinate information, determine the position of the gripper in the camera coordinate system. and the orientation angle of the gripper in the image coordinate system The direction angle The angle between the first direction pre-specified for the gripper and a specified coordinate axis in the image coordinate system; Based on the position of the gripper in the camera coordinate system and the orientation angle in the image coordinate system And the position of the target object to be captured in the camera coordinate system has been obtained. and the orientation angle in the image coordinate system The motion control information of the gripper when grasping the target object is determined, including the direction angle. The angle between the second direction pre-specified for the target object and the specified coordinate axis; The motion control information is used to control the movement of the gripper so that it can grasp the target object.

2. The method according to claim 1, characterized in that, The position of the gripper in the camera coordinate system is determined based on the label pixel coordinate information. and the orientation angle of the gripper in the image coordinate system include: The coordinates of the center pixel in the label image are mapped from the image coordinate system to the camera coordinate system to obtain the position of the label in the camera coordinate system. Based on the coordinate information of at least two pixels in the label image along a pre-specified third direction for the label, determine the angle between the third direction and the specified coordinate axis. Based on the position of the label in the camera coordinate system Using a specified transformation matrix, the position of the gripper in the camera coordinate system is determined. The specified transformation matrix is ​​used to convert the position of the label in the camera coordinate system into the position of the gripper in the camera coordinate system; According to the above Determine the orientation angle of the gripper in the image coordinate system.

3. The method according to claim 2, characterized in that, According to the above Determine the orientation angle of the gripper in the image coordinate system. include: If the third direction is parallel to the first direction, then the... The orientation angle of the gripper in the image coordinate system is determined. If the third direction has an angular deviation from the first direction, then according to the... The orientation angle of the gripper in the image coordinate system is determined by the angle deviation.

4. The method according to claim 1, characterized in that, The label is a QR code; and / or, the position of the label in the camera coordinate system. The position of the gripper in the camera coordinate system The distance between them is less than a specified distance threshold.

5. The method according to claim 2, characterized in that, The position of the label in the camera coordinate system Using a specified transformation matrix, the position of the gripper in the camera coordinate system is determined. include: The position of the gripper in the camera coordinate system is determined according to the following formula. in, This indicates the position of the gripper in the camera coordinate system; Indicates the specified transformation matrix; This indicates the position of the gripper in the label coordinate system; This represents the camera extrinsic parameter matrix, which is pre-calibrated. The rotation matrix represents the coordinate system of the label and the base coordinate system of the robotic arm; This indicates the position of the label in the camera coordinate system.

6. The method according to claim 1, characterized in that, The position of the gripper in the camera coordinate system and the orientation angle in the image coordinate system And the position of the target object to be captured in the camera coordinate system has been obtained. and the orientation angle in the image coordinate system Determining the motion control information of the gripper when grasping the target object includes: Based on the position of the target object to be captured in the camera coordinate system and the position of the gripper in the camera coordinate system Determine the linear velocity; Based on the orientation angle of the target object to be captured in the image coordinate system and the orientation angle of the gripper in the image coordinate system Determine the angular velocity; The motion control information is determined based on the linear velocity and the angular velocity.

7. The method according to claim 6, characterized in that, The position of the target object to be captured in the camera coordinate system is used as a basis. and the position of the gripper in the camera coordinate system Determining linear velocity includes: The linear velocity is determined according to the following formula: Where v represents linear velocity; K v The linear velocity gain coefficient is a predefined constant. Let be the camera extrinsic rotation matrix, and They are inverse matrices; err_pos represents the position error.

8. The method according to claim 6, characterized in that, The orientation angle of the target object to be captured in the image coordinate system is used as a basis. and the orientation angle of the gripper in the image coordinate system Determining angular velocity includes: The angular velocity is determined according to the following formula: w=K w *err_angle w represents angular velocity; K w The angular velocity gain coefficient is a predefined constant; err_angle represents the angular error.

9. A robotic arm grasping control system, characterized in that, The system includes: a robotic arm, on which a tag is installed at a designated location, the tag being used to determine the position and orientation of the gripper at the end of the robotic arm; A camera is used to capture the image of the label corresponding to the label; A processor for performing the method as described in any one of claims 1-8.

10. A robotic arm grasping control device, characterized in that, The device is applied to a robot, wherein a tag is installed at a designated location on the robot's robotic arm, the tag being used to determine the position and orientation of the gripper at the end of the robotic arm; the device includes: The acquisition module is used to obtain the coordinate information of the label pixels in the label image corresponding to the label captured by the camera; The gripper pose information determination module is used to determine the position of the gripper in the camera coordinate system based on the label pixel coordinate information. and the orientation angle of the gripper in the image coordinate system The direction angle The angle between the first direction pre-specified for the gripper and a specified coordinate axis in the image coordinate system; The motion control information determination module is used to determine the position of the gripper in the camera coordinate system. and the orientation angle in the image coordinate system And the position of the target object to be captured in the camera coordinate system has been obtained. and the orientation angle in the image coordinate system The motion control information of the gripper when grasping the target object is determined, including the direction angle. The angle between the second direction pre-specified for the target object and the specified coordinate axis; The gripping control module is used to control the movement of the gripper based on the motion control information so that the gripper can grip the target object.

11. The apparatus according to claim 10, characterized in that, The gripper pose information determination module is specifically used for: The coordinates of the center pixel in the label image are mapped from the image coordinate system to the camera coordinate system to obtain the position of the label in the camera coordinate system. Based on the coordinate information of at least two pixels in the label image along a pre-specified third direction for the label, determine the angle between the third direction and the specified coordinate axis. Based on the position of the label in the camera coordinate system Using a specified transformation matrix, the position of the gripper in the camera coordinate system is determined. The specified transformation matrix is ​​used to convert the position of the label in the camera coordinate system into the position of the gripper in the camera coordinate system; According to the above Determine the orientation angle of the gripper in the image coordinate system. According to the above Determine the orientation angle of the gripper in the image coordinate system. include: If the third direction is parallel to the first direction, then the... The orientation angle of the gripper in the image coordinate system is determined. If the third direction has an angular deviation from the first direction, then according to the... The orientation angle of the gripper in the image coordinate system is determined by the angle deviation. The label is a QR code; and / or, the position of the label in the camera coordinate system. The position of the gripper in the camera coordinate system The distance between them is less than a specified distance threshold; The position of the label in the camera coordinate system Using a specified transformation matrix, the position of the gripper in the camera coordinate system is determined. include: The position of the gripper in the camera coordinate system is determined according to the following formula. in, This indicates the position of the gripper in the camera coordinate system; Indicates the specified transformation matrix; This indicates the position of the gripper in the label coordinate system; This represents the camera extrinsic parameter matrix, which is pre-calibrated. The rotation matrix represents the coordinate system of the label and the base coordinate system of the robotic arm; This indicates the position of the label in the camera coordinate system; The motion control information determination module is specifically used for: Based on the position of the target object to be captured in the camera coordinate system and the position of the gripper in the camera coordinate system Determine the linear velocity; Based on the orientation angle of the target object to be captured in the image coordinate system and the orientation angle of the gripper in the image coordinate system Determine the angular velocity; The motion control information is determined based on the linear velocity and the angular velocity; The position of the target object to be captured in the camera coordinate system is used as a basis. and the position of the gripper in the camera coordinate system Determining linear velocity includes: The linear velocity is determined according to the following formula: Where v represents linear velocity; K v The linear velocity gain coefficient is a predefined constant. Let be the camera extrinsic rotation matrix, and They are inverse matrices; err_pos represents the position error. The orientation angle of the target object to be captured in the image coordinate system is used as a basis. and the orientation angle of the gripper in the image coordinate system Determining angular velocity includes: The angular velocity is determined according to the following formula: w=K w *err_angle w represents angular velocity; K w The angular velocity gain coefficient is a predefined constant; err_angle represents the angular error.

12. An electronic device, characterized in that, Electronic devices include: processors and memory; The memory is used to store machine-executable instructions; The processor is configured to read and execute machine-executable instructions stored in the memory to implement the method as described in any one of claims 1 to 8.

Images