Calibration method, device, terminal equipment and storage medium

Abstract

This application is applicable to the field of robotic arm technology and provides a calibration method, apparatus, terminal device, and storage medium. In an embodiment of the application, the calibration efficiency of multiple robotic arms is improved by obtaining a first calibration relationship between a preset depth camera and a preset first robotic arm; obtaining position information of a target workpiece using the depth camera; obtaining a second calibration relationship between a preset second robotic arm and the first robotic arm; and determining the relative position relationship between the second robotic arm and the target workpiece based on the first calibration relationship, the second calibration relationship, and the position information.

Description

Technical Field

[0001] The present application belongs to the field of robotic arm technology, and in particular relates to a calibration method, apparatus, terminal device and storage medium. Background Art

[0002] With the development of the robotics industry and the demands of modern manufacturing processes, robotic arms have become a widely used device within the robotics field, particularly in automation sectors such as machinery manufacturing, semiconductor processing, and medical production. Given the increasing flexibility of modern automated production lines and the increasing complexity and variability of processing tasks, achieving collaborative grasping with multiple robotic arms is a key issue in industrial automation.

[0003] However, due to the current common multi-arm grasping technology, the operator needs to manually control the joint movement of the robot arm through the teach pendant to move the robot arm to a predetermined position, thereby achieving calibration of the predetermined position, so that the robot arm can automatically repeat the task. In this process, each robot arm needs to be taught individually by humans to achieve calibration of each robot arm to the predetermined position, resulting in low calibration efficiency of multiple robot arms. Summary of the Invention

[0004] The embodiments of the present application provide a calibration method, apparatus, terminal device, and storage medium, which can solve the problem of low calibration efficiency of multiple robotic arms.

[0005] In a first aspect, an embodiment of the present application provides a calibration method, comprising:

[0006] Acquire a first calibration relationship between a preset depth camera and a preset first robotic arm;

[0007] Acquiring position information of a target workpiece using the depth camera;

[0008] Acquire a second calibration relationship between a preset second robotic arm and the first robotic arm;

[0009] The relative position relationship between the second robotic arm and the target workpiece is determined according to the first calibration relationship, the second calibration relationship and the posture information.

[0010] In a second aspect, an embodiment of the present application provides a calibration device, including:

[0011] A first calibration relationship acquisition module, configured to acquire a first calibration relationship between a preset depth camera and a preset first robotic arm;

[0012] A posture acquisition module is used to acquire the posture information of the target workpiece using the depth camera;

[0013] A second calibration relationship acquisition module, configured to acquire a second calibration relationship between a preset second robotic arm and the first robotic arm;

[0014] A relative position relationship determination module is used to determine the relative position relationship between the second robotic arm and the target workpiece according to the first calibration relationship, the second calibration relationship and the posture information.

[0015] In a third aspect, an embodiment of the present application provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor implements the steps of any one of the above calibration methods.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned calibration methods are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the calibration methods described in the first aspect above.

[0018] In the embodiment of the present application, by obtaining a first calibration relationship between a preset depth camera and a preset first robotic arm, and then using the depth camera to obtain the posture information of the target workpiece, the relative position relationship between the first robotic arm and the target workpiece can be determined according to the first calibration relationship and the posture matrix, and then the second calibration relationship between the preset second robotic arm and the first robotic arm is obtained, thereby determining the relative position relationship between the second robotic arm and the target workpiece according to the first calibration relationship, the second calibration relationship and the posture information, so as to quickly determine the relative position relationship between multiple robotic arms and the target workpiece through the calibration relationship between the robotic arms, thereby improving the calibration efficiency of multiple robotic arms. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 This is a schematic diagram of the first flow chart of the calibration method provided in the embodiment of the present application;

[0021] Figure 2 This is a second flow chart of the calibration method provided in the embodiment of the present application;

[0022] Figure 3 It is a structural diagram of the calibration plate provided in an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of overlapping scenes of the calibration method provided in an embodiment of the present application;

[0024] Figure 5 Schematic diagram of non-overlapping scenarios of the calibration method provided in an embodiment of the present application;

[0025] Figure 6 is a structural diagram of a calibration device provided in an embodiment of the present application;

[0026] Figure 7 It is a structural diagram of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0028] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0029] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0030] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0031] At present, in order to realize collaborative grasping of multiple robotic arms, each robotic arm in the industrial production area needs to be taught separately to complete the calibration of each robotic arm to the predetermined position. However, the calibration efficiency of this process is low. For this reason, the present application provides a calibration method, which quickly determines the relative position relationship between multiple robotic arms and the target workpiece through the calibration relationship between the robotic arms, thereby improving the calibration efficiency of multiple robotic arms.

[0032] Figure 1FIG. 1 is a flow chart of a calibration method in an embodiment of the present application. The execution subject of the method may be a terminal device, such as Figure 1 As shown, the above calibration method may include the following steps:

[0033] Step S101: Acquire a first calibration relationship between a preset depth camera and a preset first robotic arm.

[0034] In this embodiment, the depth camera is used to capture images of workpieces in the area to be grasped by the robotic arm. It can be positioned above the area to be grasped by the robotic arm to facilitate better observation of the area to be grasped by the robotic arm. The terminal device establishes a relationship between the first robotic arm and the area to be grasped by the robotic arm by acquiring the first calibration relationship. The first robotic arm is located in an industrial production area; the target calibration board can be a board in the form of black and white chessboard; and the depth camera can be an RGBD depth camera.

[0035] In one embodiment, the first calibration relationship needs to be determined in the above step S101, which may specifically include: by setting the above target calibration plate at the end of the first robotic arm, the terminal device then controls the depth camera to collect target calibration plate data of the target calibration plate, that is, an image of the target calibration plate, and processing the data with a preset first processing algorithm, thereby determining the end calibration relationship between the depth camera and the end of the first robotic arm. The above first processing algorithm may be a hand-eye calibration algorithm, such as the Tsai-Lenz algorithm, by which a rotation and translation matrix (denoted as RT matrix) between the depth camera and the target calibration plate is obtained, and the RT matrix is ​​the above end calibration relationship.

[0036] The coordinate system used in the automated production process uses the center point of each robotic arm base in the industrial production area to represent each robotic arm. Therefore, in order to facilitate the control work in the automated production process, the terminal device can be processed by the robotic arm control software. Specifically, the transformation matrix between the end of the first robotic arm and the base of the first robotic arm can be obtained, and then the first calibration relationship can be determined based on the end calibration relationship and the transformation matrix determined above. The first calibration relationship is the calibration relationship between the first robotic arm base and the depth camera.

[0037] For example, if the calculated end calibration relationship of the first robotic arm is matrix A1 and the transformation matrix of the first robotic arm is B1, then the first calibration relationship C1 of the first robotic arm is:

[0038] C1=A1 -1 B1

[0039] Step S102: Use a depth camera to obtain the position and posture information of the target workpiece.

[0040] In this embodiment, the depth camera can capture a target workpiece located in the robotic arm's grasping area according to control instructions from the terminal device, capture an image of the target workpiece, and transmit the image to the terminal device. After the terminal device acquires the workpiece image, it can process the acquired workpiece image using a preset second processing algorithm to determine the position and pose information of the target workpiece. In this embodiment, by using the depth camera to estimate the position and pose of the target workpiece, the accuracy of workpiece detection is improved, enabling the robotic arm to grasp the target workpiece from the correct angle, thereby improving the efficiency of workpiece processing in the assembly line.

[0041] Exemplarily, the second processing algorithm may be a Hough voting algorithm. Specifically, the terminal device may perform segmentation processing on the workpiece image, and then use the Hough voting algorithm to perform voting processing on the segmented image to determine the 3D key points of the workpiece image. Then, based on the correspondence between the 3D key points in the depth camera and the workpiece coordinate system, the least squares method is used to estimate the six degrees of freedom pose of the target workpiece, thereby obtaining a rigid transformation of the target workpiece from its workpiece coordinate system to the coordinate system corresponding to the depth camera, that is, determining a pose matrix including the rotation transformation and translation transformation of the workpiece. The pose matrix is ​​the above-mentioned pose information.

[0042] In one embodiment, after step S102 , the method may further include: determining a relative position relationship between the first robotic arm and the target workpiece according to the first calibration relationship and the posture information.

[0043] In this embodiment, the terminal device may process the first calibration relationship and the pose matrix using a preset formula to obtain the relative position relationship between the first robotic arm and the target workpiece. The preset formula is:

[0044]

[0045] Among them, the above K1 is the relative position relationship between the first robot arm and the target workpiece, and the above S1 is the posture matrix representing the posture information.

[0046] Step S103: Obtain a preset second calibration relationship between the second robotic arm and the first robotic arm.

[0047] In this embodiment, since there may be at least one second robotic arm in the above-mentioned industrial production area, in order to reduce the time required for calibration between at least one robotic arm in the industrial production area and the location of the target workpiece, a second calibration relationship between the at least one second robotic arm and the first robotic arm can be obtained, so as to determine the relative position relationship between the at least one second robotic arm and the target workpiece based on the second calibration relationship between the at least one second robotic arm and the first robotic arm. The at least one second robotic arm can simultaneously calculate the calibration relationship with the first robotic arm, or can calculate the calibration relationship with the first robotic arm in sequence according to the arrangement order. For example, There are currently three robotic arms arranged in sequence, namely robotic arm 1, robotic arm 2, and robotic arm 3, and robotic arm 1 is the first robotic arm, then robotic arm 2 and robotic arm 3 can calculate the calibration relationship with robotic arm 1 at the same time; or robotic arm 2 and robotic arm 3 can calculate the calibration relationship with robotic arm 1 in sequence, where, after calculating the calibration relationship between robotic arm 1 and robotic arm 2, robotic arm 3 can calculate the calibration relationship between itself and robotic arm 1 through robotic arm 2, that is, calculate the calibration relationship between robotic arm 3 and robotic arm 2, and then determine the calibration relationship between robotic arm 3 and robotic arm 1 based on the calibration relationship between robotic arm 3 and robotic arm 2.

[0048] In one embodiment, Figure 2 As shown, in the above step S103, the second calibration relationship needs to be determined, which may specifically include:

[0049] Step S201: Determine whether there is an overlapping field of view between a first camera and a second camera; wherein the first camera is a camera located at the end of a first robotic arm, and the second camera is a camera located at the end of a second robotic arm.

[0050] If yes, execute steps S202 to S204; if no, execute step S205.

[0051] In this embodiment, the terminal device can determine the calibration relationship between the first and second robotic arms corresponding to the cameras based on the overlapping fields of view between the first and second cameras. Therefore, the corresponding calibration relationship needs to be determined based on the result of determining whether there is an overlapping field of view between the first and second cameras. The overlapping field of view refers to the same field of view seen from the perspective of the first camera and the perspective of the second camera.

[0052] Step S202: Control the preset calibration plate to move to the overlapping field of view between the first camera and the second camera.

[0053] In this embodiment, the terminal device needs to determine the calibration relationship between the first camera and the second camera based on the same data in the overlapping field of view, so it is necessary to control the preset calibration plate to move into the overlapping field of view of the two cameras so that the terminal device can obtain the same data on the calibration plate in the overlapping field of view.

[0054] In one embodiment, step S202 may include: the terminal device controls a mobile device carrying a calibration plate to move between the first camera and the second camera to accelerate the time to reach the overlapping field of view, and during the movement, previews the calibration plate data sent by the first camera and the second camera in real time for comparison to determine whether the calibration plate data corresponding to the first camera and the second camera respectively have the same data. When the calibration plate data collected by the first camera and the second camera respectively contain the same calibration sub-plate data, it is determined that the calibration plate has moved to the overlapping field of view between the first camera and the second camera. The mobile device may be a smart car, and the terminal device may control the mobile device to move or rotate in various directions, such as forward, backward, left, right, or rotate in place. The first camera and the second camera may be RGB cameras, which may be fixedly mounted at the end of a robotic arm to facilitate calibration of the robotic arm.

[0055] It is understood that cameras can be installed at the end of each robotic arm in the industrial production area to facilitate calibration of the relationship between the robotic arms in the industrial production area. In addition, the calibration plate can be divided into various calibration sub-plates according to preset rules. The aforementioned identical calibration sub-plate data refers to all data on the divided calibration sub-plates. For example, if the calibration plate data collected by two cameras only contains partial data from a particular calibration sub-plate, the calibration sub-plates containing only partial data are not considered to be the same calibration sub-plate.

[0056] For example, Figure 3 As shown, Figure 3 The calibration plate in the example is a five-sided three-dimensional black and white chessboard. The calibration plate includes five calibration sub-plates, namely A, B, C, D, and E. In this example, for easy distinction, the calibration sub-plate A located in the center of the calibration plate is set to a 7*10 black and white chessboard format, and the calibration sub-plates B, C, D, and E located on the periphery of the calibration plate are set to a 6*9 black and white chessboard format. The angle between the calibration sub-plates B, C, D, and E and the calibration sub-plate A is set to 45°, so that cameras located at different positions on the calibration plate can obtain more corner point information on the calibration plate. When a mobile device carrying the calibration plate moves between a first camera and a second camera, the calibration plate data collected by the first camera and the second camera are compared. When the calibration plate data corresponding to the two cameras contains data from at least one of the calibration sub-plates A, B, C, D, and E, it is determined that the mobile device is currently in an overlapping field of view. The calibration sub-plate data contains data for each corner point on the calibration sub-plate.

[0057] Step S203: Acquire calibration plate data of the calibration plate collected by the first camera and the second camera respectively.

[0058] In this embodiment, when the first camera and the second camera reach the overlapping field of view, the terminal device can control the mobile device to stay at the current position for a preset time, so that the first camera and the second camera can respectively shoot the calibration plate to collect the calibration plate data of the calibration plate, so as to facilitate the subsequent processing of the calibration plate data.

[0059] In one embodiment, when there are at least three robotic arms arranged in sequence in an industrial production area, and there are overlapping fields of view between adjacent robotic arms among the at least three robotic arms, the terminal device can control the mobile device to move according to the arrangement order of the robotic arms, thereby obtaining the calibration plate data between each robotic arm in sequence, so as to obtain the calibration relationship between each robotic arm, so that after a robotic arm among the at least three robotic arms arranged in sequence determines the relative position relationship between itself and the target workpiece, according to the calibration relationship between each robotic arm, the calibration relationship between each robotic arm except the robotic arm that determines the relative position relationship and the robotic arm that determines the relative position relationship can be determined.

[0060] For example, when there are three robotic arms and cameras 1, 2, and 3 are respectively provided at the ends of the three robotic arms, when the terminal device controls the mobile device to move from camera 1 to camera 3, when the terminal device detects that the calibration plate is located in the overlapping field of view of camera 1 and camera 2, the mobile device is controlled to stop moving and sends a shooting instruction to the camera so that the camera shoots the calibration plate, obtains the calibration plate data, and stores the camera numbers 1 and 2 and the same calibration sub-plate number in the current data acquisition process. Then, the mobile device is controlled to continue moving toward camera 3. When the terminal device detects that the calibration plate is located in the overlapping field of view of camera 2 and camera 3, the mobile device is controlled to stop moving and sends a shooting instruction to the camera so that the camera shoots the calibration plate, obtains the calibration plate data, and stores the camera numbers 2 and 3 and the same calibration sub-plate number in the current data acquisition process. In this way, the data acquisition of the entire process is completed, which speeds up the time required for data acquisition. By storing various data during the movement process, relevant optimization can be performed later. The data collected by the camera can also be visualized through the display interface during the data acquisition process, such as the calibration plate data captured by camera 1.

[0061] In one embodiment, before step S203, the method may further include: the terminal device determines the relative position of the same calibration sub-plate in the calibration plate according to the same calibration sub-plate data and calibration plate data, such as Figure 3 As shown, if the same calibration sub-board is Figure 3The C calibration sub-board in the figure is the left position. The rotation direction of the preset pan-tilt is determined according to the relative position. The pan-tilt is used to fix the calibration board on the mobile device, such as Figure 3 As shown, the terminal device can control the gimbal to rotate in various directions by means of horizontal, vertical pitch, etc., and can also control the gimbal to stop during the rotation process; control the gimbal to rotate in the rotation direction within a preset rotation range, and determine the rotation angle of the gimbal when the calibration plate data collected by the first camera and the second camera respectively have the most identical calibration sub-plate data during the rotation process;

[0062] Accordingly, the above S203 may include: based on the rotation angle of the gimbal, the terminal device obtains the calibration plate data collected by the first camera and the second camera respectively, so as to obtain more identical calibration sub-plate data and improve the accuracy of determining the calibration relationship.

[0063] In one embodiment, in order to obtain more identical calibration sub-plate data in the overlapping field of view, after determining the rotation angle of the gimbal, the moving device can also be moved by the above-mentioned relative position control to determine whether the number of identical calibration sub-plates can be further increased during the movement process.

[0064] Step S204 : determining a second calibration relationship based on the same calibration sub-plate data in the calibration plate data of the first camera and the second camera.

[0065] In this embodiment, the terminal device determines the calibration relationship between the first camera and the second camera relative to the same calibration sub-board based on the same calibration sub-board data of the first camera and the second camera, and then determines the calibration relationship between the two cameras based on their calibration relationship relative to the same calibration sub-board, that is, the above-mentioned second calibration relationship.

[0066] In one embodiment, Figure 4 As shown, the above step S204 may include: the terminal device determines a first rotation and translation matrix between the first camera and the same calibration sub-board according to the same calibration sub-board data corresponding to the first camera, Figure 4 Where X1 is the first camera, R1 is the first rotation and translation matrix, and O X1 、X X1 、Y X1 , Z X1 Construct the coordinate system of the first camera X1; then determine the second rotation and translation matrix between the second camera and the same calibration sub-plate according to the same calibration sub-plate data corresponding to the second camera, Figure 4 X2 is the second camera, R2 is the second rotation and translation matrix, O X2 、X X2 、Y X2 , Z X2Construct the coordinate system of the second camera X2. The terminal device can determine the first rotation and translation matrix and the second rotation and translation matrix according to the Zhang Zhengyou calibration method. Finally, the second calibration relationship is determined based on the first rotation and translation matrix and the second rotation and translation matrix. The formula for determining the second calibration relationship is:

[0067] R 12 =R1 -1 *R2

[0068] Among them, the above R 12 is the second calibration relationship.

[0069] Step S205: Determine the target camera that is closest to the relative position of the camera without overlapping fields of view, wherein the relative position is the center position between the cameras without overlapping fields of view.

[0070] In this embodiment, if there is no overlapping field of view between the first camera and the second camera, the center position between the two is determined, and the terminal device then determines the target camera closest to the center position. The target camera can be set at the end of the robot arm or at any position between the first camera and the second camera that can collect calibration plate data. Figure 5 As shown, there is a target camera X3 between the first camera X1 and the second camera X2. Figure 3 O in X3 、X X3 、Y X3 , Z X3 Construct the coordinate system of the target camera X3.

[0071] Step S206: Determine whether the target camera has overlapping fields of view with the two cameras that do not have overlapping fields of view, wherein the two cameras that do not have overlapping fields of view are the first camera and the second camera.

[0072] If yes, execute step S207; if no, execute step S205.

[0073] In this embodiment, the terminal device can determine the calibration relationship between the camera and the two cameras based on the overlapping fields of view between the target camera and the two cameras, and then obtain the calibration relationship between the two cameras without overlapping fields of view. Therefore, it is necessary to determine whether there is an overlapping field of view between the target camera and the two cameras without overlapping fields of view.

[0074] If there is still no overlapping field of view between the target camera and the two cameras, it is necessary to continue to determine the target camera between the target camera and the cameras with no overlapping field of view, until it is finally determined that there is an overlapping field of view between the target camera and the two cameras with no overlapping field of view.

[0075] Step S207: determining calibration relationships between the target camera and the two cameras, for example, a third calibration relationship between the target camera and the first camera, and a fourth calibration relationship between the target camera and the second camera.

[0076] In this embodiment, if Figure 5 As shown in , if there is an overlapping field of view between the target camera and the two cameras, it is necessary to determine the calibration relationship between the target camera and the two cameras, that is, Figure 5 R in 13 and R 23 , the above R 13 is the third calibration relationship, the above R 23 The calibration relationships between the target camera and the two cameras can be determined according to the method of steps S202 to S204 above, thereby solving the problem of non-overlapping fields of view between cameras by determining the relationship between multiple cameras.

[0077] Step S208: determining a second calibration relationship based on the calibration relationships between the target camera and the two cameras, for example, determining the second calibration relationship based on the third calibration relationship and the fourth calibration relationship.

[0078] In this embodiment, the terminal device determines the second calibration relationship between the two cameras by converting the calibration relationship between the target camera and the two cameras. The formula for determining the second calibration relationship is:

[0079] R 12 =R 13 -1 *R 23

[0080] For example, if there is no overlapping field of view between the first camera X1 and the second camera X2, and there is a target camera X3, and it is determined in step S206 that there is no overlapping field of view between X3 and X1, then the process returns to step S205, i.e., determining the target camera X4 that is closest relative to X3 and X1, and then executing step S206 to determine whether the target camera X4 has overlapping fields of view with both X3 and X1. If it is determined that there is no overlapping field of view between X4 and X1, then the process returns to step S205 until the determined target camera has overlapping fields of view with each camera. Correspondingly, if the target camera X4 has overlapping fields of view with both X3 and X1, then step S207 is executed to determine the calibration relationships between X1 and X4, the calibration relationship between X4 and X3, and the calibration relationship between X3 and X2. Then, based on these calibration relationships, the calibration relationship between X1 and X2 is determined.

[0081] Step S104: determining the relative position relationship between the second robotic arm and the target workpiece according to the first calibration relationship, the second calibration relationship, and the posture information.

[0082] In this embodiment, after the terminal device determines the calibration relationship between each robotic arm, the relationship between the first robotic arm and the robotic arm area to be grasped where the target workpiece is present can be determined according to the second calibration relationship based on the relationship between the first robotic arm and the robotic arm area to be grasped where the target workpiece is present, that is, the above-mentioned first calibration relationship, and then the relative position relationship between the second robotic arm and the target workpiece can be determined according to the posture information; in addition, after the terminal device determines the calibration relationship between each robotic arm, the relative position relationship between the first robotic arm and the target workpiece can also be determined according to the first calibration relationship and the posture information, and then the relative position relationship between the second robotic arm and the target workpiece can be determined according to the second calibration relationship, so that the relative position relationship between each robotic arm and the target workpiece can be obtained by the above two methods, that is, the relative position relationship of each robotic arm relative to the position of the target workpiece in the industrial production area can be obtained, so that in the subsequent industrial production process, each robotic arm can realize the grasping operation of the workpiece at that position through the relative position relationship with that position.

[0083] In one embodiment, since the above-mentioned calibration relationship is a calibration relationship between the ends of the robotic arms, and since the coordinate system based on the automated production process uses the center point of the base of each robotic arm in the industrial production area to represent each robotic arm, in order to facilitate the control work in the automatic production process, it is also necessary to convert it into a calibration relationship between the bases of the robotic arms, that is, to obtain the transformation matrix between the ends of each robotic arm and its base, and then determine the processed second calibration relationship based on the second calibration relationship and the transformation matrix.

[0084] For example, if there are currently three robotic arms, and the second calibration relationships of the two second robotic arms relative to the first robotic arm are R 12 、R 13 , we can obtain the transformation matrices of the three current robot ends relative to their respective bases as P1, P2, and P3, thereby obtaining the second calibration relationship after the two second robot arms are processed:

[0085] B 12 =P1 -1 ·R 12 P2

[0086] B 13 =P1 -1 ·R 13 P3

[0087] Among them, the above B 12 、B 13 These are the second calibration relationships after processing by the two second robotic arms.

[0088] Correspondingly, after the relative position relationship K1 between the first robotic arm and the target workpiece is determined based on the first calibration relationship and the posture information, the relative position relationship between the two second robotic arms and the target workpiece is:

[0089] K2=B 12 -1 K1

[0090] K3=B 13 -1 K1

[0091] Among them, the above-mentioned K2 and K3 are the relative position relationships between the above-mentioned two second robotic arms and the target workpiece respectively.

[0092] In the embodiment of the present application, by obtaining a first calibration relationship between a preset depth camera and a preset first robotic arm, and then using the depth camera to obtain the posture information of the target workpiece, the relative position relationship between the first robotic arm and the target workpiece can be determined according to the first calibration relationship and the posture matrix, and then the second calibration relationship between the preset second robotic arm and the first robotic arm is obtained, thereby determining the relative position relationship between the second robotic arm and the target workpiece according to the first calibration relationship, the second calibration relationship and the posture information, so as to quickly determine the relative position relationship between multiple robotic arms and the target workpiece through the calibration relationship between the robotic arms, thereby improving the calibration efficiency of multiple robotic arms.

[0093] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0094] Corresponding to a calibration method described above, Figure 6 FIG. 1 is a schematic diagram of a structure of a calibration device in an embodiment of the present application. Figure 6 As shown, the calibration device may include:

[0095] The first calibration relationship acquisition module 601 is configured to acquire a first calibration relationship between a preset depth camera and a preset first robotic arm.

[0096] The posture acquisition module 602 is used to acquire the posture information of the target workpiece using a depth camera.

[0097] The second calibration relationship acquisition module 603 is used to acquire a preset second calibration relationship between the second robotic arm and the first robotic arm.

[0098] The relative position relationship determination module 604 is used to determine the relative position relationship between the second robotic arm and the target workpiece according to the first calibration relationship, the second calibration relationship and the posture information.

[0099] In one embodiment, the second calibration relationship acquisition module 603 may include:

[0100] The control submodule is used to control the preset calibration plate to move to the overlapping field of view between the first camera and the second camera when there is an overlapping field of view between the first camera and the second camera, where the first camera is the camera located at the end of the first robotic arm and the second camera is the camera located at the end of the second robotic arm.

[0101] The data acquisition submodule is used to acquire the calibration plate data of the calibration plate collected by the first camera and the second camera respectively.

[0102] The first relationship determination submodule is configured to determine a second calibration relationship based on the same calibration sub-plate data in the calibration plate data of the first camera and the second camera.

[0103] In one embodiment, the relationship determination submodule may include:

[0104] The first matrix determining unit is configured to determine a first rotation and translation matrix between the first camera and the same calibration sub-plate according to the same calibration sub-plate data corresponding to the first camera.

[0105] The second matrix determining unit is configured to determine a second rotation and translation matrix between the second camera and the same calibration sub-plate according to the same calibration sub-plate data corresponding to the second camera.

[0106] The relationship determining unit is used to determine a second calibration relationship according to the first rotation and translation matrix and the second rotation and translation matrix.

[0107] In one embodiment, the control submodule may include:

[0108] The mobile control unit is used to control a preset mobile device carrying a calibration plate to move between the first camera and the second camera.

[0109] The field of view determining unit is configured to determine that the calibration plate is moved to an overlapping field of view between the first camera and the second camera when the calibration plate data respectively acquired by the first camera and the second camera contain the same calibration sub-plate data.

[0110] In one embodiment, the control submodule may further include:

[0111] The position determining unit is used to determine the relative position of the same calibration sub-plate in the calibration plate according to the same calibration sub-plate data and calibration plate data.

[0112] The direction determination unit is used to determine the rotation direction of a preset pan-tilt platform according to the relative position, and the pan-tilt platform is used to fix the calibration plate on the mobile device.

[0113] The control unit is used to control the pan-tilt head to rotate in a rotation direction within a preset rotation range, and to determine, during the rotation process, the pan-tilt head rotation angle when the calibration plate data collected by the first camera and the second camera respectively contain the most identical calibration sub-plate data.

[0114] Accordingly, the above data acquisition submodule may include:

[0115] The data acquisition unit is used to acquire the calibration plate data collected by the first camera and the second camera respectively based on the rotation angle of the pan / tilt platform.

[0116] In one embodiment, the second calibration relationship acquisition module 603 may further include:

[0117] The camera determination submodule is used to determine the target camera closest to the relative position of the camera without overlapping field of view when there is no overlapping field of view between the first camera and the second camera, where the relative position is the center position between the cameras without overlapping field of view. The first camera is the camera located at the end of the first robotic arm, and the second camera is the camera located at the end of the second robotic arm.

[0118] The second relationship determination submodule is used to determine a third calibration relationship between the target camera and the first camera, and a fourth calibration relationship between the target camera and the second camera if there is an overlapping field of view between the target camera and the first camera and between the target camera and the second camera.

[0119] The third relationship determination submodule is configured to determine a second calibration relationship according to the third calibration relationship and the fourth calibration relationship.

[0120] In one embodiment, the first calibration relationship acquisition module 601 may include:

[0121] The fourth relationship determination submodule is used to determine the end calibration relationship between the depth camera and the end of the first robotic arm through the target calibration plate.

[0122] The matrix acquisition submodule is used to obtain the transformation matrix between the end of the first robotic arm and the base of the first robotic arm.

[0123] The fifth relationship determination submodule is used to determine the first calibration relationship according to the terminal calibration relationship and the transformation matrix.

[0124] In the embodiment of the present application, by obtaining a first calibration relationship between a preset depth camera and a preset first robotic arm, and then using the depth camera to obtain the posture information of the target workpiece, the relative position relationship between the first robotic arm and the target workpiece can be determined according to the first calibration relationship and the posture matrix, and then the second calibration relationship between the preset second robotic arm and the first robotic arm is obtained, thereby determining the relative position relationship between the second robotic arm and the target workpiece according to the first calibration relationship, the second calibration relationship and the posture information, so as to quickly determine the relative position relationship between multiple robotic arms and the target workpiece through the calibration relationship between the robotic arms, thereby improving the calibration efficiency of multiple robotic arms.

[0125] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned system embodiments and method embodiments, and will not be repeated here.

[0126] Figure 7 This is a schematic diagram of the structure of the terminal device provided in the embodiment of the present application. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0127] like Figure 7 As shown, the terminal device 7 of this embodiment includes: at least one processor 700 ( Figure 7 Only one is shown), a memory 701 connected to the processor 700, and a computer program 702 stored in the memory 701 and executable on the at least one processor 700, such as a calibration program. When the processor 700 executes the computer program 702, the steps in the above-mentioned calibration method embodiments are implemented, such as Figure 1 Alternatively, when the processor 700 executes the computer program 702, the functions of the modules in the above-mentioned device embodiments are realized, for example, Figure 6 Functions of modules 601 to 604 are shown.

[0128] Exemplarily, the computer program 702 may be divided into one or more modules, which are stored in the memory 701 and executed by the processor 700 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 702 in the terminal device 7. For example, the computer program 702 may be divided into a first calibration relationship acquisition module 601, a posture acquisition module 602, a second calibration relationship acquisition module 603, and a relative position relationship determination module 604. The specific functions of each module are as follows:

[0129] A first calibration relationship acquisition module 601 is configured to acquire a first calibration relationship between a preset depth camera and a preset first robotic arm;

[0130] A posture acquisition module 602 is used to acquire the posture information of the target workpiece using a depth camera;

[0131] A second calibration relationship acquisition module 603 is used to acquire a preset second calibration relationship between the second robotic arm and the first robotic arm;

[0132] The relative position relationship determination module 604 is used to determine the relative position relationship between the second robotic arm and the target workpiece according to the first calibration relationship, the second calibration relationship and the posture information.

[0133] The terminal device 7 may include, but is not limited to, a processor 700 and a memory 701. Those skilled in the art will appreciate that Figure 7 It is only an example of the terminal device 7 and does not constitute a limitation on the terminal device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, buses, etc.

[0134] The processor 700 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0135] In some embodiments, the memory 701 may be an internal storage unit of the terminal device 7, such as a hard disk or memory of the terminal device 7. In other embodiments, the memory 701 may also be an external storage device of the terminal device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 7. Furthermore, the memory 701 may include both an internal storage unit of the terminal device 7 and an external storage device. The memory 701 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory 701 may also be used to temporarily store data that has been output or is about to be output.

[0136] Those skilled in the art will clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here. In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For the parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0137] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For example, the division of the above modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0139] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The above-mentioned computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The above-mentioned computer-readable medium can at least include: any entity or device capable of carrying computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, mobile hard disk, magnetic disk, or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals or telecommunication signals.

[0141] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A calibration method, characterized in that: include: Acquire a first calibration relationship between a preset depth camera and a preset first robotic arm; Acquiring position information of a target workpiece using the depth camera, wherein the position information is position information in three-dimensional space; Obtaining a second calibration relationship between a preset second robotic arm and the first robotic arm, wherein the second calibration relationship is calibrated by cameras installed at an end of each of the second robotic arm and the end of the first robotic arm, and the second robotic arm is multiple; determining a relative positional relationship between each of the second robotic arms and the target workpiece according to the first calibration relationship, the second calibration relationship, and the pose information, wherein the relative positional relationship is used to control the second robotic arm to grasp the target workpiece; The step of obtaining a first calibration relationship between a preset depth camera and a preset first robotic arm includes: Determining an end calibration relationship between the depth camera and the end of the first robotic arm by using a target calibration plate provided at the end of the first robotic arm; Obtaining a transformation matrix between the first robotic arm end and the first robotic arm base; Determining the first calibration relationship according to the terminal calibration relationship and the transformation matrix; The obtaining of a second calibration relationship between the preset second robotic arm and the first robotic arm includes: When there is an overlapping field of view between a first camera and a second camera, controlling a preset calibration plate to move to the overlapping field of view between the first camera and the second camera, the first camera being a camera located at the end of the first robotic arm and the second camera being a camera located at the end of the second robotic arm; Acquire calibration plate data of the calibration plate acquired by the first camera and the second camera respectively; The second calibration relationship is determined according to the same calibration sub-plate data in the calibration plate data of the first camera and the second camera.

2. The calibration method according to claim 1, wherein: The determining the second calibration relationship according to the same calibration sub-plate data in the calibration plate data of the first camera and the second camera includes: Determine a first rotation and translation matrix between the first camera and the same calibration sub-plate according to the same calibration sub-plate data corresponding to the first camera; Determine a second rotation and translation matrix between the second camera and the same calibration sub-plate according to the same calibration sub-plate data corresponding to the second camera; The second calibration relationship is determined according to the first rotation and translation matrix and the second rotation and translation matrix.

3. The calibration method according to claim 1, wherein: The controlling the preset calibration plate to move to the overlapping field of view between the first camera and the second camera includes: Controlling a preset moving device to carry the calibration plate and move between the first camera and the second camera; When the calibration plate data respectively acquired by the first camera and the second camera contain identical calibration sub-plate data, it is determined that the calibration plate is moved to an overlapping field of view between the first camera and the second camera.

4. The calibration method according to claim 3, wherein: Before acquiring the calibration plate data respectively collected by the first camera and the second camera, the method further includes: Determining the relative position of the same calibration sub-plate in the calibration plate according to the same calibration sub-plate data and the calibration plate data; Determining a rotation direction of a preset pan / tilt platform according to the relative position, the pan / tilt platform being used to fix the calibration plate on the mobile device; Controlling the gimbal to rotate in the rotation direction within a preset rotation range, and determining during the rotation process the rotation angle of the gimbal when the calibration plate data collected by the first camera and the second camera respectively contain the most identical calibration sub-plate data; Accordingly, the obtaining of the calibration plate data respectively collected by the first camera and the second camera includes: Based on the rotation angle of the pan / tilt platform, calibration plate data respectively collected by the first camera and the second camera are obtained.

5. The calibration method according to claim 1, wherein: The obtaining of a second calibration relationship between the preset second robotic arm and the first robotic arm includes: When there is no overlapping field of view between the first camera and the second camera, determining a target camera closest to a relative position of the camera without the overlapping field of view, where the relative position is a center position between the cameras without the overlapping field of view, the first camera is a camera located at an end of the first robotic arm, and the second camera is a camera located at an end of the second robotic arm; If there is an overlapping field of view between the target camera and the first camera and between the target camera and the second camera, determining a third calibration relationship between the target camera and the first camera, and a fourth calibration relationship between the target camera and the second camera; The second calibration relationship is determined according to the third calibration relationship and the fourth calibration relationship.

6. A calibration device, characterized in that: include: A first calibration relationship acquisition module, configured to acquire a first calibration relationship between a preset depth camera and a preset first robotic arm; A posture acquisition module is used to acquire the posture information of the target workpiece using the depth camera; a second calibration relationship acquisition module, configured to acquire a second calibration relationship between a preset second robotic arm and the first robotic arm, wherein the second calibration relationship is calibrated by cameras installed at the ends of the second robotic arms and the first robotic arm, and the number of the second robotic arms is multiple; a relative position relationship determination module, configured to determine a relative position relationship between each of the second robotic arms and the target workpiece based on the first calibration relationship, the second calibration relationship, and the posture information, wherein the relative position relationship is used to control the second robotic arm to grasp the target workpiece; Wherein, the first calibration relationship acquisition module includes a fourth relationship determination submodule, a matrix acquisition submodule and a fifth relationship determination submodule, The fourth relationship determination submodule is configured to determine an end calibration relationship between the depth camera and the end of the first robotic arm through a target calibration plate; The matrix acquisition submodule is used to acquire the transformation matrix between the first robotic arm end and the first robotic arm base; The fifth relationship determination submodule is configured to determine a first calibration relationship based on the terminal calibration relationship and the transformation matrix; The second calibration relationship acquisition module includes a control submodule, a data acquisition submodule and a first relationship determination submodule. The control submodule is configured to control a preset calibration plate to move to the overlapping field of view between a first camera and a second camera when there is an overlapping field of view between the first camera and the second camera, wherein the first camera is a camera located at the end of the first robotic arm and the second camera is a camera located at the end of the second robotic arm; The data acquisition submodule is used to acquire the calibration plate data of the calibration plate collected by the first camera and the second camera respectively; The first relationship determination submodule is configured to determine a second calibration relationship based on the same calibration sub-plate data in the calibration plate data of the first camera and the second camera.

7. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of a calibration method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a calibration method according to any one of claims 1 to 5 are implemented.

Images