Robot calibration method and apparatus

Abstract

The embodiment of the present application provides a robot calibration method and device, the method is used for calibrating a robot, and the method comprises the following steps: acquiring a mechanical arm parameter, so as to determine a first conversion relationship between a first mechanical arm base and a first mechanical arm end, and a second conversion relationship between a second mechanical arm base and a second mechanical arm end according to the mechanical arm parameter; acquiring at least three groups of first calibration data, so as to determine a third conversion relationship between the first mechanical arm base and the second mechanical arm base according to the first calibration data, wherein the first calibration data comprises pose data acquired when the first mechanical arm end and the second mechanical arm end coincide in a zero force state; acquiring at least three groups of second calibration data, so as to determine a fourth conversion relationship between the first mechanical arm end and a depth camera according to the second calibration data, wherein the second calibration data comprises first image data of the second mechanical arm end photographed by the depth camera of the first mechanical arm end when the depth camera is stationary.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and more specifically, to a robot calibration method and apparatus. Background Technology

[0002] Robots, such as inspection robots, generally include an autonomous mobile chassis, a robotic arm, and a visual perception module (such as a camera). Each module (or component) works in its corresponding coordinate system. To enable the modules to work together to complete the specified functions and tasks, unifying (calibrating) the coordinate systems of the modules is a necessary prerequisite for the normal operation of the robot system.

[0003] Existing calibration methods typically employ external calibration, such as using other equipment to measure and calibrate the robot. However, this method requires processing by external equipment, making the calibration process very inconvenient. Summary of the Invention

[0004] The embodiments of this application provide a robot calibration method and apparatus, which can more conveniently calibrate robots.

[0005] The technical solution is as follows:

[0006] In a first aspect, this application provides a robot calibration method for calibrating a robot, the robot comprising the following components: a first robotic arm base, a first robotic arm end effector, a second robotic arm base, a second robotic arm end effector, and a depth camera mounted on the first robotic arm end effector. The robot calibration includes determining the transformation relationship between the coordinate systems corresponding to the robot's components. The method includes: acquiring robotic arm parameters to determine a first transformation relationship between the first robotic arm base and the first robotic arm end effector, and a second transformation relationship between the second robotic arm base and the second robotic arm end effector based on the robotic arm parameters; acquiring at least three sets of first calibration data to determine a third transformation relationship between the first robotic arm base and the second robotic arm base based on the first calibration data, the first calibration data including pose data acquired when the first robotic arm end effector and the second robotic arm end effector are in a zero-force state and their ends coincide; acquiring at least three sets of second calibration data to determine a fourth transformation relationship between the first robotic arm end effector and the depth camera based on the second calibration data, the second calibration data including first image data of the second robotic arm end effector captured by the depth camera at the first robotic arm end effector when the depth camera at the first robotic arm end effector is stationary.

[0007] Preferably, the robot further includes a movable chassis, and the method further includes: activating the robot chassis's mobile mapping function to establish an environmental map of the current environment starting from the mapping origin using a depth camera; moving the end effector of the first robotic arm so that the depth camera acquires second image data of fixed points near the mapping origin, wherein at least three fixed points are provided; and determining a fifth transformation relationship between the chassis and the depth camera based on at least three sets of three-dimensional position coordinates and second image data corresponding to at least three fixed points.

[0008] Preferably, determining the fifth transformation relationship between the chassis and the depth camera based on at least three sets of three-dimensional position coordinates corresponding to at least three fixed points and second image data includes: the fixed points are provided with QR codes, and the QR codes carry the three-dimensional position coordinates of the fixed points; the three-dimensional position coordinates of the fixed points relative to the chassis are obtained by recognizing the QR codes; the second image data is recognized to determine the predicted position coordinates of the fixed points relative to the depth camera; and the fifth transformation relationship between the chassis and the depth camera is determined based on at least three sets of three-dimensional position coordinates and predicted position coordinates corresponding to at least three fixed points.

[0009] Preferably, determining the fifth transformation relationship between the chassis and the depth camera based on at least three sets of three-dimensional position coordinates and predicted position coordinates corresponding to at least three fixed points includes: obtaining a first transformation relationship, a second transformation relationship, a third transformation relationship, and a fourth transformation relationship; obtaining at least three sets of three-dimensional position coordinates and predicted position coordinates corresponding to at least three fixed points, and performing transformations and solutions based on the first transformation relationship, the second transformation relationship, the third transformation relationship, and the fourth transformation relationship to determine the fifth transformation relationship between the chassis and the depth camera.

[0010] Preferably, the step of acquiring at least three sets of second calibration data to determine the fourth conversion relationship between the end effector of the first robotic arm and the depth camera based on the second calibration data includes: acquiring first image data of the end effector of the second robotic arm taken when the depth camera at the end effector of the first robotic arm is stationary, and acquiring a first conversion relationship, a second conversion relationship, and a third conversion relationship to form second calibration data; identifying a preset point on the end effector of the second robotic arm based on the first image data in the second calibration data, and determining the predicted coordinate value of the preset point relative to the depth camera; determining the mapped coordinate value of the preset point relative to the end effector of the first robotic arm based on the first conversion relationship, the second conversion relationship, and the third conversion relationship in the second calibration data; and determining the fourth conversion relationship between the end effector of the first robotic arm and the depth camera based on at least three sets of preset coordinate values ​​and mapped coordinate values.

[0011] Preferably, the step of acquiring at least three sets of first calibration data to determine the third conversion relationship between the first robotic arm base and the second robotic arm base based on the first calibration data includes: controlling the end effector of the first robotic arm and the end effector of the second robotic arm to be in a zero-force state, and acquiring the first pose data of the end effector of the first robotic arm and the second pose data of the end effector of the second robotic arm when the end effectors overlap, to determine a set of first calibration data to form at least three sets of first calibration data; converting and solving the first pose data and the second pose data in each set of first calibration data according to the first conversion relationship and the second conversion relationship to determine the third conversion relationship.

[0012] Preferably, the transformation relationship between the coordinate systems corresponding to the robot's components includes translation and rotation matrices.

[0013] Secondly, this application provides a robot calibration device for calibrating a robot. The robot includes the following components: a first robotic arm base, a first robotic arm end effector, a second robotic arm base, a second robotic arm end effector, and a depth camera mounted on the first robotic arm end effector. The robot calibration includes determining the transformation relationship between the coordinate systems corresponding to the robot's components. The device includes a parameter data acquisition module for acquiring robotic arm parameters to determine a first transformation relationship between the first robotic arm base and the first robotic arm end effector, and a second transformation relationship between the second robotic arm base and the second robotic arm end effector based on the robotic arm parameters. The system includes a first data acquisition module for acquiring at least three sets of first calibration data to determine a third conversion relationship between the first robotic arm base and the second robotic arm base based on the first calibration data. The first calibration data includes pose data acquired when the ends of the first and second robotic arms are in a zero-force state and coincident. The second data acquisition module is used to acquire at least three sets of second calibration data to determine a fourth conversion relationship between the end of the first robotic arm and the depth camera based on the second calibration data. The second calibration data includes first image data of the end of the second robotic arm taken by the depth camera at the end of the first robotic arm when it is stationary.

[0014] Thirdly, this application provides a network device, including: a memory, a transceiver, and a processor; wherein the memory is used to store a computer program; the transceiver is used to send and receive data under the control of the processor; and the processor is used to read the computer program in the memory and execute the method as described in the first aspect.

[0015] Fourthly, this application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0016] The beneficial effects of the technical solution provided in this application are:

[0017] The solution proposed in this application can be applied to robot calibration scenarios. The robot includes the following components: a base, a first robotic arm base, a first robotic arm end effector, a second robotic arm base, a second robotic arm end effector, and a depth camera mounted on the first robotic arm end effector. The first and second robotic arm bases are mounted on a movable base, and each component resides in a different coordinate system. The robot calibration proposed in this application includes determining the transformation relationship between the coordinate systems corresponding to the robot's components. This solution can complete the robot calibration process using the robot itself. It can calibrate the robotic arm base and the robotic arm end effector based on the robotic arm parameters, calibrate the two robotic arm end effectors based on the data when the two end effectors overlap, calibrate the depth camera by identifying a preset point on the other robotic arm end effector using the depth camera on one end effector, and calibrate the chassis by identifying fixed points on the chassis using the depth camera. This allows for the determination of the transformation relationship between any two components among the chassis, the first robotic arm base, the first robotic arm end effector, the second robotic arm base, the second robotic arm end effector, and the depth camera, thus completing the robot calibration. This solution requires no external equipment and can directly use the robot itself to complete the calibration process, making it simpler and more convenient.

[0018] Specifically, this solution can acquire robotic arm parameters to determine the first conversion relationship between the first robotic arm base and the first robotic arm end effector, and the second conversion relationship between the second robotic arm base and the second robotic arm end effector. After the robotic arm base and the robotic arm end effector are calibrated, the first robotic arm end effector and the second robotic arm end effector can be controlled to be manually adjusted (or adjusted by other means) in a zero-force state so that the first robotic arm end effector and the second robotic arm end effector coincide. The pose data of the two robotic arms are acquired as the first calibration data. Three sets of the first calibration data can be acquired, and then the third conversion relationship between the first robotic arm base and the second robotic arm base can be analyzed. After calibrating the end effector of the first robotic arm and the end effector of the second robotic arm, the depth camera at the end effector of the first robotic arm can be kept stationary, and the first image data of the preset point of the end effector of the second robotic arm can be captured as the second calibration data. The end effector of the second robotic arm can be moved to multiple positions to obtain at least three sets of calibration data. Then, the fourth conversion relationship between the end effector of the first robotic arm and the depth camera is determined, and the calibration of the first robotic arm base, the first robotic arm end effector, the second robotic arm base, the second robotic arm end effector and the depth camera is completed. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0020] Figure 1This is a schematic diagram of the structure of a robot according to an embodiment of this application;

[0021] Figure 2 This is a schematic diagram illustrating the steps of a robot calibration method according to an embodiment of this application;

[0022] Figure 3 This is a flowchart illustrating a singular value decomposition algorithm according to an embodiment of this application;

[0023] Figure 4 This is a flowchart illustrating a robot calibration method according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of the structure of a robot calibration device according to an embodiment of this application;

[0025] Figure 6 This is a structural block diagram of a network device according to an embodiment of this application;

[0026] Figure 7 This is a structural block diagram of a user equipment according to one embodiment of this application; Detailed Implementation

[0027] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals identify the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0028] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms, while “a plurality” refers to two or more, and other quantifiers are similarly understood. It should be further understood that the word “comprising” as used in this application’s specification means the presence of the stated feature, integer, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The word “and / or” as used herein describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0029] The solution proposed in this application can be applied to scenarios involving robot calibration. The calibration process can be completed using the robot itself. The robot includes: a chassis, a base for a first robotic arm (or robotic arm 1), an end effector for the first robotic arm, a base for a second robotic arm (or robotic arm 2), an end effector for the second robotic arm, and a depth camera mounted on the end effector of the first robotic arm. Robot calibration involves determining the transformation relationships between the coordinate systems corresponding to the robot's components. For the robot, the coordinate system relationships between its various components (or modules) are as follows: Figure 1 As shown, the calibration of the inspection robot involves determining the coordinate system O of the mobile chassis center. w -X w Y w Z w Robotic arm 1 base coordinate system O ab1 -X ab1 Y ab1 Z ab1 Robotic arm 2 base coordinate system O ab2 -X ab2 Y ab2 Z ab2 Robotic arm 1 end coordinate system O ae1 -X ae1 Y ae1 Z ae1 The end coordinate system O of the robotic arm 2 ae2 -X ae2 Y ae2 Z ae2 and camera coordinate system O c -X c Y c Z c The transformation relationship between the six coordinate systems is to solve for the coordinate rotation matrix R and translation matrix T between each coordinate system.

[0030] This solution calibrates the robot arm base and end effector based on robot arm parameters, calibrates the two end effectors based on data when they overlap, calibrates the depth camera by identifying preset points on the other end effector using a depth camera on one end effector, and calibrates the chassis by identifying fixed points on the chassis using the depth camera. This determines the conversion relationship between any two components among the chassis, first robot arm base, first robot arm end effector, second robot arm base, second robot arm end effector, and depth camera, thus completing the robot calibration. This solution requires no external equipment and can directly use the robot itself to complete the calibration process, making it simpler and more convenient.

[0031] Specifically, this solution can acquire robotic arm parameters to determine the first conversion relationship between the first robotic arm base and the first robotic arm end effector, and the second conversion relationship between the second robotic arm base and the second robotic arm end effector. After the robotic arm base and the robotic arm end effector are calibrated, the first robotic arm end effector and the second robotic arm end effector can be controlled to be in a zero-force state. The first robotic arm end effector and the second robotic arm end effector can be manually adjusted to make the first robotic arm end effector and the second robotic arm end effector coincide, and the pose data can be acquired as the first calibration data. Three sets of the first calibration data can be acquired, and then the third conversion relationship between the first robotic arm base and the second robotic arm base can be analyzed. After calibrating the end effectors of the first and second robotic arms, the depth camera at the end effector of the first robotic arm can be kept stationary, and first image data of preset points at the end effector of the second robotic arm can be captured as second calibration data. The end effector of the second robotic arm can be moved to multiple positions to obtain at least three sets of calibration data. Then, a fourth conversion relationship between the end effector of the first robotic arm and the depth camera is determined, completing the calibration of the first robotic arm base, the first robotic arm end effector, the second robotic arm base, the second robotic arm end effector, and the depth camera. Afterward, multiple fixed points on the chassis can be identified using the depth camera, thereby calibrating the chassis.

[0032] Specifically, in an optional example, the robot in this solution can be an inspection robot, or an intelligent inspection robot. In the dual-arm calibration system, based on the arm parameters (such as DH parameters) of the used arm, the rotation and translation matrices between the arm's base coordinate system and end effector coordinate system can be obtained, thereby determining the rotation matrix for transforming the arm's end effector coordinates to the arm's base coordinate system. ab1 R ae1 , ab2 R ae2 Translation matrix ab1 T ae1 , ab2 T ae2 ,Right now ab1 R ae1 , ab2 R ae2 , ab1 T ae1 , ab2 T ae2 It is known that we only need to determine the rotation matrix that transforms the coordinates of robot arm 1 base to the coordinates of robot arm 2 base. ab2 R ab1 Translation matrix ab2 T ab1 This completes the calibration of the dual robotic arms. During calibration, both robotic arms are placed in a zero-force teaching state. The two robotic arms are dragged until their ends are absolutely aligned. The pose data of the ends of the two robotic arms at the point of alignment are then read to obtain a set of corresponding point coordinates X in the coordinate systems of robotic arms 1 and 2.i ab1 X i ab2 Repeat this process at least three times to obtain the coordinates of three or more corresponding points. Then, use the coordinate transformation relationship: X i ab2 = ab2 R ab1 X i ab1 + ab2 T ab1 Construct a system of equations and solve for the rotation matrix using the SVD method. ab2 R ab1 Translation matrix ab2 T ab1 Among them, X i ab2 X represents the coordinates of the i-th corresponding point in the coordinate system of the robot arm 2 base. i ab1 This represents the coordinates of the i-th corresponding point in the coordinate system of robot arm 1's base. Due to sensor measurement errors during calibration, the rotation and translation matrices obtained using only three corresponding points contain certain errors. Increasing the number of corresponding points can reduce the estimation error of the coordinate transformation matrix.

[0033] In the extrinsic parameter calibration system for the depth camera, the dual-arm calibration has been completed, specifically the rotation matrix that transforms the coordinates of the robotic arm's end effector to the coordinates of its base. ab1 R ae1 , ab2 R ae2 Translation matrix ab1 T ae1 , ab2 T ae2 Given the rotation matrix for transforming the coordinates of robot arm 1 base to the coordinates of robot arm 2 base. ab2 R ab1 Translation matrix ab2 T ab1 Given that the extrinsic parameter calibration of the depth camera determines the coordinate system O of the end effector of robotic arm 1. ae1 -X ae1 Y ae1 Z ae1 Coordinate system O of the depth camera c -X c Y c Z c Transformation matrix ae1 R c , ae1 T cDuring calibration, robotic arm 1, equipped with a depth camera, remains stationary, while robotic arm 2 is moved so that its end effector is within the optimal depth measurement range of the depth camera. Using the RGB image and depth point cloud image acquired by the depth camera, the center of the robotic arm's end effector relative to the depth camera's coordinate system O is identified and acquired. c -X c Y c Z c coordinates X i c Meanwhile, the end effector of robotic arm 2 is located in the coordinate system O of the base of robotic arm 2. ab2 -X ab2 Y ab2 Z ab2 The coordinates X below i ab2 The end effector of robotic arm 1 is located in the coordinate system O of the base of robotic arm 1. ab1 -X ab1 Y ab1 Z ab1 The coordinates X below i ab1 Determine the transformation matrix from the base coordinate system to the end effector coordinate system of robotic arm 1. Transform the coordinate values ​​X... i ab2 Through X i ab1 = ab1 R ab2 X i ab2 + ab1 T ab2 Projected onto the coordinate system of robot arm 1 base, and then through X i ae1 = ae1 R ab1 X i ab1 + ae1 T ab1 X i ab1 Project the coordinates onto the end effector coordinate system of robotic arm 1. At this point, a set of corresponding point coordinates between the depth camera coordinate system and the end effector coordinate system of robotic arm 1 is obtained. Repeat the above operation to obtain three or more corresponding point coordinates, and then use X... i ae1 = ae1 R c X i c + ae1 T c By constructing a system of linear equations and then using the SVD method, the transformation matrix between the depth camera coordinate system and the end effector coordinate system of robotic arm 1 can be solved. ae1 R c and ae1 Tc Complete the extrinsic parameter calibration of the depth camera.

[0034] In the overall calibration system, the extrinsic parameters of the dual robotic arms and the depth camera have been calibrated. The next step is to determine the coordinate transformation matrices between the robot chassis base coordinate system and the coordinate systems of robotic arms 1, 2, and the depth camera. Since the coordinate transformation relationships between the depth camera, robotic arm 1, and robotic arm 2 have already been determined, the robot chassis coordinate system only needs to determine its transformation relationship with the coordinate system of one of the three modules. This approach prioritizes determining the depth camera coordinate system O. c -X c Y c Z c and the robot's mobile chassis base coordinate system O w -X w Y w Z w The coordinate transformation relationship is determined. During calibration, the mapping function of the robot's mobile chassis is first activated to complete the mapping of the robot's current environment. Near the origin of the robot's mobile chassis mapping, three or more location points are marked, and a QR code recording the robot's mobile chassis position is placed at the corresponding marked point. After mapping is completed, the robot is moved to the origin of the map (or environment map). The robotic arm 1 equipped with a depth camera is moved so that the QR code is within the optimal depth field of view of the depth camera. Using the RGB image and depth image captured by the depth camera, the robot's mobile chassis position coordinates X in the QR code are identified and read. i w The three-dimensional position coordinates (X) of the QR code relative to the camera i c At this point, a set of coordinate correspondences between the mobile chassis base coordinate system and the depth camera coordinate system are obtained. The position information and coordinates of other QR codes are then identified and read again to obtain three or more corresponding point coordinates. These coordinates are then used via X... i w = w R c X i c + w T c By constructing a system of linear equations and then using the SVD method, the transformation matrix between the robot's mobile chassis base coordinate system and the depth camera coordinate system can be solved. ae1 R c and ae1 T c .

[0035] In a more specific example, such as Figure 2As shown, the specific technical implementation scheme for the calibration of the intelligent inspection robot system is divided into three parts: calibration of the dual robotic arm system, hand-eye calibration of the robotic arm and depth camera, and joint calibration of the robot's mobile chassis with the dual robotic arms and depth camera.

[0036] For the calibration of dual robotic arms in an inspection robot system, the main issue is the coordinate system O. ab2 -X ab2 Y ab2 Z ab2 To coordinate system O ab1 -X ab1 Y ab1 Z ab1 Transformation matrix ab1 R ab2 and ab1 T ab2 The problem is to solve the following: The ends of robotic arms 1 and 2 are manually aligned using zero-force teaching, and the end coordinates (x, y, y) of robotic arms 1 and 2 are collected respectively. a y a , z a The coordinates of robotic arm 1 are stored in matrix A. a In the middle, the coordinates of robotic arm 2 are stored in matrix B. a In the middle. After repeated trials, A was obtained. a n×3 and B a n×3 Given two matrices, n≥3, the transformation matrix can be solved using the Singular Value Decomposition (SVD) algorithm. ab1 R ab2 and ab1 T ab2 The algorithm flowchart is as follows Figure 3 As shown.

[0037] For the calibration of robotic arm 1 and depth camera, the main issue is solving the camera coordinate system O. c -X c Y c Z c To the end coordinate system O of robotic arm 1 ae1 -X ae1 Y ab1 Z ae1 Transformation matrix ae1 R c , ae1 T c The problem involves solving the coordinate transformation matrix using the PnP method. The key is to determine at least three corresponding points in coordinate system O. c -X c Y c Z c and Oae1 -X ae1 Y ab1 Z ae1 The coordinate values ​​are taken as O. This scheme uses the origin of the coordinate system at the end of the robotic arm 2 (without a camera) as O. c -X c Y c Z c Coordinate system and O ae1 -X ae1 Y ab1 Z ae1 Corresponding points in two coordinate systems. The following section will detail how to calculate the coordinates of corresponding points in each coordinate system.

[0038] First, determine the coordinate origin of the end effector of robotic arm 2 in the coordinate system O of the end effector of robotic arm 1. ae1 -X ae1 Y ab1 Z ae1 Let the coordinate values ​​be the coordinates below. Let the homogeneous transformation matrix from the base coordinate system of robot arm 1 to the end-effector coordinate system of robot arm 1 be... The homogeneous transformation matrix from the coordinate system of robot arm 2 to the coordinate system of robot arm 1 is: The homogeneous transformation matrix from the end-effector coordinate system of robotic arm 2 to the base coordinate system of robotic arm 2 is: The homogeneous transformation matrix from the end-effector coordinate system of robotic arm 2 to the end-effector coordinate system of robotic arm 1 can be obtained. for:

[0039]

[0040] in,

[0041]

[0042]

[0043]

[0044] ab1 R ae1 , ab1 T ae1 It is obtained from DH of robotic arm 1. ab1 R ab2 , ab1 T ab2 The result is obtained from the calibration in the first step. ab2 R ae2 , ab2 R ae2 It is obtained from the DH parameters of robotic arm 2.

[0045] Based on the homogeneous transformation matrix have:

[0046]

[0047] in, ae1 T ae2 This refers to the coordinates of the origin of the end-effector coordinate system of robotic arm 2 in the coordinate system of the origin of the end-effector coordinate system of robotic arm 1.

[0048] Secondly, the coordinates of the origin of the robotic arm's end effector in the camera coordinate system are determined. The origin of the coordinate system at the robotic arm's end effector is typically the center of the circular surface at the end. A deep learning network model is constructed based on a convolutional neural network, and images of the robotic arm's end effector are acquired under different lighting and background conditions. Rectangular bounding boxes are used to mark the circular end effector in the images. In the images, the circular end effector generally appears as either a circle or an ellipse, and the center of the marked rectangular bounding box is the intersection of the center of the circle and the major and minor axes of the ellipse. The acquired images are trained using the constructed deep learning network model to obtain the weights of the convolutional neural network. Based on the deep learning model and the trained weight parameters, the robotic arm's end effector is detected online. Based on the center of the rectangular bounding box in the detection result, the pixel value (u) of the center of the robotic arm's end effector in the image is obtained. x u y Based on the camera's pinhole imaging model and the camera's intrinsic parameters, we have:

[0049]

[0050] in, It is the depth value at the pixel (ux, uy) where the origin of the coordinate system of the end of the robotic arm 2 is located under the i-th observation at a different position. It is also the z-coordinate value of the origin of the coordinate system of the end of the robotic arm 2 in the camera coordinate system. This value can be obtained directly by the depth camera. and These are the x and y coordinates of the origin of the coordinate system at the end of the robotic arm 2 in the camera coordinate system, respectively, under the i-th observation at a different position. u0, v0, d x d y f and f are both known camera intrinsic parameters.

[0051] At this point, the coordinates of the specified corresponding point in both the camera coordinate system and the end effector coordinate system of robotic arm 1 can be determined. Through multiple observations from different positions, matrix A, consisting of the coordinates of the corresponding point in the camera coordinate system, can be obtained. c n×3 The matrix B formed by the coordinates of the corresponding points in the coordinate system of the robotic arm's end effector. ae n×3 The camera coordinate system O can be solved using the SVD algorithm. c -X c Y c Z c To the end coordinate system O of robotic arm 1 ae1 -X ae1 Yab1 Z ae1 Transformation matrix ae1 R c , ae1 T c Complete the external parameter calibration of the camera.

[0052] Based on the solution ae1 R c , ae1 T c By defining the calibration error through reprojection, let the coordinates of the origin of the end effector coordinate system of robotic arm 2 in the camera coordinate system and the end effector coordinate system of robotic arm 1 in a new observation be respectively... and Then we have:

[0053]

[0054] By setting the calibration error threshold err threshold Under certain conditions, automatic online calibration of this part can be achieved.

[0055] Since the extrinsic parameter calibration of the depth camera is performed based on the calibration using two robotic arms, and the transformation matrix between the two arms is estimated using linear least squares, the calibration process introduces positioning errors of the robotic arms and transformation matrix estimation errors, which increases the overall error of the depth camera extrinsic parameter estimation. Therefore, a linear method is used to determine the transformation matrix. ae1 R c , ae1 T c Afterwards, ae1 R c , ae1 T c As the initial value, Equation (1.4) is used as the objective function. Using the obtained corresponding point coordinates, the transformation matrix is ​​optimized again nonlinearly by the Gauss-Newton method to reduce the positioning error of the robotic arm, the estimation error of the transformation matrix between the two robotic arms, and the influence of camera depth observation noise on camera extrinsic parameter estimation, thereby improving the accuracy of camera extrinsic parameter estimation.

[0056] For the calibration of the mobile chassis, robotic arm, and depth camera, only the calibration of the mobile chassis and depth camera is needed to determine the coordinate transformation relationship of the mobile chassis relative to robotic arms 1 and 2. In this part of the calibration, a QR code label is introduced as the corresponding point between the mobile chassis base coordinate system and the camera coordinate system. The QR code label contains the coordinate value of the QR code label center relative to the mobile robot chassis base coordinate system. The solution for the coordinates of the QR code center relative to the camera coordinate system is the same as the solution for the origin of the coordinate system of the end of robotic arm 1 relative to the camera coordinate system. First, images of the QR code are collected from different angles, under different lighting conditions, and under different backgrounds. The QR codes in the images are then marked, and the convolutional neural network weights of the QR code are trained using a deep learning model. Then, during the calibration process, the content of the QR code is read in real time, and the center of the QR code is detected. The coordinates X of the QR code center in the camera coordinate system are solved using equation (1.3). i c Since the relative positions of the bases of robotic arms 1 and 2 and the centers of the movable chassis remain constant, the solved coordinate values ​​are projected onto the coordinate system of the base of robotic arm 1. This yields the coordinates of the corresponding points.

[0057]

[0058] After obtaining the coordinates of three or more corresponding points, the transformation matrix between the mobile robot chassis coordinate system and the robot arm 1 base coordinate system can be solved using the SVD algorithm. w R ab1 , w T ab1 .

[0059] Again ae1 R c , ae1 T c Using the initial values ​​and equation (1.4) as the objective function, the transformation matrix is ​​then modified again using the obtained coordinates of the corresponding points through the Gauss-Newton method. w R ab1 , w T ab1 Nonlinear optimization is performed to reduce the impact of robot positioning errors, transformation matrix estimation errors between the two robots, camera depth observation noise, and mobile robot positioning errors on the transformation matrices of the mobile robot chassis coordinate system and the robot arm 1 base coordinate system, thereby improving the accuracy of camera extrinsic parameter estimation. This scheme optimizes the transformation matrix by minimizing the reprojection error. Besides the Gauss-Newton method, gradient descent, Levenberg-Marquardt, and other methods can also be used.

[0060] The homogeneous form of the transformation matrix from the base coordinate system of robotic arm 1 to the mobile chassis coordinate system is: The homogeneous transformation matrix from the end-effector coordinate system of robotic arm 1 to the base coordinate system of the mobile chassis can be obtained. for:

[0061]

[0062] The homogeneous transformation matrix from the camera coordinate system to the mobile chassis base coordinate system can be obtained. for:

[0063]

[0064] The homogeneous transformation matrix from the end-effector coordinate system of robotic arm 2 to the mobile chassis coordinate system can be obtained. for

[0065]

[0066] In this scheme, besides calibrating the dual-arm robotic system, the transformation matrix estimated by the linear method is used as the initial value. By minimizing the reprojection error, the optimal transformation matrix under certain measurement noise and calibration error is estimated. This scheme eliminates the need for a calibration target and external systems and calibration boards to calibrate the extrinsic parameters of the dual-arm robotic system and depth camera. The calibration process is simple and requires no complex manual operations. This scheme can be applied to the calibration of multi-functional robotic systems, providing a quick and easy calibration solution for complex multi-functional inspection robots. Furthermore, this scheme requires fewer observations, utilizes multiple calibration systems in conjunction, and improves the efficiency of nonlinear optimization through linear and nonlinear estimation of the rotation matrix, reducing the impact of sensor measurement noise and process estimation errors on the accuracy of the final transformation matrix estimation.

[0067] Specifically, this application provides a robot calibration method for calibrating a robot. The robot includes the following components: a first robotic arm base, a first robotic arm end effector, a second robotic arm base, a second robotic arm end effector, and a depth camera mounted on the first robotic arm end effector. The robot calibration includes determining the transformation relationships between the coordinate systems corresponding to the robot's components. These transformation relationships include translation and rotation matrices. Figure 4 As shown, the method includes:

[0068] Step 402: Obtain the robot arm parameters to determine the first conversion relationship between the first robot arm base and the first robot arm end effector, and the second conversion relationship between the second robot arm base and the second robot arm end effector based on the robot arm parameters.

[0069] Step 404: Obtain at least three sets of first calibration data to determine the third conversion relationship between the first robotic arm base and the second robotic arm base based on the first calibration data. The first calibration data includes the pose data obtained when the ends of the first robotic arm and the ends of the second robotic arm are in a zero-force state and coincide.

[0070] Step 406: Obtain at least three sets of second calibration data to determine the fourth conversion relationship between the end effector of the first robotic arm and the depth camera based on the second calibration data. The second calibration data includes first image data of the end effector of the second robotic arm taken when the depth camera at the end effector of the first robotic arm is stationary.

[0071] The solution proposed in this application can be applied to scenarios where robots are calibrated. The robot includes the following components: a base, a first robotic arm base, a first robotic arm end effector, a second robotic arm base, a second robotic arm end effector, and a depth camera mounted on the first robotic arm end effector. The first robotic arm base and the second robotic arm base are mounted on a movable base. The coordinate systems of each component are different. The calibration of the robot in this solution includes determining the transformation relationship between the coordinate systems corresponding to the robot's components.

[0072] This solution allows the robot to complete the calibration process itself. It calibrates the robot arm base and end effector based on robot arm parameters, calibrates the two end effectors based on data when they overlap, calibrates the depth camera by identifying preset points on the other end effector using a depth camera on one end effector, and calibrates the chassis by identifying fixed points on the chassis using the depth camera. This establishes the conversion relationships between any two components among the chassis, first robot arm base, first robot arm end effector, second robot arm base, second robot arm end effector, and depth camera, thus completing the robot calibration. This solution requires no external equipment and can directly use the robot itself for calibration, making the process simpler and more convenient.

[0073] Specifically, this solution can acquire robotic arm parameters to determine the first conversion relationship between the first robotic arm base and the first robotic arm end effector, and the second conversion relationship between the second robotic arm base and the second robotic arm end effector. After the robotic arm base and the robotic arm end effector are calibrated, the first and second robotic arm end effectors can be controlled to coincide under zero force by manual adjustment, so that the first and second robotic arm end effectors coincide. The pose data (parameters of the robotic arm base and the robotic arm end effector) is acquired as the first calibration data. Three sets of the first calibration data can be acquired, and then the third conversion relationship between the first and second robotic arm bases can be analyzed. After calibrating the end effector of the first robotic arm and the end effector of the second robotic arm, the depth camera at the end effector of the first robotic arm can be kept stationary, and the first image data of the preset point of the end effector of the second robotic arm can be captured as the second calibration data. The end effector of the second robotic arm can be moved to multiple positions to obtain at least three sets of calibration data. Then, the fourth conversion relationship between the end effector of the first robotic arm and the depth camera is determined, and the calibration of the first robotic arm base, the first robotic arm end effector, the second robotic arm base, the second robotic arm end effector and the depth camera is completed.

[0074] This solution can calibrate not only the robotic arm base, robotic arm end effector, and depth camera, but also the movable chassis. The depth camera can be used to identify fixed points on the chassis to determine the conversion relationship between the chassis and the depth camera. Specifically, as an optional embodiment, the robot also includes a movable chassis. The method further includes: activating the robot chassis's mobile mapping function to create an environment map of the current environment starting from the mapping origin using the depth camera; moving the first robotic arm end effector so that the depth camera acquires second image data of fixed points near the mapping origin, where at least three fixed points are set; and determining a fifth conversion relationship between the chassis and the depth camera based on at least three sets of three-dimensional position coordinates and the second image data corresponding to the at least three fixed points. A QR code can be set at a fixed point on the chassis, where the fixed point can be the center of the QR code. The QR code can carry the three-dimensional position coordinates of the fixed point relative to the chassis. The three-dimensional position coordinates can be identified by recognizing the QR code, and the predicted position coordinates of the fixed point relative to the depth camera can be determined by depth recognition. Then, the first conversion relationship can be determined using the calibrated conversion relationship. Specifically, as an optional embodiment, determining the fifth transformation relationship between the chassis and the depth camera based on at least three sets of three-dimensional position coordinates corresponding to at least three fixed points and second image data includes: the fixed points are provided with QR codes, and the QR codes carry the three-dimensional position coordinates of the fixed points; by recognizing the QR codes, the three-dimensional position coordinates of the fixed points relative to the chassis are obtained; the second image data is recognized to determine the predicted position coordinates of the fixed points relative to the depth camera; and the fifth transformation relationship between the chassis and the depth camera is determined based on at least three sets of three-dimensional position coordinates and predicted position coordinates corresponding to at least three fixed points.

[0075] The first, second, third, and fourth transformation relationships can be used to transform the three-dimensional position coordinates and the predicted position coordinates, and then solve for and determine the fifth transformation relationship. Specifically, as an optional embodiment, determining the fifth transformation relationship between the chassis and the depth camera based on at least three sets of three-dimensional position coordinates and predicted position coordinates corresponding to at least three fixed points includes: obtaining the first, second, third, and fourth transformation relationships; obtaining at least three sets of three-dimensional position coordinates and predicted position coordinates corresponding to at least three fixed points, and transforming and solving for them based on the first, second, third, and fourth transformation relationships to determine the fifth transformation relationship between the chassis and the depth camera.

[0076] For the calibration of the depth camera, the conversion relationship between the depth camera and the end effector of the first robotic arm can be determined, thereby deriving the conversion relationship between the depth camera and other components. Specifically, as an optional embodiment, acquiring at least three sets of second calibration data to determine the fourth conversion relationship between the end effector of the first robotic arm and the depth camera based on the second calibration data includes: acquiring first image data of the end effector of the second robotic arm taken when the depth camera at the end effector of the first robotic arm is stationary, and acquiring the first conversion relationship, the second conversion relationship, and the third conversion relationship to form the second calibration data; identifying a preset point on the end effector of the second robotic arm based on the first image data in the second calibration data, and determining the predicted coordinate value of the preset point relative to the depth camera; determining the mapped coordinate value of the preset point relative to the end effector of the first robotic arm based on the first conversion relationship, the second conversion relationship, and the third conversion relationship in the second calibration data; and determining the fourth conversion relationship between the end effector of the first robotic arm and the depth camera based on at least three sets of preset coordinate values ​​and mapped coordinate values.

[0077] For the calibration of two robotic arm bases, the end effectors of the two robotic arms can be made to coincide, thereby using pose data for analysis to determine the transformation relationship between the robotic arm bases. Specifically, as an optional embodiment, acquiring at least three sets of first calibration data to determine the third transformation relationship between the first and second robotic arm bases based on the first calibration data includes: controlling the end effectors of the first and second robotic arms to be in a zero-force state, and acquiring the first pose data of the first robotic arm end effector and the second pose data of the second robotic arm end effector when the end effectors coincide, to determine one set of first calibration data, thus forming at least three sets of first calibration data; transforming and solving the first pose data and second pose data in each set of first calibration data according to the first and second transformation relationships to determine the third transformation relationship. When the end effectors of the two robotic arms coincide, the positions of the end effectors of the robotic arms ultimately determined by the pose data should be corresponding (only identical after mapping through a coordinate transformation matrix). Therefore, the positions of the end effectors of the robotic arms can be determined through the first pose data and the second pose data, thereby calibrating the end effectors of the two robotic arms.

[0078] Specifically, based on the above embodiments, this application also provides a robot calibration device. The device is used to calibrate a robot, which includes the following components: a first robotic arm base, a first robotic arm end effector, a second robotic arm base, a second robotic arm end effector, and a depth camera mounted on the first robotic arm end effector. The robot calibration includes determining the transformation relationships between the coordinate systems corresponding to the robot's components, such as... Figure 5 As shown, the device includes:

[0079] The parameter data acquisition module 502 is used to acquire the parameters of the robotic arm, so as to determine the first conversion relationship between the first robotic arm base and the first robotic arm end and the second conversion relationship between the second robotic arm base and the second robotic arm end based on the parameters of the robotic arm.

[0080] The first data acquisition module 504 is used to acquire at least three sets of first calibration data to determine the third conversion relationship between the first robotic arm base and the second robotic arm base based on the first calibration data. The first calibration data includes the pose data acquired when the ends of the first robotic arm and the ends of the second robotic arm are in a zero-force state and the ends overlap.

[0081] The second data acquisition module 506 is used to acquire at least three sets of second calibration data to determine the fourth conversion relationship between the end effector of the first robotic arm and the depth camera based on the second calibration data. The second calibration data includes first image data of the end effector of the second robotic arm taken when the depth camera at the end effector of the first robotic arm is stationary.

[0082] The implementation methods of this application are similar to those of the above method embodiments. For specific implementation methods, please refer to the specific implementation methods of the above method embodiments, which will not be repeated here.

[0083] The solution proposed in this application can be applied to scenarios where robots are calibrated. The robot includes the following components: a base, a first robotic arm base, a first robotic arm end effector, a second robotic arm base, a second robotic arm end effector, and a depth camera mounted on the first robotic arm end effector. The first robotic arm base and the second robotic arm base are mounted on a movable base. The coordinate systems of each component are different. The calibration of the robot in this solution includes determining the transformation relationship between the coordinate systems corresponding to the robot's components.

[0084] This solution allows the robot to complete the calibration process itself. It calibrates the robot arm base and end effector based on robot arm parameters, calibrates the two end effectors based on data when they overlap, calibrates the depth camera by identifying preset points on the other end effector using a depth camera on one end effector, and calibrates the chassis by identifying fixed points on the chassis using the depth camera. This establishes the conversion relationships between any two components among the chassis, first robot arm base, first robot arm end effector, second robot arm base, second robot arm end effector, and depth camera, thus completing the robot calibration. This solution requires no external equipment and can directly use the robot itself for calibration, making the process simpler and more convenient.

[0085] Specifically, this solution can acquire robotic arm parameters to determine the first conversion relationship between the first robotic arm base and the first robotic arm end effector, and the second conversion relationship between the second robotic arm base and the second robotic arm end effector. After the robotic arm base and the robotic arm end effector are calibrated, the first robotic arm end effector and the second robotic arm end effector can be controlled to be in a zero-force state. The first robotic arm end effector and the second robotic arm end effector can be manually adjusted to make the first robotic arm end effector and the second robotic arm end effector coincide, and the pose data can be acquired as the first calibration data. Three sets of the first calibration data can be acquired, and then the third conversion relationship between the first robotic arm base and the second robotic arm base can be analyzed. After calibrating the end effector of the first robotic arm and the end effector of the second robotic arm, the depth camera at the end effector of the first robotic arm can be kept stationary, and the first image data of the preset point of the end effector of the second robotic arm can be captured as the second calibration data. The end effector of the second robotic arm can be moved to multiple positions to obtain at least three sets of calibration data. Then, the fourth conversion relationship between the end effector of the first robotic arm and the depth camera is determined, and the calibration of the first robotic arm base, the first robotic arm end effector, the second robotic arm base, the second robotic arm end effector and the depth camera is completed.

[0086] It should be noted that the division of units and / or modules in the embodiments of this application is illustrative and only represents a logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional units and / or modules in the various embodiments of this application can be integrated into one processing unit and / or module, or each unit and / or module can exist physically separately, or two or more units and / or modules can be integrated into one unit and / or module. The integrated units and / or modules described above can be implemented in hardware or as software functional units and / or modules.

[0087] If the integrated units and / or modules are implemented as software functional units and / or modules and sold or used as independent products, they can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] Furthermore, the data transmission apparatus and data transmission method provided in the above embodiments are based on the same application concept. Since the methods and apparatus solve problems in similar principles, the implementation of the apparatus and methods can refer to each other, and repeated parts will not be described again.

[0089] Figure 6 A structural block diagram of a network device is shown according to an exemplary embodiment.

[0090] like Figure 6 As shown, the network device 1100 includes at least: a processor 1110, a memory 1120, and a transceiver 1130.

[0091] The transceiver 1130 is used to receive and send data under the control of the processor 1110.

[0092] exist Figure 6 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 1110 and memory represented by memory 1120 together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 1130 can be multiple elements, including transmitters and receivers, providing units and / or modules for communicating with various other devices over transmission media, including wireless channels, wired channels, optical fibers, and other transmission media.

[0093] The processor 1110 is responsible for managing the bus architecture and general processing, and the memory 1120 can store the data used by the processor 1110 when performing operations.

[0094] Optionally, the processor 1110 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD). The processor 1110 may also adopt a multi-core architecture. The processor 1110 and the memory 1120 may also be physically separated.

[0095] The processor 1110 calls the computer program stored in the memory 1120 to execute any of the cell wireless network temporary identifier allocation methods provided in the above embodiments of this application according to the obtained executable instructions.

[0096] Figure 7 A structural block diagram of a user equipment is shown according to an exemplary embodiment.

[0097] like Figure 7 As shown, the user equipment 1300 includes at least: a processor 1310, a memory 1320, and a transceiver 1330.

[0098] The transceiver 1330 is used to receive and send data under the control of the processor 1310.

[0099] exist Figure 7 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 1310 and memory represented by memory 1320 together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 1330 can be multiple elements, including transmitters and receivers, providing units and / or modules for communicating with various other devices over a transmission medium, including wireless channels, wired channels, optical fibers, etc. For different user equipment, the user interface 1340 can also be an interface capable of connecting external or internal devices, including but not limited to keypads, displays, speakers, microphones, joysticks, etc.

[0100] The processor 1310 is responsible for managing the bus architecture and general processing, and the memory 1320 can store the data used by the processor 1310 when performing operations.

[0101] Optionally, the processor 1310 can be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a CPLD (Complex Programmable Logic Device). The processor 1310 can also adopt a multi-core architecture. The processor 1310 and the memory 1320 can also be physically separated.

[0102] The processor 1310 calls the computer program stored in the memory 1320 to execute any of the cell wireless network temporary identifier allocation methods provided in the above embodiments of this application according to the obtained executable instructions.

[0103] It should be noted that the apparatus provided in this application embodiment can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0104] Furthermore, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the data transmission methods described in the above embodiments. The storage medium can be any available medium or data storage device accessible to the processor, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0105] This application provides a program product, such as an FPGA chip or a DSP chip, which includes executable instructions stored in a storage medium. A processor reads the executable instructions from the storage medium, causing the processor to execute the executable instructions to implement the data transmission methods described in the above embodiments.

[0106] The solution proposed in this application can be applied to scenarios where robots are calibrated. The robot includes the following components: a base, a first robotic arm base, a first robotic arm end effector, a second robotic arm base, a second robotic arm end effector, and a depth camera mounted on the first robotic arm end effector. The first robotic arm base and the second robotic arm base are mounted on a movable base. The coordinate systems of each component are different. The calibration of the robot in this solution includes determining the transformation relationship between the coordinate systems corresponding to the robot's components.

[0107] This solution allows the robot to complete the calibration process itself. It calibrates the robot arm base and end effector based on robot arm parameters, calibrates the two end effectors based on data when they overlap, calibrates the depth camera by identifying preset points on the other end effector using a depth camera on one end effector, and calibrates the chassis by identifying fixed points on the chassis using the depth camera. This establishes the conversion relationships between any two components among the chassis, first robot arm base, first robot arm end effector, second robot arm base, second robot arm end effector, and depth camera, thus completing the robot calibration. This solution requires no external equipment and can directly use the robot itself for calibration, making the process simpler and more convenient.

[0108] Specifically, this solution can acquire robotic arm parameters to determine the first conversion relationship between the first robotic arm base and the first robotic arm end effector, and the second conversion relationship between the second robotic arm base and the second robotic arm end effector. After the robotic arm base and the robotic arm end effector are calibrated, the first robotic arm end effector and the second robotic arm end effector can be controlled to be in a zero-force state. The first robotic arm end effector and the second robotic arm end effector can be manually adjusted to make the first robotic arm end effector and the second robotic arm end effector coincide, and the pose data can be acquired as the first calibration data. Three sets of the first calibration data can be acquired, and then the third conversion relationship between the first robotic arm base and the second robotic arm base can be analyzed. After calibrating the end effector of the first robotic arm and the end effector of the second robotic arm, the depth camera at the end effector of the first robotic arm can be kept stationary, and the first image data of the preset point of the end effector of the second robotic arm can be captured as the second calibration data. The end effector of the second robotic arm can be moved to multiple positions to obtain at least three sets of calibration data. Then, the fourth conversion relationship between the end effector of the first robotic arm and the depth camera is determined, and the calibration of the first robotic arm base, the first robotic arm end effector, the second robotic arm base, the second robotic arm end effector and the depth camera is completed.

[0109] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0110] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-executable instructions. These computer-executable instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the processor-readable memory produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These processors can execute instructions that can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0114] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A robot calibration method, characterized in that, The method is used to calibrate a robot, which includes the following components: a first robotic arm base, a first robotic arm end effector, a second robotic arm base, a second robotic arm end effector, and a depth camera mounted on the first robotic arm end effector. The calibration of the robot includes determining the transformation relationships between the coordinate systems corresponding to the robot's components. The method includes: Obtain the parameters of the robotic arm to determine the first conversion relationship between the first robotic arm base and the first robotic arm end effector, and the second conversion relationship between the second robotic arm base and the second robotic arm end effector based on the parameters of the robotic arm. At least three sets of first calibration data are acquired to determine the third conversion relationship between the first robotic arm base and the second robotic arm base based on the first calibration data. The first calibration data includes the pose data acquired when the ends of the first robotic arm and the ends of the second robotic arm are in a zero-force state and coincide. At least three sets of second calibration data are acquired to determine a fourth conversion relationship between the end effector of the first robotic arm and the depth camera based on the second calibration data. The second calibration data includes first image data of the end effector of the second robotic arm taken when the depth camera at the end effector of the first robotic arm is stationary. The step of acquiring at least three sets of first calibration data to determine a third conversion relationship between the first robotic arm base and the second robotic arm base based on the first calibration data includes: controlling the ends of the first and second robotic arms to be in a zero-force state, and acquiring the first pose data of the first robotic arm end and the second pose data of the second robotic arm end when the ends overlap, to determine a set of first calibration data, thereby forming at least three sets of first calibration data; converting and solving the first pose data and the second pose data in each set of first calibration data according to the first conversion relationship and the second conversion relationship, to determine the third conversion relationship; The step of acquiring at least three sets of second calibration data to determine a fourth conversion relationship between the end effector of the first robotic arm and the depth camera includes: acquiring first image data of the end effector of the second robotic arm taken when the depth camera at the end effector of the first robotic arm is stationary, and acquiring a first conversion relationship, a second conversion relationship, and a third conversion relationship to form second calibration data; identifying a preset point on the end effector of the second robotic arm based on the first image data in the second calibration data, and determining the predicted coordinate value of the preset point relative to the depth camera; determining the mapped coordinate value of the preset point relative to the end effector of the first robotic arm based on the first conversion relationship, the second conversion relationship, and the third conversion relationship in the second calibration data; and determining a fourth conversion relationship between the end effector of the first robotic arm and the depth camera based on at least three sets of preset coordinate values ​​and mapped coordinate values.

2. The method according to claim 1, characterized in that, The robot also includes a movable chassis, and the method further includes: Activate the robot's chassis to move and map, and use the depth camera to create an environment map of the current environment starting from the mapping origin; The end effector of the first robotic arm is moved so that the depth camera acquires second image data of fixed points near the mapping origin, wherein at least three fixed points are provided; Based on at least three sets of three-dimensional position coordinates and second image data corresponding to at least three fixed points, the fifth transformation relationship between the chassis and the depth camera is determined.

3. The method according to claim 2, characterized in that, The determination of the fifth transformation relationship between the chassis and the depth camera based on at least three sets of three-dimensional position coordinates corresponding to at least three fixed points and second image data includes: The fixed point is equipped with a QR code, and the QR code carries the three-dimensional position coordinates of the fixed point; By scanning the QR code, the three-dimensional position coordinates of the fixed point relative to the chassis can be obtained; The second image data is used to identify the predicted position coordinates of the fixed point relative to the depth camera. Based on at least three sets of three-dimensional position coordinates and predicted position coordinates corresponding to at least three fixed points, the fifth transformation relationship between the chassis and the depth camera is determined.

4. The method according to claim 3, characterized in that, The determination of the fifth transformation relationship between the chassis and the depth camera based on at least three sets of three-dimensional position coordinates corresponding to at least three fixed points and predicted position coordinates includes: Obtain the first transformation relation, the second transformation relation, the third transformation relation, and the fourth transformation relation; Obtain at least three sets of three-dimensional position coordinates and predicted position coordinates corresponding to at least three fixed points, and perform transformation and solution based on the first transformation relationship, the second transformation relationship, the third transformation relationship and the fourth transformation relationship to determine the fifth transformation relationship between the chassis and the depth camera.

5. The method according to claim 1, characterized in that, The transformation relationships between the coordinate systems corresponding to the robot's components include translation and rotation matrices.

6. A robot calibration device, characterized in that, The device is used to calibrate a robot, which includes the following components: a first robotic arm base, a first robotic arm end effector, a second robotic arm base, a second robotic arm end effector, and a depth camera mounted on the first robotic arm end effector. The calibration of the robot includes determining the transformation relationships between the coordinate systems corresponding to the robot's components. The device includes: The parameter data acquisition module is used to acquire the parameters of the robotic arm, so as to determine the first conversion relationship between the first robotic arm base and the first robotic arm end and the second conversion relationship between the second robotic arm base and the second robotic arm end based on the parameters of the robotic arm. The first data acquisition module is used to acquire at least three sets of first calibration data to determine the third conversion relationship between the first robotic arm base and the second robotic arm base based on the first calibration data. The first calibration data includes pose data acquired when the ends of the first robotic arm and the ends of the second robotic arm are in a zero-force state and coincide. The second data acquisition module is used to acquire at least three sets of second calibration data to determine the fourth conversion relationship between the end of the first robotic arm and the depth camera based on the second calibration data. The second calibration data includes first image data of the end of the second robotic arm taken when the depth camera at the end of the first robotic arm is stationary. The step of acquiring at least three sets of first calibration data to determine a third conversion relationship between the first robotic arm base and the second robotic arm base based on the first calibration data includes: controlling the ends of the first and second robotic arms to be in a zero-force state, and acquiring the first pose data of the first robotic arm end and the second pose data of the second robotic arm end when the ends overlap, to determine a set of first calibration data, thereby forming at least three sets of first calibration data; converting and solving the first pose data and the second pose data in each set of first calibration data according to the first conversion relationship and the second conversion relationship, to determine the third conversion relationship; The step of acquiring at least three sets of second calibration data to determine a fourth conversion relationship between the end effector of the first robotic arm and the depth camera includes: acquiring first image data of the end effector of the second robotic arm taken when the depth camera at the end effector of the first robotic arm is stationary, and acquiring a first conversion relationship, a second conversion relationship, and a third conversion relationship to form second calibration data; identifying a preset point on the end effector of the second robotic arm based on the first image data in the second calibration data, and determining the predicted coordinate value of the preset point relative to the depth camera; determining the mapped coordinate value of the preset point relative to the end effector of the first robotic arm based on the first conversion relationship, the second conversion relationship, and the third conversion relationship in the second calibration data; and determining a fourth conversion relationship between the end effector of the first robotic arm and the depth camera based on at least three sets of preset coordinate values ​​and mapped coordinate values.

7. A network device, characterized in that, include: The system includes a memory, a transceiver, and a processor; wherein the memory is used to store computer programs; and the transceiver is used to send and receive data under the control of the processor. The processor is configured to read a computer program from the memory and execute the method as described in any one of claims 1-5.

8. A storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the method as described in any one of claims 1-5.

Images