Calibration method, device, electronic device and storage medium

Abstract

The present application provides a calibration method, device, electronic device, and storage medium. The method includes: determining two first conversion matrices respectively; determining two second conversion matrices respectively; determining a third conversion matrix for converting a third coordinate system to a second coordinate system based on the two first conversion matrices and the two second conversion matrices; determining N first coordinates respectively, and determining first point cloud data based on the N first coordinates, where N ≥ 2 and N is an integer; determining N second coordinates respectively, and determining second point cloud data based on the N second coordinates; determining a fourth conversion matrix for converting the third coordinate system to a second coordinate system based on the first point cloud data and the second point cloud data; and determining a calibration result for calibrating the conversion relationship between the third coordinate system and the second coordinate system based on the third conversion matrix and the fourth conversion matrix. This solution can calibrate the conversion relationship between the coordinate system of the microscope and the coordinate system of the end of the robotic arm.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a calibration method, device, electronic device, and storage medium. Background Art

[0002] Optical microscopes can observe diseased tissue and adjacent anatomical structures within the surgical target area at high magnification. Combined with other intraoperative auxiliary instruments, they can significantly improve surgical treatment outcomes. With the development of surgical navigation robots and their precise positioning advantages, combining optical microscopes with surgical navigation robots allows the microscope's position to be automatically adjusted during surgery based on navigation information, reducing the surgeon's workload and improving surgical efficiency and precision. The optical microscope can be supported by the multi-degree-of-freedom robotic arm of the surgical navigation robot, allowing it to adjust its position and angle at any time during surgery.

[0003] For example, after completing patient registration during surgery, the multi-degree-of-freedom robotic arm is guided to move the optical microscope to the appropriate position according to the preoperative planned path, so that the optical axis of the optical microscope can be aligned with the patient's surgical target area, thereby facilitating observation of the surgical target area through the optical microscope.

[0004] To achieve this goal, since optical microscopes are mostly standalone devices, and they and multi-DOF robotic arms are two independent systems, it is necessary to calibrate the conversion relationship between the coordinate system of the optical microscope and the coordinate system of the end of the multi-DOF robotic arm. The more accurate the calibration result, the better the precision of controlling the multi-DOF robotic arm to move the optical microscope. Based on this, a new technical solution is needed to obtain more accurate calibration results. Summary of the Invention

[0005] In order to solve the above problems, embodiments of the present application provide a calibration method, device, electronic device and storage medium to at least partially solve the above problems.

[0006] According to one aspect of the present application, a calibration method is provided, which includes:

[0007] Determining two first transformation matrices for transforming a second coordinate system into a first coordinate system when the end of the robotic arm is located at a first position and a second position, respectively, wherein the first coordinate system is a coordinate system of a base of the robotic arm and the second coordinate system is a coordinate system of the end of the robotic arm;

[0008] determining two second transformation matrices for transforming the fourth coordinate system into the third coordinate system when the microscope is located at a third position and a fourth position, respectively, wherein the third coordinate system is a coordinate system of the microscope, and the fourth coordinate system is a coordinate system of a marking plate, the marking plate is disposed outside the robotic arm and the microscope, the microscope is fixedly connected to an end of the robotic arm, and when the end of the robotic arm moves from the first position to the second position, the microscope follows the end of the robotic arm in moving from the third position to the fourth position;

[0009] determining a third transformation matrix for transforming the third coordinate system into the second coordinate system based on the two first transformation matrices and the two second transformation matrices;

[0010] Determine N first coordinates of the origin of the first coordinate system in the second coordinate system when the end of the robotic arm is located at N fifth positions in the trajectory moving from the first position to the second position, and determine first point cloud data based on the N first coordinates, where N ≥ 2 and N is an integer;

[0011] determining N second coordinates of the first marking point on the marking plate in the third coordinate system when the microscope is at N sixth positions in the trajectory moving from the third position to the fourth position, respectively, and determining second point cloud data based on the N second coordinates, wherein when the end of the robotic arm is at different fifth positions, the microscope is at different sixth positions;

[0012] determining, based on the first point cloud data and the second point cloud data, a fourth transformation matrix for transforming the third coordinate system into the second coordinate system;

[0013] A calibration result of calibrating the conversion relationship between the third coordinate system and the second coordinate system is determined based on the third conversion matrix and the fourth conversion matrix.

[0014] In some optional embodiments, determining two first transformation matrices for transforming the second coordinate system into the first coordinate system when the end of the robotic arm is located at the first position and the second position, respectively, includes:

[0015] respectively obtaining the rotation angles of the joints of the robotic arm when the end of the robotic arm is located at the first position and the second position;

[0016] Finding a kinematics forward solution for the robotic arm based on the rotation angles of the joints when the end of the robotic arm is located at the first position, and determining a first transformation matrix when the end of the robotic arm is located at the first position;

[0017] According to the rotation angles of the joints when the end of the robotic arm is located at the second position, a kinematics forward solution is obtained for the robotic arm to determine a first transformation matrix when the end of the robotic arm is located at the second position.

[0018] In some optional embodiments, determining two second transformation matrices for transforming the fourth coordinate system into the third coordinate system when the microscope is located at the third position and the fourth position, respectively, includes:

[0019] acquiring a first image obtained by photographing a first marking point on the marking plate when the microscope is at the third position, determining, based on the first image, position coordinates of the first marking point on the marking plate in a third coordinate system when the microscope is at the third position, and determining, based on the position coordinates, a second transformation matrix when the microscope is at the third position;

[0020] Acquire a second image obtained by photographing the first marking point on the marking plate when the microscope is at the fourth position, determine the position coordinates of the first marking point on the marking plate in the third coordinate system when the microscope is at the fourth position based on the second image, and determine a second transformation matrix of the microscope when it is at the fourth position based on the position coordinates.

[0021] In some optional embodiments, determining a third transformation matrix for transforming the third coordinate system into the second coordinate system according to the two first transformation matrices and the two second transformation matrices includes:

[0022] The third transformation matrix is ​​determined by the following formula:

[0023] M 2_2 -1 M 2_1 *M3=M3*M 1_2 *M 1_1 -1

[0024] Among them, M 1_1 The first transformation matrix used to characterize the end of the robotic arm when it is at the first position, M 1_2 The first transformation matrix used to characterize the end of the robotic arm when it is at the second position, M 2_1 The second transformation matrix used to characterize the microscope when it is located at the third position, M 2_2 M is used to characterize the second transformation matrix when the microscope is located at the fourth position, and M3 is used to characterize the third transformation matrix.

[0025] In some optional embodiments, determining, based on the first point cloud data and the second point cloud data, a fourth transformation matrix for transforming the third coordinate system into the second coordinate system includes:

[0026] Based on a one-to-one correspondence between the N first coordinates in the first point cloud data and the N second coordinates in the second point cloud data, an affine transformation is performed on the first point cloud data and the second point cloud data to determine a fourth transformation matrix.

[0027] In some optional embodiments, determining a calibration result of calibrating the conversion relationship between the third coordinate system and the second coordinate system based on the third conversion matrix and the fourth conversion matrix includes:

[0028] determining first error data for transforming the third coordinate system into the second coordinate system using a third transformation matrix;

[0029] determining second error data for transforming the third coordinate system into the second coordinate system using a fourth transformation matrix;

[0030] If the first error data is smaller than the second error data, the third conversion matrix is ​​determined as the calibration result; or if the first error data is larger than the second error data, the fourth conversion matrix is ​​determined as the calibration result.

[0031] According to another aspect of the embodiments of the present application, a calibration device is provided, comprising:

[0032] a first determining module, configured to respectively determine two first transformation matrices for transforming a second coordinate system into a first coordinate system when the end of the robotic arm is located at a first position and a second position, wherein the first coordinate system is a coordinate system of a base of the robotic arm, and the second coordinate system is a coordinate system of the end of the robotic arm;

[0033] a second determination module, for respectively determining two second transformation matrices for transforming a fourth coordinate system into a third coordinate system when the microscope is located at a third position and a fourth position, wherein the third coordinate system is a coordinate system of the microscope, and the fourth coordinate system is a coordinate system of a marking plate, the marking plate is disposed outside the manipulator and the microscope, the microscope is fixedly connected to an end of the manipulator, and when the end of the manipulator moves from the first position to the second position, the microscope follows the end of the manipulator to move from the third position to the fourth position;

[0034] a third determining module, configured to determine a third conversion matrix for converting the third coordinate system into the second coordinate system based on the two first conversion matrices and the two second conversion matrices;

[0035] a fourth determining module, configured to respectively determine N first coordinates of the origin of the first coordinate system in the second coordinate system when the end of the robotic arm is located at N fifth positions in the trajectory moving from the first position to the second position, and determine first point cloud data based on the N first coordinates, where N is an integer.

[0036] a fifth determining module, configured to respectively determine N second coordinates of the first marking point on the marking plate in the third coordinate system when the microscope is located at N sixth positions in the trajectory moving from the third position to the fourth position, and determine second point cloud data based on the N second coordinates, wherein when the end of the robotic arm is located at different fifth positions, the microscope is located at different sixth positions;

[0037] a sixth determining module, configured to determine, based on the first point cloud data and the second point cloud data, a fourth transformation matrix for transforming the third coordinate system into the second coordinate system;

[0038] The seventh determining module is configured to determine a calibration result of calibrating the conversion relationship between the third coordinate system and the second coordinate system based on the third conversion matrix and the fourth conversion matrix.

[0039] According to another aspect of the embodiments of the present application, an electronic device is provided, including: the electronic device includes a memory and a processor, the memory is used to store a computer executable program, and the processor is used to run the computer executable program to implement any of the calibration methods described above.

[0040] According to another aspect of the embodiments of the present application, a computer storage medium is provided, wherein the computer storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the calibration method according to any of the above items.

[0041] According to another aspect of the embodiments of the present application, a computer program product is provided, which includes a computer program, wherein the computer program implements any of the calibration methods described above when executed by a processor.

[0042] In summary, since the microscope is fixedly connected to the end of the robotic arm in the calibration method of the present application, two first transformation matrices for converting the second coordinate system to the first coordinate system can be determined respectively when the end of the robotic arm is located at the first position and the second position, the first coordinate system is the coordinate system of the base of the robotic arm, and the second coordinate system is the coordinate system of the end of the robotic arm. Two second transformation matrices for converting the fourth coordinate system to the third coordinate system can be determined respectively when the microscope is located at the third position and the fourth position, the third coordinate system is the coordinate system of the microscope, and the fourth coordinate system is the coordinate system of the marking plate. According to the two first transformation matrices and the two second transformation matrices, the third transformation matrix for converting the third coordinate system to the second coordinate system is determined. When the end of the robotic arm is located at N fifth positions in the trajectory moving from the first position to the second position, the N first coordinates of the origin of the first coordinate system in the second coordinate system are determined respectively, and the first point cloud data is determined based on the N first coordinates, and the microscope is determined respectively when it is located at the third position When moving to the N sixth positions in the trajectory of the fourth position, the first marking point on the marking plate is at the N second coordinates in the third coordinate system, and the second point cloud data is determined based on the N second coordinates. According to the first point cloud data and the second point cloud data, the fourth transformation matrix for converting the third coordinate system to the second coordinate system is determined. Based on the third transformation matrix and the fourth transformation matrix, the calibration result for calibrating the conversion relationship between the third coordinate system and the second coordinate system is determined. Therefore, the data processing scheme can obtain different transformation matrices for converting the third coordinate system to the second coordinate system according to different calculation methods, and thus can ultimately obtain a more accurate and reliable calibration result for calibrating the conversion relationship between the third coordinate system (i.e., the coordinate system of the microscope) and the second coordinate system (i.e., the coordinate system of the end of the robotic arm) based on different transformation matrices. Based on this more accurate and reliable calibration result, the accuracy of controlling the multi-degree-of-freedom robotic arm to drive the movement of the optical microscope can be effectively improved, the complexity of microsurgery can be reduced, and the surgical accuracy and efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0044] Figure 1 A flow chart of an exemplary calibration method according to the present application is shown.

[0045] Figure 2 An optional sub-flowchart of step S101 according to the present application is shown.

[0046] Figure 3 A schematic diagram of an exemplary robotic arm, microscope, and marking plate according to the present application is shown.

[0047] Figure 4 An optional sub-flowchart of step S102 according to the present application is shown.

[0048] Figure 5 An optional sub-flowchart of step S107 according to the present application is shown.

[0049] Figure 6 A block diagram of an exemplary calibration device according to the present application is shown.

[0050] Figure 7 A block diagram of an exemplary electronic device according to the present application is shown. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0052] A calibration method is provided in an embodiment of the present application. The method can be executed by a calibration device. The calibration device may include at least one processor capable of performing data processing, such as a CPU, MCU, DSP, FPGA, etc., or may be a computer device, server, etc. capable of performing data processing, which is not limited here.

[0053] Figure 1 FIG. 1 shows a flow chart of an exemplary calibration method according to the present application. Figure 1 As shown in the flowchart, the calibration method includes the following steps S101, S102, S103, S104, S105, S106 and S107, specifically:

[0054] Step S101 : respectively determining two first transformation matrices for transforming a second coordinate system into a first coordinate system when the end of the robot arm is located at a first position and a second position.

[0055] In the present application, the first coordinate system is the coordinate system of the base of the robotic arm, and the second coordinate system is the coordinate system of the end of the robotic arm.

[0056] Optionally, the robotic arm may be part of a surgical navigation robot. In the present application, the robotic arm may be a multi-degree-of-freedom robotic arm, such as a 6-degree-of-freedom robotic arm, a 7-degree-of-freedom robotic arm, and the like. Taking a 6-degree-of-freedom robotic arm as an example, it may include 6 joints, and the 6 joints rotate according to their respective rotation angles, so that the end of the robotic arm can be in different postures. The same applies to other situations (such as a 7-degree-of-freedom robotic arm). Of course, the number of joints of the multi-degree-of-freedom robotic arm is not limited in the present application, but for the convenience of illustrating this embodiment, the following examples are taken as examples in which the robotic arm includes 6 joints, that is, the robotic arm is a 6-degree-of-freedom robotic arm.

[0057] In the present application, the end of the robotic arm moves from a first position to a second position while being controlled to move.

[0058] In this application, the first coordinate system is the coordinate system of the base of the robotic arm, that is, the base coordinate system of the robotic arm. The second coordinate system is the coordinate system of the end of the robotic arm. Optionally, the end of the robotic arm can be the end flange of the robotic arm, that is, the second coordinate system can be the end flange coordinate system of the robotic arm. In this application, for ease of description, the first coordinate system is denoted as J, the second coordinate system is denoted as F, and the first transformation matrix is ​​denoted as M1.

[0059] The second coordinate system F can be transformed into the first coordinate system J through the first transformation matrix M1. The second coordinate system F will change with the movement of the end of the robot arm, and the first transformation matrix M1 can change with the movement of the end of the robot arm.

[0060] In the present application, when the end of the robotic arm is located at a first position, a first conversion matrix is ​​obtained, and when the end of the robotic arm is located at a second position, another first conversion matrix is ​​obtained.

[0061] This application does not limit the specific implementation of step S101. In some optional embodiments, refer to Figure 2 In the flowchart shown, step S101 includes sub-steps S1011, S1012 and S1013.

[0062] Sub-step S1011: respectively obtaining the rotation angles of the joints of the robotic arm when the end of the robotic arm is located at the first position and the second position.

[0063] For example, the robot arm has 6 degrees of freedom, i.e., it includes 6 joints, and the end of the robot arm is located at the first position. When the end of the robot arm is at the initial first position, the rotation angles of the 6 joints can be obtained: θ 1_1 ,θ 1_2 ,θ 1_3 ,θ 1_4 ,θ 1_5 ,θ 1_6For example, when the end of the robotic arm is at the second position, the rotation angles of the six joints can be obtained: θ 2_1 ,θ 2_2 ,θ 2_3 ,θ 2_4 ,θ 2_5 ,θ 2_6 The rest can be deduced by analogy and will not be described in detail. For the first position and the second position, two groups of rotation angles of each joint can be obtained, and the first position and the second position correspond to a group of rotation angles of each joint respectively.

[0064] Sub-step S1012: according to the rotation angles of the joints when the end of the robotic arm is located at the first position, a kinematic forward solution is obtained for the robotic arm to determine a first transformation matrix when the end of the robotic arm is located at the first position.

[0065] The kinematic solution of the robotic arm is obtained by using the rotation angles of each joint when the end of the robotic arm is in the first position, and the first transformation matrix M1 for transforming the second coordinate system F to the first coordinate system J when the end of the robotic arm is in the first position is obtained.

[0066] Sub-step S1013: according to the rotation angles of the joints when the end of the robotic arm is located at the second position, a kinematic forward solution is obtained for the robotic arm to determine a first transformation matrix when the end of the robotic arm is located at the second position.

[0067] The rotation angles of the joints obtained when the end of the robotic arm is in the second position are used to obtain a kinematic solution for the robotic arm, thereby obtaining a first transformation matrix M1 for transforming the second coordinate system F into the first coordinate system J when the end of the robotic arm is in the second position. Thus, two first transformation matrices can be obtained in this manner.

[0068] Based on this, in this application, through the above steps S1011 to S1013, the results of the two first conversion matrices can be calculated more accurately, so that more accurate processing results can be obtained when the two first conversion matrices are used for data processing later.

[0069] Step S102: respectively determining two second transformation matrices for transforming the fourth coordinate system into the third coordinate system when the microscope is located at the third position and the fourth position.

[0070] In this application, the third coordinate system is the coordinate system of the microscope, and the fourth coordinate system is the coordinate system of the marking plate. The marking plate is arranged on the outside of the robotic arm and the microscope. The microscope is fixedly connected to the end of the robotic arm. When the end of the robotic arm moves from the first position to the second position, the microscope follows the end of the robotic arm to move from the third position to the fourth position.

[0071] When the end of the robotic arm is located at the first position, the microscope is located at the third position, and when the end of the robotic arm is located at the second position, the microscope is located at the fourth position.

[0072] The microscope of the present application can be an optical microscope. The third coordinate system is the coordinate system of the microscope. For example, it can be pre-established with the optical axis of the microscope as one of the coordinate axes. For ease of representation, the third coordinate system is denoted as C below. The microscope moves with the movement of the end of the robotic arm, and the third coordinate system C changes with the movement of the microscope.

[0073] The marking board in this application can be used to assist in calibration, and can include multiple marking points on it. One of the multiple marking points can be used as the first marking point. In some examples, the marking board can include multiple black blocks and multiple white blocks, and a marking point is formed at the intersection of the black block and the white block (the intersection can also be called a corner point). Figure 3 An example of a marking board is shown in FIG. 8 , which should be understood as not limiting the present application in any way.

[0074] The fourth coordinate system is the coordinate system of the marker plate, which can be predetermined based on the position of the marker plate. For ease of representation, the fourth coordinate system is denoted as T. The position of the marker plate is fixed outside the robot arm and the microscope, and the fourth coordinate system T is fixed.

[0075] In this application, the second transformation matrix is ​​denoted as M2, and the fourth coordinate system T can be transformed into the third coordinate system C via the second transformation matrix M2. The third coordinate system C changes with the movement of the microscope, and the second transformation matrix M2 can also change with the movement of the microscope. In this application, when the microscope is in the third position, a second transformation matrix M2 is obtained, and when the microscope is in the fourth position, another second transformation matrix M2 is obtained.

[0076] Figure 3 Schematic diagram of an exemplary robotic arm, microscope and marking board according to the present application is shown. Figure 3 Further understanding, the microscope 20 is fixedly connected to the end of the robot arm 10 and can move with the end of the robot arm 10. The marking plate 30 is fixedly set on the outside of the robot arm 10 and the microscope 20. The microscope 20 can take pictures of the first calibration point 31 on the marking plate 30. The robot arm 10 is installed on the base 11 of the robot arm 10. This form is consistent with the eye-on-hand form in hand-eye calibration. It should be understood that Figure 3 The examples are not intended to limit the present application in any way.

[0077] Optionally, refer to Figure 4 As shown, step S102 includes sub-step S1021 and sub-step S1022.

[0078] Sub-step S1021: Acquire a first image obtained by photographing the first marking point on the marking plate when the microscope is located at the third position, determine the position coordinates of the first marking point on the marking plate in the third coordinate system when the microscope is located at the third position based on the first image, and determine the second transformation matrix of the microscope when it is located at the third position based on the position coordinates.

[0079] Sub-step S1022: Obtain a second image obtained by photographing the first marking point on the marking plate when the microscope is located at the fourth position, determine the position coordinates of the first marking point on the marking plate in the third coordinate system when the microscope is located at the fourth position based on the second image, and determine the second transformation matrix of the microscope when it is located at the fourth position based on the position coordinates.

[0080] Optionally, when the microscope is located at the third position, the first marking point of the marking plate can be photographed by the camera of the microscope to obtain a first image, and then the position coordinates of the first marking point in the third coordinate system C can be directly calculated using the first image and the camera parameters of the microscope (such as camera intrinsic parameters, camera extrinsic parameters, etc.), and then the second transformation matrix M2 of the microscope when it is located at the third position is determined based on the position coordinates.

[0081] Optionally, when the microscope is at the fourth position, the first marking point of the marking plate can be photographed by the microscope's camera to obtain a second image, and then the position coordinates of the first marking point in the third coordinate system C can be directly calculated using the second image and the microscope's camera parameters (such as camera intrinsic parameters, camera extrinsic parameters, etc.), and then the second transformation matrix M2 of the microscope when it is at the fourth position is determined based on the position coordinates.

[0082] It should be understood that calculating the position coordinates of a spatial point (such as the first marking point) in the coordinate system of the microscope (i.e., the third coordinate system C) by using the images captured by the microscope's camera (such as the first image and the second image) and camera parameters (such as camera intrinsic parameters, camera extrinsic parameters, etc.) is a mature technology and will not be elaborated here.

[0083] Based on this, in this application, through the above steps S1021 to S1022, the results of the two second conversion matrices can be calculated more accurately, so that more accurate processing results can be obtained when the two second conversion matrices are used for data processing later.

[0084] Step S103: determining a third transformation matrix for transforming the third coordinate system into the second coordinate system according to the two first transformation matrices and the two second transformation matrices.

[0085] In this application, the third transformation matrix is ​​denoted as M3. Ideally, the third coordinate system C (i.e., the coordinate system of the microscope) can be transformed into the second coordinate system F (i.e., the coordinate system of the end of the robotic arm) via the third transformation matrix M3. During the movement of the end of the robotic arm, the third transformation matrix M3 remains unchanged. In this step S103, the third transformation matrix M3 can be calculated using the two first transformation matrices M1 obtained in step S101 and the two second transformation matrices M2 obtained in step S102.

[0086] In the present application, the third transformation matrix M3 can be used to more reliably determine the calibration result of the conversion relationship between the third coordinate system and the second coordinate system in subsequent steps (that is, the conversion relationship between the coordinate system of the microscope and the coordinate system of the end of the robotic arm). In some implementations, the third transformation matrix M3 can also be directly used as the calibration result when the needs are met.

[0087] In some optional embodiments, step S103 includes: determining a third conversion matrix M3 according to the following formula 1:

[0088] M 2_2 -1 M 2_1 *M3=M3*M 1_2 *M 1_1 -1 (Formula 1)

[0089] Among them, M 1_1 The first transformation matrix used to characterize the end of the robotic arm when it is at the first position, M 1_2 The first transformation matrix used to characterize the end of the robotic arm when it is at the second position, M 2_1 The second transformation matrix used to characterize the microscope when it is located at the third position, M 2_2 M is used to characterize the second transformation matrix when the microscope is located at the fourth position, and M3 is used to characterize the third transformation matrix.

[0090] Based on this, the present application can more accurately determine the third transformation matrix M3 based on the two first transformation matrices and the two second transformation matrices through the above method, so as to finally determine the calibration result of the transformation relationship between the third coordinate system (i.e., the coordinate system of the microscope) and the second coordinate system (i.e., the coordinate system of the end of the robotic arm).

[0091] The following is a brief explanation of the principle of determining the third transformation matrix M3 using the above formula 1. In this application, M0 is used to represent the zeroth transformation matrix used to transform the fourth coordinate system T (i.e., the coordinate system of the marking plate) to the first coordinate system J (i.e., the coordinate system of the base of the manipulator). During the movement of the end of the manipulator, the zeroth transformation matrix M0 remains unchanged. Therefore, the transformation relationship between the zeroth transformation matrix M0, the first transformation matrix M1, the second transformation matrix M2, and the third transformation matrix M3 satisfies the following relationship:

[0092] M0=M2*M3*M1

[0093] For example, the first position can be denoted as f1, the second position as f2; the third position can be denoted as c1, and the fourth position as c2. If the end of the robotic arm moves from the first position f1 to the second position f2, and the microscope moves from the third position c1 to the fourth position c2, the above relationship, when the end of the robotic arm is at the first position and the microscope is at the third position, can be:

[0094] M0=M 2_1 *M3*M 1_1

[0095] When the end of the robotic arm is in the second position and the microscope is in the fourth position, it may be:

[0096] M0=M 2_2 *M3*M 1_2

[0097] Then we can get: M 2_1 *M3*M 1_1 =M 2_2 *M3*M 1_2

[0098] After transformation, we can get: M 2_1 *M3=M 2_2 *M3*M 1_2 *M 1_1 -1

[0099] After further transformation, we can get the above formula 1: M 2_2 -1 M 2_1 *M3=M3*M 1_2 *M 1_1 -1

[0100] Based on this, the two first transformation matrices M calculated in step S101 are used 1_1 and M 1_2 , and the two second transformation matrices M calculated in step S102 2_1 and M2_2 , the third conversion matrix M3 can be determined according to the above formula 1.

[0101] The third transformation matrix M3 includes a translation component tX and a rotation component RX for translationally transforming and rotating the third coordinate system C to the second coordinate system F. To facilitate solving the third transformation matrix M3 using Formula 1, optionally, a two-step method can be used to solve the third transformation matrix M3. By separating the relevant rotation component and translation component from Formula 1, the rotation component RX in the third transformation matrix M3 is first calculated, and then the translation component tX in the third transformation matrix M3 is calculated, thereby ultimately obtaining the determination result of the third transformation matrix M3.

[0102] In this embodiment, let M in the above formula 1 be 2_2 -1 M 2_1 =A,M 1_2 *M 1_1 -1 =B, M3=X, then the above formula 1 can be rewritten as the following equation:

[0103] AX=XB

[0104] Separating the rotational and translational components of this equation yields:

[0105]

[0106] Among them, R A is the rotation component of A, t A is the translation component of A, R B is the rotation component of B, t B is the translation component of B, R X is the rotation component of X (i.e., the third transformation matrix M3), t X is the translation component of X (i.e., the third transformation matrix M3), and the above formula is further transformed to obtain:

[0107]

[0108] This results in the following two equations 1 and 2:

[0109] R A R X =R X R B (Equation 1)

[0110] R A tx+t A =R X t B +t X (Equation 2)

[0111] According to the properties of Kronecker product, the following conversion equation is obtained: Among them, vec() means vectorization, Represents the Kronecker product operator.

[0112] Using the properties of the Kronecker product, we can transform Equation 1 and Equation 2 to obtain Equation 3 and Equation 4 respectively:

[0113] vec(R A R X i)=vec(iR X R B ) (Equation 3)

[0114]

[0115] The i in Equations 3 and 4 above is used to represent the unit vector. Subtracting both sides of the equal sign in Equation 4 above yields:

[0116]

[0117] Then order Then Equation 5 can be rewritten as:

[0118] Kvec(R X )=0 (Equation 6)

[0119] The above equation 6 satisfies the equation system of the form "Ax = 0", and the rotation component E can be calculated by the singular value decomposition method. X .

[0120] Specifically, the parameter K is a 9*9 matrix obtained by the Kronecker product (assuming that there are n positions in the movement process of the end of the manipulator during the calibration process, and one K can be calculated for two adjacent positions, with a total of n-1 Ks. The matrices are stacked to obtain a 9(n-1)*9 matrix. Corresponding to this embodiment, the positions of the end movement process of the manipulator are the first position and the second position, a total of two positions, n=2, and the parameter K is a 9*9 matrix.), and then the singular value decomposition is performed to finally obtain the rotation component R of X (that is, the third transformation matrix M3) X .

[0121] In finding the rotation component R X Then solve the translation component t of X (i.e. the third transformation matrix M3) X , based on the above equations 1 and 2, we get the following equation 7:

[0122] (R A -i)t X =R X t B -t A(Equation 7)

[0123] Let R A -i=D,R X t B -t A =E, then Equation 7 satisfies the linear equation: Dt X =E, where D is a 3*3 matrix, t X and E are both three-dimensional column vectors (assuming that there are n positions in the movement process of the end of the manipulator during the calibration process, and a set of values ​​can be calculated for two adjacent positions. After vector stacking, D is a 3(n-1)*3 matrix and E is a 3(n-1) column vector. Corresponding to this embodiment, the positions of the end of the manipulator in the movement process are the first position and the second position, n=2, then D is a 3*3 matrix and E is a three-dimensional column vector). The translation component t of X (i.e., the third transformation matrix M3) is obtained by solving X .

[0124] At this point, the rotation component R of the third transformation matrix M3 X and the translation component t X All calculations are completed, and the third conversion matrix M3 is determined. In this application, through the above process, it is possible to more accurately calculate the value of the first conversion matrix M3 according to the two first conversion matrices (i.e., M 1_1 and M 1_2 ) and two second transformation matrices (i.e., M 2_1 and M 2_2 ) determines the third transformation matrix M3, so as to finally determine the calibration result of calibrating the transformation relationship between the third coordinate system C (i.e., the coordinate system of the microscope) and the second coordinate system F (i.e., the coordinate system of the end of the robotic arm).

[0125] Of course, the above method is only an exemplary description of the process of determining the third conversion matrix M3, and not any limitation of the present application. Those skilled in the art may also use other reasonable methods to solve the third conversion matrix M3, which is not limited in the embodiments of the present application.

[0126] Step S104: respectively determine N first coordinates of the origin of the first coordinate system in the second coordinate system when the end of the robotic arm is located at N fifth positions in the trajectory moving from the first position to the second position, and determine the first point cloud data based on the N first coordinates.

[0127] In the present application, when the end of the manipulator moves from the first position to the second position, the movement path of the end of the manipulator forms a trajectory, from which N fifth positions can be selected as needed, and the origin of the first coordinate system J (i.e., the coordinate system of the base of the manipulator) is determined to be in the second coordinate system F (i.e., the coordinate system of the end of the manipulator) when the end of the manipulator is located at the N fifth positions. In the present application, N ≥ 2 and N is an integer. The value of N can be selected as needed. For example, in some optional embodiments, N can be selected as 4, that is, 4 fifth positions are selected to obtain 4 first coordinates.

[0128] Optionally, the first position and the second position may be used as two of the N fifth positions, and the first position and the second position may be used as the first fifth position and the last fifth position of the N fifth positions respectively.

[0129] The first coordinate system C is pre-established based on the base of the robotic arm, and its origin is fixed. The first coordinate of the origin of the first coordinate system C in the second coordinate system F (i.e., the coordinate system of the end of the robotic arm) can be directly calculated in real time through the robotic arm's underlying interface. A first coordinate is obtained for each of the N fifth positions of the robotic arm's end, for a total of N first coordinates. Finally, the first point cloud data is obtained based on these N first coordinates.

[0130] Step S105: respectively determining N second coordinates of the first marking point on the marking plate in the third coordinate system when the microscope is located at N sixth positions in the trajectory moving from the third position to the fourth position, and determining second point cloud data based on the N second coordinates.

[0131] When the end of the robotic arm is located at a different fifth position, the microscope is located at a different sixth position.

[0132] In the present application, as the microscope moves from the third position to the fourth position, the movement path of the microscope following the end of the robotic arm forms a trajectory, from which N sixth positions can be selected as needed, and the N second coordinates of the first marking point on the marking plate in the third coordinate system C (i.e., the coordinate system of the microscope) are determined when the microscope is located at these N sixth positions. As mentioned above, in the present application, N ≥ 2 and N is an integer. The value of N can be selected as needed. For example, in some optional embodiments, N can be selected as 4, that is, 4 sixth positions are selected to obtain 4 second coordinates. The N fifth positions correspond one-to-one to the N sixth positions.

[0133] Alternatively, the third position and the fourth position can be used as two of the N sixth positions, and the third position and the fourth position can be used as the first sixth position and the last sixth position of the N sixth positions, respectively. Therefore, the two position coordinates obtained in steps S1021 and S1022 can also be used as the two second coordinates.

[0134] Alternatively, when the microscope is at each sixth position, the first marking point on the marking plate can be photographed by the microscope's camera to obtain a captured image. The captured image and the microscope's camera parameters (e.g., camera intrinsic parameters, camera extrinsic parameters, etc.) can then be used to directly calculate the second coordinate of the first marking point in the third coordinate system C (i.e., the microscope's coordinate system). A second coordinate is obtained at each of the N sixth positions of the microscope, for a total of N second coordinates. Finally, second point cloud data is obtained based on the N second coordinates.

[0135] Step S106: Determine a fourth transformation matrix for transforming the third coordinate system into the second coordinate system based on the first point cloud data and the second point cloud data.

[0136] In the present application, the fourth transformation matrix is ​​denoted as M4. Ideally, the third coordinate system C (i.e., the coordinate system of the microscope) can be transformed into the second coordinate system F (i.e., the coordinate system of the end of the robotic arm) through the fourth transformation matrix M4. During the movement of the end of the robotic arm, the fourth transformation matrix M4 remains unchanged. In step S106 of the present application, the first point cloud data and the second point cloud data obtained can be used for data processing to determine the fourth transformation matrix M4 for transforming the third coordinate system C (i.e., the coordinate system of the microscope) into the second coordinate system (i.e., the coordinate system of the end of the robotic arm).

[0137] In the present application, the fourth transformation matrix M4 can be used to more reliably determine the calibration result of the conversion relationship between the third coordinate system and the second coordinate system in subsequent steps (that is, the conversion relationship between the coordinate system of the microscope and the coordinate system of the end of the robotic arm). In some implementations, the fourth transformation matrix M4 can also be directly used as the calibration result when the requirements are met.

[0138] Optionally, step S106 in the present application may be to perform an affine transformation on the first point cloud data and the second point cloud data based on a one-to-one correspondence between the N first coordinates in the first point cloud data and the N second coordinates in the second point cloud data, so as to determine a fourth transformation matrix M4 for transforming the third coordinate system C into the second coordinate system F. Based on this, the fourth transformation matrix M4 can be reliably calculated in this manner in the present application.

[0139] For example, taking N as 4 (i.e., each first point cloud data includes the first coordinates of four points, and the second point cloud data includes the second coordinates of four points), in order, the first first coordinate of the four first coordinates in the first point cloud data corresponds to the first second coordinate of the four second coordinates in the second point cloud data, the second first coordinate of the four first coordinates in the first point cloud data corresponds to the second second coordinate of the four second coordinates in the second point cloud data, the third first coordinate of the four first coordinates in the first point cloud data corresponds to the third second coordinate of the four second coordinates in the second point cloud data, and the fourth first coordinate of the four first coordinates in the first point cloud data corresponds to the fourth second coordinate of the four second coordinates in the second point cloud data. The transformation relationship between the first point cloud data and the second point cloud data is calculated using the point cloud affine transformation algorithm, thereby obtaining a fourth transformation matrix M4 for transforming the third coordinate system C into the second coordinate system F. It should be understood that the example of N being 4 does not constitute any limitation to the present application.

[0140] Step S107: determining a calibration result of calibrating the conversion relationship between the third coordinate system and the second coordinate system based on the third conversion matrix and the fourth conversion matrix.

[0141] In the present application, since the microscope is fixedly connected to the end of the robotic arm in the calibration method, two first transformation matrices for converting the second coordinate system to the first coordinate system can be determined respectively when the end of the robotic arm is located at the first position and the second position, the first coordinate system is the coordinate system of the base of the robotic arm, and the second coordinate system is the coordinate system of the end of the robotic arm. Two second transformation matrices for converting the fourth coordinate system to the third coordinate system can be determined respectively when the microscope is located at the third position and the fourth position, the third coordinate system is the coordinate system of the microscope, and the fourth coordinate system is the coordinate system of the marking plate. According to the two first transformation matrices and the two second transformation matrices, the third transformation matrix for converting the third coordinate system to the second coordinate system is determined. When the end of the robotic arm is located at N fifth positions in the trajectory moving from the first position to the second position, the N first coordinates of the origin of the first coordinate system in the second coordinate system are determined respectively, and the first point cloud data is determined according to the N first coordinates. The microscope is determined respectively when it is located at the N fifth positions in the trajectory moving from the first position to the second position. When there are N sixth positions in the trajectory of the fourth position, the first marking point on the marking plate is at the N second coordinates in the third coordinate system, and the second point cloud data is determined based on the N second coordinates. According to the first point cloud data and the second point cloud data, the fourth transformation matrix for converting the third coordinate system to the second coordinate system is determined. Based on the third transformation matrix and the fourth transformation matrix, the calibration result for calibrating the conversion relationship between the third coordinate system and the second coordinate system is determined. Therefore, the data processing scheme can obtain different transformation matrices for converting the third coordinate system to the second coordinate system according to different calculation methods, and thus can ultimately obtain a more accurate and reliable calibration result for calibrating the conversion relationship between the third coordinate system (i.e., the coordinate system of the microscope) and the second coordinate system (i.e., the coordinate system of the end of the robotic arm) based on different transformation matrices. Based on this more accurate and reliable calibration result, the accuracy of controlling the multi-degree-of-freedom robotic arm to drive the movement of the optical microscope can be effectively improved, the complexity of microsurgery can be reduced, and the surgical accuracy and efficiency can be improved.

[0142] In step S107 of the present application, the reason why the calibration result for calibrating the conversion relationship between the third coordinate system and the second coordinate system is determined based on the third conversion matrix and the fourth conversion matrix, rather than directly calculating only the third conversion matrix as the calibration result or only calculating only the fourth conversion matrix as the calibration result, is because in some cases the accuracy of the third conversion matrix is ​​higher, and in other cases the accuracy of the fourth conversion matrix is ​​higher. Therefore, the third conversion matrix and the fourth conversion matrix can be analyzed, and the one with higher accuracy can be used as the calibration result, so that the calibration result for calibrating the conversion relationship between the third coordinate system (i.e., the coordinate system of the microscope) and the second coordinate system (i.e., the coordinate system of the end of the robotic arm) can be obtained more accurately and reliably.

[0143] In some optional embodiments, referring to Figure 5In the flowchart shown, step S107 includes sub-steps S1071, S1072 and S1073. Specifically:

[0144] Sub-step S1071: Determine first error data for transforming the third coordinate system into the second coordinate system using a third transformation matrix.

[0145] The first error data can be used to indicate the accuracy of converting the third coordinate system C to the second coordinate system F using the third transformation matrix. The larger the first error data, the lower the accuracy of converting the third coordinate system C to the second coordinate system F using the third transformation matrix. Optionally, step S1071 may include the following process: when the end of the robotic arm reaches M precision-determined positions (for example, in this process, the end of the robotic arm may be manually moved to the precision-determined position by a staff member), for each of the M precision-determined positions, determine the first actual coordinate of the endpoint of the end of the robotic arm at the precision-determined position in the first coordinate system J (i.e., the coordinate system of the base of the robotic arm), and obtain the first actual coordinates corresponding to the M precision-determined positions, where M≥1 and M is an integer; for each of the M precision-determined positions, calculate using the third transformation matrix M3 to obtain the end of the robotic arm controlled by the third transformation matrix M3 to move to the position. When the precision position is determined, the endpoint of the end of the manipulator is in the first predicted coordinates in the first coordinate system J (i.e., the coordinate system of the base of the manipulator), and a total of M first predicted coordinates corresponding to the precision determined positions are obtained; for each of the M precision determined positions, the first distance error between the first actual coordinate corresponding to the precision determined position and the first predicted coordinate is calculated (optionally, the straight-line distance between the two coordinates can be directly calculated as the first distance error), and a total of M first distance errors are obtained; the first error data is determined based on the M first distance errors (optionally, when M=1, the first distance error can be used as the first error data, and when M>1, the average value of the M first distance errors can be calculated as the first error data). In this way, the first error data can be reliably calculated. It should be understood that this is only an optional embodiment of the present application and is not a limitation to the present application.

[0146] Sub-step S1072: Determine second error data obtained by transforming the third coordinate system into the second coordinate system using a fourth transformation matrix.

[0147] The second error data can be used to indicate the accuracy of converting the third coordinate system C to the second coordinate system F using the fourth conversion matrix. The larger the first error data, the lower the accuracy of converting the third coordinate system C to the second coordinate system F using the third conversion matrix. Optionally, based on the M precision-determined positions and the M first actual coordinates in the aforementioned step S1071, step S1072 may include the following process: for each of the M precision-determined positions, use the fourth transformation matrix M4 to perform calculations to obtain the second predicted coordinates of the end point of the end of the robotic arm in the first coordinate system J (i.e., the coordinate system of the base of the robotic arm) when the end of the robotic arm is controlled by the fourth transformation matrix M4 to move to the precision-determined position, and obtain a total of M second predicted coordinates corresponding to the precision-determined positions; for each of the M precision-determined positions, calculate the second distance error between the first actual coordinate corresponding to the precision-determined position and the second predicted coordinate (optionally, the straight-line distance between the two coordinates can be directly calculated as the second distance error), and obtain a total of M second distance errors; determine second error data based on the M second distance errors (optionally, when M=1, the second distance error can be used as the second error data, and when M>1, the average value of the M second distance errors can be calculated as the second error data). In this way, the second error data can be reliably calculated. It should be understood that this is merely an optional embodiment of the present application and is not intended to limit the present application.

[0148] Sub-step S1073: If the first error data is smaller than the second error data, the third conversion matrix is ​​determined as the calibration result; or if the first error data is larger than the second error data, the fourth conversion matrix is ​​determined as the calibration result.

[0149] If the first error data is smaller than the second error data, it means that the accuracy of the third conversion matrix M3 is greater than the accuracy of the fourth conversion matrix M4. Taking the third conversion matrix M3 as the calibration result can better meet the actual needs of accurately controlling the robotic arm to drive the microscope to align with the target area, so the third conversion matrix M3 is determined as the calibration result; and if the first error data is greater than the second error data, it means that the accuracy of the fourth conversion matrix M4 is greater than the accuracy of the third conversion matrix M3. Taking the fourth conversion matrix M4 as the calibration result can better meet the actual needs of accurately controlling the robotic arm to drive the microscope to align with the target area, so the fourth conversion matrix M4 is determined as the calibration result.

[0150] Optionally, when M = 1, the magnitude relationship between the first distance error and the second distance error is determined. If the first distance error is greater than the second distance error, the first error data is greater than the second error data; if the first distance error is less than the second distance error, the first error data is less than the second error data. When M > 1, the magnitude relationship between the average of M first distance errors and the average of M second distance errors is determined. If the average of the first distance error is greater than the average of the second distance error, the first error data is greater than the second error data; if the average of the first distance error is less than the average of the second distance error, the first error data is less than the second error data. It should be understood that this is merely an optional embodiment of the present application and is not intended to limit the present application.

[0151] When the first error data is equal to the second error data, the third conversion matrix M3 or the fourth conversion matrix M4 may be determined as the calibration result.

[0152] Based on the method in the above steps S1071 to S1073, the calibration result of the conversion relationship between the third coordinate system (i.e., the coordinate system of the microscope) and the second coordinate system (i.e., the coordinate system of the end of the robotic arm) can be accurately and reliably obtained.

[0153] It can be understood that the above embodiments are merely some illustrative embodiments of the embodiments of the present application, and do not impose any limitation on the calibration method in the embodiments of the present application.

[0154] Based on the same inventive concept as the aforementioned calibration method, according to another aspect of the present application, referring to Figure 6 A block diagram of a calibration device 100 is provided, which includes:

[0155] A first determining module 101 is configured to determine two first transformation matrices for transforming a second coordinate system into a first coordinate system when the end of the robotic arm is located at a first position and a second position, respectively, wherein the first coordinate system is a coordinate system of a base of the robotic arm and the second coordinate system is a coordinate system of the end of the robotic arm;

[0156] a second determining module 102, configured to respectively determine two second transformation matrices for transforming a fourth coordinate system into a third coordinate system when the microscope is located at a third position and a fourth position, wherein the third coordinate system is a coordinate system of the microscope, and the fourth coordinate system is a coordinate system of a marking plate, the marking plate being disposed outside the manipulator and the microscope, the microscope being fixedly connected to an end of the manipulator, and when the end of the manipulator moves from the first position to the second position, the microscope follows the end of the manipulator to move from the third position to the fourth position;

[0157] A third determining module 103 is configured to determine a third transformation matrix for transforming the third coordinate system into the second coordinate system based on the two first transformation matrices and the two second transformation matrices;

[0158] a fourth determining module 104, configured to respectively determine N first coordinates of the origin of the first coordinate system in the second coordinate system when the end of the robotic arm is located at N fifth positions in the trajectory moving from the first position to the second position, and determine first point cloud data based on the N first coordinates, where N is an integer.

[0159] a fifth determining module 105, configured to respectively determine N second coordinates of the first marking point on the marking plate in the third coordinate system when the microscope is located at N sixth positions in the trajectory moving from the third position to the fourth position, and determine second point cloud data based on the N second coordinates, wherein when the end of the robotic arm is located at different fifth positions, the microscope is located at different sixth positions;

[0160] A sixth determining module 106 is configured to determine a fourth transformation matrix for transforming the third coordinate system into the second coordinate system based on the first point cloud data and the second point cloud data;

[0161] The seventh determining module 107 is configured to determine a calibration result of calibrating the conversion relationship between the third coordinate system and the second coordinate system based on the third conversion matrix and the fourth conversion matrix.

[0162] In some optional embodiments, the first determination module 101 is specifically used to: obtain the rotation angles of each joint of the robotic arm when the end of the robotic arm is located at the first position and the second position, respectively; calculate the kinematic solution of the robotic arm based on the rotation angles of each joint when the end of the robotic arm is located at the first position, and determine the first transformation matrix when the end of the robotic arm is located at the first position; calculate the kinematic solution of the robotic arm based on the rotation angles of each joint when the end of the robotic arm is located at the second position, and determine the first transformation matrix when the end of the robotic arm is located at the second position.

[0163] In some optional embodiments, the second determination module 102 is specifically used to: obtain a first image obtained by photographing the first marking point on the marking plate when the microscope is located at the third position, determine the position coordinates of the first marking point on the marking plate in the third coordinate system when the microscope is located at the third position based on the first image, and determine the second transformation matrix of the microscope when it is located at the third position according to the position coordinates; obtain a second image obtained by photographing the first marking point on the marking plate when the microscope is located at the fourth position, determine the position coordinates of the first marking point on the marking plate in the third coordinate system when the microscope is located at the fourth position based on the second image, and determine the second transformation matrix of the microscope when it is located at the fourth position according to the position coordinates.

[0164] In some optional embodiments, the third determining module 103 is specifically configured to determine the third conversion matrix using the following formula:

[0165] M 2_2 -1 M 2_1 *M3=M3*M 1_2 *M 1_1 -1

[0166] Among them, M 1_1 The first transformation matrix used to characterize the end of the robotic arm when it is at the first position, M 1_2 The first transformation matrix used to characterize the end of the robotic arm when it is at the second position, M 2_1 The second transformation matrix used to characterize the microscope when it is located at the third position, M 2_2 M is used to characterize the second transformation matrix when the microscope is located at the fourth position, and M3 is used to characterize the third transformation matrix.

[0167] In some optional embodiments, the sixth determination module 106 is specifically used to: perform an affine transformation on the first point cloud data and the second point cloud data based on a one-to-one correspondence between the N first coordinates in the first point cloud data and the N second coordinates in the second point cloud data to determine a fourth transformation matrix.

[0168] In some optional embodiments, the seventh determination module 107 is specifically used to: determine the calibration result of calibrating the conversion relationship between the third coordinate system and the second coordinate system based on the third conversion matrix and the fourth conversion matrix, including: determining the first error data of converting the third coordinate system to the second coordinate system using the third conversion matrix; determining the second error data of converting the third coordinate system to the second coordinate system using the fourth conversion matrix; if the first error data is smaller than the second error data, determining the third conversion matrix as the calibration result; or, if the first error data is larger than the second error data, determining the fourth conversion matrix as the calibration result.

[0169] The calibration device 100 in the embodiment of the present application corresponds to the calibration method in the aforementioned embodiments. The relevant contents of the calibration device 100 can be understood with reference to the aforementioned calibration method and will not be described in detail here.

[0170] According to another aspect of the embodiment of the present application, referring to Figure 7 In the block diagram, the embodiment of the present application provides an electronic device 700. Figure 7 As shown, the electronic device 700 includes a memory 701 and a processor 702, the memory 701 is used to store a computer executable program, and the processor 702 is used to run the computer executable program to implement the calibration method described in any one of the above embodiments.

[0171] According to another aspect of the embodiments of the present application, a computer storage medium is provided, wherein the computer storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the calibration method according to any of the above items.

[0172] According to another aspect of the embodiments of the present application, a computer program product is provided, which includes a computer program, wherein the computer program implements any of the calibration methods described above when executed by a processor.

[0173] As for the calibration device, electronic device, computer storage medium, and computer program product embodiments, since they are basically similar to the above-mentioned calibration method embodiments, the description is relatively simple. For relevant details, please refer to the partial description of the above-mentioned calibration method embodiments, and no further details will be given here.

[0174] It can be understood that the above embodiments are merely experimental examples of some examples in the embodiments of the present application, and do not constitute any limitation on the calibration methods, devices, electronic devices, computer storage media, and computer program products in the embodiments of the present application.

[0175] It should be noted that the term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0176] It should be noted that the modifications of "one" and "multiple" mentioned in this application are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present application, and not to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A calibration method, characterized in that: include: Determining two first transformation matrices for transforming a second coordinate system into a first coordinate system when the end of the robotic arm is located at a first position and a second position, respectively, wherein the first coordinate system is a coordinate system of a base of the robotic arm and the second coordinate system is a coordinate system of the end of the robotic arm; determining two second transformation matrices for transforming the fourth coordinate system into the third coordinate system when the microscope is located at a third position and a fourth position, respectively, wherein the third coordinate system is a coordinate system of the microscope, and the fourth coordinate system is a coordinate system of a marking plate, the marking plate is disposed outside the robotic arm and the microscope, the microscope is fixedly connected to an end of the robotic arm, and when the end of the robotic arm moves from the first position to the second position, the microscope follows the end of the robotic arm in moving from the third position to the fourth position; determining a third transformation matrix for transforming the third coordinate system into the second coordinate system based on the two first transformation matrices and the two second transformation matrices; Determine N first coordinates of the origin of the first coordinate system in the second coordinate system when the end of the robotic arm is located at N fifth positions in the trajectory moving from the first position to the second position, and determine first point cloud data based on the N first coordinates, where N ≥ 2 and N is an integer; determining N second coordinates of the first marking point on the marking plate in the third coordinate system when the microscope is at N sixth positions in the trajectory moving from the third position to the fourth position, respectively, and determining second point cloud data based on the N second coordinates, wherein when the end of the robotic arm is at different fifth positions, the microscope is at different sixth positions; determining, based on the first point cloud data and the second point cloud data, a fourth transformation matrix for transforming the third coordinate system into the second coordinate system; A calibration result of calibrating the conversion relationship between the third coordinate system and the second coordinate system is determined based on the third conversion matrix and the fourth conversion matrix.

2. The method according to claim 1, characterized in that The method of determining two first transformation matrices for transforming the second coordinate system into the first coordinate system when the end of the robotic arm is located at the first position and the second position, respectively, includes: respectively obtaining the rotation angles of the joints of the robotic arm when the end of the robotic arm is located at the first position and the second position; Finding a kinematics forward solution for the robotic arm based on the rotation angles of the joints when the end of the robotic arm is located at the first position, and determining a first transformation matrix when the end of the robotic arm is located at the first position; According to the rotation angles of the joints when the end of the robotic arm is located at the second position, a kinematics forward solution is obtained for the robotic arm to determine a first transformation matrix when the end of the robotic arm is located at the second position.

3. The method according to claim 1, characterized in that The determining of two second transformation matrices for transforming the fourth coordinate system into the third coordinate system when the microscope is located at the third position and the fourth position, respectively, comprises: acquiring a first image obtained by photographing a first marking point on the marking plate when the microscope is at the third position, determining, based on the first image, position coordinates of the first marking point on the marking plate in a third coordinate system when the microscope is at the third position, and determining, based on the position coordinates, a second transformation matrix when the microscope is at the third position; Acquire a second image obtained by photographing the first marking point on the marking plate when the microscope is at the fourth position, determine the position coordinates of the first marking point on the marking plate in the third coordinate system when the microscope is at the fourth position based on the second image, and determine a second transformation matrix of the microscope when it is at the fourth position based on the position coordinates.

4. The method according to any one of claims 1 to 3, characterized in that The determining, based on the two first transformation matrices and the two second transformation matrices, a third transformation matrix for transforming the third coordinate system into the second coordinate system comprises: The third transformation matrix is ​​determined by the following formula: M 2_2 -1 M 2_1 *M3=M3*M 1_2 *M 1_1 -1 Among them, M 1_1 The first transformation matrix used to characterize the end of the robotic arm when it is at the first position, M 1_2 The first transformation matrix used to characterize the end of the robotic arm when it is at the second position, M 2_1 The second transformation matrix used to characterize the microscope when it is located at the third position, M 2_2 M is used to characterize the second transformation matrix when the microscope is located at the fourth position, and M3 is used to characterize the third transformation matrix.

5. The method according to any one of claims 1 to 3, characterized in that The determining, based on the first point cloud data and the second point cloud data, a fourth transformation matrix for transforming the third coordinate system into the second coordinate system comprises: Based on a one-to-one correspondence between the N first coordinates in the first point cloud data and the N second coordinates in the second point cloud data, an affine transformation is performed on the first point cloud data and the second point cloud data to determine a fourth transformation matrix.

6. The method according to any one of claims 1 to 3, characterized in that The determining, based on the third conversion matrix and the fourth conversion matrix, a calibration result of calibrating the conversion relationship between the third coordinate system and the second coordinate system includes: determining first error data for transforming the third coordinate system into the second coordinate system using a third transformation matrix; determining second error data for transforming the third coordinate system into the second coordinate system using a fourth transformation matrix; If the first error data is smaller than the second error data, the third conversion matrix is ​​determined as the calibration result; or if the first error data is larger than the second error data, the fourth conversion matrix is ​​determined as the calibration result.

7. A calibration device, characterized in that: include: a first determining module, configured to respectively determine two first transformation matrices for transforming a second coordinate system into a first coordinate system when the end of the robotic arm is located at a first position and a second position, wherein the first coordinate system is a coordinate system of a base of the robotic arm, and the second coordinate system is a coordinate system of the end of the robotic arm; a second determination module, for respectively determining two second transformation matrices for transforming a fourth coordinate system into a third coordinate system when the microscope is located at a third position and a fourth position, wherein the third coordinate system is a coordinate system of the microscope, and the fourth coordinate system is a coordinate system of a marking plate, the marking plate is disposed outside the manipulator and the microscope, the microscope is fixedly connected to an end of the manipulator, and when the end of the manipulator moves from the first position to the second position, the microscope follows the end of the manipulator to move from the third position to the fourth position; a third determining module, configured to determine a third conversion matrix for converting the third coordinate system into the second coordinate system based on the two first conversion matrices and the two second conversion matrices; a fourth determining module, configured to respectively determine N first coordinates of the origin of the first coordinate system in the second coordinate system when the end of the robotic arm is located at N fifth positions in the trajectory moving from the first position to the second position, and determine first point cloud data based on the N first coordinates, where N is an integer. a fifth determining module, configured to respectively determine N second coordinates of the first marking point on the marking plate in the third coordinate system when the microscope is located at N sixth positions in the trajectory moving from the third position to the fourth position, and determine second point cloud data based on the N second coordinates, wherein when the end of the robotic arm is located at different fifth positions, the microscope is located at different sixth positions; a sixth determining module, configured to determine, based on the first point cloud data and the second point cloud data, a fourth transformation matrix for transforming the third coordinate system into the second coordinate system; The seventh determining module is configured to determine a calibration result of calibrating the conversion relationship between the third coordinate system and the second coordinate system based on the third conversion matrix and the fourth conversion matrix.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory is used to store a computer executable program, and the processor is used to run the computer executable program to implement the calibration method according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that The computer storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the calibration method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises a computer program, wherein when the computer program is executed by a processor, the computer program implements the calibration method according to any one of claims 1 to 6.

Images