Surgical robotic registration method, apparatus, device, and medium

Abstract

The application discloses a surgical robot registration method, device, equipment and medium. The method comprises the following steps: determining a mechanical arm base coordinate system constructed based on a mechanical arm base center of a surgical mechanical arm and a three-dimensional image coordinate system constructed based on a scanning center of a C-shaped arm; determining a coarse registration matrix between the mechanical arm base coordinate system and the three-dimensional image coordinate system; determining a target mechanical arm shape and position of the surgical mechanical arm, collecting a target image through the C-shaped arm; determining a first coordinate set of a first optical target set in the three-dimensional image coordinate system according to target target information, and determining a second coordinate set of the first optical target set in the mechanical arm base coordinate system under the target mechanical arm shape and position through a target joint motion model; determining a registration error matrix between the first coordinate set and the second coordinate set through a registration error equation, and determining a fine registration matrix according to the registration error matrix and the coarse registration matrix to perform fine registration. The application can improve the efficiency and accuracy of surgical robot registration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer application, and in particular to a surgical robot registration method, device, equipment and medium. BACKGROUND

[0002] In the image-guided surgical robot application, in order to ensure the operation accuracy of the surgical robot, the registration and calibration between the surgical robot operating object and the surgical robot are crucial.

[0003] In the related art, the technical personnel usually performs registration calculation on the structure data of the surgical robot and the pre-obtained image data, and manually registers the two through the registration data obtained by calculation, but the registration efficiency and accuracy are often difficult to meet the user's demand. SUMMARY

[0004] The present application provides a surgical robot registration method, device, equipment and medium to solve the technical problem of low registration efficiency and accuracy of the surgical robot.

[0005] According to an aspect of the present application, a surgical robot registration method is provided, wherein the method comprises:

[0006] In the case that the surgical robot arm and the C-arm are rigidly connected, a mechanical arm base coordinate system based on the mechanical arm base center of the surgical robot arm is determined, and a three-dimensional image coordinate system based on the scanning center of the C-arm is determined;

[0007] A coarse registration matrix between the mechanical arm base coordinate system and the three-dimensional image coordinate system is determined;

[0008] A target mechanical arm pose of the surgical robot arm is determined, and a target image is collected by the C-arm, wherein the target image includes target target information of a first optical target set of a mechanical arm end mounted on the surgical robot arm by a connecting piece in the target mechanical arm pose;

[0009] A first coordinate set of the first optical target set in the three-dimensional image coordinate system is determined according to the target target information, and a second coordinate set of the first optical target set in the mechanical arm base coordinate system in the target mechanical arm pose is determined through a target joint motion model;

[0010] A registration error matrix between the first coordinate set and the second coordinate set is determined through a registration error equation, an accurate registration matrix is determined according to the registration error matrix and the coarse registration matrix, and the three-dimensional image coordinate system and the mechanical arm base coordinate system are accurately registered based on the accurate registration matrix.

[0011] According to another aspect of the present application, there is provided a registration device of a surgical robot, wherein the device comprises:

[0012] a rigid connection module configured to determine a robot base coordinate system constructed based on a robot base center of a surgical robot and a three-dimensional image coordinate system constructed based on a scan center of a C-arm when the surgical robot and the C-arm are rigidly connected;

[0013] a coarse conversion module configured to determine a coarse registration matrix between the robot base coordinate system and the three-dimensional image coordinate system;

[0014] an image acquisition module configured to determine a target robot pose of the surgical robot and acquire a target image through the C-arm, wherein the target image comprises target marker information of a first set of optical markers mounted on a robot end of the surgical robot through a connecting member at the target robot pose;

[0015] a coordinate set determination module configured to determine a first coordinate set of the first set of optical markers in the three-dimensional image coordinate system according to the target marker information, and determine a second coordinate set of the first set of optical markers in the robot base coordinate system at the target robot pose through a target joint motion model;

[0016] a fine registration module configured to determine a registration error matrix between the first coordinate set and the second coordinate set through a registration error equation, determine a fine registration matrix according to the registration error matrix and the coarse registration matrix, and perform fine registration of the three-dimensional image coordinate system and the robot base coordinate system based on the fine registration matrix.

[0017] According to another aspect of the present application, there is provided an electronic device, comprising:

[0018] at least one processor; and

[0019] a memory connected with the at least one processor; wherein,

[0020] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the registration method of the surgical robot according to any one of the embodiments of the present application.

[0021] According to another aspect of the present application, there is provided a computer readable storage medium storing computer instructions for enabling a processor to perform the registration method of the surgical robot according to any one of the embodiments of the present application when executed by the processor.

[0022] The technical scheme of the embodiment of the application is characterized in that, in the case that a surgical robot arm and a C-arm are rigidly connected, a robot arm base coordinate system is determined based on a center of a robot arm base of the surgical robot arm, and a three-dimensional image coordinate system is determined based on a scanning center of the C-arm; a coarse registration matrix between the robot arm base coordinate system and the three-dimensional image coordinate system is determined; a target robot arm pose of the surgical robot arm is determined, and a target image is acquired by the C-arm, wherein the target image comprises target target information of a first optical target set of a robot arm end mounted on the surgical robot arm by a connecting piece in the target robot arm pose; a first coordinate set of the first optical target set in the three-dimensional image coordinate system is determined according to the target target information, a second coordinate set of the first optical target set in the robot arm base coordinate system in the target robot arm pose is determined through a target joint motion model; a registration error matrix between the first coordinate set and the second coordinate set is determined through a registration error equation, a fine registration matrix is determined according to the registration error matrix and the coarse registration matrix, and the three-dimensional image coordinate system and the robot arm base coordinate system are fine registered based on the fine registration matrix, so that automatic registration of the surgical robot is realized, and the efficiency and accuracy of the registration of the surgical robot are improved.

[0023] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the application, nor is it used to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0025] Figure 1 is a flow chart of a registration method of a surgical robot according to an embodiment of the application;

[0026] Figure 2 is a scene diagram of a surgical robot arm and a C-arm rigidly connected to realize the embodiment of the application;

[0027] Figure 3 is a flow chart of a registration method of a surgical robot according to an embodiment of the application;

[0028] Figure 4 is a whole flow chart of a registration method of a surgical robot according to an embodiment of the application;

[0029] Figure 5 is a conversion relationship diagram between coordinate systems according to an embodiment of the present application;

[0030] Figure 6 is a structural schematic diagram of a registration device of a surgical robot according to the third embodiment of the present application;

[0031] Figure 7 is a structural schematic diagram of an electronic device for implementing a registration method of a surgical robot according to the present application. DETAILED DESCRIPTION

[0032] In order to make the personnel in the technical field better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0033] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0034] Embodiment one

[0035] Figure 1 A flowchart of a surgical robot registration method is provided for the first embodiment of the present application. The present embodiment can be applicable to the calibration between a surgical robot and an operating object of the surgical robot by precise registration of coordinate systems of a surgical mechanical arm and a C-shaped arm. The method can be executed by a surgical robot registration device, which can be realized in the form of hardware and / or software, and can be configured in a computer. As shown in the figure, the method comprises: Figure 1

[0036] ​S110, determine a mechanical arm base coordinate system constructed based on a mechanical arm base center of the surgical robot and a three-dimensional image coordinate system constructed based on a scanning center of the C-arm, in a case where the surgical robot is rigidly connected with the C-arm.

[0037] Figure 2 is a scene diagram of rigid connection of a surgical robot and a C-arm in an embodiment of the application. As shown in the figure, A represents the C-arm, B represents the mechanical arm base, C represents the mechanical arm end, and D represents the optical positioner, on which a binocular camera is mounted. In the embodiment of the application, the coordinate system constructed based on the scanning center of the C-arm is taken as the three-dimensional image coordinate system, the coordinate system constructed based on the mechanical arm base center is taken as the mechanical arm base coordinate system, the coordinate system constructed based on the mechanical arm end is taken as the mechanical arm end coordinate system, and the midpoint of the line connecting the optical centers of the binocular camera of the optical positioner is taken as the vision coordinate system. Figure 2

[0038] In the embodiment of the application, the mechanical arm base center and the scanning center of the C-arm are determined based on the hardware structure parameters of the surgical robot, which are not specifically limited here.

[0039] S120, determine a coarse registration matrix between the mechanical arm base coordinate system and the three-dimensional image coordinate system.

[0040] In the embodiment of the application, the coarse registration matrix can be directly measured based on common measurement methods, which are not specifically limited here.

[0041] S130, determine a target mechanical arm pose of the surgical robot, and acquire a target image by the C-arm, wherein the target image includes target target information of a first optical target set mounted on the mechanical arm end of the surgical robot by the connecting piece in the target mechanical arm pose.

[0042] The target mechanical arm pose can be understood as a kind of pose of the surgical robot. In the embodiment of the application, the target mechanical arm pose can be a random kind of mechanical arm pose.

[0043] The target image can be understood as an image of the surgical robot in the target mechanical arm pose.

[0044] The first optical target set can include a plurality of first optical targets. The first optical target set can be mounted on the mechanical arm end of the surgical robot based on the connecting piece. The distance of each first optical target from the mechanical arm end can be different or the same.

[0045] The target target information can include position information of the first optical target set. ​

[0046] S140, determining a first coordinate set of the first optical target set in a three-dimensional image coordinate system according to the target target information, determining a second coordinate set of the first optical target set in the mechanical arm base coordinate system in the target mechanical arm pose through a target kinematic model.

[0047] Optionally, the kinematic model can be a D-H (Denavit-Hartenberg) model, which can be used to describe parameters of kinematic characteristics of the robot arm.

[0048] In the embodiment of the present application, the first coordinate set of the first optical target set in the three-dimensional image coordinate system can be directly determined based on the target target information in the target image.

[0049] The target kinematic model can be used for data conversion calculation between the mechanical arm end coordinate system and the mechanical arm base coordinate system.

[0050] Optionally, the second coordinate set of the first optical target set in the mechanical arm base coordinate system in the target mechanical arm pose is determined through the target kinematic model, comprising:

[0051] determining a mechanical arm end coordinate system constructed based on the mechanical arm end of the surgical mechanical arm and first connector information of a connector connecting the mechanical arm end and the first optical target set;

[0052] determining an end coordinate set of the first optical target set in the mechanical arm end coordinate system based on the first connector information;

[0053] converting the input end coordinate set through a target kinematic model to obtain the second coordinate set of the first optical target set in the mechanical arm base coordinate system.

[0054] In the embodiment of the present application, the connector can include a plurality of connection branches, each connection branch being used to connect one of the first optical targets to the mechanical arm end. The length of each connection branch can be different or the same. The first connector information can include the length of the connection branch of the connector corresponding to each first optical target, that is, the distance between each first optical target and the mechanical arm end, that is, the distance between each first optical target and the origin of the mechanical arm end coordinate system.

[0055] The end coordinate set can be understood as the coordinate set of the first optical target set in the mechanical arm end coordinate system.

[0056] S150, determine a registration error matrix between the first coordinate set and the second coordinate set through a registration error equation, determine a fine registration matrix according to the registration error matrix and the coarse registration matrix, and fine register the three-dimensional image coordinate system and the robot base coordinate system based on the fine registration matrix.

[0057] The registration error equation can be used to calculate the registration error between the robot base coordinate system and the three-dimensional image coordinate system R. Optionally, the registration error equation can be as shown in the following formula:

[0058]

[0059] The registration error matrix can be represented as ΔR; Δα, Δβ, Δγ, Δx, Δy, Δz can represent error parameters of the registration error equation; R x The rotation operator around the x-axis can be represented as R y The rotation operator around the y-axis can be represented as R z The rotation operator around the z-axis can be represented as R x The translation operator around the X-axis can be represented as D y The translation operator around the y-axis can be represented as D z The translation operator around the z-axis can be represented as D.

[0060] Specifically, the calculation formula of determining the fine registration matrix according to the registration error matrix and the coarse registration matrix can be as shown in the following formula:

[0061]

[0062] The fine registration matrix can be represented as R The coarse registration matrix can be represented as R The registration error matrix can be represented as ΔR.

[0063] Optionally, before the registration error matrix between the first coordinate set and the second coordinate set is determined through the registration error equation, the method further includes:

[0064] controlling the surgical robot arm to move to each first robot pose in the preset first robot pose sequence in turn and acquiring a pose image sequence through the C-arm, wherein the pose image includes first target information of a first optical target set of a robot end installed on the surgical robot arm in the first robot pose;

[0065] Optionally, before the surgical robot arm is controlled to move to each first robot pose in the preset first robot pose sequence in turn, the method further includes:

[0066] ​​determine the first mechanical arm pose sequence based on the preset imaging area of the C-arm and the preset operating space of the surgical mechanical arm, determine the image coordinate set sequence of the first optical target set in the three-dimensional image coordinate system based on the first target information in each of the shape and position image sequence,

[0067] determine the base coordinate set sequence of the first optical target set in the mechanical arm base coordinate system under the first mechanical arm pose sequence based on the target joint motion model;

[0068] adjust the error parameters of the pre-constructed initial error equation according to the image coordinate set sequence and the base coordinate set sequence, and obtain a registration error equation.

[0069] The shape and position image sequence includes a plurality of shape and position images.

[0070] The technical scheme of the embodiment of the application determines the mechanical arm base coordinate system constructed based on the center of the mechanical arm base of the surgical mechanical arm and the three-dimensional image coordinate system constructed based on the scanning center of the C-arm under the condition that the surgical mechanical arm of the surgical robot is rigidly connected with the C-arm, determines the coarse registration matrix between the mechanical arm base coordinate system and the three-dimensional image coordinate system, determines the target mechanical arm pose of the surgical mechanical arm, and acquires a target image by the C-arm, wherein the target image includes target target information of a first optical target set of the mechanical arm end mounted on the surgical mechanical arm by the connecting piece under the target mechanical arm pose, determines a first coordinate set of the first optical target set in the three-dimensional image coordinate system according to the target target information, determines a second coordinate set of the first optical target set in the mechanical arm base coordinate system under the target mechanical arm pose by a target joint motion model, determines a registration error matrix between the first coordinate set and the second coordinate set by a registration error equation, determines a fine registration matrix according to the registration error matrix and the coarse registration matrix, and performs fine registration of the three-dimensional image coordinate system and the mechanical arm base coordinate system based on the fine registration matrix, thereby realizing automatic registration of the surgical robot and improving the efficiency and accuracy of the registration of the surgical robot.

[0071] Embodiment Two

[0072] Figure 3 A flowchart of a surgical robot registration method provided for the second embodiment of the application, and the embodiment is additionally provided for determining the second coordinate set of the first optical target set in the mechanical arm base coordinate system under the target mechanical arm pose by a target joint motion model in the above-mentioned embodiment.

[0073] As shown in Figure 3 , the method comprises:

[0074] S210, determine an optical positioning instrument and a visual coordinate system constructed based on a midpoint of an optical center line of a binocular camera based on the optical positioning instrument.

[0075] The optical positioning instrument can be used to identify position information of an optical target, and a surgical mechanical arm, a C-arm, and a placement position of the optical positioning instrument can be as shown in a scene diagram. Figure 2

[0076] S220, determine a second optical target set connected to the center of the mechanical arm base based on the connecting member, and determine the target joint motion model based on the mechanical arm base coordinate system, the mechanical arm end coordinate system, the visual coordinate system, and the second optical target set.

[0077] The second optical target set can include a plurality of second optical targets. The second optical target set can be installed at the center of the mechanical arm base based on the connecting member. The distance of each second optical target from the center of the mechanical arm base can be different or the same.

[0078] Optionally, the determination of the target joint motion model based on the mechanical arm base coordinate system, the mechanical arm end coordinate system, the visual coordinate system, and the second optical target set comprises:

[0079] determining a homogeneous transformation matrix based on the mechanical arm base coordinate system, the mechanical arm end coordinate system, and the second optical target set;

[0080] determining the target joint motion model based on the homogeneous transformation matrix and a preliminary joint motion model constructed in advance.

[0081] Specifically, a mechanical arm base coordinate system is constructed based on the center of the mechanical arm base of the surgical mechanical arm and is denoted as {R}, a connecting member is installed at the center of the mechanical arm base, and a plurality of spherical second optical targets are fixed on the connecting member; the distance of each second optical target to the origin of the mechanical arm base coordinate system is determined based on the length information of the connecting branch of the connecting member, and the three-dimensional coordinate point set of each second optical target in the mechanical arm base coordinate system is denoted as R P, a visual coordinate system is constructed based on the midpoint of the optical center line between the binocular cameras of the optical positioning instrument as the origin and is denoted as {C}, the position information of each second optical target ball in the visual coordinate system is obtained, and the three-dimensional coordinate point set C P, the nonlinear optimization iteration solution of R and t is obtained through a target function constructed in advance based on the rigid transformation principle, and the homogeneous transformation matrix between the visual coordinate system and the mechanical arm base coordinate system corresponding to the minimum value of the target function is obtained The above target function and the determined homogeneous transformation matrix can be as follows:

[0082] ​

[0083] wherein R can represent a rotation matrix of a vision coordinate system to a robot base coordinate system, and t can represent a translation vector of the vision coordinate system to the robot base coordinate system, R P i denotes a three-dimensional coordinate of the i-th second optical target ball in the vision coordinate system, C P i denotes a three-dimensional coordinate of the i-th second optical target ball in the robot base coordinate system, may represent a homogeneous transformation matrix.

[0084] Optionally, the determining the target joint motion model based on the homogeneous transformation matrix and a preliminary joint motion model constructed in advance comprises:

[0085] determining a plurality of groups of preset joint parameters corresponding to the preliminary joint motion model, and determining an intermediate joint motion model corresponding to the preliminary joint motion model based on the preset joint parameters;

[0086] controlling the surgical robot arm to sequentially move to each second robot pose in a preset second robot pose sequence and collect a sequence of encoded positions of the robot end through an encoder;

[0087] determining a sequence of theoretical positions of the sequence of encoded positions in the robot base coordinate system through the preliminary joint motion model;

[0088] determining a sequence of actual positions of the sequence of encoded positions in the robot base coordinate system through the optical positioner and the homogeneous transformation matrix;

[0089] determining a theoretical-actual error based on the sequence of theoretical positions and the sequence of actual positions, adjusting model parameters of the intermediate joint motion model based on the theoretical-actual error, and obtaining the target joint motion model.

[0090] The preset joint parameters can be understood as model parameters corresponding to the joint motion model and related to the joints of the robot arm.

[0091] The second robot pose sequence can be a sequence comprising a plurality of robot poses determined based on a preset imaging region of the C-arm and a preset operating space of the surgical robot arm. The second robot pose sequence and the first robot pose sequence can be different or the same.

[0092] The coded position sequence can include a plurality of coded positions, the coded position being a position of the robot end effector in the second robot configuration, and the coded position sequence can be obtained based on an encoder mounted on the robot end effector. The theoretical position sequence can be understood as a position sequence determined by theoretical calculation through a kinematic model. The actual position sequence can be understood as a position sequence determined by actual positioning through the optical positioner.

[0093] The theoretical actual error can be understood as an error between the theoretical position sequence and the actual position sequence.

[0094] Optionally, the determining, by the optical positioner and the homogeneous transformation matrix, the actual position sequence of the coded position sequence in the robot base coordinate system, includes:

[0095] For the second robot configuration sequence, determining, by the optical positioner, a visual position sequence of the second optical target set in the visual coordinate system;

[0096] Determining, based on the homogeneous transformation matrix and the visual position sequence, the actual position sequence of the coded position sequence in the robot base coordinate system.

[0097] Optionally, the determining, based on the preset joint parameters, the intermediate kinematic model corresponding to the preliminary kinematic model, includes:

[0098] For each set of preset joint parameters, determining, by the preliminary kinematic model, a theoretical transformation matrix between the robot base coordinate system and the robot end effector coordinate system under the preset joint parameters;

[0099] Adjusting model parameters of the preliminary kinematic model based on a plurality of theoretical transformation matrices to obtain an intermediate kinematic model.

[0100] Specifically, the theoretical transformation matrix can be calculated as follows:

[0101] T i i-1 = R x (α i-1 )D x (a i-1 )R z (θ i )D z (d i )

[0102] wherein, T i i-1 = can represent a theoretical transformation matrix; α i-1 , (a i-1 ), θ i), d i may represent a preset joint parameter; R x may represent a rotation operator around the x-axis; R z may represent a rotation operator around the z-axis; D x may represent a translation operator around the X-axis; D z may represent a translation operator around the z-axis.

[0103] S230, in the case that the surgical robot arm and the C-arm are rigidly connected, determining a robot base coordinate system constructed based on a robot base center of the surgical robot arm and a three-dimensional image coordinate system constructed based on a scanning center of the C-arm.

[0104] S240, determining a coarse registration matrix between the robot base coordinate system and the three-dimensional image coordinate system.

[0105] S250, determining a target robot pose of the surgical robot arm, and acquiring a target image by the C-arm.

[0106] S260, determining a first coordinate set of the first optical target set in the three-dimensional image coordinate system according to the target target information, and determining a second coordinate set of the first optical target set in the robot base coordinate system in the target robot pose by the target joint motion model.

[0107] S270, determining a registration error matrix between the first coordinate set and the second coordinate set by a registration error equation, determining a fine registration matrix according to the registration error matrix and the coarse registration matrix, and performing fine registration on the three-dimensional image coordinate system and the robot base coordinate system based on the fine registration matrix.

[0108] The technical scheme of the embodiment of the application determines a visual coordinate system constructed based on a midpoint of a line connecting optical centers of binocular cameras of an optical positioner and the optical positioner, determines a second optical target set connected to the robot base center by a connecting piece, and determines the target joint motion model based on the robot base coordinate system, the robot end coordinate system, the visual coordinate system, and the second optical target set. The application adopts the visual coordinate system constructed based on the midpoint of the line connecting the optical centers of the binocular cameras of the optical positioner as an intermediate medium, constructs the target joint motion model having the conversion relationship function between the robot base coordinate system and the robot end coordinate system, and can improve the model precision of the constructed target joint motion model.

[0109] On the basis of the above embodiment, Figure 4 is a whole flowchart of a registration method of a surgical robot according to the embodiment of the application. Figure 5is a conversion relationship diagram between coordinate systems according to an embodiment of the present application. Wherein, {R} represents a mechanical arm base coordinate system, {C} represents a vision coordinate system, and {E} represents a mechanical arm end coordinate system. As shown in Figure 4 and Figure 5 As shown in the following, the overall flow of the surgical robot registration method of the present application is described. The surgical robot of the present application can be composed of a mechanical arm body and an electrical control system, and the surgical robot can be a six-degree-of-freedom robot arm including one translational joint and five rotational joints. The surgical robot can be rigidly connected with a C-arm, and the present application can directly measure the conversion relationship between the three-dimensional image coordinate system constructed based on the scan center of the C-arm and the mechanical arm base coordinate system constructed based on the center of the mechanical arm base of the surgical robot, so as to perform fine registration.

[0110] The optical target in the present application can be an object that can be recognized by the optical positioner and the C-arm at the same time. For example, the optical target can be made of a material with a high CT (Computed tomography) value in the scan image of the C-arm and covered with fluorescent powder on the surface of the optical target, so as to facilitate tracking and recognition of the optical positioner.

[0111] Specifically, the mechanical arm base coordinate system is constructed based on the center of the mechanical arm base of the surgical robot and denoted as {R}, a connecting piece is installed at the center of the mechanical arm base, and a plurality of spherical second optical targets are fixed on the connecting piece; the distance from each second optical target to the origin of the mechanical arm base coordinate system is determined based on the length information of the connecting branch of the connecting piece, and the three-dimensional coordinate point set of each second optical target in the mechanical arm base coordinate system is denoted as R P, the vision coordinate system is constructed based on the midpoint of the optical center line between the binocular cameras of the optical positioner as the origin and denoted as {C}, the position information of each second optical target ball in the vision coordinate system is obtained, and the three-dimensional coordinate point set is obtained C P, the homogeneous transformation matrix between the vision coordinate system and the mechanical arm base coordinate system corresponding to the minimum value of the objective function is obtained by performing nonlinear optimization iteration on R and t based on the objective function constructed based on the rigid transformation principle in advance The target mechanical arm forward kinematics model (target joint motion model) is established based on the joint motion model.

[0112] The transformation matrices established between the joint coordinate systems in the order of the joints of the surgical robot arm are sequentially right multiplied, and the transformation matrix of the mechanical arm end in the mechanical arm base coordinate system is calculated

[0113] In the range of each joint of the surgical robot arm, Monte Carlo method is used to traverse, and the preset N groups of random joint variables are substituted into the preliminary joint motion model to obtain the reachable workspace of the surgical robot arm, determine the small error value of the kinematic parameters and record it as δa i-1 ,δa i-1 ,δθ i ,δd i , based on the error values, the model parameters of the preliminary DH equation are adjusted to obtain and differentiate the geometric parameter error model (the intermediate joint motion model).

[0114] The surgical robot arm is controlled to move to the randomly preset second robot arm shape and position in sequence, and the coded position sequence of the joints of the surgical robot arm is recorded. According to the established preliminary joint motion model, the theoretical position sequence Q R of the robot arm end in the robot arm base coordinate system is calculated. An optical positioner is used to track the first optical target installed at the end of the robot arm, and the visual position sequence Q C of the robot arm end in the visual coordinate system is calculated. The actual position sequence of the robot arm end in the robot arm base coordinate system is determined by using the homogeneous transformation matrix between the robot arm base coordinate system and the visual coordinate system.

[0115]

[0116] Wherein, may represent the actual position of the robot arm end in the robot arm base coordinate system, Q C may represent the position of the robot arm end in the visual coordinate system, may represent the homogeneous transformation matrix between the robot arm base coordinate system and the visual coordinate system.

[0117] The theoretical actual error between the actual position and the theoretical position can be determined based on the following formula:

[0118]

[0119] Wherein, ΔQ may represent the theoretical actual error, may represent the actual position of the robot arm end in the robot arm base coordinate system, Q R may represent the theoretical position of the robot arm end in the robot arm base coordinate system.

[0120] To solve the multiple unknowns in the geometric error model, not less than 200 poses are selected for measurement to construct an overdetermined equation set, and the approximate solution of the geometric parameter error is obtained. A nonlinear optimization algorithm is used to identify the error parameters. The direct compensation method is used to substitute the identified error values into the established error model to calculate the pose deviation. The compensated pose can be obtained by subtracting the pose deviation from the expected pose.

[0121] In the case of rigid connection between the surgical robot arm and the C-arm, a coarse registration matrix between the three-dimensional image coordinate system constructed based on the scan center of the C-arm and the robot base coordinate system constructed based on the robot base center of the surgical robot arm is obtained through coarse measurement, but due to a large number of uncertain factors leading to errors in the coarse measurement process, the accuracy of the coarse registration through coarse measurement cannot meet the user's requirements. The coarse registration matrix determined by coarse measurement can be as follows:

[0122]

[0123] Wherein, The coarse registration matrix can be represented as The coarse rotation matrix between the three-dimensional image coordinate system and the robot base coordinate system can be represented as The coarse translation vector between the three-dimensional image coordinate system and the robot base coordinate system can be represented as

[0124] A more accurate conversion matrix between the shape and position image and the robot arm is obtained using joint calibration. Specifically, a certain number of first robot arm shapes and positions of the surgical robot arm are randomly generated within the imaging area of the C-arm and the operating space of the surgical robot arm, and the robot arm is controlled to move to each first robot arm shape and position. When the end of the surgical robot arm moves to the target point, the C-arm is controlled to capture the target image without interference with the surgical robot arm. The captured shape and position image sequence is filtered and denoised, and the center of each first optical target in the shape and position image sequence is extracted using the Hough transform method and the threshold method. The three-dimensional image coordinate system of the first optical target at the end of the surgical robot arm and the coordinate value of the center of the first optical target in the three-dimensional image coordinate system {I} are obtained using a three-dimensional reconstruction algorithm after processing the shape and position image sequence, denoted as I P. The position of the first optical target in the robot base coordinate system {R} is denoted as R P. The preliminary error equation and the objective function can be as follows:

[0125]

[0126]

[0127] Wherein, The registration error matrix can be represented as The coarse registration matrix can be represented as R The coordinates of the first optical target in the robot base coordinate system can be represented as I The coordinates of the first optical target in the three-dimensional image coordinate system can be represented as

[0128] After the data acquisition of multiple sets of shape and position is completed, a Jacobian matrix of the objective function with respect to error parameters is constructed, and the values of errors Delta alpha, Delta beta, Delta gamma, Delta x, Delta y and Delta z are obtained by iterative solution, so as to obtain the fine registration matrix after joint calibration.

[0129] The application uses an optical target as a medium to complete coordinate conversion registration and calibration of an image and a surgical robot arm when the robot arm is fixed with the C-shaped arm, without the need of processing and manufacturing calibration tools, and the method has high automation degree and can effectively reduce errors caused by personnel misoperation.

[0130] The overall joint calibration process of the application can be completed before the surgical robot is put into practical application, without the need of re-calibration as long as the robot arm system and the C-shaped arm do not have large deformation, so that errors caused by various factors during surgery can be effectively reduced.

[0131] The application uses an optical positioner to perform robot arm calibration and fine registration between a surgical robot operating object and a robot arm when the surgical operating arm is fixed with the X-ray imaging equipment.

[0132] The application uses an optical positioner to perform joint calibration when the surgical operating arm is fixed with the X-ray imaging equipment (which can be a C-shaped arm), so that the registration efficiency before the surgical robot is applied can be improved.

[0133] The application uses the joint calibration method to obtain a more accurate coordinate conversion matrix and improve the solution efficiency of the optimization process.

[0134] Embodiment three

[0135] Figure 6 A structural schematic diagram of a surgical robot registration device provided by the embodiment three of the application is shown in the figure. Figure 6 As shown in the figure, the device comprises a rigid connection module 310, a coarse conversion module 320, an image acquisition module 330, a coordinate set determination module 340 and a fine registration module 350.

[0136] The rigid connection module 310 is used for determining a mechanical arm base coordinate system constructed based on a mechanical arm base center of the surgical robot and a three-dimensional image coordinate system constructed based on a scanning center of the C-shaped arm in the case that the surgical robot arm and the C-shaped arm are rigidly connected.

[0137] The technical scheme of the embodiment of the application realizes the automatic registration of the surgical robot and improves the efficiency and accuracy of the registration of the surgical robot.

[0138] The registration device of the surgical robot further comprises an image sequence acquisition module, an image coordinate identification module, a model calculation module and an error equation determination module.

[0139] The image sequence acquisition module is configured to control the surgical robot arm to sequentially move to each first robot arm pose in a preset first robot arm pose sequence and acquire a pose image sequence through the C-shaped arm before the registration error matrix between the first coordinate set and the second coordinate set is determined through the registration error equation, wherein the first target information of a first optical target set of a robot arm end mounted on the surgical robot arm in the first robot arm pose is included in the pose image.

[0140] The image coordinate identification module is configured to determine a first optical target set image coordinate set sequence in the three-dimensional image coordinate system based on the first target information in each of the pose images of the pose image sequence.

[0141] The model calculation module is configured to determine a base coordinate set sequence of the first optical target set in the robot arm base coordinate system in the first robot arm pose sequence through the target joint motion model.

[0142] The error equation determination module is configured to adjust an error parameter of a pre-constructed initial error equation according to the image coordinate set sequence and the base coordinate set sequence to obtain a registration error equation.

[0143] Optionally, the coordinate set determination module is specifically configured to:

[0144] determine a robot arm end coordinate system constructed based on the robot arm end of the surgical robot arm and first connector information of a connector connecting the robot arm end and the first optical target set.

[0145] determine an end coordinate set of the first optical target set in the robot arm end coordinate system based on the first connector information.

[0146] convert the input end coordinate set through a target joint motion model to obtain a second coordinate set of the first optical target set in the robot arm base coordinate system.

[0147] Optionally, the surgical robot registration device further comprises an optical positioning module and a target model determination module.

[0148] The optical positioning module is configured to determine a visual coordinate system constructed based on a midpoint of an optical center line of an optical positioning instrument and a binocular camera before the second coordinate set of the first optical target set in the robot arm base coordinate system in the target robot arm pose is determined through the target joint motion model.

[0149] a target model determination module configured to determine a second optical target set connected to the center of the base of the robot arm based on the connecting member, and determine the target joint motion model based on the base coordinate system of the robot arm, the end coordinate system of the robot arm, the vision coordinate system, and the second optical target set.

[0150] Optionally, the target model determination module comprises a homogeneous transformation unit and a target model determination unit.

[0151] The homogeneous transformation unit is configured to determine a homogeneous transformation matrix based on the base coordinate system of the robot arm, the end coordinate system of the robot arm, and the second optical target set.

[0152] The target model determination unit is configured to determine the target joint motion model based on the homogeneous transformation matrix and a preliminary joint motion model constructed in advance.

[0153] Optionally, the target model determination unit comprises an intermediate model determination subunit, an encoded position acquisition subunit, a theoretical position determination subunit, an actual position determination subunit, and a model parameter adjustment subunit.

[0154] The intermediate model determination subunit is configured to determine a plurality of groups of preset joint parameters corresponding to the preliminary joint motion model, and determine an intermediate joint motion model corresponding to the preliminary joint motion model based on the preset joint parameters.

[0155] The encoded position acquisition subunit is configured to control the surgical robot arm to move to each second robot arm pose in a preset second robot arm pose sequence in sequence, and acquire an encoded position sequence of the end of the robot arm through an encoder.

[0156] The theoretical position determination subunit is configured to determine a theoretical position sequence of the encoded position sequence in the base coordinate system of the robot arm through the preliminary joint motion model.

[0157] The actual position determination subunit is configured to determine an actual position sequence of the encoded position sequence in the base coordinate system of the robot arm through the optical positioner and the homogeneous transformation matrix.

[0158] The model parameter adjustment subunit is configured to determine a theoretical-actual error based on the theoretical position sequence and the actual position sequence, adjust model parameters of the intermediate joint motion model based on the theoretical-actual error, and obtain the target joint motion model.

[0159] Optionally, the actual position determination subunit is specifically configured to:

[0160] For the second robot arm pose sequence, determine a vision position sequence of the second optical target set in the vision coordinate system through the optical positioner.

[0161] determine an actual position sequence of the coded position sequence in the mechanical arm base coordinate system based on the homogeneous transformation matrix and the visual position sequence.

[0162] Optionally, the intermediate model determining subunit is specifically used for:

[0163] For each group of preset joint parameters, a theoretical transformation matrix between the mechanical arm base coordinate system and the mechanical arm end coordinate system under the preset joint parameters is determined through the preliminary joint motion model.

[0164] Based on the plurality of theoretical transformation matrices, the model parameters of the preliminary joint motion model are adjusted to obtain an intermediate joint motion model.

[0165] The surgical robot registration device provided in the embodiments of the present application can perform the surgical robot registration method provided in any embodiment of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0166] Embodiment four

[0167] Figure 7 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0168] As shown in Figure 7 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0169] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0170] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the coordinate system fine registration method of the surgical robot.

[0171] In some embodiments, the coordinate system fine registration method of the surgical robot can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the coordinate system fine registration method of the surgical robot described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the coordinate system fine registration method of the surgical robot by any other appropriate means, such as by means of firmware.

[0172] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0173] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, can cause instructions defined in the flow charts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, and partially on a remote machine or entirely on a remote machine or server.

[0174] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0175] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0176] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0177] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0178] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in series, or executed in different orders, as long as the desired results of the technical solutions of the present disclosure can be achieved, and the present disclosure is not limited herein.

[0179] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A surgical robotic registration method, characterized by, Comprise: In the case of rigid connection between the surgical robot and the C-arm, determine the mechanical arm base coordinate system constructed based on the mechanical arm base center of the surgical robot and the three-dimensional image coordinate system constructed based on the scanning center of the C-arm; Determine the coarse registration matrix between the mechanical arm base coordinate system and the three-dimensional image coordinate system; Determine the target mechanical arm pose of the surgical robot, and collect target images through the C-arm, wherein the target images include target target information of a first optical target set mounted on the mechanical arm end of the surgical robot through the connecting piece under the target mechanical arm pose; Determine the first coordinate set of the first optical target set in the three-dimensional image coordinate system according to the target target information, and determine the second coordinate set of the first optical target set in the mechanical arm base coordinate system under the target mechanical arm pose through the target joint motion model; Determine the registration error matrix between the first coordinate set and the second coordinate set through the registration error equation, determine the fine registration matrix according to the registration error matrix and the coarse registration matrix, and perform fine registration on the three-dimensional image coordinate system and the mechanical arm base coordinate system based on the fine registration matrix.

2. The method of claim 1, wherein, Before the registration error matrix between the first coordinate set and the second coordinate set is determined through the registration error equation, it further comprises: Control the surgical robot to move to each first mechanical arm pose in the preset first mechanical arm pose sequence in turn and collect a sequence of pose images through the C-arm, wherein the first target information of the first optical target set mounted on the mechanical arm end of the surgical robot under the first mechanical arm pose is included in the pose image; Determine the image coordinate set sequence of the first optical target set in the three-dimensional image coordinate system based on the first target information in each of the sequence of pose images; Determine the base coordinate set sequence of the first optical target set in the mechanical arm base coordinate system under the first mechanical arm pose sequence through the target joint motion model; Adjust the error parameters of the initial error equation constructed in advance according to the image coordinate set sequence and the base coordinate set sequence to obtain the registration error equation.

3. The method of claim 1, wherein, The determination of the second coordinate set of the first optical target set in the mechanical arm base coordinate system under the target mechanical arm pose through the target joint motion model comprises: Determine the mechanical arm end coordinate system constructed based on the mechanical arm end of the surgical robot and the first connecting piece information of the connecting piece connecting the mechanical arm end and the first optical target set; Determine the end coordinate set of the first optical target set in the mechanical arm end coordinate system based on the first connecting piece information; Convert the input end coordinate set through the target joint motion model to obtain the second coordinate set of the first optical target set in the mechanical arm base coordinate system.

4. The method of claim 1, wherein, Before the determination of the second coordinate set of the first optical target set in the mechanical arm base coordinate system under the target mechanical arm pose through the target joint motion model, it further comprises: determining a visual coordinate system constructed based on a midpoint of a line connecting optical centers of an optical positioner and a binocular camera based on the optical positioner; determining a second optical target set connected to a center of a base of the mechanical arm based on a connecting member, and determining the target joint motion model based on the base coordinate system of the mechanical arm, the end coordinate system of the mechanical arm, the visual coordinate system, and the second optical target set.

5. The method of claim 4, wherein, The determining the target joint motion model based on the base coordinate system of the mechanical arm, the end coordinate system of the mechanical arm, the visual coordinate system, and the second optical target set comprises: determining a homogeneous transformation matrix based on the base coordinate system of the mechanical arm, the end coordinate system of the mechanical arm, and the second optical target set; determining the target joint motion model based on the homogeneous transformation matrix and a preliminary joint motion model constructed in advance.

6. The method of claim 5, wherein, The determining the target joint motion model based on the homogeneous transformation matrix and the preliminary joint motion model constructed in advance comprises: determining a plurality of groups of preset joint parameters corresponding to the preliminary joint motion model, and determining an intermediate joint motion model corresponding to the preliminary joint motion model based on the preset joint parameters; controlling the surgical mechanical arm to move to each second mechanical arm pose in a preset second mechanical arm pose sequence in sequence and collecting a coded position sequence of the end of the mechanical arm by an encoder; determining a theoretical position sequence of the coded position sequence in the base coordinate system of the mechanical arm by the preliminary joint motion model; determining an actual position sequence of the coded position sequence in the base coordinate system of the mechanical arm by the optical positioner and the homogeneous transformation matrix; determining a theoretical-actual error based on the theoretical position sequence and the actual position sequence, adjusting model parameters of the intermediate joint motion model based on the theoretical-actual error, and obtaining the target joint motion model.

7. The method of claim 6, wherein, The determining the actual position sequence of the coded position sequence in the base coordinate system of the mechanical arm by the optical positioner and the homogeneous transformation matrix comprises: for the second mechanical arm pose sequence, determining a visual position sequence of the second optical target set in the visual coordinate system by the optical positioner; determining the actual position sequence of the coded position sequence in the base coordinate system of the mechanical arm based on the homogeneous transformation matrix and the visual position sequence.

8. The method of claim 6, wherein, The determining the intermediate joint motion model corresponding to the preliminary joint motion model based on the preset joint parameters comprises: for each group of preset joint parameters, determining a theoretical transformation matrix between the base coordinate system of the mechanical arm and the end coordinate system of the mechanical arm under the preset joint parameters by the preliminary joint motion model; adjusting model parameters of the preliminary joint motion model based on a plurality of theoretical transformation matrices, and obtaining an intermediate joint motion model.

9. A surgical robotic registration device, characterized by, comprises: a rigid connection module configured to, in a case where a surgical mechanical arm of a surgical robot is rigidly connected with a C-arm, determine a base coordinate system of the mechanical arm constructed based on a center of a base of the surgical mechanical arm and a three-dimensional image coordinate system constructed based on a scanning center of the C-arm; a coarse conversion module configured to determine a coarse registration matrix between a robot base coordinate system and a three-dimensional image coordinate system; an image acquisition module configured to determine a target robot pose of the surgical robot, and acquire a target image through the C-arm, wherein the target image includes target target information of a first set of optical targets mounted on a robot end of the surgical robot through a connecting member in the target robot pose; a coordinate set determination module configured to determine a first coordinate set of the first set of optical targets in the three-dimensional image coordinate system according to the target target information, and determine a second coordinate set of the first set of optical targets in the robot base coordinate system in the target robot pose through a target joint motion model; a fine registration module configured to determine a registration error matrix between the first coordinate set and the second coordinate set through a registration error equation, determine a fine registration matrix according to the registration error matrix and the coarse registration matrix, and perform fine registration on the three-dimensional image coordinate system and the robot base coordinate system based on the fine registration matrix.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to implement the surgical robot registration method in any one of claims 1-8 when executed.

Images