Endoscopic Surgery Control System

Abstract

The present invention discloses an endoscopic surgery control system, which includes: at least one robotic arm is equipped with an endoscope; at least one robotic arm is equipped with surgical instruments; the operation handle is operated by a doctor; the positioning marker is rigidly connected to the object to be measured, and is used to obtain the current pose of the object to be measured, and the object to be measured is an endoscope, surgical instruments and an operation handle; the positioning device obtains the current pose of the object to be measured through the positioning marker and transmits it to the main controller; the main controller is used to calculate the target pose of the surgical device according to the current pose of the operation handle, drive the robotic arm to control the surgical device to reach the target pose; and determine whether the current pose of the surgical device is consistent with the target pose, if so, stop driving, wherein the surgical device is an endoscope and surgical instruments. The present invention does not adopt an encoder, has small errors, simple hardware, small occupied space, and flexible installation methods.

Description

Technical Field

[0001] The present invention relates to the technical field of medical device control, and particularly to an endoscopic surgical control system. Background Art

[0002] This section aims to provide background or context for the embodiments of the present invention described in the claims. The descriptions herein are not admitted to be prior art merely because they are included in this section.

[0003] The existing endoscopic surgical robot system consists of three parts: a surgeon's console, a bedside robotic arm system, and an imaging system.

[0004] In the surgeon's console, the doctor sits in the console outside the sterile area of the operating room and uses both hands (by operating two master controllers) and feet (by a foot pedal) to control the instruments and a three-dimensional high-definition endoscope.

[0005] The bedside robotic arm system is the operating component of the surgical robot, and its main function is to provide support for the instrument arm and the camera arm. The assistant doctor works around the bedside robotic arm system in the sterile area, responsible for replacing the instruments and the endoscope, and assisting the doctor to complete the operation.

[0006] The imaging system is equipped with the core processor and image processing equipment of the surgical robot. During the operation, it is located outside the sterile area, can be operated by the circulating nurse, and can place various auxiliary surgical equipment.

[0007] However, the existing endoscopic surgical robot control system has a complex structure, a large target positioning error, and high manufacturing and use costs. It is mainly reflected in:

[0008] 1. This system uses multiple series-connected encoders to obtain the pose of the operating handle through complex algorithms and maps it to the bedside robotic arm system. This method has a large error and a high overall cost of the system.

[0009] 2. The bedside robotic arm system occupies a large space, encroaching on the limited surgical operation space of the doctor. Summary of the Invention

[0010] Embodiments of the present invention provide an endoscopic surgical control system that does not use encoders, has small errors, simple hardware, small occupied space, and flexible installation methods. The system includes: at least one robotic arm, at least one endoscope, at least one surgical instrument, at least one operating handle, at least one positioning marker, at least one positioning device, and a master controller; wherein,

[0011] At least one robotic arm is equipped with an endoscope;

[0012] At least one robotic arm is equipped with a surgical instrument;

[0013] The operation handle is operated by a doctor;

[0014] The positioning mark is rigidly connected to the object to be measured, and is used to obtain the current pose of the object to be measured, where the object to be measured is an endoscope, a surgical instrument, and an operation handle;

[0015] The positioning device obtains the current pose of the object to be measured through the positioning mark and transmits it to the main controller;

[0016] The main controller is used to calculate the target pose of the surgical device according to the current pose of the operation handle, drive the robotic arm to control the surgical device to reach the target pose; and determine whether the current pose of the surgical device is consistent with the target pose. If so, stop driving, where the surgical device is an endoscope and a surgical instrument.

[0017] In the embodiment of the present invention, the positioning device and the positioning mark are used to replace the encoder to directly measure the poses of the operation handle and the surgical device: a large amount of hardware is omitted, the work of the software algorithm is reduced, and the error amplification of the series system composed of the encoders is avoided. In addition, by measuring the poses of the robotic arms through the positioning device, the robotic arms are no longer limited to a same base, and the installation method is more flexible. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings. In the drawings:

[0019] Figure 1 is one of the schematic diagrams of the endoscope surgical control system in the embodiment of the present invention;

[0020] Figure 2 is another schematic diagram of the endoscope surgical control system in the embodiment of the present invention;

[0021] Figure 3 is the third schematic diagram of the endoscope surgical control system in the embodiment of the present invention;

[0022] Figure 4 is one of the schematic diagrams of the degrees of freedom of the robotic arm in the embodiment of the present invention;

[0023] Figure 5 is another schematic diagram of the degrees of freedom of the robotic arm in the embodiment of the present invention;

[0024] Figure 6 is the schematic diagram of the principle of the operation handle provided with a pose clutch in the embodiment of the present invention;

[0025] Figure 7Schematic diagram of fusing the poses of multiple positioning devices in an embodiment of the present invention;

[0026] Figure 8 Overall flowchart of control by the endoscopic surgery control system in an embodiment of the present invention;

[0027] Figure 9 Flowchart of determining the relative position of the robotic arm base using the mechanical measurement method in an embodiment of the present invention;

[0028] Figure 10 One of the flowcharts of determining the relative position of the robotic arm base using the positioning device measurement method in an embodiment of the present invention;

[0029] Figure 11 Another flowchart of determining the relative position of the robotic arm base using the positioning device measurement method in an embodiment of the present invention;

[0030] Figure 12 Flowchart of realizing gravity compensation in an embodiment of the present invention;

[0031] Figure 13 One of the schematic diagrams of the principle of measuring the gravity direction during the movement of the operating table in an embodiment of the present invention;

[0032] Figure 14 Another schematic diagram of the principle of measuring the gravity direction during the movement of the operating table in an embodiment of the present invention;

[0033] Figure 15 Flowchart of the main controller calculating the target pose of the surgical device in an embodiment of the present invention;

[0034] Figure 16 Schematic diagram of multiple positioning devices measuring the pose of the operating handle in an embodiment of the present invention;

[0035] Figure 17 Flowchart of performing pose mapping using the incremental method in an embodiment of the present invention;

[0036] Figure 18 Flowchart of performing pose mapping using the point-to-point mapping method in an embodiment of the present invention;

[0037] Figure 19 Flowchart of measuring delay correction of the positioning device in an embodiment of the present invention;

[0038] Figure 20 Flowchart of calculating the target pose of the surgical device in an embodiment of the present invention. Specific implementation manners

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clearly understood, the following further describes the embodiments of the present invention in detail with reference to the accompanying drawings. Herein, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention.

[0040] Figure 1 FIG. 1 is one of the schematic diagrams of the endoscopic surgery control system in the embodiments of the present invention, including:

[0041] At least one robotic arm 11, at least one endoscope 13, at least one surgical instrument 14, at least one operating handle 12, at least one positioning device 15, at least one positioning marker (not shown), and a main controller (not shown);

[0042] At least one robotic arm 11 is equipped with an endoscope 13;

[0043] At least one robotic arm 11 is equipped with a surgical instrument 14;

[0044] The operating handle 12 is operated by a doctor;

[0045] The positioning marker is rigidly connected to the object to be measured, and is used to obtain the current pose of the object to be measured, where the object to be measured is the endoscope, the surgical instrument, and the operating handle 12;

[0046] The positioning device 15 obtains the current pose of the object to be measured through the positioning marker and transmits it to the main controller;

[0047] The main controller is used to calculate the target pose of the surgical device according to the current pose of the operating handle, drive the robotic arm to control the surgical device to reach the target pose; and determine whether the current pose of the surgical device is consistent with the target pose. If so, stop driving, where the surgical device is the endoscope or the surgical instrument.

[0048] Among them, the surgical instrument 14 can be a surgical instrument 14 with a multi-joint end.

[0049] Figure 1 In FIG. 2, there are two positioning devices 15, where Figure 1 The left positioning device 15 is used to obtain the current pose of the surgical device, Figure 1 The right positioning device 15 is used to obtain the current pose of the operating handle.

[0050] The positioning marker is rigidly connected to the object to be measured. Herein, the rigid connection is relative to the flexible connection. Only through the rigid connection can the pose of the object to be measured be measured through the marker.

[0051] In the embodiments of the present invention, the pose includes the position in space (for example, which can be represented by three coordinate axes X, Y, and Z) and the attitude in different navigation coordinate systems.

[0052] In Figure 1 this, the robotic arm has 7 degrees of freedom, two operating handles 12, an endoscope 13, two surgical instruments 14 with multi-joint ends, and an optical positioning device 15 (for obtaining positioning marks rigidly connected to the surgical device) and an electromagnetic positioning device 15 (for obtaining positioning marks rigidly connected to the operating handle).

[0053] In one embodiment, the robotic arm is mounted on the bedside rail of the operating bed; or the robotic arm is fixed to the operating bed.

[0054] Through the above settings, the robotic arm can follow the movement of the operating bed, and the doctor can adjust the position and posture of the operating bed to achieve better exposure of the lesion.

[0055] In one embodiment, the end of the endoscope is a fixed structure, or consists of multiple movable joints;

[0056] The end of the surgical instrument is a fixed structure, or consists of multiple movable joints.

[0057] In one embodiment, the endoscope and the surgical instrument can be disassembled and used independently.

[0058] That is, the endoscope and the surgical instrument can be separated from the robotic arm. The separation point is before the drive motor. After separation, the endoscope and the surgical instrument can be held by the doctor for use, thus providing a more convenient application method; wherein, the drive motor is used to drive the robotic arm.

[0059] Figure 2 This is one of the schematic diagrams of the degrees of freedom of the robotic arm in the embodiments of the present invention. In one embodiment, the robotic arm controls the surgical device in a passive remote fixed point or hybrid remote fixed point manner, where the hybrid remote fixed point includes a hybrid remote active fixed point and a hybrid remote passive fixed point;

[0060] The robotic arm has 7 degrees of freedom,

[0061] The robotic arm includes a first joint 111, a second joint 112, a third joint 113, a fourth joint 114, a fifth joint 115, a sixth joint 116, and a seventh joint 117 connected in sequence. Among them, the first joint 111, the second joint 112, the third joint 113, the fourth joint 114, and the seventh joint 117 are active joints that can rotate actively, and the fifth joint 115 and the sixth joint 116 are passive joints that can only follow;

[0062] The fifth joint 115 and the sixth joint 116 are orthogonal; the fifth joint 115, the sixth joint 116, and the seventh joint 117 intersect at a point.

[0063] Among them, the active fixed point is defined as: the robotic arm driving the controlled object contains at least a pair of non-linearly related intersecting degrees of freedom, and the controlled object moves with the above intersection point as the center point.

[0064] Among them, the passive fixed point is defined as: the controlled object is constrained at a certain point by an external constraint, and the controlled object moves with this point as the center point under the drive of the robotic arm. The robotic arm has at least 5 degrees of freedom, and at least 3 of them are degrees of freedom that can move actively.

[0065] Among them, the hybrid remote fixed point is defined as: at least one reference axis of the degrees of freedom of the drive structure of the controlled object intersects with the external constraint point, and the controlled object moves with the above point as the center point under the drive of the robotic arm.

[0066] Among them, through the redundant joint 113, the robotic arm can achieve a more flexible positioning to facilitate cooperation with other robotic arms.

[0067] Figure 3 This is the second schematic diagram of the degrees of freedom of the robotic arm in the embodiment of the present invention. In one embodiment, the robotic arm controls the surgical device in the way of using a virtual remote fixed point;

[0068] The robotic arm has at least 6 degrees of freedom that can move actively;

[0069] Take Figure 3 as an example, it is a robotic arm with 6 degrees of freedom that can move actively. The robotic arm includes a first joint 1211, a second joint ( Figure 3 not marked), a third joint ( Figure 3 not marked), a fourth joint ( Figure 3 not marked), a fifth joint ( Figure 3 not marked), a sixth joint 1216 and a seventh joint 1217 connected in sequence. Among them, the first joint, the second joint, the third joint, the fourth joint, the fifth joint, and the sixth joint are active joints that can rotate actively, and the fifth joint and the sixth joint are passive joints that can only follow, which are used to move the position of the surgical device and make the end of the surgical device always point to the abdominal wall entry point;

[0070] The seventh joint is used to adjust the direction of the surgical device to keep the picture direction of the endoscope always consistent with the hand-eye coordination direction of the operator.

[0071] Among them, the virtual remote fixed point is defined as: through the linkage of the joints of the multi-degree-of-freedom robotic arm, the controlled object always moves around a non-existent center point without other constraints.

[0072] Figure 4This is a schematic diagram of the operating handle in the embodiment of the present invention provided with a pose clutch. In one embodiment, the operating handle is provided with a pose clutch;

[0073] The positioning device is used to actively capture the positioning mark on the operating handle when the pose clutch of the operating handle is open.

[0074] Among them, the pose clutch can be triggered mechanically or non-mechanically; the pose clutch can be installed on the operating handle, on the foot pedal or other suitable positions. Preferably, when installed on the operating handle, it is set at the position operated by the index finger, ring finger and / or little finger.

[0075] Since the operating handle has no mechanical structure support, it is necessary to control whether to collect the pose of the operating handle through an additional pose clutch to ensure that the robotic arm and the instrument do not move according to the movement of the operating handle when not needed, avoiding the positioning device from always capturing the pose of the operating handle, thereby realizing on-demand control.

[0076] Figure 5 This is the second schematic diagram of the endoscopic surgery control system in the embodiment of the present invention, Figure 5 The positioning device in it is an optical positioning device 15, which is used to obtain the current pose of the operating handle through the positioning mark rigidly connected to the operating handle.

[0077] Figure 6 This is the third schematic diagram of the endoscopic surgery control system in the embodiment of the present invention, Figure 6 The positioning device in it is an electromagnetic positioning device 15, which is used to obtain the current pose of the operating handle through the positioning mark rigidly connected to the operating handle.

[0078] Figure 7 This is the overall flowchart of the control of the endoscopic surgery control system in the embodiment of the present invention, including two parts. The first part A1-A2 is preoperative calibration, and the second part B1-B5 is intraoperative control.

[0079] In one embodiment, the main controller is further used for:

[0080] Step A1: Before the operation, calibrate the RCM position of the robotic arm;

[0081] Step A2: Determine the relative position of the robotic arm base according to the RCM position of the robotic arm to complete the preoperative calibration.

[0082] In the above embodiment, after recording the RCM position of the robotic arm, a prompt message can be given to indicate that the current robotic arms have been calibrated. After calculating the relative positions of the robotic arm bases, a prompt message can also be given to indicate that the calculation of the relative positions of the robotic arm bases is completed.

[0083] It should be noted that the relative position of the robotic arm base is determined here, rather than the relative pose. The relative position can be directly applied later. This is because when all robotic arms are connected to the same reference, such as being installed on surfaces at the same height or on the rails of the operating table, since their postures are consistent, only the relative position (without the need for posture) of the robotic arm base needs to be measured to calculate the relative poses of all robotic arms.

[0084] In addition, the above operations can ensure that when the doctor operates under the endoscopic view, the movement of the surgical device carried by the robotic arm is consistent with the movement of the operating handle in terms of the movement direction and speed ratio, that is, if viewed from the operation screen, the movement of the operating handle is synchronized with the movement of the surgical device, achieving eye-hand coordination.

[0085] In the embodiment of the present invention, for Figure 7 step B3, there are two methods to obtain the current pose of the surgical device (the end).

[0086] First, the master controller obtains the positioning mark of the surgical device through the positioning device to obtain the current pose of the end of the surgical device;

[0087] Second, the master controller calculates the current pose of the end of the surgical device through the relative position of the robotic arm base.

[0088] Then, the master controller determines whether the current pose of the end of the surgical device is consistent with the target pose.

[0089] The following gives the specific process of calculating the current pose of the end of the surgical device through the relative position of the robotic arm base of the master controller.

[0090] In one embodiment, the master controller is specifically used for:

[0091] Adopt the following steps to calculate the current pose of the end of the surgical device through the relative position of the robotic arm base:

[0092] Obtain the position of the end of the robotic arm calculated through the encoder of the active joint of the robotic arm;

[0093] Obtain the position of the entry point of the surgical device measured during preoperative calibration;

[0094] Calculate the overall pose of the surgical device according to the relative position of the robotic arm base, the position of the end of the robotic arm, and the position of the entry point of the surgical device;

[0095] Obtain the pose of the end of the surgical device relative to the overall surgical device calculated through the encoder of the active joint inside the surgical device;

[0096] Overlay the overall pose of the surgical device and the pose of the end of the surgical device relative to the overall surgical device to obtain the current pose of the end of the surgical device.

[0097] Among them, the surgical device entry points include, but are not limited to, the abdominal wall and the chest cavity. As can be seen from the above process, the relative position of the robotic arm base determined by preoperative calibration can be applied to subsequent calculations. In step A2, there are three methods to determine the relative position of the robotic arm base, which will be introduced in detail below.

[0098] For Figure 7 step A2 of, one of the methods to determine the relative position of the robotic arm base:

[0099] Figure 8 This is a flowchart of using the mechanical measurement method to determine the relative position of the robotic arm base in an embodiment of the present invention. In one embodiment, the main controller is specifically configured to:

[0100] Use the following mechanical measurement method to determine the relative position of the robotic arm base:

[0101] Determine at least one point that all robotic arms can reach (if there are multiple points, the relative positions of these points should be known) as the measurement point for the relative position of the robotic arms;

[0102] After each robotic arm reaches the measurement point, record the current pose of the robotic arm, where each robotic arm is dragged to the above measurement point;

[0103] Through the built-in sensors of each robotic arm, according to the current pose of the robotic arm, inversely deduce the relative position of each robotic arm base relative to the above measurement point.

[0104] For Figure 7 step A2 of, another method to determine the relative position of the robotic arm base:

[0105] Figure 9 This is one of the flowcharts of using the positioning device measurement method to determine the relative position of the robotic arm base in an embodiment of the present invention. In one embodiment, the main controller is specifically configured to:

[0106] Use the following positioning device measurement method to determine the relative position of the robotic arm base:

[0107] Obtain the position of the probe on the robotic arm base measured by the probe of the positioning device;

[0108] According to the position of the probe on the robotic arm base, obtain the RCM position of the robotic arm;

[0109] According to the RCM position of the robotic arm, calculate the relative position of the robotic arm base.

[0110] In the above embodiments, after recording the RCM positions of the robotic arms, a prompt message may be given indicating that the RCM positions of the robotic arms have been recorded, that is, the current robotic arms have been calibrated. After calculating the relative positions of the bases of the robotic arms, a prompt message may also be given indicating that the calculation of the relative positions of the bases of the robotic arms is completed.

[0111] For Figure 7 Step A2, the third method for determining the relative position of the robotic arm base:

[0112] Figure 10 This is the second flowchart of using the positioning device measurement method to determine the relative position of the robotic arm base in the embodiments of the present invention. In one embodiment, the main controller is specifically configured to:

[0113] Use the following positioning device measurement method to determine the relative position of the robotic arm base:

[0114] Obtain the RCM positions of the robotic arms obtained by the positioning device through the positioning marks;

[0115] According to the RCM positions of the robotic arms, calculate the relative positions of the bases of the robotic arms.

[0116] In the above embodiments, first, the positioning marks, such as an optical positioning mark array, need to be rigidly connected to the robotic arms. After calculating the relative positions of the bases of the robotic arms, a prompt message may also be given indicating that the calculation of the relative positions of the bases of the robotic arms is completed.

[0117] Refer to Figure 7 , in step B2, the specific process of the main controller driving the robotic arm to control the surgical device to reach the target pose is: driving the robotic arm 11 to make the entire surgical device 13 reach the target pose, and driving the end of 13 to reach the target pose relative to the entire surgical device 13 through the internal joints of the surgical device 13.

[0118] Refer to Figure 7 , in step B1, the main controller needs to calculate the target pose of the surgical device according to the current pose of the operating handle. Figure 11 This is the flowchart of the main controller calculating the target pose of the surgical device in the embodiments of the present invention. The main controller is further configured to: calculate the target pose of the surgical device according to the current pose of the operating handle, including:

[0119] Step C1: Obtain the current pose in the coordinate system of the master hand pose measurement CS.M.ME (coordinate system of measuring device for master); the master hand pose measurement coordinate system is the coordinate system that uses the output result of the positioning device; when using an optical positioning device, the image coordinate system output by the camera is CS.M.ME; when using an electromagnetic positioning device, the positioning result coordinate system output by the coil is CS.M.ME;

[0120] Step C2: Map the current pose in the master hand pose measurement coordinate system to the target pose of the surgical device in the slave hand pose measurement coordinate system.

[0121] Among them, the current pose of the operating handle in the master coordinate system CS.M (coordinate system of master) is measured by one or several of the inertial measurement unit double integration, optical positioning device, or electromagnetic positioning device.

[0122] For the pose mapping in Step C2, the embodiments of the present invention provide two methods.

[0123] The first is the incremental method. Figure 12 This is the flow chart of using the incremental method for pose mapping in the embodiments of the present invention. In one embodiment, the main controller is further used for:

[0124] Adopt the incremental method to calculate the target pose of the surgical device according to the current pose of the operating handle:

[0125] According to the current pose of the operating handle actively captured by the positioning device (the first sampling) and the pose after one sampling period, the pose change of the operating handle after one sampling period is ΔPO.M. Among them, the pose change of the operating handle after one sampling period is the pose change in the master hand pose measurement coordinate system;

[0126] Adopt a transformation matrix to map the pose change of the operating handle after one sampling period to the pose change ΔPO.S of the surgical device after one sampling period. Among them, the pose change of the surgical device after one sampling period is the pose change in the slave hand pose measurement coordinate system; specifically, it is mapped through the transformation matrix T.K.MS, and the formula is ΔPO.S(ΔPosition.Slave)=ΔPO.M×T.K.MS(Transfer coefficient Master to slave); T.K.MS is a transformation matrix, and its function is to amplify or reduce the output of the master hand to achieve fast or precise operations, and to fine-tune each output quantity of the master hand to compensate for its errors, etc.;

[0127] Determine the target pose of the surgical device based on the current pose of the endoscope actively captured by the positioning device and the pose change of the surgical device after a sampling period.

[0128] In the above embodiment, the pose change of the surgical device after a sampling period can be directly output to drive the surgical device.

[0129] For step C2, the second one is the point-to-point mapping method. Figure 13 This is a flowchart of using the point-to-point mapping method for pose mapping in an embodiment of the present invention. In one embodiment, the main controller is further configured to:

[0130] Adopt the point-to-point mapping method to calculate the target pose of the surgical device according to the current pose of the operating handle:

[0131] After the operating handle is activated (at time T0), record the relative position P1.M of the operating handle in the master hand pose measurement coordinate system CS.M.ME, and generate the first coordinate system CS.P1.M at this relative position; based on the relative position P1.S of the operating handle in the slave hand pose measurement coordinate system CS.S.ME of the endoscope system, generate the second coordinate system CS.P1.S at this relative position;

[0132] Obtain the pose of the current pose of the operating handle in the first coordinate system;

[0133] Adopt the transformation matrix T.K.MS to map the pose of the current pose of the operating handle in the first coordinate system to the target pose of the surgical device in the second coordinate system CS.P1.S;

[0134] Convert the target pose of the surgical device in the second coordinate system CS.P1.S to the target pose of the surgical device in the slave hand pose measurement coordinate system CS.S.ME.

[0135] Figure 7 In, after the pose of the operating table is adjusted, the attitude of the robotic arm needs to be adjusted. At this time, step B6 is triggered, that is, gravity compensation is performed, so as to accurately calculate the target position of the surgical device and eliminate the error in gravity caused by the adjustment of the pose of the operating table. Specifically, Figure 14 This is a flowchart of realizing gravity compensation in an embodiment of the present invention. Figure 15 This is one of the schematic diagrams of the principle of measuring the gravity direction during the movement of the operating table in an embodiment of the present invention. B1 is the gravity direction of the robotic arm before the operating table rotates, B2 is the operating table rotates by an angle, and B3 is the gravity direction of the robotic arm after the operating table rotates. Figure 16 This is the second schematic diagram of the principle of measuring the gravity direction during the movement of the operating table in an embodiment of the present invention. The pose of the operating table or the component fixed to the operating table is directly measured by an external positioning device to calculate the direction of gravity.

[0136] In one embodiment, the main controller is further configured to:

[0137] After the attitude adjustment of the operating table, obtain the pose change of the robotic arm through the built-in sensors or positioning devices of the robotic arm;

[0138] Calculate the gravity compensation of the robotic arm according to the pose change of the robotic arm;

[0139] Calculate the target pose of the surgical device according to the gravity compensation of the robotic arm and the current pose of the operating handle.

[0140] In the above steps, after the attitude adjustment of the operating table, the attitude of the robotic arm body changes accordingly. In this way, the pose change of the robotic arm can be obtained through the built-in sensors or positioning devices of the robotic arm to meet the requirement of surgical area exposure.

[0141] In one embodiment, the positioning device is an electromagnetic navigation device, and the positioning marker is a coil or a coil group;

[0142] Or, the positioning device is an optical navigation device, and the positioning marker is an optical positioning marker;

[0143] Or, the positioning device is an inertial navigation device, and the positioning marker is an inertial measurement unit.

[0144] In the embodiments of the present invention, the measurement results of a single navigation device may be affected by environmental interference and generate incorrect information, that is, there is a problem of insecurity in a single data link. The embodiments of the present invention propose to use multiple positioning devices to solve this problem. Figure 17 This is the schematic diagram of the pose acquisition of multiple positioning devices in the embodiments of the present invention. The specific scheme includes two types:

[0145] The first method, the pose correction method, the steps include:

[0146] When there are multiple positioning devices that capture the current pose of the operating handle, the main controller obtains the current poses of the operating handle captured by the multiple positioning devices;

[0147] Taking one of the current poses of the operating handle captured by the multiple positioning devices as a reference, and taking the other two of the current poses of the operating handle captured by the multiple positioning devices as calibration;

[0148] Calculate the difference between the reference and the calibration;

[0149] When the difference exceeds the threshold, discard the current pose;

[0150] When the difference does not exceed the threshold, take the reference as the final current pose of the operating handle.

[0151] For example, taking the current pose of the operating handle captured by the optical navigation device as the reference and the current pose of the operating handle captured by the inertial navigation device as the calibration, calculate the difference between the reference and the calibration. When the difference exceeds the threshold, discard the current pose. When the difference does not exceed the threshold, take the current pose of the operating handle captured by the optical navigation device as the final current pose of the operating handle.

[0152] The second method, the pose fusion method, the steps include:

[0153] When there are multiple positioning devices that capture the current pose of the operating handle, the main controller obtains the current poses of the operating handle captured by the multiple positioning devices;

[0154] According to the current poses of the multiple operating handles, use a fusion algorithm to calculate the final current pose of the operating handle;

[0155] When there are multiple positioning devices that capture the current poses of the robotic arm and the surgical device, obtain the current poses of the robotic arm and the surgical device captured by the multiple positioning devices;

[0156] According to the current poses of the multiple robotic arms and surgical devices, use a fusion algorithm to calculate the final current poses of the robotic arm and the surgical device.

[0157] In the above embodiments, there are various fusion algorithms. Different fusion algorithms are to find relatively accurate values among the state values with inaccuracies. For example, the basic Mahony, KF, EKF, adaptive KF, etc. are optimization algorithms proposed in a specific scenario. Of course, there can be other fusion algorithms, which are not limited here.

[0158] Through the above two methods, using multiple positioning devices eliminates the problem of inaccurate data of a single navigation device, solves the security problem of a single data link, and at the same time uses multiple navigation devices to avoid the disadvantages of a single navigation device. For example, the inertial measurement unit can make up for the problem of large target errors or even loss of the electromagnetic navigation device and the optical navigation device under interference, as well as the problem of slow response.

[0159] See Figure 7 , taking the operating handle as an example, the positioning markers rigidly connected to the operating handle include 3 electromagnetic positioning coils (provided in the electromagnetic positioning unit), optical positioning markers, and an inertial measurement unit. The optical navigation device is a camera; the electromagnetic navigation device is a magnetic field generator. After different positioning devices obtain the pose of the operating handle, they send it to the main controller.

[0160] Figure 18Schematic diagram of pose measurement of an operating handle by multiple positioning devices in an embodiment of the present invention. Among them, the operating handle is activated by a pose clutch 124, and then the current pose of the operating handle is measured by one or several of the following three methods: the double integration of an inertial measurement unit (with an output value of current, converted into acceleration and angular acceleration) 123, an optical positioning device 121, or an electromagnetic positioning device 122. Then, the current poses of different positioning devices are integrated together through a fusion algorithm.

[0161] Figure 7 When the operating handle suddenly moves, step B7 is triggered at this time, that is, measurement delay correction is performed, so as to accurately calculate the target position of the surgical device and eliminate the error caused by the sudden movement of the operating handle. Specifically, Figure 19 Flowchart of measurement delay correction of a positioning device in an embodiment of the present invention. In one embodiment, the main controller is further configured to:

[0162] When the operating handle suddenly moves at time T0, obtain the movement direction of the operating handle;

[0163] Drive the surgical device to move at an initial speed in the movement direction of the operating handle;

[0164] According to the movement speed of the operating handle, calculate the pose calculation value PO1' of the operating handle at time T1, where the time T1 is separated from the time T0 by one or more sampling periods;

[0165] At time T1, measure the accurate pose PO1 of the operating handle through the positioning device;

[0166] Calculate the difference △PO1 between the pose calculation value of the operating handle at time T1 and the accurately measured pose;

[0167] Map the difference to the pose difference △PO2 of the surgical device;

[0168] According to the pose difference of the surgical device, correct the target pose of the surgical device.

[0169] Drive the surgical device to reach the target pose.

[0170] In the above embodiment, the positioning mark can be an inertial unit. After the above correction, the error caused by the sudden movement of the operating handle is eliminated, and the accuracy of the target pose of the surgical device can be ensured.

[0171] Figure 7 When non-subjective jitter occurs, step B8 is triggered at this time, that is, non-subjective jitter elimination is performed, so as to accurately calculate the target position of the surgical device and eliminate the error caused by the doctor's non-subjective jitter. Figure 20 Flowchart of calculating the target pose of a surgical device in an embodiment of the present invention. In one embodiment, the main controller is further configured to:

[0172] Sample once every several sampling periods (i.e., delay the responses of the robotic arm and surgical device), and obtain the jitter data of the current pose of the operating handle. The jitter data includes amplitude and frequency;

[0173] Compare the jitter data of the current pose with the jitter data of a preset typical non-subjective jitter to obtain a comparison result;

[0174] When the differences between the jitter data of the current pose and the jitter data of the preset typical non-subjective jitter are all within the threshold range (i.e., close to the typical non-subjective jitter) among the comparison results obtained from multiple samplings, determine that the jitter of the current pose is non-subjective jitter, and discard the current pose (not applied to the surgical device).

[0175] Among them, the jitter data of the typical non-subjective jitter can establish different jitter data models of the typical non-subjective jitter according to different doctors, and through model training, obtain the jitter data of the typical non-subjective jitter.

[0176] Specifically, when comparing the jitter data of the current pose with the jitter data of the preset typical non-subjective jitter, being close to the typical non-subjective jitter means that the amplitude and frequency should not be too large or too small, and when within the threshold range, determine that the jitter of the current pose is non-subjective jitter.

[0177] In addition, to avoid determining it as non-subjective jitter just because it appears within the threshold range only once, the comparison results of multiple samplings are required to ensure the accuracy of the non-subjective jitter judgment. Through the above steps, the error caused by the non-subjective jitter of the doctor is eliminated.

[0178] In summary, the system proposed in the embodiment of the present invention has the following beneficial effects:

[0179] First, by using the positioning device and positioning markers instead of the encoder, directly measure the poses of the operating handle and surgical device, omit a large amount of hardware, reduce the work of software algorithms, and avoid the error amplification of the series system composed of encoders.

[0180] Second, the operating handle and positioning device do not need to be close to the operating table. Measure the poses of each robotic arm through the positioning device, so that the robotic arms are no longer limited to the same base, and the installation method is more flexible;

[0181] Third, the robotic arms are installed on the bedside rail of the operating table or fixed to the operating table, and can move with the operating table. The doctor can adjust the pose of the operating table to achieve better exposure of the lesion.

[0182] Fourth, the measurement results of a single navigation device may be interfered by the environment to generate incorrect information, resulting in the problem of insecure single data link. The pose correction method and pose fusion method using multiple positioning devices solve this problem.

[0183] Fifth, preoperative calibration was performed to determine the relative position of the robotic arm base to calculate the current pose of the end of the surgical device. The above operations can ensure that when the doctor operates under the endoscopic view, the movement of the robotic arm carrying the surgical device is consistent with the movement of the operating handle in terms of the movement direction and speed ratio, that is, if viewed from the operation screen, the movement of the operating handle is synchronized with the movement of the surgical device, achieving eye-hand coordination.

[0184] Sixth, the error caused by the sudden movement of the operating handle was solved by measuring delay correction.

[0185] Seventh, the error caused by the doctor's involuntary tremor was eliminated by involuntary tremor cancellation.

[0186] Eighth, through gravity compensation, the problem that occurred when the pose of the robotic arm body changed with the adjustment of the surgical bed posture was solved, and finally the need for surgical area exposure was met.

[0187] The specific embodiments described above further elaborate on the purpose, technical solutions, and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An endoscopic surgery control system, characterized in that, Comprising: At least one robotic arm, at least one endoscope, at least one surgical instrument, at least one operating handle, at least one positioning marker, at least one positioning device, and a main controller; wherein, At least one robotic arm is equipped with an endoscope; At least one robotic arm is equipped with a surgical instrument; The operating handle is operated by a doctor; The positioning marker is rigidly connected to the object to be measured, and is used to obtain the current pose of the object to be measured, and the object to be measured is an endoscope, a surgical instrument, and an operating handle; The positioning device obtains the current pose of the object to be measured through the positioning marker and transmits it to the main controller; The main controller is used to calculate the target pose of the surgical device according to the current pose of the operating handle, drive the robotic arm to control the surgical device to reach the target pose; judge whether the current pose of the surgical device is consistent with the target pose, and if so, stop driving, wherein the surgical device is an endoscope and a surgical instrument; Calculating the target pose of the surgical device according to the current pose of the operating handle includes: Based on the current pose of the operating handle actively captured by the positioning device and the pose after one sampling period, obtaining the pose change of the operating handle after one sampling period; Using a transformation matrix to map the pose change of the operating handle after one sampling period to the pose change of the surgical device after one sampling period; Based on the current pose of the endoscope actively captured by the positioning device and the pose change of the surgical device after one sampling period, determining the target pose of the surgical device.

2. The system according to claim 1, wherein The robotic arm controls the surgical device in a passive remote fixed point or a hybrid remote fixed point manner, wherein the hybrid remote fixed point includes a hybrid remote active fixed point and a hybrid remote passive fixed point; The robotic arm has 7 degrees of freedom; The robotic arm includes a first joint, a second joint, a third joint, a fourth joint, a fifth joint, a sixth joint, and a seventh joint connected in sequence, wherein the first joint, the second joint, the third joint, the fourth joint, and the seventh joint are active joints that can rotate actively, and the fifth joint and the sixth joint are passive joints that can only follow; The fifth joint and the sixth joint are orthogonal; the fifth joint, the sixth joint, and the seventh joint intersect at a point.

3. The system according to claim 1, characterized in that, The robotic arm controls the surgical device in a virtual remote fixed point manner; The robotic arm has at least 6 degrees of freedom that can move actively.

4. The system according to claim 1, wherein The operating handle is provided with a pose clutch; The positioning device is used to actively capture the positioning marker on the operating handle when the pose clutch of the operating handle is open.

5. The system according to claim 1, wherein The robotic arm is installed on the bedside rail of the operating bed; or the robotic arm is fixed to the operating bed.

6. The system according to claim 1, wherein The endoscope and the surgical instrument can be disassembled and used independently.

7. The system according to claim 1, wherein The main controller is further used for: Before the operation, calibrate the RCM position of the robotic arm; According to the RCM position of the robotic arm, determine the relative position of the robotic arm base to complete the pre-operative calibration.

8. The system according to claim 1, characterized in that, The main controller is further used for: Calculate the current pose of the end of the surgical device through the relative position of the robotic arm base; Judge whether the current pose of the end of the surgical device is consistent with the target pose.

9. The system according to claim 8, wherein Specifically, the main controller is used for: Adopting the following steps to calculate the current pose of the end of the surgical device through the relative position of the robotic arm base: Obtain the position of the end of the robotic arm calculated by the encoder of the active joint of the robotic arm; Obtain the position of the entry point of the surgical device measured during preoperative calibration; Calculate the overall pose of the surgical device based on the relative position of the robotic arm base, the position of the end of the robotic arm, and the position of the entry point of the surgical device; Obtain the pose of the end of the surgical device relative to the overall surgical device calculated by the encoder of the active joint inside the surgical device; Superimpose the overall pose of the surgical device and the pose of the end of the surgical device relative to the overall surgical device to obtain the current pose of the end of the surgical device.

10. The system according to claim 8, wherein The main controller is specifically used for: Determine the relative position of the robotic arm base by using the following mechanical measurement method: Determine at least one point that all robotic arms can reach as the measurement point for the relative position of the robotic arm; After each robotic arm reaches the measurement point, record the current pose of the robotic arm, where each robotic arm is dragged to the above measurement point; Through the built-in sensors of each robotic arm, inversely deduce the relative position of each robotic arm base relative to the above measurement point according to the current pose of the robotic arm.

11. The system according to claim 9, wherein The main controller is specifically used for: Determine the relative position of the robotic arm base by using the following positioning device measurement method: Obtain the position of the probe on the robotic arm base measured by using the probe of the positioning device; Obtain the RCM position of the robotic arm according to the position of the probe on the robotic arm base; Calculate the relative position of the robotic arm base according to the RCM position of the robotic arm.

12. The system according to claim 9, wherein The main controller is specifically used for: Determine the relative position of the robotic arm base by using the following positioning device measurement method: Obtain the RCM position of the robotic arm obtained by the positioning device through the positioning mark; Calculate the relative position of each robotic arm base according to the RCM position of the robotic arm.

13. The system according to claim 1, wherein The pose change of the operating handle after one sampling period is the pose change in the master hand pose measurement coordinate system; The pose change of the surgical device after one sampling period is the pose change in the slave hand pose measurement coordinate system.

14. The system according to claim 1, wherein, The main controller is also used for: After the posture of the operating table is adjusted, obtain the pose change of the robotic arm through the built-in sensor of the robotic arm or the positioning device; Calculate the gravity compensation of the robotic arm according to the pose change of the robotic arm; Calculate the target pose of the surgical device according to the gravity compensation of the robotic arm and the current pose of the operating handle.

15. The system according to claim 1, wherein The positioning device is a magnetic navigation device, and the positioning mark is a coil or a coil group; Or, the positioning device is an optical navigation device, and the positioning mark is an optical positioning mark; Or, the positioning device is an inertial navigation device, and the positioning mark is an inertial measurement unit.

16. The system according to claim 15, wherein The main controller is also used for: When there are multiple positioning devices capturing the current pose of the operating handle, obtain the current poses of the operating handle captured by the multiple positioning devices; Take one of the current poses of the operating handle captured by the multiple positioning devices as the reference, and take the other of the current poses of the operating handle captured by the multiple positioning devices as the calibration; Calculate the difference between the reference and the calibration; When the difference exceeds the threshold, discard the current pose; When the difference does not exceed the threshold, take the reference as the final current pose of the operating handle.

17. The system according to claim 15, wherein The main controller is also used for: When there are multiple positioning devices for capturing the current pose of the operating handle, obtain the current poses of the operating handle captured by the multiple positioning devices; According to the current poses of the operating handle captured by the multiple positioning devices, use a fusion algorithm to calculate the final current pose of the operating handle; When there are multiple positioning devices for capturing the current poses of the robotic arm and the surgical device, obtain the current poses of the robotic arm and the surgical device captured by the multiple positioning devices; According to the current poses of the robotic arm and the surgical device captured by the multiple positioning devices, use a fusion algorithm to calculate the final current poses of the robotic arm and the surgical device.

18. The system according to claim 1, characterized in that, The main controller is further configured to: When the operating handle suddenly moves at time T0, obtain the moving direction of the operating handle; Drive the surgical device to move at an initial speed in the moving direction of the operating handle; According to the moving speed of the operating handle, calculate the pose calculation value of the operating handle at time T1, where the time T1 is separated from the time T0 by one or more sampling periods; At time T1, measure the accurate pose of the operating handle through the positioning device; Calculate the difference between the pose calculation value of the operating handle at time T1 and the measured accurate pose; Map the difference to the pose difference of the surgical device; According to the pose difference of the surgical device, correct the target pose of the surgical device; Drive the surgical device to reach the target pose.

19. The system according to claim 1, characterized in that The main controller is further configured to: Perform sampling once every several delayed sampling periods to obtain the amplitude and frequency of the jitter of the current pose of the operating handle; Compare the jitter data of the current pose with the jitter data of a preset typical non-subjective jitter to obtain a comparison result, where the jitter data includes amplitude and frequency; When the differences between the jitter data of the current pose and the jitter data of the preset typical non-subjective jitter are all within the threshold range in the comparison results obtained from multiple samplings, determine that the jitter of the current pose is non-subjective jitter and discard the current pose.

Images