Method and apparatus for table pose tracking using fiducial markers - Patents.com

Abstract

This invention provides a method and system for determining the location of an operating table and estimating its position. [Solution] A method for aligning a manipulator assembly 26 and an independently positionable operating table 14 includes the steps of: reading reference markers on the operating table using sensors associated with the manipulator assembly; and determining the positions of the manipulator assembly and the operating table relative to a common reference frame. The method may further include the step of converting the 3D configuration of the operating table to a 2D reference frame in order to estimate the 3D orientation of the operating table relative to the manipulator assembly for use in coordinated movement between the manipulator assembly and the operating table.

Description

[Technical field]

[0001] The present invention relates to a method and apparatus for table pose tracking using fiducial markers. [Background technology]

[0002] This application claims priority to U.S. Provisional Patent Application No. 61 / 954,559, filed March 17, 2014, the entire contents of which are incorporated herein by reference for all purposes.

[0003] This application is related to U.S. Provisional Patent Application No. 61 / 954,538, entitled “Methods and Devices for Tele-Surgical Table Registration,” filed March 17, 2014 (Attorney Docket No. ISRG05550PROV / US), which is hereby incorporated by reference in its entirety for all purposes.

[0004] Minimally invasive medical techniques are intended to reduce the amount of irrelevant tissue damaged during diagnostic or surgical procedures, thereby reducing patient recovery time, discomfort, and adverse side effects. One effect of minimally invasive surgery, for example, is to reduce hospital recovery time after surgery. Because the average length of stay for standard surgery is typically significantly longer than the average length of stay for a similar minimally invasive procedure, increasing the use of minimally invasive techniques can save millions of dollars in hospital costs each year. Many of the surgeries performed each year in the United States could potentially be performed in a minimally invasive manner, but due to the limitations of minimally invasive surgical instruments and the additional surgical training involved in mastering these instruments, only a portion of current surgeries use these advantageous techniques.

[0005] Minimally invasive or telesurgery systems have been developed to improve the dexterity of surgeons and avoid some of the limitations of traditional minimally invasive techniques. In telesurgery, the surgeon uses some form of remote control (e.g., servo mechanisms, etc.) to steer the movement of surgical instruments rather than directly holding and moving the instruments with their hands. In telesurgery systems, an image of the surgical site can be provided to the surgeon at the surgical workstation. While viewing a two-dimensional or three-dimensional image of the surgical site on a display, the surgeon performs a surgical procedure on the patient by steering a master controller, which in turn controls the movement of the instruments mechanically actuated by the servos.

[0006] Servomechanisms used in telesurgery often accept input from two master controllers (one for each hand of the surgeon) and may include two or more telesurgical arms, with each surgical arm having a surgical instrument attached. Communication between the master controller and associated manipulator arms and instrument assemblies is typically accomplished through a control system. The control system typically includes at least one processor that relays input commands, such as in the case of force feedback, from the master controller to the associated manipulator arms and instrument assemblies, and conversely from the instrument and arm assemblies to the associated master controller. One example of a telesurgery system is the DA VINCI® system available from Intuitive Surgical, Inc. of Sunnyvale, California.

[0007] A variety of structural arrangements can be used to support a surgical instrument at a surgical site during telesurgery. The driven linkage or "slave" is often referred to as a telesurgery manipulator, and exemplary linkage configurations used as telesurgery manipulators during minimally invasive telesurgery are described in U.S. Patent Nos. 7,594,912, 6,758,843, 6,246,200, and 5,800,423, the complete disclosures of which are incorporated herein by reference. These linkages often utilize a parallelogram arrangement to hold an instrument having a shaft. Such manipulator structures can constrain the motion of the instrument such that it pivots about a remote center of steering located in space along the length of the rigid shaft. By aligning the remote center of manipulation with the incision point for an internal surgical site (e.g., using a trocar or cannula in the abdominal wall during laparoscopic surgery), the end effector of the surgical instrument can be safely positioned by moving the proximal end of the shaft using the manipulator's linkage without exerting potentially dangerous forces against the abdominal wall. Alternative manipulator structures are described, for example, in U.S. Patent Nos. 7,763,015, 6,702,805, 6,676,669, 5,855,583, 5,808,665, 5,445,166, and 5,184,601, the complete disclosures of which are incorporated herein by reference.

[0008] A variety of structural arrangements can be used to support and position the telesurgical manipulators and surgical instruments at the surgical site during telesurgery. Support linkages, sometimes referred to as set-up joints or set-up joint arms, are often used to position and align each manipulator with a respective incision point on the patient's body. The support linkages facilitate alignment of the surgical manipulators with the desired surgical incision points and target anatomical structures. Exemplary support linkages are described in U.S. Patent Nos. 6,246,200 and 6,788,018, the complete disclosures of which are incorporated herein by reference. Summary of the Invention [Problem to be solved by the invention]

[0009] While such new telesurgery systems and devices provide a wide range of configurations and coordinated motions between highly maneuverable manipulators and have proven to be very effective and advantageous, it has proven to be challenging to localize such motions in a surgical environment. Further improvements are therefore desirable. It would be particularly beneficial if these improved techniques improved the efficiency and ease of use of telesurgery systems. For example, it would be particularly beneficial to improve maneuverability, increase space utilization in the operating room, provide quick and easy set-up, reduce manipulator collisions during use, and / or reduce the mechanical complexity and size of these new surgical systems. [Means for solving the problem]

[0010] The following presents a simplified summary of some embodiments of the invention in order to provide a basic understanding of the invention. This summary is not an extensive overview of the invention. It is not intended to identify key / critical elements of the invention or to delineate the scope of the invention. Its sole purpose is to present some embodiments of the invention in a simplified form as a prelude to the more detailed description that is presented hereinafter.

[0011] The present invention provides improved telesurgery devices, systems, and methods in general, and methods and systems for locating and estimating the orientation of a surgical table in particular. The telesurgery system includes a kinematic linkage and associated control system that is particularly useful in performing minimally invasive surgical procedures on a patient. Such procedures often utilize interrelated and coordinated motions between multiple manipulators, each of which is highly configurable and has a range of alternative configurations for a specified end effector position within the surgical environment. For various reasons, it may be desirable to position a patient in a particular position and / or orientation for a particular procedure. Also, in some procedures, it may be further desirable to change the patient's position and / or orientation during the procedure. For example, a particular patient position and / or orientation may be particularly useful in accessing a particular area within a surgical workspace, or it may be desirable to position the patient in a particular alignment (e.g., tilted along one or more axes) during the procedure for various physiological reasons. Because many telesurgery systems utilize an operating table that is separate from the manipulator system and that is often independently positionable along multiple degrees of freedom, the various positions of the operating table can present particular challenges during the operation of the telesurgery manipulator, especially in systems having multiple manipulators. It is therefore desirable for such manipulator systems to have a means for "localizing" the operating table together with the manipulator assembly, thereby determining the spatial relationship between the operating table and the manipulator assembly, and which can be utilized in calculating the motion of the surgical manipulator. In one aspect, it is desirable if such localization can be achieved without direct contact between the manipulator assembly and the operating table.

[0012] In one aspect, the present invention provides a method for localizing a surgical table using one or more fiducial markers. The localization method includes the steps of reading one or more fiducial markers, such as a 2D barcode, located on the surgical table; transforming a 3D pose of the surgical table into a 2D reference frame common to the manipulator assembly; and determining a spatial relationship between the surgical table and the manipulator assembly. Such a method may utilize a sensor, such as an optical sensor or a camera, located in the base of the manipulator assembly, which may locate the surgical table relative to the manipulator assembly with respect to a ground plane when the surgical table is positioned proximate to the manipulator assembly, and may determine a spatial relationship between the pose of the surgical table and the manipulator assembly. This is advantageous, since such a pose estimation may be utilized for coordinated motion of the manipulator and the surgical table, for example, in controlled movements of the surgical table to compensate for patient movements such as heartbeat and breathing, or to facilitate the movement of certain organs of the patient during a procedure due to gravity.

[0013] In one aspect, a method for estimating the orientation of a surgical table includes reading one or more fiducial markers on a base of the surgical table with a camera associated with a manipulator assembly adjacent to the surgical table; and locating the surgical table relative to the manipulator assembly on a common plane based on the reading of the one or more markers. The one or more fiducial markers are typically placed at various fixed locations on the surgical table at known and / or predetermined positions and / or orientations relative to the surgical table. The method may include affixing one or more markers to the surgical table and / or forming one or more markers at selected locations on the surgical table at specific orientations relative to the surgical table. Forming one or more markers on the surgical table may include etching, engraving, and / or embossing one or more markers on the surgical table.

[0014] In some embodiments, reading the one or more markers includes reading at least one of the one or more markers before and / or during the surgical procedure while the patient is supported on the surgical table. In one aspect, the common plane is a ground plane on which the surgical table and the manipulator assembly are placed. In many embodiments, the surgical table is positionable along one or more degrees of freedom (DOF).

[0015] In some aspects, the method includes converting a 6 degree of freedom 3D pose of the positionable operating table to a 3 degree of freedom 2D pose on a common plane; and estimating the 3D pose of the operating table relative to the manipulator assembly based on a localization on the common plane of a base of the operating table relative to the manipulator assembly. In some embodiments, the method includes extracting a position of the operating table within a visualization image acquired with a camera; and providing a representation or display of the operating table relative to a camera view of the camera.

[0016] In another aspect, the method can include reading one or more markers and identifying the make, model, or manufacturer of the operating table based on the reading of the markers. Then, the method can include checking compatibility between the operating table and the manipulator assembly and / or providing authorization for use of the manipulator assembly with the operating table.

[0017] In some embodiments, the one or more markers are multiple markers disposed at multiple locations on the operating table, and reading the one or more markers comprises reading at least one of the one or more markers. In one aspect, reading the one or more markers comprises reading a single marker of the one or more markers, and the localization of the operating table is based on the reading of the single marker.

[0018] In another aspect, the method includes the steps of reading at least one of the one or more markers after an initial reading of the one or more markers and determining a location based on the initial reading; and determining a position of the operating table relative to the manipulator assembly on a common plane based on a subsequent reading of the one or more markers to update and / or verify the initial location determination.

[0019] In one aspect, the present invention provides a telesurgery system, the system including: a manipulator assembly; a surgical table disposed proximate to the manipulator assembly, the surgical table having one or more fiducial markers; and a sensor configured to read the one or more markers on the surgical table when the surgical table is positioned in close proximity to the manipulator assembly. In some embodiments, the surgical table is positionable along one or more degrees of freedom. The sensor can include a camera associated with the manipulator assembly. In some embodiments, the one or more fiducial markers can include a bar code, an RFID tag, a light, or any combination thereof. The one or more markers can be fixedly attached and / or formed on a selected portion of the surgical table. In one aspect, a marker is fixedly attached and / or formed on a selected portion and at a selected position in a specific orientation relative to the surgical table, whereby an orientation of the surgical table relative to a ground surface can be determined by reading at least one of the one or more markers. In some embodiments, the one or more fiducial markers include one or more 2D barcodes that extend around the base of the operating table and / or along an edge or side of the operating table top surface of the operating table.

[0020] In another aspect, the system includes a processor configured to read one or more markers on a base of the operating table with a camera associated with a manipulator assembly adjacent to the operating table; and identify a position of the operating table relative to the manipulator assembly on a common plane based on the reading of the one or more markers. In some embodiments, the operating table is positionable along six degrees of freedom, and the processor is further configured to convert a 6-degree-of-freedom 3D pose of the positionable operating table to a 3-degree-of-freedom 2D pose on the common plane. The processor can be further configured to estimate a 3D pose of the operating table relative to the manipulator assembly based on the positioning of the base of the operating table relative to the manipulator assembly on the common plane. In some embodiments, the processor is further configured to extract a position of the operating table within a visualization image acquired with the sensor and provide a representation or display of the operating table relative to a camera view of the camera. In some embodiments, the processor is further configured to identify a make, model, or manufacturer of the operating table based on the reading of the markers. The processor may be configured to check compatibility aspects between the surgical table and the manipulator assembly and / or, based on such identification, grant permission for use of the manipulator assembly with the surgical table.

[0021] In another aspect, the present invention provides a table, typically an operating table for use in a remote surgery system. In some embodiments, the table includes a substrate having a patient support surface; a support structure supporting the substrate, the support structure being movable to allow the patient support surface to be positioned along one or more degrees of freedom; and one or more fiducial markers disposed on the operating table at selected positions and / or selected orientations to allow the orientation of the operating table relative to a ground surface to be determined by reading the one or more markers. The one or more markers can include multiple markers disposed along an outer edge or side of the substrate and / or the base of the operating table. In some embodiments, the one or more markers are disposed at selected positions and / or orientations are predetermined to allow the orientation of the table to be determined by reading at least one of the one or more markers. In one aspect, the support structure is configured to allow the table substrate to be positioned along multiple degrees of freedom, for example six degrees of freedom, and the one or more markers can include a series of 2D barcodes extending around the operating table.

[0022] In some embodiments, the surgical table includes a sensor configured to read one or more markers on the surgical table when the surgical table is positioned together with and communicatively coupled to a remote surgical system having a manipulator assembly. In other embodiments, the sensor is associated with another device external to the surgical table, such as the manipulator assembly. The sensor may be a light-sensitive detector, such as a camera, an RFID detector, an echo location detector, a magnetic sensor, a laser detector, or other sensor suitable for locating at least one of the one or more markers relative to a common reference.

[0023] For a more complete understanding of the nature and advantages of the present invention, reference should be made to the following detailed description and accompanying drawings, in which: Other aspects, objects and advantages of the present invention will become apparent from the drawings and the following detailed description. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 is a plan view of a minimally invasive telesurgery system used to perform surgery in accordance with many embodiments. [Diagram 2] FIG. 1 is a perspective view of a surgeon control console of a telesurgery system according to many embodiments. [Diagram 3] FIG. 1 is a perspective view of an electronics cart of a telesurgical system according to many embodiments. [Figure 4] FIG. 1 is a schematic diagram of a telesurgery system according to many embodiments. [Figure 5A] FIG. 1 illustrates a partial view of a patient side cart (surgical robot) of a telesurgery system according to many embodiments. [Figure 5B] FIG. 1 is a front view of a telesurgery tool according to many embodiments. [Figure 6] FIG. 1 is a schematic perspective view of a telesurgery system according to many embodiments. [Figure 7] FIG. 1 is a schematic perspective view of another telesurgery system according to many embodiments. [Figure 8] FIG. 8 illustrates a telesurgery system according to various embodiments consistent with the schematic diagram of FIG. 7. [Figure 9] FIG. 9 illustrates the rotational limitations of the setup linkage relative to the orientation platform of the telesurgery system of FIG. [Figure 10] FIG. 1 illustrates a center of gravity diagram associated with rotational limits of a boom assembly of a telesurgical system according to many embodiments. [Figure 11A] FIG. 1 illustrates an exemplary manipulator assembly with a localization camera and a positionable surgical table with fiducial markers, in accordance with aspects of the present invention. [Figure 11B] 1 illustrates an exemplary manipulator assembly and a positionable surgical table including a drape affixed over the surgical table, in accordance with aspects of the present invention. [Figure 12] FIG. 1 illustrates an exemplary operating table that is positionable along six degrees of freedom. [Figure 13A] FIG. 1 illustrates an exemplary operating table that is positionable along six degrees of freedom. [Figure 13B] FIG. 1 illustrates an exemplary operating table that is positionable along six degrees of freedom. [Figure 13C] FIG. 1 illustrates an exemplary operating table that is positionable along six degrees of freedom. [Figure 14A] FIG. 1 illustrates an exemplary operating table having a base including fiducial markers in accordance with an aspect of the present invention. [Figure 14B] FIG. 1 illustrates an exemplary operating table having a base including fiducial markers in accordance with an aspect of the present invention. [Figure 15] FIG. 14C is a diagram showing a detailed example of the fiducial marker shown in FIG. 14B. [Figure 16] 1 is a schematic diagram of locating an object having fiducial markers using a sensor according to an aspect of the present invention. [Figure 17] FIG. 13 is a diagram of a camera sensor reading fiducial markers in a proximal base of a manipulator assembly in accordance with an aspect of the present invention. [Figure 18A] 1 is a plot demonstrating optical pose estimation based on reading fiducial markers, in accordance with an aspect of the present invention; [Figure 18B] 1 is a plot demonstrating optical pose estimation based on reading fiducial markers, in accordance with an aspect of the present invention; [Figure 19A] FIG. 1 is a plot illustrating the accuracy of a pose estimation method according to an aspect of the present invention. [Figure 19B] FIG. 1 is a plot illustrating the accuracy of a pose estimation method according to an aspect of the present invention. [Figure 20] FIG. 13 is a schematic diagram illustrating the dependency of set image parameters for reading fiducial markers as they vary with sensor distance according to an aspect of the present invention. [Figure 21] FIG. 2 is a schematic diagram illustrating integration of pose estimation according to an aspect of the present invention. [Figure 22] FIG. 2 is a schematic diagram illustrating integration of pose estimation according to an aspect of the present invention. [Diagram 23] 1 is a schematic diagram of an exemplary method according to an aspect of the present invention. [Figure 24] 1 is a schematic diagram of an exemplary method according to an aspect of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] In the following detailed description, various embodiments of the present invention are described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that the present invention may be practiced without the specific details. Additionally, well-known features may be omitted or simplified so as not to obscure the embodiments being described.

[0026] The kinematic link structure and control system described herein are particularly useful in assisting a system user in positioning a telesurgery manipulator structure on a particular patient. In conjunction with an actively driven manipulator used to interact with tissues and the like during treatment, a telesurgery system can have one or more kinematic link systems configured to help support and align the manipulator structure to a surgical work site. Although the highly configurable capabilities of these kinematic systems provide many advantages and advanced capabilities, it can be difficult to locate the manipulator mechanism of the manipulator assembly relative to a separate component such as a surgical table, especially when the surgical table is positionable separately from the manipulator assembly. Since it is often useful to position a patient in various orientations or alignments during or after surgery, it is desirable to be able to locate the manipulator assembly relative to the surgical table either during initialization or during a procedure, so that the position and / or orientation of the surgical table relative to the manipulator assembly can be determined and potentially utilized for calculated manipulator movements or surgical table movements (either automatically or user driven). Such localization and pose estimation methods allow further utilization of various computed motions of the manipulator, including but not limited to various null space motions and collision avoidance motions, as described in the related applications, and can be further used to determine the position and / or orientation of the surgical table relative to any manipulator or associated components of the manipulator assembly. The pose estimation can also be used according to various other features, such as any of those described in U.S. patent application Ser. No. 14 / 101,769, entitled "Collision Avoidance During Controlled Movement of Image Capturing Device and Manipulatable Device Movable Arms," ​​filed Dec. 10, 2013, which is incorporated by reference in its entirety or any of its references are incorporated by reference herein for all purposes.The systems, devices, and methods described herein apply to these particular surgical systems, but may be used with a variety of different types of manipulator systems in accordance with aspects of the invention described herein.

[0027] Minimally Invasive Telesurgery

[0028] Referring now to the drawings, like reference numerals represent like parts throughout the several views. FIG. 1 is a plan view of a Minimally Invasive Telesurgery (MIRS) system 10, which is typically used to perform a minimally invasive diagnostic or surgical procedure on a patient 12 lying on an operating table 14. The system may include a surgeon console 16 that is used by a surgeon 18 during the procedure. One or more assistants 20 may also participate in the procedure. The MIRS system 10 may further include a patient side cart 22 (surgical robot) and an electronics cart 24. The patient side cart 22 may steer at least one detachably coupled tool assembly 26 (hereafter simply referred to as a "tool") through a minimally invasive incision within the patient 12 while the surgeon 18 views the surgical site through the console 16. Images of the surgical site may be acquired by an endoscope 28, such as a stereo endoscope, which may be steered by the patient side cart 22 to orient the endoscope 28. The electronics cart 24 can be used to process images of the surgical site for later display to the surgeon 18 via the surgeon console 16. The number of surgical tools 26 used at one time will generally depend on the diagnostic or surgical procedure and the space constraints within the operating room, among other factors. When it is necessary to change one or more tools 26 used during surgery, the assistant 20 can remove a tool 26 from the patient side cart 22 and replace it with another tool 26 from a tray 30 in the operating room.

[0029] FIG. 2 is a perspective view of the Surgeon's Console 16. The Surgeon's Console 16 includes a left eye display 32 and a right eye display 34 for displaying a calibrated stereoscopic view of the surgical site to the Surgeon 18, enabling depth perception. The Console 16 further includes one or more input controls 36, which in turn operate one or more tools on the Patient Side Cart 22 (shown in FIG. 1A). The input controls 36 may provide the same degrees of freedom as their associated tools 26 (shown in FIG. 1A), providing telepresence to the surgeon, or may provide a sense that the input controls 36 are one with the tools 26, such that the surgeon has a highly discernible sense of direct control of the tools 26. To this end, position, force, and tactile feedback sensors (not shown) may be used to transmit position, force, and tactile sensations from the tools 26 through the input controls 36 back to the surgeon's hands.

[0030] The surgeon console 16 is typically located in the same room as the patient so that the surgeon can directly monitor the procedure, be physically present if necessary, and speak directly to an assistant rather than speaking over the telephone or other communication medium, however, the surgeon can be located in another room, an entirely different building, or other location remote from the patient to allow for remote surgical procedures.

[0031] 3 is a perspective view of the electronics cart 24. The electronics cart 24 may be coupled to an endoscope 28 and may include a processor for processing the captured images for subsequent display to the surgeon on the surgeon console or another suitable display located locally and / or remotely. For example, if a stereoscopic endoscope is used, the electronics cart 24 may process the captured images to present the surgeon with a calibrated stereoscopic image of the surgical site. Such calibration may include alignment between opposing images and may also include adjustment of the stereoscopic working distance of the stereoscopic endoscope. As another example, image processing may include the use of previously determined camera calibration parameters to compensate for imaging errors of the image capture device, such as optical aberrations.

[0032] FIG. 4 illustrates a schematic of a telesurgery system 50 (such as the MIRS system 10 of FIG. 1). As discussed above, a surgeon console 52 (such as the surgeon console 16 of FIG. 1) can be used by a surgeon to control a patient side cart (surgical robot) 54 (such as the patient side cart 22 of FIG. 1) during a minimally invasive procedure. The patient side cart 54 can capture images of the surgical site using an imaging device such as a stereo endoscope and output the captured images to an electronics cart 56 (such as the electronics cart 24 of FIG. 1). As discussed above, the electronics cart 56 can process the captured images in various ways before subsequent display. For example, the electronics cart 56 can overlay the captured images on a virtual control interface before displaying the combined image to the surgeon via the surgeon console 52. The patient side cart 54 can output the captured images for processing outside of the electronics cart 56. For example, the patient side cart 54 can output the captured images to a processor 58, which can be used to process the captured images. Images may also be processed by a combination of electronics cart 56 and processor 58, which may be coupled together to process captured images together, sequentially, and / or combinations thereof. One or more separate displays 60 may also be coupled to the processor 58 and / or electronics cart 56 for local and / or remote display of images, such as images of the surgical site or other related images.

[0033] The processor 58 typically includes a combination of software and hardware, where the software includes tangible media embodying computer readable code instructions for functionally performing the control method steps described herein. The hardware typically includes one or more data processing boards that may be co-located, but will most likely have components distributed among the manipulator structures described herein. The software most likely includes non-volatile media, and may also include monolithic code, but will more typically include a number of subroutines executed as needed in any of a wide variety of distributed data processing architectures.

[0034] 5A and 5B show the Patient Side Cart 22 and the surgical tool 62, respectively. The surgical tool 62 is an example of a surgical tool 26. The Patient Side Cart 22 shown is provided for steering three surgical tools 26 and an imaging device 28, such as a stereo endoscope, used to capture images of the surgical site. Steering is provided by a manipulator mechanism having multiple joints. The imaging device 28 and the surgical tools 26 can be positioned at and steered through the incision in the patient, maintaining a kinematic remote center at the incision to minimize the size of the incision. The images of the surgical site may include images of the tips of the surgical tools 26 when they are positioned within the field of view of the imaging device 28.

[0035] The surgical tool 26 is inserted into the patient's body by inserting a tubular cannula 64 through a minimally invasive access opening, such as an incision, a natural orifice, percutaneous penetration, etc. The cannula 64 is attached to a manipulator arm such that the shaft of the surgical tool 26 passes through the lumen of the cannula. The manipulator arm can send a signal indicating that the cannula has been attached.

[0036] Telesurgery system and modular manipulator support

[0037] 6 is a schematic perspective view of a telesurgery system 70 according to many embodiments. The surgical system 70 includes a mounting base 72, a support linkage 74, an orientation platform 76, a plurality of external set-up linkages 78 (two shown), a plurality of internal set-up linkages 80 (two shown), and a plurality of surgical instrument manipulators 82. Each manipulator 82 is operable to selectively articulate a surgical instrument attached to the manipulator 82 and is insertable into a patient along an insertion axis. Each manipulator 82 is attached to and supported by one of the set-up linkages 78, 80. Each external set-up linkage 78 is rotatably coupled to and supported by the orientation platform 76 by a first set-up linkage joint 84. Each internal set-up linkage 80 is fixedly attached to and supported by the orientation platform 76. Orientation platform 76 is rotatably coupled to and supported by support linkage 74 , which in turn is fixedly mounted to and supported by mounting base 72 .

[0038] In many embodiments, the mounting base 72 is movable and supported on a floor, thereby allowing the entire surgical system 70 to be selectively repositioned, for example, within a surgical room. The mounting base 72 may include a steerable wheel assembly and / or other suitable support mechanism that provides both selective repositioning as well as selectively preventing the mounting base 72 from moving from a selected position. The mounting base 72 may also have other suitable configurations, such as, for example, a ceiling mount, a floor fixed / pedestal mount, a wall mount, or an interface configured to be supported by other suitable mounting surfaces.

[0039] The support linkage 74 is operable to selectively position and / or orient the orientation platform 76 relative to the mounting base 72. The support linkage 74 includes a pedestal 86, a translatable column member 88, a shoulder joint 90, a boom base member 92, a boom first stage member 94, a boom second stage member 96, and a wrist joint 98. The pedestal 86 is fixedly attached to the mounting base 72. The translatable column member 88 is slidably coupled to the pedestal 86 for translation relative to the pedestal 86. In many embodiments, the translatable column member 88 translates relative to the pedestal 86 along a vertically oriented axis. The boom base member 92 is rotatably coupled to the translatable column member 88 by the shoulder joint 90. The shoulder joint 90 is operable to selectively orient the boom base member 92 in a horizontal plane relative to the translatable column member 88, the boom base member having a fixed angular orientation relative to the column 86 and the mounting base 72. The boom first stage member 94 is selectively translatable in a horizontal direction relative to the boom base member 92, and in many embodiments, the horizontal direction is aligned with both the boom base member 92 and the boom first stage member 94. The boom second stage member 96 is similarly selectively translatable in a horizontal direction relative to the boom first stage member 94, and in many embodiments, the horizontal direction is aligned with both the boom first stage member 94 and the boom second stage member 96. Thus, the support linkage 74 is operable to selectively set the distance between the shoulder joint 90 and the tip of the boom second stage member 96. A wrist joint 98 rotatably couples the tip of the boom second stage member 96 to the orientation platform 76. The wrist joint 98 is operable to selectively set the angular orientation of the orientation platform 76 relative to the mounting base 72 .

[0040] Each setup linkage 78, 80 is operable to selectively position and / or orient an associated manipulator 82 relative to the orienting platform 76. Each setup linkage 78, 80 includes a setup linkage base link 100, a setup linkage extension link 102, a setup linkage parallelogram link 104, a setup linkage vertical link 106, a second setup linkage joint 108, and a manipulator support link 110. In each setup linkage, the base link 100 of the outer setup linkage 78 can be selectively oriented relative to the orienting platform 76 by operation of the first setup linkage joint 84. In the embodiment shown, the base link 100 of each setup linkage of the inner setup linkage is fixedly attached to the orienting platform 76. Each internal setup linkage 80 may also be rotatably attached to the orientation platform 76 as well as the external setup linkage by an additional first setup linkage joint 84. The extension link 102 of each setup linkage is horizontally translatable relative to the base link 100 of the associated setup linkage, and in many embodiments, its horizontal orientation is aligned with the extension link 102 of the associated setup linkage. The parallelogram link 104 of each setup linkage is configured and operable to selectively translate the vertical link 106 of the setup linkage vertically while maintaining the vertical link 106 of the setup linkage vertically. In an exemplary embodiment, the parallelogram link 104 of each setup linkage includes a first parallelogram joint 112, a connecting link 114, and a second parallelogram joint 116. A first parallelogram joint 112 rotatably couples a connecting link 114 to the extension link 102 of the set-up linkage. A second parallelogram joint 116 rotatably couples a vertical link 106 of the set-up linkage to the connecting link 114.The first parallelogram joint 112 is rotatably coupled to the second parallelogram joint 116 such that a rotation of the connecting link 114 relative to the setup linkage extension link 102 is matched to a reaction rotation of the setup linkage vertical link 106 relative to the connecting link 114 so as to maintain the setup linkage vertical link 106 in a vertical orientation while selectively translating the setup linkage vertical link 106 vertically. The second setup linkage joint 108 is operable to selectively orient the manipulator support link 110 relative to the setup linkage vertical link 106, thereby selectively orienting the associated mounted manipulator 82 relative to the setup linkage vertical link 106.

[0041] FIG. 7 is a schematic perspective view of a telesurgery system 120 according to many embodiments. Because the surgical system 120 includes similar components to those of the surgical system 70 of FIG. 6, the same reference numbers are used for similar components, and corresponding descriptions of the similar components described above are applicable to the surgical system 120 and are omitted here to avoid redundancy. The surgical system 120 includes a mounting base 72, a support linkage 122, an orientation platform 124, a number of set-up linkages 126 (four shown), and a number of surgical instrument manipulators 82. Each manipulator 82 is operable to selectively articulate a surgical instrument attached to the manipulator 82 and is insertable into the patient's body along an insertion axis. Each manipulator 82 is attached to and supported by one of the set-up linkages 126. Each set-up linkage 126 is rotatably coupled to and supported by the orientation platform 124 by a first set-up linkage joint 84. The orientation platform 124 is rotationally coupled to and supported by the support linkage 122, which in turn is fixedly mounted to and supported by the mounting base 72.

[0042] The support linkage 122 is operable to selectively position and / or orient the orienting platform 124 relative to the mounting base 72. The support linkage 122 includes a column 86, a translatable column member 88, a shoulder joint 90, a boom base member 92, a boom first stage member 94, and a wrist joint 98. The support linkage 122 is operable to selectively set a distance between the shoulder joint 90 and a tip of the boom first stage member 94. The wrist joint 98 rotatably couples the tip of the boom first stage member 94 to the orienting platform 124. The wrist joint 98 is operable to selectively set an angular orientation of the orienting platform 124 relative to the mounting base 72.

[0043] Each setup linkage 126 is operable to selectively position and / or orient an associated manipulator 82 relative to the orienting platform 124. Each setup linkage 126 includes a setup linkage base link 100, a setup linkage extension link 102, a setup linkage vertical link 106, a second setup linkage joint 108, a tornado mechanism support link 128, and a tornado mechanism 130. The setup linkage base link 100 of each setup linkage 126 can be selectively oriented relative to the orienting platform 124 by operation of the associated first setup linkage joint 84. The setup linkage vertical link 106 of each setup linkage is selectively translatable vertically relative to the associated setup linkage extension link 102. The second set-up linkage joint 108 is operable to selectively orient the tornado mechanism support link 128 relative to the set-up linkage vertical link 106.

[0044] Each tornado mechanism 130 includes a tornado joint 132, a connecting link 134, and a manipulator support 136. The connecting link 134 fixedly connects the manipulator support 136 to the tornado joint 132. The tornado joint 130 is operable to rotate the manipulator support 136 about a tornado axis 136 relative to the support link 128 of the tornado mechanism. The tornado mechanism 130 is configured to position and orient the manipulator support 136 such that a remote center of steering (RC) of the manipulator 82 intersects with the tornado axis 136. Thus, movement of the tornado joint 132 can be used to reorient the associated manipulator 82 relative to the patient without moving the associated remote center of steering (RC) relative to the patient.

[0045] Figure 8 is a simplified diagram of a telesurgery system 140 according to many embodiments that corresponds to the schematic diagram of telesurgery system 120 of Figure 7. Because surgical system 140 corresponds to telesurgery system 120 of Figure 7, the same reference numbers are used for like components, and corresponding descriptions of like components discussed above are applicable to surgical system 140 and will be omitted here to avoid redundancy.

[0046] The support linkage 122 is configured to selectively position and orient the orientation platform 124 relative to the mounting base 72 by relative movement between the links of the support linkage 122 along a plurality of set-up structure axes. The translatable column member 88 is selectively repositionable relative to the pedestal 86 along a first set-up structure (SUS) axis 142, which in many embodiments is oriented vertically. The shoulder joint 90 is operable to selectively orient the boom base member 92 relative to the translatable column member 88 about a second SUS axis 144, which in many embodiments is oriented vertically. The boom first stage member 94 is selectively repositionable relative to the boom base member 92 along a third SUS axis 146, which in many embodiments is oriented horizontally. The wrist joint 98 is operable to selectively orient the orientation platform 124 relative to the boom first stage member 94 about a fourth SUS axis 148, which in many embodiments is oriented vertically.

[0047] Each setup linkage 126 is configured to selectively position and orient an associated manipulator 82 relative to the orienting platform 124 through relative movement between the links of the setup linkage 126 along a plurality of setup joint (SUJ) axes. The joints 84 of each first setup linkage are operable to selectively orient a base link 100 of the associated setup linkage relative to the orienting platform 124 about a first SUJ axis 150, which in many embodiments is oriented vertically. The extension link 102 of each setup linkage can be selectively repositioned relative to a base link 100 of the associated setup linkage along a second SUJ axis 152, which in many embodiments is oriented horizontally. Each set-up linkage vertical link 106 can be selectively repositioned relative to the associated set-up link extension link 102 along a third SUJ axis 154, which in many embodiments is oriented vertically. Each second set-up linkage joint 108 is operable to selectively orient the tornado mechanism support link 128 relative to the set-up linkage vertical link 106 about the third SUJ axis 154. Each tornado joint 132 is operable to rotate the associated manipulator 82 about the associated tornado axis 138.

[0048] FIG. 9 illustrates the rotational limits of the set-up linkages 126 relative to the orienting platform 124 according to many embodiments. Each set-up linkage 126 is shown with a clockwise limit orientation relative to the orienting platform 124. A corresponding counterclockwise limit orientation is shown mirrored in FIG. 9 with respect to a vertically oriented mirror surface. As shown, each of the two inner set-up linkages 126 can be oriented from 5 degrees in one direction from the vertical reference 156 to 75 degrees in the opposite direction from the vertical reference 156. As shown, each of the two outer set-up linkages can be oriented from 15 degrees to 95 degrees in a corresponding direction from the vertical reference 156.

[0049] In use, it will often be desirable for a surgical assistant, surgeon, technical support, or other user to configure some or all of the linkages of the telesurgery system 140 for surgery to include the linkages, set-up joints, and / or manipulators of the set-up configuration. The tasks involved in configuring these linkages include positioning the orientation platform 124 relative to the first stage member 94 about the fourth SUS axis 148 in the direction of the wrist joint 98. A joint drive motor 121 and / or brake system 123 are coupled to the wrist joint 98, and in one exemplary embodiment includes both a drive motor 121 and a brake 123. Additionally, a joint sensor system typically senses the angular setting or position of the wrist joint 98.

[0050] Exemplary user interfaces, systems, and methods for manually configuring the system for use are described herein with reference to manual articulation of the wrist joint 98 about the fourth SUS axis 148 of the orientation platform 124, as shown diagrammatically by arrow 127. It should be understood that alternative embodiments, including one or more alternative joints of the setup structure, one or more setup joints, or one or more joints of the manipulator linkage, may be used to articulate one or more alternative joints of the overall kinematic system. Use of the exemplary embodiment for articulating the motorized wrist joint allows the user to efficiently position the manipulator 82. Manual articulation of the wrist joint 98 as described herein may improve speed and ease of use during manual docking of the manipulators 82 to their associated cannulas 64, as shown in FIG. 5B.

[0051] 10 illustrates a center of gravity diagram associated with rotational limits of a support linkage of a telesurgical system 160 according to many embodiments. With the components of the telesurgical system 160 positioned and oriented to maximally shift the center of gravity 162 of the telesurgical system 160 to one side relative to the support linkage 164 of the surgical system 160, the shoulder joint of the support linkage 164 can be configured to limit rotation of the support structure 164 about a shoulder joint axis 166 of the set-up structure (SUS) to prevent exceeding a predetermined stability limit of the mounting base.

[0052] FIG. 11 shows an overview of an exemplary system including a Patient Side Cart 22 having multiple manipulator arms 82 supported by linkages 126 of an associated setup structure, under which a surgical table 200 is positioned. In certain aspects, the surgical table 200 is a separate structure from the Patient Side Cart, whereby the surgical table is positionable separately from, and in most cases independently from, the Patient Side Cart. However, it is understood that in certain other aspects, the localization and pose estimation methods described herein allow a separately positionable surgical table to be controlled in conjunction with the calculated movements of the manipulators, whereby the surgical table, while separately positionable, may no longer be considered independently positionable as such movements are coordinated by the system. In many embodiments, the surgical table 200 includes a surgical table patient support surface 210 supported by columns 204 attached to a support base 202. The system further includes fiducial markers 300 that enable the system to align the operating table relative to the patient side cart so that the spatial relationship between the manipulators of the patient side cart and the patient surface 210 of the operating table can be determined and utilized for calculated manipulator movements or commanded operating table translations.

[0053] FIG. 12 illustrates an exemplary operating table 200 for use with the surgical manipulator system. The operating table 200 can include one or more joints (not shown) that, when attached, move the top surface of the operating table to a desired position and / or orientation. The one or more joints can include driven joints, manual articulation joints, or combinations thereof. Such joints can include translatable joints, such as hydraulic, as well as rotatable and pivoting joints, such as any described herein. The one or more joints can be adjusted by a patient assistant or anesthesiologist, as needed, or can be configured to be adjusted by a more remote user, such as a physician at a surgeon console, or by a system following an autonomous algorithm, or according to one or more calculated movements, such as movements that compensate for physiological movements, such as the patient's breathing.

[0054] The operating table 200 includes an operating table patient support surface 210 supported on columns 204 extending vertically from a support base 202. Typically, the operating table 200 is positionable along at least one degree of freedom, preferably along multiple degrees of freedom, and more preferably along six degrees of freedom. As shown in FIG. 12, the exemplary operating table 200 can translate in three different mutually orthogonal directions, along the x-axis, the y-axis, and vertically along the z-axis, and can pivot about an axis 214 extending along the length of the patient support surface 210, pivot about an axis 216 extending along the width of the patient support surface 210, and pivot about an axis 212 extending vertically. These pivotal movements are illustrated in FIGS. 13A-13C. Thus, the exemplary operating table 200 is positionable along six degrees of freedom. It should be understood that these pivot axes do not necessarily intersect. Various positions and / or orientations of the patient support surface 210 allowed by these six degrees of freedom can be utilized during initial setup to achieve a desired position, orientation, or tilt of the patient, and can be used during a procedure depending on the need to reposition the patient for any reason. It is understood that the pivoting motion need not necessarily be centered along these particular axes illustrated, and that the operating table can provide such pivoting motion in the same direction along various other axes, thereby allowing the operating table top to provide pivoting motion for various positions on or off the operating table top. In some embodiments, the operating table is configured to provide such motion about an isocenter or near a cannula through which instruments are inserted into a minimally invasive opening.

[0055] While this high degree of configurability of the operating table provides many advantages and versatility in positioning the patient, this configurability may still pose additional challenges in calculating the motions of the manipulator arms and associated tools. For example, when positioning the operating table at an angle, certain movements of the tools or associated manipulators supporting the tools may result in collisions with the patient support surface. Various methods may be used to avoid such collisions, but are particularly useful when determining the position of the operating table relative to the manipulators of the Patient Side Cart, such that the movements of the manipulators are calculated taking into account the position of the operating table and / or are calculated to compensate for the movement and / or repositioning of the operating table during the procedure. To enable such determinations, methods and systems according to aspects of the invention provide a table pose estimate of the operating table relative to the Patient Side Cart, thereby determining the spatial relationship between the operating table and the Patient Side Cart, and can be utilized for various calculated movements as needed. In another aspect, the methods of the invention may be used to identify an operating table for use with a particular manipulator assembly and / or to establish communication with the operating table to enable various calculated coordinated movements between those particular manipulators and the operating table.

[0056] In one aspect, the relative pose of the operating table and the robot is needed since both the robot and the operating table articulate the object with multiple degrees of freedom. By determining the pose of the operating table relative to the manipulator assembly, various advantageous functions can be realized by performing coordinated motion between the operating table and the manipulator assembly. Since each degree of freedom of the operating table and the manipulator assembly can be sensed by a sensor, determining the relative pose between any two parts of the components is sufficient. In one aspect, the method determines the position and orientation of the base of the operating table and enables the system to determine the relative positions of various components of each of the operating table and the manipulator assembly with respect to each other.

[0057] In a particular embodiment, the operating table is controlled to rotate around the center of gravity of the port to minimize the overall movement of the port (isocenter concept). This is useful for various functions, especially "port dragging". Active actuation of the robot arm is required to follow the movement of the operating table, when port dragging leads to very large forces on the patient's body wall. Knowledge of the patient placement (through the placement of the operating table) relative to the robot helps the robot optimize its control to avoid potential collisions with the patient. There are several ways to measure the relative pose between two objects. There are some constraints regarding the selection of an optical solution as the preferred method.

[0058] In certain aspects, the method of the present invention provides various advantageous features over conventional manipulator systems. In one aspect, the method includes determining a 6 degree of freedom 3D pose (translation and rotation) between the surgical robot and the surgical table using a camera attached to the robot and fiducial markers on the exterior surface of the surgical table. The 3D pose is composed of a 3 degree of freedom 2D pose on a ground plane, with both components being placed on a common ground plane. The fiducial markers can be subdivided into components that can be recognized and identified independently, or a subset of the fiducial markers can be recognized and identified relative to the entire marker. The fiducial markers are embedded with redundancy and correction schemes, such as checksums or Reed-Solomon error checks. In certain aspects, the fiducial markers include visual features that can be precisely located in the image used for pose estimation. The fiducial markers may be composed of multiple unique 2D barcodes, which can be Data Matrix codes. The fiducial markers may be composed of multiple unique 1D barcodes that include vertical bars. In another aspect, the fiducial marker may consist of multiple single point light emitting elements in a flashing pattern (eg, Morse code) and may also be combined with color to form a unique identifier for the device.

[0059] It is understood that embodiments may include one or more markers including any aspect of the fiducial markers described herein. For example, a surgical table may include one or more markers including a light emitting device, an RFID tag, or a 1D, 2D, or 3D barcode in combination with any aspect described herein. Also, such markers may be included at various different locations on the surgical table to allow sensing of each marker by one or more corresponding sensors associated with one or more other external devices. It is further understood that the method of identifying the position of the surgical table relative to the manipulator assembly may be applied to non-surgical applications, such as, for example, simulation, testing, and / or setup of a remote surgical system, as well as various other applications including, but not limited to, various industrial robotics applications.

[0060] In certain other aspects, the fiducial markers can identify the make, model, and serial number of the operating table used with a particular manipulator assembly. The marker pattern may include a unique identifier for each operating table. The software can be configured to detect the fiducial markers in the image and extract the location of the visual feature. The unique identifiers of the corresponding authorized fiducial markers are established by using the known 3D locations of the markers. The corresponding 3D and 2D observations are used to calculate the 3D pose of the operating table relative to the camera. In certain embodiments, multiple surgical robots or disjoint surgical robot components are equipped with respective cameras. In one aspect, the relative pose is calculated by observing a common operating table, including the fiducial markers on its exterior surface, by the cameras. It is understood that the cameras are not required to have a common view. Among the many advantages of these methods is that in many embodiments, no physical connection between the manipulator assembly and the operating room operating table is required for localization. As long as a line of sight is maintained between a sensor associated with the manipulator and at least one marker pattern on the operating table, the localization is not affected by the environment (e.g., EMI). In some aspects, once localization is established, line of sight does not need to be maintained unless the table is repositioned. In other aspects, a movable support structure on which the top surface of the table is positioned communicates wirelessly with the manipulator assembly, whereby changes in the position of the table after localization are accounted for without the need to reposition the table using one or more fiducial markers.

[0061] 14A-14B show an exemplary operating table according to an embodiment of the present invention. Each figure shows a positionable operating table 200 having a patient support surface 210, a vertical support column 204, and a movable support base 202 having fiducial markers 300 thereon. In one embodiment, the base is metal and the fiducial markers are laser etched, stamped, or embossed into the operating table, for example, on the side of the base or top substrate of the operating table. This method of manufacturing the fiducial markers is advantageous because the base or top of the operating table can be easily sterilized, as opposed to various other types of markers. Furthermore, such markers are not easily removed so that the markers can be used to ensure proper identification of the operating table for use with a particular manipulator assembly.

[0062] FIG. 15 shows a detailed example of the fiducial marker shown in FIG. 14B. Such markers may also include 3D capabilities such that the distance of the markers may be determined by a sensor associated with the manipulator assembly (typically a sensor or camera located at the base of the manipulator assembly). Distance may also be determined by the size and amount of markers read within the reading area of ​​the sensor reader (shown as a circle in FIG. 15). In one aspect, the markers may be in the form of any marker suitable for determining the relative position and / or orientation of the surgical base with respect to the manipulator assembly. Typically, the marker scheme and sensor module are configured to estimate the 2D pose of the object on the ground using the markers attached to the object. In one aspect, the markers include error checking redundancy provided by a bar code. A durable label material that can withstand chemical cleaning is preferred. The markers may be configured as a unique operating table ID as part of a machine-readable marker. In certain embodiments, the markers are typically between 2 cm and 12 cm in height and width and may include a series of markers along the surface of the operating table base, often around the entire operating table. In one aspect, the markers are configured such that they are displayed within a horizontal field of view of 60 degrees or greater, such that they can be read by a sensor in the base of the manipulator assembly at a variety of different positions and / or orientations. These aspects can be further understood with reference to the schematic diagram of FIG.

[0063] In one aspect, the markers and sensors are configured according to various range of motion considerations, including any or all of the following: The sensor module performs pose estimation when the distance between the sensor and the front of the target is in the range of 5cm to 100cm. Note that the sensor module may be configured to operate when the target is either in the center or off-center of the sensor's field of view.

[0064] A sensor module that operates at an extended range. The sensor module is configured to perform pose estimation when the object is at any angle to the sensor.

[0065] In one aspect, the markers and sensors are configured according to various lighting considerations, including any or all of the following:

[0066] · Environments can range from very low light (5 lux) to bright operating room lighting (2000 lux).

[0067] If necessary, additional lighting can be placed closer to the sensor, possibly near infrared.

[0068] Additional lighting can be configured to be activated, if desired, when image data collection is taking place.

[0069] Estimating the posture of an operating table in an operating room

[0070] In one aspect, one distinguishing feature of a manipulator assembly that benefits from these methods is the ability to move the patient table during a procedure without undocking and redocking. To accommodate the movement of the table during surgery, an operating room (OR) table can be used that pivots about a point in 3D. In some embodiments, the pivot point coincides with the center of gravity of all ports, thereby minimizing the overall displacement of all ports. This may require that the relative orientation of the Patient Side Cart (PSC) and the OR table be known. Since full kinematic information is available for both the PSC and the OR table, the orientation between any part on the PSC and any part on the OR table is sufficient. These methods address various issues associated with conventional marker reading and localization techniques, including any or all of the following: no mechanical link between the OR table and the PSC; no cables between the OR table and the PSC. These methods address the challenges associated with the OR environment, including low lighting associated with robotic surgery that does not require light, particularly bright standard operating rooms, possible interference, cauterization noise, and the presence of IR and / or RF links.

[0071] In one aspect, the markers and sensors are configured to provide sufficient accuracy. In some embodiments, the accuracy requirement is that the center of the operating table is measured within + / - 10 mm, which is primarily driven by the size of the cannula. Even with zero error, the manipulator has some offset at each port (e.g., the pivot point is the center of gravity of all ports), thus allowing for large errors.

[0072] In another aspect, the sensor module must fit within the space allotted in the PSC base. In some embodiments, the space allotted for the sensor module in the base of the manipulator assembly is approximately 3 inches by 3 inches by 5 inches, with a distance from the optical window to the frontmost face being approximately 2 inches. FIG. 17 shows one possible placement of the sensor camera in the base of the manipulator assembly. In some embodiments, shock absorbing material is included in the sensor or camera mount.

[0073] In another aspect, the sensor and marker configuration is configured to have a range to be read to allow sufficient working distance between the operating table and the manipulator assembly, which working distance is the working range of the position and / or orientation of the operating table. For example, the markers and sensors can be configured to read the markers within a horizontal field of view close to 60 degrees, and the working distance to read the markers is between 1 inch and 2 feet or more. The operating table can also include markers on both sides of the support base to allow full rotation of the operating table when needed for a particular procedure.

[0074] In certain aspects, the markers and sensors may be configured to track one or more markers using any or all of the following: electromagnetic tracking, optical tracking, laser-based, shape sensors, ultrasonic means, or any combination thereof. However, due to drawbacks associated with these particular approaches, methods typically utilize optical tracking of markers to perform pose estimation as described herein. One advantage of optical tracking is that it is not affected by unseen obstacles (e.g., electromagnetic (EM) interference). A downside is line-of-sight issues, since the fiducial marker must be visible. However, unlike EM interference, this is much easier for the user to understand what to correct (e.g., keep debris away between the camera and the marker). The basic principle of optical tracking is to calculate the pose of an object or fiducial marker using a known geometry from observing its image. In another aspect, a monocular or pair of stereo cameras can be used.

[0075] In one embodiment, multiple passive retroreflective balls or active blinking LEDs form a geometrically unique configuration. In one embodiment, the fiducial markers include a 2D barcode. Any 2D barcode can be used as a marker as long as the reader provides image coordinates of several fiducial points (ideally >=4) of the barcode. In another embodiment, active point markers may be used. Multiple active point markers (e.g., LEDs) are placed on the periphery of the operating table base. Their blinking patterns are unique to make them easily self-identifiable. If three or more points are displayed in the field of view, the pose can be calculated. Since the markers are light-emitting fiducial markers, the contrast is higher than passive markers and no light source is required for the PSC. The overall power consumption of the illumination is very low. Unlike passive markers, which require powered and controlled markers, the passive markers are less desirable. If all fiducial markers are at the same level, a 1000 CCD sensor can be used.

[0076] optical pose estimation

[0077] 18A-B show plots illustrating pose estimation using optical tracking. The plots shown above show the theory of pose estimation in a simplified setup, where the center of the camera is at the entrance of the pattern and the camera is oriented horizontally. The pattern in the image is mirrored by the horizon, so only the top part is considered. The x-coordinate of p determines its angle of the ray in the top view. The coordinate of p(h) determines the distance (d). The 3D rotation of the marker is mapped 1:1 to the image rotation of the trapezoid side. With these three quantities in the image, the pose can be fully determined in the top view with 3 degrees of freedom. In this example, there are 8 measurements (4 coordinates in the image). This is an over-determined system, and the remaining measurements are used to make the estimation more accurate.

[0078] Such an estimation formula is as follows:

[0079] d=(H*f) / h where H is the physical half-height of the marker and f is the focal length in pixels (f=(resolution / 2) / tan(field of view / 2)). It is clear that the relationship between h and d is non-linear. (By comparison, an overhead camera provides a linear relationship and maximized utilization of image resolution.) The depth resolution can be calculated by the derivative of d with respect to h:

[0080] d'=-H*f / (h''2)

[0081] The plots in Figures 19A-B show the depth resolution. In this example, an image error of one pixel translates into a distance error of 10 mm (-2') at a distance of 600 mm, roughly meeting the accuracy requirements. In some embodiments, the pose estimation involves determining the 3D pose (3 translations and 3 rotations) of the object using a known geometry from a calibrated monocular camera. The module is also capable of tool tracking. In another aspect, the pose estimation is performed on a surface (e.g., the ground plane) where only one surface is needed to estimate the three degrees of freedom (2 translations and 1 rotation). In such cases, there are at least two options: Perform 3D pose estimation and project the 3D pose back to a 2D pose.

[0082] Parameterize the 2D pose and perform 2D pose optimization.

[0083] In some embodiments, the fiducial markers are configured to wrap around and completely cover the base of the operating table. All areas can be covered by markers to maximize the probability that at least one marker is visible during the procedure. The size may be determined by constraining both the near and far ends of the working range. The width-height ratio of the barcode is designed to be larger on one side due to the fact that horizontal foreshortening is larger than vertical foreshortening due to rotation. A more quantitative analysis is provided in the next paragraph. Self-identifiable sub-markers are one way to deal with partial occlusions when the viewpoint is unknown. Each sub-marker has a unique ID and can be associated with a model. One sub-marker provides sufficient constraints for pose estimation, but more sub-markers improve accuracy.

[0084] In one aspect, the size of the barcodes includes two factors: a lower limit is needed to make them detectable at the maximum distance and at the maximum rotation angle; and an upper limit is needed to ensure that at least the entire pattern is displayed at the minimum working distance. In another aspect, the markers and sensors are configured to maintain line of sight to the markers. As can be seen by referring to FIG. 11B, the operating table is draped during the procedure to hide much of the operating table from view. It is therefore particularly useful to configure the fiducial markers and sensors such that the line of sight between them and the sensor remains unobscured when the operating table is draped. One means to achieve this is to position the fiducial markers at the base of the operating table and the sensor in the base of the manipulator assembly. It should be noted that theoretically no personnel or equipment should be between the two parts (fiducial markers and sensor) after the system is docked. Alternatives may include infrared, RFID tags, or various other means that could potentially penetrate draping materials or other such obstructions. An alternative might be to include multiple sensors so that if the line of sight is blocked by one sensor, another sensor can be used.

[0085] 21 and 22 show two alternative approaches that can be used to incorporate cameras and barcode readers into existing teleoperated surgical manipulator systems. In one aspect, the pose of the camera frame needs to be transformed into the PSC frame to be useful. This can include two transformations that are chained together: transforming from internal camera to camera housing; transforming from camera housing to PSC. In one aspect, it is particularly useful if the second one mentioned above (transforming from camera housing to PSC) can be done accurately so that no calibration is required. This allows all calibration to be done at the factory so that no field calibration is required in the event that the camera needs to be replaced.

[0086] 23-24 illustrate an exemplary method according to an embodiment of the present invention. In one example, the method of pose estimation includes converting a 6-DOF 3D pose of a positionable operating table to a 3-DOF 2D pose on a ground surface; reading markers on a base of the operating table with a camera associated with the manipulator assembly; locating the operating table relative to the manipulator assembly on the ground surface based on the reading of the fiducial markers; and estimating the 3D pose of the operating table relative to the manipulator assembly based on the locating of the base of the operating table relative to the manipulator assembly on the ground surface. Some methods further include extracting the position of the operating table in a visualized image of the camera; and providing a representation or display of the operating table relative to the camera view. In another example, the method includes reading markers on a base of the operating table with a camera associated with the manipulator assembly; and identifying the make, model, or manufacturer of the operating table based on the reading of the markers. Some methods further include checking compatibility aspects between the operating table and the manipulator assembly and / or providing permission to use the manipulator assembly with the operating table. Such a method may include outputting the make, model, or make of the operating table and / or a calculation of coordinated motion between the operating table and the manipulator assembly for calculation of a pose estimate of the operating table.

[0087] Other variations are within the spirit of the invention. Thus, while the invention is susceptible to various modifications and alternative constructions, specific example embodiments thereof are shown in the drawings and have been described above in detail. It should be understood, however, that there is no intention to limit the invention to the particular form(s) disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention as defined by the appended claims.

[0088] The use of "a," "an," "the," and similar referents in the context of describing the present invention (particularly in the context of the claims below) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by the context of this specification. The terms "comprising," "having," "including," and "containing" are to be construed as open-ended terms (i.e., meaning, but not limited to, "including"), unless otherwise indicated. The term "connected" is to be construed as being partially or completely contained in, attached to, or joined together with something, even if there is something intervening therebetween. The recitation of ranges of values ​​herein is merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if each separate value were set forth individually herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or clearly contradicted by the context of the specification. The use of any and all examples or exemplary language (e.g., "etc.") described herein is intended merely to better illustrate embodiments of the invention and does not limit the scope of the invention unless otherwise noted. Language in the specification should be construed to indicate any non-claimed element essential to the practice of the invention.

[0089] All references mentioned in this specification, including publications, patent applications, and patent documents, are incorporated by reference herein to the same extent as if each was individually indicated to be incorporated by reference and was set forth in its entirety herein.

[0090] The contents of the claims as originally filed are described below as examples. [Example 1] 1. A method for estimating a posture of an operating table, the method comprising: reading one or more fiducial markers on a base of the surgical table using a camera associated with a manipulator assembly adjacent to the surgical table; and determining a position on a common surface of the operating table relative to the manipulator assembly based on reading of the one or more markers. method. [Example 2] 2. The method of example 1, wherein the one or more markers include one or more fiducial markers at fixed positions on the operating table. [Example 3] The method of example 1, comprising the steps of attaching the one or more markers to the operating table and / or forming the one or more markers at selected positions on the operating table in a specific orientation relative to the operating table. [Example 4] 2. The method of claim 1, wherein forming the one or more markers on the operating table comprises etching, engraving, and / or embossing the one or more markers on the operating table. [Example 5] The method of example 1, wherein reading the one or more markers includes reading at least one of the one or more markers before and / or during a surgical procedure while a patient is supported on the operating table. [Example 6] 2. The method of claim 1, wherein the common plane is a ground surface on which the operating table and the manipulator assembly are placed. [Example 7] The method of example 1, wherein the operating table is positionable along one or more degrees of freedom (DOF). [Example 8] 8. The method of example 7, further comprising converting a 6 degree of freedom 3D pose of the positionable operating table into a 3 degree of freedom 2D pose on the common plane. [Example 9] 2. The method of example 1, further comprising estimating a 3D orientation of the operating table relative to the manipulator assembly based on localization on the common plane of the base of the operating table relative to the manipulator assembly. [Example 10] The method of example 1, further comprising the steps of extracting a position of the operating table within a visualization image acquired using the camera, and providing a representation or display of the operating table relative to a camera view of the camera. [Example 11] 2. The method of example 1, further comprising identifying the make, model, or manufacturer of the operating table based on reading the marker. [Example 12] The method described in Example 11, further comprising a step of checking compatibility between the operating table and the manipulator assembly, and / or a step of providing permission to use the manipulator assembly together with the operating table. [Example 13] 2. The method of example 1, wherein the one or more markers include a plurality of markers disposed at a plurality of positions on the operating table, and reading the one or more markers includes reading at least one of the one or more markers. [Example 14] 14. The method of example 13, wherein the step of reading the one or more markers includes the step of reading a single marker of the one or more markers, and the location of the operating table is based on reading the single marker. [Example 15] reading at least one of the one or more markers after an initial reading of the one or more markers and determining a location based on the initial reading; The method of example 13, further comprising: determining a position on the common surface of the operating table relative to the manipulator assembly based on subsequent readings of the one or more markers to update and / or verify the initial position determination. [Example 16] 1. A telesurgery system, comprising: A manipulator assembly; a surgical table disposed proximate to the manipulator assembly, the surgical table having one or more fiducial markers; a sensor configured to read the one or more markers on the surgical table when the surgical table is positioned proximate to a manipulator assembly. system. [Example 17] The system of example 16, wherein the operating table is positionable along one or more degrees of freedom (DOF). [Example 18] The system of example 16, wherein the sensor includes a camera of the manipulator assembly. [Example 19] The system of example 16, wherein the one or more fiducial markers include a barcode, an RFID tag, a light, or any combination thereof. [Example 20] The system of Example 16, wherein the one or more markers are fixedly attached and / or formed to selected portions of the operating table. [Example 21] The system of Example 16, wherein the one or more markers are fixedly attached and / or formed at selected portions and selected positions in a specific orientation relative to the operating table such that the orientation of the operating table relative to a ground surface can be determined by reading at least one of the one or more markers. [Example 22] The system of Example 16, wherein the one or more fiducial markers include one or more 2D barcodes extending around the base of the operating table and / or along an edge or side of the operating table top surface of the operating table. [Example 23] The system comprises: reading one or more markers on a base of the surgical table with a camera associated with a manipulator assembly adjacent to the surgical table; and a processor configured to determine a position on a common surface of a surgical table relative to a manipulator assembly based on reading of the one or more markers. The system described in Example 16. [Example 24] The surgical table is positionable along six degrees of freedom, and the processor: The system of Example 23, further configured to convert a 6-degree-of-freedom 3D posture of the positionable operating table into a 3-degree-of-freedom 2D posture on the common plane. [Example 25] The processor, The system of example 23, further configured to estimate a 3D orientation of the operating table relative to the manipulator assembly based on localization of the base of the operating table relative to the manipulator assembly on the common plane. [Example 26] The processor, Extracting the position of the operating table in a visualized image acquired using the sensor; The system of example 23, further configured to provide a representation or display of the operating table relative to a camera view of the camera. [Example 27] The system of example 16, further comprising a processor configured to identify the make, model, or manufacturer of the operating table based on reading the marker. [Example 28] The processor, The system described in Example 27, further configured to check compatibility between the operating table and the manipulator assembly and / or grant permission to use the manipulator assembly with the operating table. [Example 29] There is an operating table, a substrate having a patient support surface; a support structure supporting the substrate, the support structure being movable to enable the patient support surface to be positionable along one or more degrees of freedom; one or more fiducial markers arranged on the operating table at selected positions and / or orientations such that an orientation of the operating table relative to a ground surface can be determined by reading the one or more markers; Operating table. [Example 30] A surgical table as described in Example 29, wherein the one or more markers include a plurality of markers arranged along the outer edge or sides of the substrate and / or base of the surgical table. [Example 31] A surgical table as described in Example 29, wherein the selected position and / or selected orientation is predetermined so as to enable the posture of the surgical table to be determined by reading at least one of the one or more markers. [Example 32] The operating table of Example 29, further comprising a sensor configured to read the one or more markers of the operating table when the operating table is positioned together with and communicatively coupled to a remote surgical system having a manipulator assembly. [Example 33] 33. A surgical table as described in Example 32, wherein the sensor includes a photosensitive detector or a camera. [Example 34] An operating table as described in Example 32, wherein the support structure is configured to enable the substrate to be positioned along six degrees of freedom. [Example 35] The operating table of Example 32, wherein the one or more markers include a series of 2D barcodes extending around the operating table.

Claims

1. A surgical system, comprising: a manipulator assembly including a mounting base supported on a floor and a sensor disposed on the mounting base and configured to read a fiducial marker; a surgical table adjacent to the manipulator assembly, the surgical table including a support base, a patient support surface, and a support structure disposed between the support base and the patient support surface for moving the patient support surface along one or more degrees of freedom (DOF), the surgical table including one or more fiducial markers on the support base; a processor configured to read the one or more fiducial markers with the sensor to determine a distance between the one or more fiducial markers on the support base of the surgical table and the sensor to identify a position of the surgical table relative to the manipulator assembly. Surgery system.

2. The surgical system of claim 1 , wherein the one or more fiducial markers are positioned at fixed positions on the support base.

3. The surgical system of claim 1 or 2, wherein the one or more fiducial markers are affixed to the support base and / or formed at selected positions on the support base with a specific orientation relative to the surgical table.

4. The surgical system of claim 3 , wherein the selected position and orientation are predetermined such that a posture of the surgical table can be determined by reading the one or more fiducial markers.

5. The surgical system of claim 1 , wherein the one or more fiducial markers include one or more 2D barcodes extending around the support base of the surgical table.

6. The surgical system of claim 1 , wherein the one or more fiducial markers include one or more 2D barcodes.

7. The surgical system of claim 1 , wherein the one or more fiducial markers include an RFID tag or a light.

8. 8. The surgical system of claim 1, wherein the processor is configured to locate the manipulator assembly on a common plane of the operating table, the common plane being a ground surface on which the operating table and the manipulator assembly are placed.

9. 1. A method of operating a manipulator assembly, the method comprising: a sensor disposed on a mounting base supported on a floor of the manipulator assembly reading one or more fiducial markers on a support base of a surgical table adjacent to the manipulator assembly, the surgical table including the support base, a patient support surface, and a support structure disposed between the support base and the patient support surface for moving the patient support surface along one or more degrees of freedom (DOF); and a processor connected to the manipulator assembly determining a distance between the sensor and the one or more fiducial markers on the support base of the surgical table based on the sensor reading of the one or more fiducial markers. method.

10. The method of claim 9 , wherein the one or more fiducial markers are located at fixed positions on the support base.

11. The method of claim 10, wherein the one or more fiducial markers comprise one or more 2D barcodes extending around the support base of the surgical table.

12. 10. The method of claim 9, wherein the one or more fiducial markers are affixed to the support base or etched, engraved and / or embossed into the support base at a specific orientation relative to the operating table.

13. 10. The method of claim 9, further comprising the step of the processor determining a position on a common plane of the operating table relative to the manipulator assembly based on the result of reading the one or more fiducial markers, the common plane being a ground surface on which the operating table and the manipulator assembly are placed.

14. 14. The method of claim 13, wherein the step of the sensor reading the one or more fiducial markers comprises reading a single fiducial marker of the one or more fiducial markers, and the localization of the operating table is based on the reading of the single fiducial marker.

Images