3D and 2D Image Registration for Surgical Navigation and Robot Guidance without Using Radiopaque References within an Image

Abstract

To provide a system and method of registering a medical image of a patient in an imaging space to the patient in a physical space, preferably without the use of any embedded radiopaque fiducials in medical images.SOLUTION: In one method, intraoperative 2D medical images are used to register a preoperative unregistered 3D medical image. The 2D medical images are registered based on simultaneous tracking of tracking markers on an imaging device and on the patient by a tracking device at the time of image capture. The 2D images are matched to corresponding simulated 2D images generated from the preoperative 3D image volume. Thus, the registration of the preoperative 3D image to the patient is accomplished without performing another 3D scan of the patient.SELECTED DRAWING: Figure 28

Description

Technical Field

[0001] Cross - Reference to Related Applications This application is a continuation of U.S. Patent Application No. 18 / 184,192, filed on March 15, 2023, which claims priority to U.S. Patent Application No. 63 / 389,691, filed on July 15, 2022, and all of these applications are incorporated herein by reference.

[0002] This application is also related to, but does not claim priority to, (1) U.S. Patent Application No. 15 / 180,126 (U.S. Patent No. 10,842,453), filed on June 13, 2016, and (2) U.S. Patent Application No. 15 / 157,444 (U.S. Patent Publication No. 2016 / 0256225), filed on May 18, 2016, and all of these applications are incorporated herein by reference.

[0003] The present invention relates to a navigation system, and more particularly, to a system and method for aligning a medical image of a patient in an imaging space with the patient in a physical space.

Background Art

[0004] Surgical navigation has revolutionized minimally invasive spinal surgery by enabling surgeons to accurately and reproducibly place implant hardware with reduced radiation and surgical time, as opposed to conventional surgical techniques. In surgical navigation, position sensors are used to track the full rigid - body motion of surgical instruments relative to a medical image aligned with a patient - reference frame. The most common form of position sensing is optical tracking, specifically, near - infrared (NIR) passive retro - reflective markers or active NIR LEDs arranged in a pattern called an array. The position of the instrument being tracked is typically displayed to the surgeon user as a CAD model of the instrument overlaid on the medical image.

[0005] Robot guidance is a technique used to position a robotic arm on a desired trajectory and to guide an instrument on an accurate and repeatable trajectory. With the release of ExcelsiusGPS ("eGPS") from Globus Medical, Inc. (Audubon, PA), robot guidance has been integrated with surgical navigation in a single system called robotic navigation. Robotic navigation provides the benefits of both technologies in a single, streamlined system.

[0006] All surgical navigation, robot guidance, and robotic navigation systems require the alignment of medical images to a patient's anatomical structure through the use of a system's reference frame. Alignment of an optical navigation system typically involves the use of an alignment fixture that includes an array of tracking markers placed in known locations relative to an array of embedded radiopaque fiducials. The fixture is attached to surrounding structures such as the patient and the imaging system so that the fiducials are embedded in the medical image and the tracking markers are visible to a position sensor (e.g., the stereo cameras of a tracking device). After the image is captured, software algorithms use computer vision and image processing to identify the radiopaque fiducial locations within the image. Since the location of the fiducials relative to the tracking markers on the fixture is known and the cameras can identify the position of the tracking markers within the fixture array relative to the patient reference array (also known as the dynamic reference array or DRB), the system can then calculate a number of transformations to align the medical image to the tracking system.

[0007] There are several drawbacks to using alignment fixtures during surgery. The main drawback is the need to capture the image volume with radiopaque fiducials already in place on the patient, which means that pre-operative scans of the patient cannot be used for navigation. A further drawback is the loss of usable image space by including the fixture fiducials within the image volume / region. In addition, the fiducials can sometimes obstruct the view of important anatomical structures. When capturing images using alignment fixtures, special care must be taken to ensure that the required number of fiducials are included and are visible with sufficient contrast against the background. Finally, alignment fixtures are additional hardware components that must be stored, cleaned, draped / sterilized, and installed during surgery, leading to an increase in time and complexity for the surgical staff.

[0008] Accordingly, several companies (e.g., Medtronic O-arm and SteathStation) have developed methods (automatic alignment) for obtaining aligned intraoperative images without the need for alignment fixtures. To achieve this, these systems have a tracking array integrated or removable at a known location of the imaging system. The associated navigation system can then be used to track the position of the imaging system navigation array relative to the patient reference array and complete the image alignment.

[0009] However, existing solutions are limited in flexibility to adapt to surgical navigation and robotic workflows in a streamlined system. The use of intraoperative scanners to align tracking to a medical image volume requires re-irradiating the patient to capture new images. A solution that can use existing medical image volumes taken without fiducials is still needed.

[0010] Furthermore, the main failure state of navigated surgery is the loss of alignment due to the movement of the patient's anatomical structure relative to the patient reference array. If the alignment is lost during the surgical procedure, the surgical team needs to return the 3D imaging system to the surgical site and capture a newly aligned image. This need for additional 3D images exposes the patient to a significant amount of additional radiation, increases the time under anesthesia, and reduces hospital efficiency.

[0011] Therefore, it is desirable to provide a system and method for aligning medical images to a patient's anatomical structure without using implanted radiopaque fiducials. Furthermore, it is desirable to provide a system and method for quickly restoring alignment without using another full 3D scan of the patient if the alignment is lost during surgery.

Summary of the Invention

[0012] According to one aspect of the present invention, a system and method for aligning a preoperative 3D medical image of a patient in an imaging space to the patient in a tracking space are disclosed. The method receives a 3D image of the patient's anatomical structure taken preoperatively by an imaging device. At this point, the 3D image is not aligned with the patient lying on the operating table. Once the patient is prepared and lying on the operating table, the imaging device is rolled, and an intraoperative 2D image of the patient's anatomical structure is captured by an imaging device having imaging tracking markers that are trackable by a tracking device (e.g., an optical or electromagnetic tracking system).

[0013] The system receives the corresponding optical image of the patient from the tracking device at the time of image capture, and the optical image includes the imaging tracking markers and a dynamic reference base (DRB) including patient tracking markers that are trackable by the tracking device.

[0014] Next, the 2D image is matched with the corresponding 2D simulated medical image, which is a synthetically generated medical image at a selected orientation and position, digitally reconstructed from the preoperative 3D image. In the case of an X-ray medical imaging device, the simulated image is a DRR (Digital Reconstructed Radiograph) of the preoperative 3D image. In the illustrated embodiment, the DRR is a simulated 2D fluoroscopic image at a selected orientation and angle, digitally reconstructed from the preoperative 3D image (e.g., a set / stack of 2D slices of the 3D image volume). In the case of ultrasound, it is a synthetically generated 2D image representing an ultrasound scan at a specific orientation and position from the preoperative 3D image.

[0015] Next, the system determines the alignment of the position of the received preoperative 3D image with respect to the dynamic reference base based on the matched DRR and the patient tracking markers and imaging tracking markers included in the received optical image. Since the pose of the matched DRR corresponds to the tracked pose of the actual 2D image, the alignment is achieved.

[0016] Next, the method displays on the display the aligned 3D image and any selected 2D DRR of the 3D image, together with the tracked surgical instrument, its planned trajectory, and the end effector, superimposed on the 3D image displayed for visual navigation assistance.

[0017] Advantageously, the alignment method does not require the presence of a radiopaque reference in the medical image and does not require attaching an alignment fixture, as was previously required, to the imaging device. As a result, the method of the present invention can substantially shorten the treatment time and enhance patient safety.

[0018] According to another aspect of the present invention, a system and method are disclosed for aligning an intraoperative 3D image (such as a 3D CT or MRI image) of a patient in an imaging space with a physical patient in a tracked physical space without embedding a reference (e.g., a radiopaque marker) in the image. The system receives an intraoperative image of the patient's anatomical structure captured by an imaging device having an imaging tracking marker that can be tracked by a tracking device, and the patient has a dynamic reference base including a patient tracking marker that can be tracked by the tracking device. The system also receives an optical image of the patient from the tracking device at the time of image capture, and the optical image includes the patient tracking marker and the imaging tracking marker.

[0019] Next, the method determines transforms A, B, and C. Transform A, which represents the pose of the imaging device relative to the dynamic reference base, is determined based on the received optical image. Transform B, which represents the pose of the received image of the patient's anatomical structure relative to the imaging device, is determined based on the received optical image. Transform C, which represents the pose of the received image relative to the dynamic reference base, is determined by multiplying transform A and transform B. Transform C represents the alignment of the patient image in the imaging space with the physical patient in the physical space, and all of the transforms are performed without using any reference.

[0020] Next, the method displays the aligned image on a display together with a tracked surgical instrument, its planned trajectory, and an end effector overlaid on the displayed 3D image for visual navigation assistance.

[0021] Advantageously, since the alignment method does not require a radiopaque reference in the medical image, the method can substantially shorten the treatment time and enhance patient safety. Also, since no preoperative scan of the patient is required, costs can be saved and unnecessary radiation exposure to the patient can be eliminated.

[0022] According to another aspect of the present invention, a system and method are provided for restoring the alignment of a 3D image of a patient in an imaging space to a physical patient in a physical space. The system receives a 3D image of the patient's anatomical structure and aligns the pose of the received 3D image with a dynamic reference base including patient tracking markers. The aligned 3D image is then used during a surgical procedure.

[0023] However, if the alignment is lost, the alignment is re-established without performing another complete 3D scan of the patient. The system receives two or more intraoperative 2D images (e.g., fluoroscopy or ultrasound) of the patient's anatomical structure in different orientations captured by an imaging device having imaging tracking markers that are trackable by a tracking device (e.g., optical or electromagnetic). The system also receives a corresponding optical image of the patient from the tracking device at the time of image capture, and the optical image includes the patient tracking markers and the imaging tracking markers.

[0024] The received 2D images are matched with corresponding simulated 2D images (e.g., DRRs) of the 3D image. The alignment of the 3D image is re-established based on the matching corresponding DRRs and the patient tracking markers and imaging tracking markers included in the optical image.

[0025] The method then displays, for visual navigation assistance, the aligned 3D image on a display together with a tracked surgical instrument, its planned trajectory, and an end effector overlaid on the displayed 3D image.

[0026] Advantageously, since the alignment is restored using only a few 2D images without performing another complete 3D scan of the patient, the method can substantially shorten the procedure time and enhance patient safety.

[0027] According to another aspect of the present invention, a system and method for aligning intraoperative 2D medical images of a patient in an imaging space with the physical patient in a physical space are provided. The system receives intraoperative 2D images of patient anatomy in different orientations captured by an imaging device having an imaging tracking marker that is trackable by a tracking device, and the patient has a dynamic reference base including a patient tracking marker that is trackable by the tracking device. The system also receives a corresponding optical image of the patient from the tracking device at the time of image capture, and the optical image includes the patient tracking marker and the imaging tracking marker.

[0028] For each received 2D image, the method then determines the transformations A, B, and C as described above. Transformation C represents aligning the patient image in the imaging space with the physical patient in the physical space, and all of the transformations are performed without using a radiopaque reference.

[0029] The method then displays the aligned 2D image on a display, along with a tracked surgical instrument, its planned trajectory, and an end effector, superimposed on the displayed 2D image, for visual navigation assistance.

[0030] According to another aspect of the present invention, a system and method for aligning intraoperative 2D images of a patient in an imaging space with the physical patient in a physical space are provided. The system receives intraoperative 2D images of patient anatomy in different orientations captured by an imaging device having an imaging tracking marker that is trackable by a tracking device, and the patient has a dynamic reference base including a patient tracking marker that is trackable by the tracking device. The system also receives a corresponding optical image of the patient from the tracking device at the time of image capture, and the optical image includes the patient tracking marker and the imaging tracking marker.

[0031] Based on the received 2D images and a general 3D model, the method creates a customized 3D model. For each received 2D image, the method determines transforms A, B, and C as described above to align the 2D image. The 2D images are matched to corresponding DRRs of the customized 3D model, whereby the customized model can be aligned based on the matched DRRs.

[0032] The method then displays, for visual navigation assistance, the aligned 3D image on a display together with a tracked surgical instrument, its planned trajectory, and an end effector overlaid on the displayed 2D image. This method enables navigation with a synthetically created 3D model even when access to a 3D scanning imaging device is not available.

Brief Description of the Drawings

[0033]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9A

Figure 9B

Figure 9C

Figure 9D

Figure 9E

Figure 9F

Figure 9G

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16A

Figure 16B

Figure 16C

Figure 17

Figure 18A

Figure 18B

Figure 18C

Figure 19

Figure 20

Figure 21A

Figure 21B

Figure 22A

Figure 22B

Figure 22C

Figure 23A

Figure 23B

Figure 23C

Figure 23D

Figure 24A

Figure 24B

Figure 24C

Figure 24D

Figure 24E

Figure 25

Figure 26A

Figure 26B

Figure 27A

Figure 27B

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

DETAILED DESCRIPTION OF THE INVENTION

[0034] For the purposes of this application, the terms "code", "software", "program", "application", "software code", "software module", "module" and "software program" are used interchangeably to mean software instructions executable by a processor. A "user" can be a physician or other medical professional.

[0035] FIG. 1 is a schematic diagram showing an imaging system 10 such as a computed tomography (CT) X-ray scanner according to an embodiment of the present invention. The imaging system 10 includes a movable station 60 and a gantry 56. The movable station includes a vertical shaft 59 and a gantry mount 58 rotatably attached to the vertical shaft. The movable station 60 includes two front omnidirectional wheels 62 and two rear omnidirectional wheels 64, both of which move the movable station 60 in any direction within the X-Y plane. The omnidirectional wheels 62, 64 can be obtained, for example, from Active Robots Limited of Somerset, U.K. A pair of handles 13 attached to the housing of the movable station 60 allows a user to manually operate the station.

[0036] A motor 66 attached to the vertical shaft 59 is designed to rotate the gantry mount 58 completely 360 degrees about the X axis, and a motor 67 moves the gantry mount 58 vertically along the z axis under the control of a control module 51.

[0037] The gantry 56 includes a first C-arm 70 slidably coupled to the gantry mount 58 and a second C-arm 72 slidably coupled to the first C-arm. In the illustrated embodiment, the first and second C-arms 70, 72 are an outer and an inner C-arm, respectively. In the illustrated embodiment, the outer and inner C-arms 70, 72 are circular in shape and rotate circumferentially about a central axis to enable imaging of a patient without the need to move the patient lying on the bed 16.

[0038] An imaging signal transmitter 74, such as an X-ray beam transmitter, is attached to one side of the second C-arm 72, and an imaging sensor 74, such as an X-ray detector array, is attached to the other side of the second C-arm and faces the transmitter. During operation, the X-ray transmitter 74 transmits an X-ray beam that is received by the X-ray detector 76 after passing through a relevant part of a patient (not shown).

[0039] In one embodiment, the system 10 is a multimodality X-ray imaging system designed with surgery in mind. The three imaging modalities include fluoroscopy, 2D radiography, and cone beam CT. Fluoroscopy is a medical imaging technique that shows continuous X-ray images on a monitor, much like an X-ray movie. 2D radiography is an imaging technique that uses X-rays to view the internal structure of an opaque object with a non-uniform composition, such as the human body. CBCT (cone beam 3D imaging or cone beam computed tomography), also known as C-arm CT, is a medical imaging technique that consists of X-ray computed tomography in which the X-rays diverge to form a cone.

[0040] The movable station 60 includes an imaging controller system 40 that serves a dual function of (1) controlling the movement of the omnidirectional wheels 62, 64, the gantry mount 58, and the gantry 56 to position the imaging signal transmitter 74 relative to the patient, and (2) controlling the imaging function for imaging the patient after the gantry 56 is properly positioned.

[0041] Referring now to FIG. 2, the imaging controller system 40 of the present invention is connected to a communication link 52 via an I / O interface 42, such as a USB (Universal Serial Bus) interface, which receives information from the communication link 52 and transmits information via the communication link 52. The imaging controller system 40 includes a memory storage device 44, such as a RAM (Random Access Memory), a processor (CPU) 46, a program storage device 48, such as a ROM or EEPROM (registered trademark), and a data storage device 50, such as a hard disk, all of which are commonly connected to each other via a bus 53. The program storage device 48 stores, among other things, an imaging control module 54 and a movement control module 51, each including software executed by the processor 46. The movement control module 51 executed by the processor 46 controls the wheels 62, 64 of the movable station 60 and various motors within the gantry mount 58 and the gantry 56 to position the station 60 near the patient and position the gantry in an appropriate position for imaging the relevant part of the patient.

[0042] The imaging control module 54 executed by the processor 46 controls the imaging signal transmitter 74 and the detector array 76 to image the patient's body. In one embodiment, the imaging control module images different planar layers of the body and stores them in the memory 44. In addition, the imaging control module 54 can process a stack of images stored in the memory 44 to generate a three-dimensional image. Alternatively, the stored images can be transmitted to a host system (not shown) for image processing.

[0043] The motion control module 51 and the imaging control module 54 include a user interface module that interacts with the user via input devices such as the display devices 11a and 11b, and the keyboard and buttons 12 and the joystick 14. The strain gauge 13 attached to the handle 15 is coupled to the I / O device 42 and provides convenient movement of the movable station 12 in any direction (X, Y, swing) while the user is holding the handle 15 by hand, as will be described in more detail below. The user interface module assists the user in positioning the gantry 56. Any of the software program modules in the program storage device 48 and the data from the data storage device 50 can be transferred to the memory 44 as needed and executed by the CPU 46. The display device 11a is attached to the housing of the movable station 60 near the gantry mount 58, and the display device 11b is coupled to the movable station via three rotatable display arms 16, 18, and 20. The first display arm 16 is rotatably attached to the movable station 60, the second display arm 18 is rotatably attached to the first arm 16, and the third display arm 20 is rotatably attached to the second display arm. The display devices 11a, 11b can have touchscreens that also function as input devices through the use of the user interface module in the modules 51 and 54 to provide maximum flexibility to the user.

[0044] The navigation marker 68 disposed on the gantry mount 58 is connected to the imaging control system 40 via the link 52. Under the control of the motion control module 51, the marker 68 enables automatic or semi-automatic positioning of the gantry 56 relative to the patient bed or the OR (operating room) table via a navigation system (not shown). The marker 68 can be optical, electromagnetic, etc.

[0045] Information is provided by the navigation system and can command the gantry 56 or the system 10 to an exact location. One example would be for the surgeon to hold a navigated probe in a desired orientation and instruct the imaging system 10 to acquire a fluoroscopic or radiographic image along its designated trajectory. Advantageously, this eliminates the need for scout shots and thus reduces X-ray exposure to the patient and the OR staff. The navigation marker 68 on the gantry 56 also enables automatic alignment of 2D or 3D images acquired by the system 10. The marker 68 also enables accurate repositioning of the system 10 if the patient moves.

[0046] In the illustrated embodiment, the system 10 provides a large range of motion in all six degrees of freedom (「DOF」). Under the control of the motion control module 51, there are two main motion modes, namely, positioning of the movable station 60 and positioning of the gantry 56.

[0047] The positioning of the movable station 60 is achieved via four omnidirectional wheels 62, 64. These wheels 62, 64 enable the movable station 60 to be positioned in all three DOFs around the horizontal plane (X, Y, swing). "Swing" is the rotation of the system 10 about the vertical axis (Z-axis), "X" is the front-to-back positioning of the system along the X-axis, and "Y" is the lateral movement of the system 10 along the Y-axis. Under the control of the control module 51, the system 10 can be positioned with an unlimited range of motion in any combination of X, Y, and swing (swing about any Z-axis using the omnidirectional wheels 62, 64). In particular, the omnidirectional wheels 62, 64 enable positioning in narrow spaces, narrow corridors, or accurately moving up and down the length of an operating table or patient bed.

[0048] The positioning of the gantry 56 is achieved for (Z, tilt, rotation). "Z" is the vertical positioning of the gantry 56, "tilt" is the rotation about a horizontal axis parallel to the X-axis as described above, and "rotation" is the rotation about a horizontal axis parallel to the Y-axis as described above.

[0049] Along with the positioning of the movable station 60 and the gantry 56, the system 10 provides a range of movement in all 6 DOFs (X, Y, swing, Z, tilt, and rotation) to accurately position the movable station 60 as well as the imaging transmitter 74 and sensor 76 where they are needed. Advantageously, 3D imaging can be performed regardless of whether the patient is standing, sitting, or lying in bed, and without the need to move the patient.

[0050] The exact position of the system 10 is stored in the memory 50 and can be called up at any time by the motion control module 51. This includes not only the positioning of the gantry 56 but also the positioning of the system 10 by the omnidirectional wheels 62, 64.

[0051] As shown in FIG. 3, each of the gantry mount 58, the outer C-arm 70, and the inner C-arm 72 has a pair of side frames 86, 88, 90 facing each other. A plurality of rollers 84 arranged at equal intervals are attached inside the side frame 86 of the gantry mount 58. The outer C-arm 70 has a pair of guide rails 78 outside the side frame 88. The roller 84 is coupled to the guide rail 78. As shown, the roller 84 and the guide rail 78 are designed such that the outer C-arm 78 can slide telescopically along the gantry mount 58, and the C-arm can rotate at least 180 degrees about its central axis with respect to the gantry mount.

[0052] A plurality of rollers 80 arranged at equal intervals are attached inside the side frame 88 of the outer C-arm 70. The inner C-arm 72 has a pair of guide rails 82 outside the side frame 90. The roller 80 is coupled to the guide rail 82. As shown, the roller 80 and the guide rail 82 are designed such that the inner C-arm 72 can slide telescopically along the outer C-arm 70, and the C-arm can rotate at least 180 degrees about its central axis with respect to the outer C-arm.

[0053] Therefore, the present invention disclosed herein advantageously enables the gantry 56 to rotate a full 360 degrees about its central axis, providing maximum flexibility when positioning the imaging system 10 while minimizing patient interference.

[0054] In another aspect of the present invention, a unique cable arrangement is provided to make the imaging system 10 more compact and visually appealing. As shown in FIGS. 5 and 6, the cable carrier / harness 92 includes electrical cables for carrying signals between the imaging controller system 40 and various motors within the gantry 56, the X-ray transmitter 74, the imaging sensor 76, and various electronic circuits. A first cable router 94 is attached to the outer surface of the outer C-arm 70, and a second cable router 96 is attached to the outer surface of the inner C-arm 72. Each cable router 94, 96 has through-holes 95, 97 through which the cable carrier 92 passes.

[0055] The cable carrier 92 extends from the gantry mount 56 over the outer surface of the first C-arm 70, passes through the through-hole 95 of the first cable router 94, and extends over the outer surface of the second C-arm 72. The cable carrier 92 covering the first C-arm 70 extends in a first circumferential direction (clockwise as shown) 98, enters the first cable router 94 in a second circumferential direction opposite to the first circumferential direction (counterclockwise as shown) 99, and forms a 180-degree service loop on the outer surface of the first C-arm.

[0056] From there, the cable carrier 92 extends in the first circumferential direction 98, enters the second cable router in the second circumferential direction 99, and forms another service loop on the outer surface of the second C-arm 72.

[0057] The specific locations of the first and second cable routers 94, 96 in combination with the service loops allow for slack in the cable carrier 92 and provide a full 360-degree rotation to the gantry 56 without causing tangling or stress in the cable carrier. In the illustrated embodiment, the routers are attached near the midpoint of the C-arm.

[0058] FIG. 8 shows an embodiment of a motor assembly 101 that can be used to rotatably extend and retract the outer C-arm 70 relative to the gantry mount 58 and the inner C-arm 72 relative to the outer C-arm. Each motor assembly 101 includes a servo motor 103 having encoder feedback, a gearbox 105 for changing the rotation ratio, a drive pulley 107, an idler pulley 108, and a belt 111 passed between the drive pulley and the idler pulley. One motor assembly 101 is mounted to the gantry mount to move the outer C-arm 70 relative to the gantry mount, and another motor assembly is mounted to the outer C-arm 70 near the center of the arm to move the inner C-arm 70 relative to the outer C-arm 70.

[0059] FIGS. 9A-9G show the gantry 56 rotated 360 degrees in 60-degree increments in the counterclockwise direction. FIG. 9A represents the 0-degree position of the imaging sensor 76 and the transmitter 74. FIG. 9B represents the 60-degree rotation / position of the gantry 56. Each time the gantry 56 rotates 60 degrees, the motor assembly 101 rotates the inner C-arm 72 30 degrees counterclockwise and the outer C-arm 70 also 30 degrees counterclockwise under the control of the motion control module 51 for a 60-degree compound rotation. FIG. 9G represents a full 360-degree rotation of the gantry 56. As can be seen from the figures, the outer C-arm 70 and the inner C-arm 72 have each moved 180 degrees from their original 0-degree positions in FIG. 9A.

[0060] As described in detail above, the present invention in various embodiments provides the following advantages: (1) movement of the system in any X-Y direction with rocking about any Z-axis by use of the omnidirectional wheels 62, 64, (2) a double nested C-gantry for rotating the imaging beam a full 360 degrees, (3) imaging while supine, seated, or standing, such as for upright CBCT, (4) memory and recall of the positions of the system 10 and the gantry 56, (5) quasi-simultaneous multi-slice x-ray imaging, (6) recall of positions via robotic or navigation coordinates.

[0061] Referring now to the drawings, FIGS. 10 and 11 illustrate a surgical robot system 100 according to an exemplary embodiment. The surgical robot system 100 can include, for example, a surgical robot 102, one or more robot arms 104, a base 106, a display 110, an end effector 112 including, for example, a guide tube 114, and one or more tracking markers 118. The surgical robot system 100 can also include a patient tracking device 116 that includes one or more tracking markers 118 adapted to be directly fixed to a patient 210 (e.g., to the bone of the patient 210). The surgical robot system 100 can also utilize, for example, a camera 200 positioned on a camera stand 202. The camera stand 202 can have any suitable configuration for moving, orienting, and supporting the camera 200 to a desired position. The camera 200 can include any suitable camera(s), such as one or more infrared cameras (e.g., bifocal or stereophotogrammetric cameras), and can identify active and passive tracking markers 118 in a given measurement volume visible from the viewpoint of the camera 200. The camera 200 can scan a given measurement volume, identify the position of the markers 118 in three dimensions, and detect light from the markers 118 to make a determination. For example, the active marker 118 can include an infrared emitting marker activated by an electrical signal (e.g., a light emitting diode (LED)), and the passive marker 118 can include, for example, a retroreflective marker that reflects infrared light emitted by an illuminator on the camera 200 or another suitable device (e.g., reflects incident IR radiation in the direction of the incident light).

[0062] Figures 10 and 11 show potential configurations regarding the placement of the surgical robot system 100 in an operating room environment. For example, the robot 102 may be positioned near or adjacent to the patient 210. Although shown near the head of the patient 210, it will be understood that the robot 102 can be positioned at any suitable location near the patient 210 depending on the area of the patient 210 undergoing surgery. The camera 200 may be separated from the robot system 100 and positioned at the foot of the patient 210. This location enables the camera 200 to have a direct line of sight to the surgical field 208. Again, it is contemplated that the camera 200 can be placed at any suitable location having a line of sight to the surgical field 208. In the illustrated configuration, the surgeon 120 may be positioned opposite the robot 102 but can still operate the end effector 112 and the display 110. The surgical assistant 126 may also be positioned opposite the surgeon 120 so as to have access to both the end effector 112 and the display 110. If necessary, the locations of the surgeon 120 and the assistant 126 may be reversed. The conventional areas of the anesthesiologist 122 and the nurse or surgical technician 124 are not restricted by the locations of the robot 102 and the camera 200.

[0063] Regarding other components of the robot 102, the display 110 can be mounted on the surgical robot 102, and in other exemplary embodiments, the display 110 can be removed from the surgical robot 102 either in an operating room of the surgical robot 102 or at a remote location. The end effector 112 is coupled to the robotic arm 104 and can be controlled by at least one motor. In an exemplary embodiment, the end effector 112 can include a guide tube 114 that can receive and orient a surgical instrument 608 (further described herein) used to perform surgery on the patient 210. As used herein, the term "end effector" is used interchangeably with the terms "end effectuator" and "effector element". Although generally shown with the guide tube 114, it will be understood that the end effector 112 can be replaced with any suitable instruments suitable for use in surgery. In some embodiments, the end effector 112 can include any known structure for effecting movement of the surgical instrument 608 in a desired manner.

[0064] The surgical robot 102 is capable of controlling the translation and orientation of the end effector 112. The robot 102 can, for example, move the end effector 112 along the x, y, and z axes. The end effector 112 can be configured to selectively rotate about one or more of the x, y, and z axes, as well as the Z-frame axis (such that one or more of the Euler angles (e.g., roll, pitch, and / or yaw) associated with the end effector 112 can be selectively controlled). In some exemplary embodiments, the selective control of the translation and orientation of the end effector 112 can enable the performance of medical procedures with significantly improved accuracy compared to conventional robots that utilize, for example, a robotic arm with six degrees of freedom including only rotational axes. For example, the surgical robot system 100 can be used to operate on the patient 210, the robotic arm 104 can be positioned over the patient 210's body, and the end effector 112 is selectively angled with respect to the z axis towards the patient 210's body.

[0065] In some exemplary embodiments, the position of the surgical instrument 608 can be dynamically updated such that the surgical robot 102 can always recognize the location of the surgical instrument 608 during the procedure. As a result, in some exemplary embodiments, the surgical robot 102 can quickly move the surgical instrument 608 to a desired position without further assistance from the physician (except if the physician so desires). In some further embodiments, the surgical robot 102 can be configured to correct the path of the surgical instrument 608 if the surgical instrument 608 deviates from a selected pre-planned trajectory. In some exemplary embodiments, the surgical robot 102 can be configured to enable the stopping, modification, and / or manual control of the movement of the end effector 112 and / or the surgical instrument 608. Thus, during use, in an exemplary embodiment, the physician or other user can operate the system 100 and has the option to stop, modify, or manually control the autonomous movement of the end effector 112 and / or the surgical instrument 608. Further details of the surgical robot system 100, including the control and movement of the surgical instrument 608 by the surgical robot 102, can be found in co-pending U.S. patent application Ser. No. 13 / 924,505, which is hereby incorporated by reference in its entirety.

[0066] The robotic surgical system 100 can include one or more tracking markers 118 configured to three-dimensionally track the movement of the robotic arm 104, the end effector 112, the patient 210, and / or the surgical instrument 608. In an exemplary embodiment, the plurality of tracking markers 118 can be attached (or otherwise fixed) on the outer surface of the robot 102, such as, without limitation, on the base 106 of the robot 102, on the robotic arm 104, or on the end effector 112. In an exemplary embodiment, at least one of the plurality of tracking markers 118 can be attached to or otherwise fixed to the end effector 112. One or more tracking markers 118 can further be attached to (or otherwise fixed to) the patient 210. In an exemplary embodiment, the plurality of tracking markers 118 can be positioned on the patient 210 spaced apart from the surgical field 208 to reduce the likelihood of being obscured by the surgeon, the surgical tool, or other parts of the robot 102. Further, one or more tracking markers 118 can be further attached to (or otherwise fixed to) the surgical tool 608 (such as a screwdriver, a dilator, an implant inserter, etc.). In this way, the tracking markers 118 enable each of the marked objects (such as the end effector 112, the patient 210, and the surgical tool 608) to be tracked by the robot 102. In an exemplary embodiment, the system 100 can use the tracking information collected from each of the marked objects to calculate, for example, the orientation and location of the relative positions of the end effector 112, the surgical instrument 608 (such as positioned within the tube 114 of the end effector 112), and the patient 210.

[0067] In an exemplary embodiment, one or more of the markers 118 may be optical markers. In some embodiments, positioning one or more tracking markers 118 on the end effector 112 may maximize the accuracy of position measurement by serving to check or verify the position of the end effector 112. Further details of the surgical robot system 100, including the control, movement, and tracking of the surgical robot 102 and the surgical instrument 608, can be found in co-pending U.S. patent application Ser. No. 13 / 924,505, which is incorporated herein by reference in its entirety.

[0068] Exemplary embodiments include one or more markers 118 coupled to the surgical instrument 608. In an exemplary embodiment, for example, these markers 118 coupled to the patient 210 and the surgical instrument 608, and the markers 118 coupled to the end effector 112 of the robot 102, may include conventional infrared light emitting diodes (LEDs), or Optotrak® diodes that can be tracked using a commercially available infrared optical tracking system such as Optotrak®. Optotrak® is a registered trademark of Northern Digital Inc., Waterloo, Ontario, Canada. In other embodiments, the marker 118 may include a conventional reflective sphere that can be tracked using a commercially available optical tracking system such as Polaris Spectra. Polaris Spectra is also a registered trademark of Northern Digital, Inc. In an exemplary embodiment, the marker 118 coupled to the end effector 112 is an active marker that includes an infrared light emitting diode that can be turned on and off, and the markers 118 coupled to the patient 210 and the surgical instrument 608 include passive reflective spheres.

[0069] In an exemplary embodiment, the light emitted from and / or reflected by marker 118 can be detected by camera 200 and used to monitor the location and movement of the marked object. In an alternative embodiment, marker 118 can include a radio frequency and / or an electromagnetic reflector or transceiver, and camera 200 can include or can be replaced by a radio frequency and / or an electromagnetic transceiver.

[0070] Similar to surgical robot system 100, FIG. 12 shows a surgical robot system 300 in a docking configuration and a camera stand 302 that conforms to an exemplary embodiment of the present disclosure. Surgical robot system 300 can include a robot 301 that includes a display 304, an upper arm 306, a lower arm 308, an end effector 310, a vertical column 312, casters 314, a cabinet 316, a tablet drawer 318, a connector panel 320, a control panel 322, and a ring of information 324. Camera stand 302 can include a camera 326. These components are described in more detail with respect to FIG. 14. FIG. 12 shows, for example, a surgical robot system 300 in a docking configuration where camera stand 302 nests with robot 301 when not in use. It will be understood by those skilled in the art that camera 326 and robot 301 can be separated from each other and positioned at any suitable location during a surgical procedure, as shown, for example, in FIGS. 10 and 11.

[0071] FIG. 13 shows a base 400 that conforms to an exemplary embodiment of the present disclosure. Base 400 can be a part of surgical robot system 300 and can include cabinet 316. Cabinet 316 can house certain components of surgical robot system 300, including but not limited to battery 402, a power distribution module 404, a platform interface board module 406, a computer 408, a handle 412, and a tablet drawer 414. The connections and relationships between these components are described in more detail with respect to FIG. 14.

[0072] FIG. 14 shows a block diagram of certain components of an exemplary embodiment of a surgical robot system 300. The surgical robot system 300 may include a platform subsystem 502, a computer subsystem 504, a motion control subsystem 506, and a tracking subsystem 532. The platform subsystem 502 may further include a battery 402, a power distribution module 404, a platform interface board module 406, and a tablet charging station 534. The computer subsystem 504 may further include a computer 408, a display 304, and a speaker 536. The motion control subsystem 506 may further include driver circuits 508, motors 510, 512, 514, 516, 518, ballasts 520, 522, 524, 526, an end effector 310, and a controller 538. The tracking subsystem 532 may further include a position sensor 540 and a camera converter 542. The system 300 may also include a foot pedal 544 and a tablet 546.

[0073] Input power is supplied to the system 300 via a power source 548 that may provide power to the power distribution module 404. The power distribution module 404 is configured to receive the input power and generate different power supply voltages that are provided to other modules, components, and subsystems of the system 300. The power distribution module 404 may be configured to provide different voltage supplies to the platform interface module 406, and the different voltage supplies are provided to other components such as the computer 408, the display 304, the speaker 536, the driver 508, etc., and may supply power to, for example, the motors 512, 514, 516, 518 and the end effector 310, the motor 510, the ring 324, the camera converter 542, and other components of the system 300, such as a fan for cooling electrical components within the cabinet 316.

[0074] The power distribution module 404 can also supply power to other components such as a tablet charging station 534 that may be located within the tablet drawer 318. The tablet charging station 534 can communicate wirelessly or wired with the tablet 546 to charge the tablet 546. The tablet 546 is consistent with the present disclosure and can be used by the surgeon described herein.

[0075] The power distribution module 404 may also be connected to a battery 402 that functions as a temporary power source when the power distribution module 404 does not receive power from the input power 548. At other times, the power distribution module 404 can help charge the battery 402 as needed.

[0076] Other components of the platform subsystem 502 may also include a connector panel 320, a control panel 322, and a ring 324. The connector panel 320 can help connect different devices and components to the system 300 and / or associated components and modules. The connector panel 320 may include one or more ports for receiving lines or connections from different components. For example, the connector panel 320 may have a ground terminal port for grounding the system 300 to other devices, a port for connecting the foot pedal 544 to the system 300, and a port for connecting to a tracking subsystem 532 that may include a position sensor 540, a camera converter 542, and a camera 326 associated with the camera stand 302. The connector panel 320 may also include other ports for enabling USB, Ethernet®, HDMI® communication to other components such as a computer 408.

[0077] The control panel 322 may provide various buttons or indicators that control the operation of the system 300 and / or provide information regarding the system 300. For example, the control panel 322 may include a button for turning the power of the system 300 on or off, a button for raising and lowering the vertical column 312, and a button for raising and lowering stabilizers 520-526 designed to engage the casters 314 to lock the system 300 so that it does not move physically. Other buttons that may remove all motor power and apply a mechanical brake to stop all movement from occurring may stop the system 300 in an emergency. The control panel 322 may also have indicators that notify the user of specific system states such as the line power indicator of the battery 402 or the state of charge.

[0078] The ring 324 may be a visual indicator for notifying the user of the system 300 of the various modes in which the system 300 is operating and specific warnings to the user.

[0079] The computer subsystem 504 includes a computer 408, a display 304, and a speaker 536. The computer 504 includes an operating system and software for operating the system 300. The computer 504 may receive and process information from other components (e.g., the tracking subsystem 532, the platform subsystem 502, and / or the motion control subsystem 506) for display to the user. Additionally, the computer subsystem 504 may also include a speaker 536 for providing audio to the user.

[0080] The tracking subsystem 532 may include a position sensor 504 and a converter 542. The tracking subsystem 532 may correspond to a camera stand 302 that includes a camera 326 as described with respect to FIG. 12. The position sensor 504 may be the camera 326. The tracking subsystem may track the location of specific markers located on different components of the system 300 and / or instruments used by the user during a surgical procedure. This tracking may be performed in a manner consistent with the present disclosure, including the use of infrared technology to track the location of active or passive elements such as LEDs or reflective markers, respectively. The location, orientation, and position of structures having these types of markers may be provided to the computer 408 and shown to the user on the display 304. For example, a surgical instrument 608 having these types of markers and tracked in this manner (which may be referred to as a navigation space) may be shown to the user in relation to a three-dimensional image of the patient's anatomical structure.

[0081] The motion control subsystem 506 may be configured to physically move the vertical column 312, the upper arm 306, and the lower arm 308, or to rotate the end effector 310. The physical movement may be performed through the use of one or more motors 510-518. For example, the motor 510 may be configured to raise and lower the vertical column 312 vertically. The motor 512 may be configured to move the upper arm 308 laterally about the engagement point with the vertical column 312, as shown in FIG. 12. The motor 514 may be configured to move the lower arm 308 laterally about the engagement point with the upper arm 308, as shown in FIG. 12. The motors 516 and 518 may be configured to move the end effector 310 in such a way that one may control roll and the other may control pitch, thereby providing a plurality of angles by which the end effector 310 may be moved. These movements may be achieved by a controller 538 that is disposed on the end effector 310 and controls these movements via load cells activated by the user and engages these load cells to move the system 300 in a desired manner.

[0082] Furthermore, the system 300 can provide automatic movement of the vertical column 312, the upper arm 306, and the lower arm 308 by the user indicating on the display 304 (which may be a touch screen input device) the location of the surgical instrument or component on the three-dimensional image of the patient's anatomical structure on the display 304. The user can initiate this automatic movement by stepping on the foot pedal 544 or by some other input means.

[0083] FIG. 15 shows a surgical robot system 600 consistent with an exemplary embodiment. The surgical robot system 600 can include an end effector 602, a robotic arm 604, a guide tube 606, an instrument 608, and a robot base 610. The instrument tool 608 can be attached to a tracking array 612 that includes one or more tracking markers (such as marker 118) and can have an associated trajectory 614. The trajectory 614 can represent a path of movement configured such that when the instrument tool 608 is positioned through or fixed within the guide tube 606, the instrument tool 608 travels, for example, an insertion path of the instrument tool 608 into the patient. In an exemplary surgery, the robot base 610 can be configured to communicate electronically with the robotic arm 604 and the end effector 602 such that the surgical robot system 600 can assist a user (e.g., a surgeon) during surgery on the patient 210. The surgical robot system 600 can be consistent with the aforementioned surgical robot systems 100 and 300.

[0084] The tracking array 612 can be attached to the instrument 608 to monitor the location and orientation of the instrument tool 608. The tracking array 612 can be mounted on the instrument 608 and can include tracking markers 804. As best seen in FIG. 17, the tracking markers 804 can be, for example, light emitting diodes and / or other types of reflective markers (e.g., marker 118 as described elsewhere in this specification). The tracking device can be one or more line-of-sight devices associated with the surgical robot system. As an example, the tracking device can be one or more cameras 200, 326 associated with the surgical robot systems 100, 300, and can also track the tracking array 612 with respect to the defined range or relative orientation of the robotic arm 604, the robotic base 610, the end effector 602, and / or the instrument 608 with respect to the patient 210. The tracking device can conform to the structure described in connection with the camera stand 302 and the tracking subsystem 532.

[0085] FIGS. 16A, 16B, and 16C respectively show a top view, a front view, and a side view of an end effector 602 consistent with an exemplary embodiment. The end effector 602 can include one or more tracking markers 702. The tracking markers 702 can be light emitting diodes or other types of active and passive markers such as the aforementioned tracking marker 118. In an exemplary embodiment, the tracking marker 702 is an active marker that emits infrared light and is activated by an electrical signal (e.g., an infrared light emitting diode (LED)). Thus, the tracking marker 702 can be activated such that the infrared marker 702 is visible to the cameras 200, 326, or deactivated such that the infrared marker 702 is not visible to the cameras 200, 326. Thus, when the marker 702 is active, the end effector 602 can be controlled by the systems 100, 300, 600, and when the marker 702 is deactivated, the end effector 602 can be locked in place and cannot be moved by the systems 100, 300, 600.

[0086] Marker 702 may be disposed on or within end effector 602 such that marker 702 is visible to one or more cameras 200, 326 or other tracking devices associated with surgical robot systems 100, 300, 600. Cameras 200, 326 or other tracking devices may track end effector 602 by following the movement of tracking marker 702 as end effector 602 moves to different positions and viewing angles. The location of marker 702 and / or end effector 602 may be displayed on displays 110, 304 associated with surgical robot systems 100, 300, 600, for example, display 110 shown in FIG. 11 and / or display 304 shown in FIG. 12. This display 110, 304 may enable the user to ensure that end effector 602 is in a desired position relative to robot arm 604, robot base 610, patient 210, and / or the user.

[0087] For example, as shown in FIG. 16A, marker 702 may be disposed around the surface of end effector 602 such that a tracking device, disposed away from the surgical field 208 and facing robots 102, 301 and cameras 200, 326, can view at least three of marker 702 through a range of common orientations of end effector 602 relative to tracking devices 100, 300, 600. For example, by dispersing marker 702 in this way, end effector 602 can be monitored by the tracking device as end effector 602 is translated and rotated within surgical field 208.

[0088] In addition, in an exemplary embodiment, end effector 602 may be equipped with an infrared (IR) receiver that can detect when external cameras 200, 326 are ready to read marker 702. Upon such detection, end effector 602 may then illuminate marker 702. When it is detected by the IR receiver that external cameras 200, 326 are ready to read marker 702, it may be signaled that it is necessary to synchronize the duty cycle of marker 702, which may be a light-emitting diode, to external cameras 200, 326. This may also enable lower power consumption for the overall robotic system, and marker 702 will be illuminated only at appropriate times instead of being continuously illuminated. Further, in an exemplary embodiment, marker 702 may be powered off to prevent interference with other navigation tools such as different types of surgical instruments 608.

[0089] FIG. 17 shows one type of surgical instrument 608 that includes a tracking array 612 and a tracking marker 804. Tracking marker 804 can be of any type described herein, including but not limited to a light-emitting diode or a retroreflective sphere. Marker 804 is monitored by a tracking device associated with surgical robotic systems 100, 300, 600 and can be one or more of vision cameras 200, 326. Cameras 200, 326 can track the location of instrument 608 based on the location and orientation of tracking array 612 and marker 804. A user, such as surgeon 120, can orient instrument 608 so that tracking array 612 and marker 804 are adequately recognized by the tracking device or cameras 200, 326, for example, to display instrument 608 and marker 804 on a display 110 of an exemplary surgical robotic system.

[0090] The method by which surgeon 120 places instrument 608 within guide tube 606 of end effector 602 and can adjust instrument 608 is apparent in FIG. 17. The hollow tubes or guide tubes 114, 606 of end effectors 112, 310, 602 are sized and configured to receive at least a portion of surgical instrument 608. Guide tubes 114, 606 are configured to be oriented by robotic arm 104 such that the insertion and trajectory of surgical instrument 608 can reach a desired anatomical target within or on the body of patient 210. Surgical instrument 608 may include at least a portion of a generally cylindrical instrument. Although a screwdriver is illustrated as surgical tool 608, it will be understood that any suitable surgical tool 608 can be positioned by end effector 602. By way of example, surgical instrument 608 may include one or more of a guide wire, cannula, retractor, drill, reamer, screwdriver, insertion tool, removal tool, and the like. Hollow tubes 114, 606 are generally shown as having a cylindrical configuration, but it will be understood by those skilled in the art that guide tubes 114, 606 can have any suitable shape, size, and configuration desired to accommodate surgical instrument 608 and access the surgical site.

[0091] FIGS. 18A - 18C show a portion of end effector 602 and robotic arm 604 consistent with an exemplary embodiment. End effector 602 may further include body 1202 and clamp 1204. Clamp 1204 may include handle 1206, ball 1208, spring 1210, and lip 1212. Robotic arm 604 may further include recess 1214, mounting plate 1216, lip 1218, and magnet 1220.

[0092] The end effector 602 can mechanically interfere with and / or engage the surgical robot system and the robotic arm 604 through one or more couplings. For example, the end effector 602 can engage the robotic arm 604 through a positioning coupling and / or a reinforcement coupling. With these couplings, the end effector 602 can be fastened to the robotic arm 604 outside the flexible sterile barrier. In an exemplary embodiment, the positioning coupling can be a magnetic kinematic mount, and the reinforcement coupling can be a five-bar over-center clamp linkage.

[0093] Regarding the positioning coupling, the robotic arm 604 can include mounting plates 1216, which can be made of a non-magnetic material, one or more recesses 1214, lips 1218, and magnets 1220. The magnets 1220 are each mounted under a respective recess 1214. The portion of the clamp 1204 includes a magnetic material and can be attracted by one or more magnets 1220. Due to the magnetic attraction between the clamp 1204 and the robotic arm 604, the balls 1208 seat within their respective recesses 1214. For example, the balls 1208 as shown in FIG. 18B will seat within the recesses 1214 as shown in FIG. 18A. This seating can be regarded as a magnetically assisted kinematic coupling. The magnets 1220 can be configured to have sufficient strength to support the entire weight of the end effector 602 regardless of the orientation of the end effector 602. The positioning coupling can be any form of kinematic mount that uniquely constrains six degrees of freedom.

[0094] Regarding the reinforcement connection, the portion of the clamp 1204 may be configured to be a fixed grounding link, and thus, the clamp 1204 can serve as a five-bar linkage. When the lips 1212 and 1218 engage the clamp 1204 to fix the end effector 602 and the robotic arm 604, by closing the clamp handle 1206, the end effector 602 can be fastened to the robotic arm 604. When the clamp handle 1206 is closed, while the clamp 1204 is in the locked position, the spring 1210 can be stretched or stressed. The locked position may be a position that provides a linkage beyond the center. Since the closed position is beyond the center, the linkage will not be released unless a force is applied to the clamp handle 1206 to release the clamp 1204. Thus, in the locked position, the end effector 602 can be firmly fixed to the robotic arm 604.

[0095] The spring 1210 may be a tensioned curved beam. The spring 1210 may be composed of a material that exhibits high rigidity and high yield strain, such as virgin PEEK (polyetheretherketone). The linkage between the end effector 602 and the robotic arm 604 can provide a sterile barrier between the end effector 602 and the robotic arm 604 without preventing the fastening of the two connections.

[0096] The reinforcement connection may be a linkage having a plurality of spring members. The reinforcement connection may be latched with a cam or friction-based mechanism. The reinforcement connection may also be a sufficiently powerful electromagnet that supports fastening the end effector 102 to the robotic arm 604. The reinforcement connection may be a multi-piece collar that is completely separated from either the end effector 602 and / or the robotic arm 604, slides on the interface surface between the end effector 602 and the robotic arm 604, and is tightened with a screw mechanism, an over-center linkage, or a cam mechanism.

[0097] Referring to FIGS. 10 and 11, prior to or during a surgical procedure, a specific alignment procedure may be performed to track objects and the target anatomical structures of patient 210 in both the navigation space and the image space. To perform such an alignment, an alignment system 1400 as shown in FIG. 19 may be used.

[0098] To track the position of patient 210, the patient tracking device 116 may include a patient fixture 1402 that is fixed to the rigid anatomical structure of patient 210, and a dynamic reference base (DRB) 1404 may be firmly attached to the patient fixture 1402. For example, the patient fixture 1402 may be inserted into the opening 1406 of the dynamic reference base 1404. The dynamic reference base 1404 may include markers 1408 that are visible to a tracking device such as the tracking subsystem 532. These markers 1408 may be optical markers or reflective spheres such as the tracking marker 118 as described above herein.

[0099] The patient fixture 1402 may be attached to the hard anatomical structure of patient 210 and remain attached throughout the surgical procedure. In an exemplary embodiment, the patient fixture 1402 is attached to a hard area of patient 210, such as a bone located away from the target anatomical structure undergoing the surgical procedure. To track the target anatomical structure, the dynamic reference base 1404 is associated with the target anatomical structure by using an alignment fixture that is temporarily placed on or near the target anatomical structure to align the dynamic reference base 1404 at the location of the target anatomical structure.

[0100] The alignment fixture 1410 is attached to the patient fixture 1402 by using a pivotal arm 1412. The pivotal arm 1412 is attached to the patient fixture 1402 by inserting the patient fixture 1402 through the opening 1414 of the alignment fixture 1410. The pivotal arm 1412 is attached to the alignment fixture 1410 by inserting a knob 1416 through the opening 1418 of the pivotal arm 1412, for example.

[0101] The alignment fixture 1410 may be placed over the target anatomical structure using the pivot arm 1412, and the location may be determined within the image space and the navigation space using the tracking markers 1420 and / or the fiducials 1422 on the alignment fixture 1410. The alignment fixture 1410 may include a set of markers 1420 that are visible within the navigation space (e.g., the markers 1420 may be detectable by the tracking subsystem 532). The tracking markers 1420 may be optical markers that are visible with infrared light, as previously described herein. The alignment fixture 1410 may also include a set of fiducials 1422, such as bearing balls, that are visible within the imaging space (e.g., a three-dimensional CT image). As will be described in more detail with respect to FIG. 20, the target anatomical structure may be associated with the dynamic reference base 1404 using the alignment fixture 1410, thereby enabling the depiction of an object within the navigation space to be overlaid on an image of the anatomical structure. The dynamic reference base 1404 disposed at a location remote from the target anatomical structure may serve as a reference point, thereby enabling the alignment fixture 1410 and / or the pivot arm 1412 to be removed from the surgical area.

[0102] FIG. 20 shows an exemplary method 1500 for alignment consistent with the present disclosure. Method 1500 begins at step 1502, where a graphical representation (or image(s)) of the target anatomical structure may be imported into the system 100, 300, 600, such as the computer 408. The graphical representation may be a three-dimensional CT or fluoroscopic scan of the patient 210's target anatomical structure that includes a detectable imaging pattern of the alignment fixture 1410 and the fiducials 1420.

[0103] At step 1504, the imaging pattern of the fiducials 1420 is detected, aligned within the imaging space, and stored in the computer 408. Optionally, at this point, at step 1506, a graphical representation of the alignment fixture 1410 may be overlaid on the image of the target anatomical structure.

[0104] In step 1508, by recognizing marker 1420, the navigation pattern of alignment fixture 1410 is detected and aligned. Marker 1420 may be an optical marker recognized within the navigation space through infrared light by tracking subsystem 532 via position sensor 540. Thus, the location, orientation, and other information of the target anatomical structure are aligned within the navigation space. Thus, alignment fixture 1410 can be recognized in both the image space by use of reference 1422 and the navigation space by use of marker 1420. In step 1510, the alignment of alignment fixture 1410 in the image space is transferred to the navigation space. This transfer is performed, for example, by using the relative position of the imaging pattern of reference 1422 compared to the position of the navigation pattern of marker 1420.

[0105] In step 1512, the alignment of alignment fixture 1410 in the navigation space (aligned with the image space) is further transferred to the navigation space of dynamic alignment array 1404 mounted on patient fixture 1402. Thus, since the navigation space is associated with the image space, alignment fixture 1410 may be removed, and dynamic reference base 1404 may be used to track the target anatomical structure in both the navigation space and the image space.

[0106] In steps 1514 and 1516, the navigation space may be overlaid with an image space and an object having a marker visible within the navigation space (e.g., surgical instrument 608 having optical marker 804). The object can be tracked through the graphical representation of surgical instrument 608 on the image of the target anatomical structure.

[0107] Figures 21A - 21B show an imaging device 1304 that can be used in conjunction with robotic systems 100, 300, 600 to acquire preoperative, intraoperative, postoperative, and / or real - time image data of patient 210. Any suitable object can be imaged using the imaging system 1304 for any suitable procedure. The imaging system 1304 can be any imaging device such as an imaging device 1306 and / or a C - arm 1308 device. In an X - ray system, it may be desirable to take X - rays of patient 210 from a number of different positions without the need for frequent manual repositioning of patient 210 as may be required in some X - ray systems. As shown in FIG. 21A, the imaging system 1304 may be in the form of a C - arm 1308 that includes an elongated C - shaped member terminating at opposing distal ends 1312 in a "C" shape. The C - shaped member 1130 may further include an X - ray source 1314 and an image receptor 1316. The space within the C - arm 1308 of the arm can provide room for a physician to access the patient without substantial interference from the X - ray support structure 1318. As shown in FIG. 21B, the imaging system can include an imaging device 1306 having a gantry housing 1324 mounted to a support structure imaging device support structure 1328 such as a wheeled mobile cart 1330 with wheels 1332 that can enclose an image capture portion (not shown). The image capture portion includes an X - ray source and / or a light - emitting portion and an X - ray receiving portion and / or an image receiving portion, which are arranged approximately 180 degrees apart from each other and may be attached to a rotor (not shown) with respect to the orbit of the image capture portion. The image capture portion may be operable to rotate 360 degrees during image acquisition. The image capture portion may rotate about a central point and / or axis, enabling image data of patient 210 to be acquired from multiple directions or within multiple planes. While a particular imaging system 1304 is illustrated herein, it will be understood that any suitable imaging system may be selected by one of ordinary skill in the art.

[0108] Referring now to FIGS. 22A - 22C, surgical robot systems 100, 300, 600 rely on the accurate positioning of end effectors 112, 602, surgical instruments 608, and / or the patient 210 (e.g., patient tracking device 116) relative to a desired surgical field. In the embodiments shown in FIGS. 22A - 22C, tracking markers 118, 804 are rigidly attached to a portion of the instrument 608 and / or end effector 112.

[0109] FIG. 22A shows a portion of a surgical robot system 100 with a robot 102 including a base 106, a robotic arm 104, and an end effector 112. Other elements not shown, such as a display and a camera, may also be present as described herein. FIG. 22B shows an enlarged view of the end effector 112 having a guide tube 114 and a plurality of tracking markers 118 rigidly attached thereto. In this embodiment, the plurality of tracking markers 118 are attached to the guide tube 112. FIG. 22C shows an instrument 608 (in this case, a probe 608A) having a plurality of tracking markers 804 rigidly attached thereto. As described elsewhere herein, the instrument 608 can include any suitable surgical instrument, including but not limited to, a guide wire, a cannula, a trocar, a drill, a reamer, a screwdriver, an insertion tool, a removal tool, or the like.

[0110] When tracking instrument 608, end effector 112, or other objects tracked in 3D, the array of tracking markers 118, 804 can be rigidly attached to a part of tool 608 or end effector 112. Preferably, the tracking markers 118, 804 are attached such that the markers 118, 804 do not get in the way (e.g., do not interfere with surgery, visibility, etc.). The markers 118, 804 may be attached to the instrument 608, end effector 112, or other objects tracked using, for example, array 612. Typically, three or four markers 118, 804 are used with array 612. The array 612 may include linear portions, cross members, and may be asymmetric such that the markers 118, 804 are in different relative positions and locations with respect to each other. For example, as shown in FIG. 22C, a probe 608A with a four-marker tracking array 612 is shown, and FIG. 22B shows an end effector 112 with a different four-marker tracking array 612.

[0111] In FIG. 22C, the tracking array 612 functions as the handle 620 of the probe 608A. Accordingly, the four markers 804 are attached to the handle 620 of the probe 608A such that they do not interfere with the shaft 622 and the tip 624. Stereophotogrammetric tracking of these four markers 804 enables the instrument 608 to be tracked as a rigid body and enables the tracking systems 100, 300, 600 to accurately determine the position of the tip 624 and the orientation of the shaft 622 while the probe 608A is being moved around in front of the tracking cameras 200, 326.

[0112] To enable automatic tracking of one or more tools 608, end effectors 112, or other objects (e.g., multiple rigid bodies) tracked in 3D, markers 118, 804 such as each tool 608, end effector 112 are asymmetrically arranged at known marker-to-marker intervals. The reason for the asymmetric alignment is to clarify which markers 118, 804 correspond to specific locations on the rigid body and whether the markers 118, 804 are viewed from the front or the back, i.e., whether they are mirrored. For example, if the markers 118, 804 are arranged in a square on the tool 608 or end effector 112, it is unclear to the systems 100, 300, 600 which markers 118, 804 correspond to which corners of the square. For example, in the probe 608A, it is unclear which marker 804 is closest to the shaft 622. Therefore, it is unclear in which direction the shaft 622 extends from the array 612. Therefore, each array 612, and thus each tool 608, end effector 112, or other object to be tracked should have a unique marker pattern to enable it to be identified from other tools 608 or other objects being tracked. The asymmetry and unique marker pattern allow the systems 100, 300, 600 to detect the individual markers 118, 804 and then check the marker intervals against a stored template to determine which tool 608, end effector 112, or other object they represent. Next, the detected markers 118, 804 can be automatically sorted and assigned to each tracking target in the correct order. Without this information, important geometric information such as the alignment of the tool tip 624 and the shaft 622, for example, cannot be extracted by performing rigid body calculations unless the user manually specifies which detected markers 118, 804 correspond to which positions on each rigid body. These concepts are generally known to those skilled in the art of 3D optical tracking methods.

[0113] Referring now to FIGS. 23A - 23D, an alternative version of end effector 912 having movable tracking markers 918A - 918D is shown. In FIG. 23A, an array having movable tracking markers 918A - 918D is shown in a first configuration, and in FIG. 23B, the movable tracking markers 918A - 918D are shown in a second configuration that is angled with respect to the first configuration. FIG. 23C shows a template of the tracking markers 918A - 918D as seen, for example, by cameras 200, 326 in the first configuration of FIG. 23A, and FIG. 23D shows a template of the tracking markers 918A - 918D as seen, for example, by cameras 200, 326 in the second configuration of FIG. 23B.

[0114] In this embodiment, not all of the markers 918A - 918D are in fixed positions relative to the rigid body. Instead, one or more of the array markers 918A - 918D are adjusted, for example during testing, to provide updated information about the rigid body being tracked without interrupting the process for automatic detection and sorting of the tracked markers 918A - 918D. A 4 - marker array tracking is contemplated that can give updated information regarding the rigid body being tracked.

[0115] When tracking any tool, such as guide tube 914, connected to end effector 912 of robot systems 100, 300, 600, the primary purpose of the tracking array is to update the position of end effector 912 in the camera coordinate system. For example, as shown in FIG. 22B, when using a rigid system, the array 612 of reflective markers 118 extends rigidly from guide tube 114. Since the tracking markers 118 are rigidly connected, knowing the marker locations in the camera coordinate system also provides the exact locations of the centerline, tip, and rear end of guide tube 114 in the camera coordinate system. Typically, information about the position of end effector 112 from such an array 612, and information about the location of a target trajectory from another tracked source, are used to calculate the necessary movements that must be input to each axis of robot 102 to move guide tube 114 to align with the trajectory and move the tip to a specific location along the trajectory vector.

[0116] Sometimes, the desired trajectory is in a difficult or inaccessible location, but it can be reached if the guide tube 114 can be rotated. For example, a very steep trajectory pointing away from the base 106 of the robot 102 may be reachable if the guide tube 114 can be rotated upward beyond the limits of the pitch (vertical angle of the wrist) axis, but may not be reachable if the guide tube 114 is attached parallel to the plate connecting it to the end of the wrist. To reach such a trajectory, the base 106 of the robot 102 may be moved, or a different end effector 112 with a different guide tube attachment may be exchanged for the working end effector. Both of these solutions can be time-consuming and cumbersome.

[0117] As best shown in FIGS. 23A and 23B, when the array 908 is configured such that one or more of the markers 918A-918D are not in fixed positions and instead one or more of the markers 918A-918D can be adjusted, rotated, swiveled, or moved, the robot 102 can provide updated information regarding the object being tracked without interrupting the detection and tracking process. For example, one of the markers 918A-918D may be fixed in place, and the other markers 918A-918D may be movable. Two of the markers 918A-918D may be fixed in place, and the other markers 918A-918D may be movable. Three of the markers 918A-918D may be fixed in place, and the other markers 918A-918D may be movable. Or, all of the markers 918A-918D may be movable.

[0118] In the embodiments shown in FIGS. 23A and 23B, markers 918A, 918B are rigidly connected directly to the base 906 of the end effector 912, and markers 918C, 918D are rigidly connected to the tube 914. Similar to array 612, an array 908 may be provided for attaching markers 918A - 918D to the end effector 912, the instrument 608, or other object being tracked. However, in this case, the array 908 consists of a plurality of separate components. For example, markers 918A, 918B may be connected to the base 906 in a first array 908A, and markers 918C, 918D may be connected to the guide tube 914 in a second array 908B. Marker 918A may be attached to the first end of the first array 908A, and marker 918B may be separated by a linear distance and attached to the second end of the first array 908A. The first array 908 is substantially linear, but the second array 908B has a bent or V - shaped configuration, with each proximal end connected to the guide tube 914 and branching V - shaped distally therefrom, with marker 918C at one distal end and marker 918D at the other distal end. Although a specific configuration is illustrated herein, it will be understood that other asymmetric designs are contemplated, including different numbers and types of arrays 908A, 908B, and different arrangements, numbers, and types of markers 918A - 918D.

[0119] The guide tube 914 may be movable, pivotable, or rotatable relative to the base 906, for example, across a hinge 920 or other connector to the base 906. Thus, markers 918C, 918D are movable such that when the guide tube 914 pivots, rotates, or moves, markers 918C, 918D also pivot, rotate, or move. As best shown in FIG. 23A, the guide tube 914 has a longitudinal axis 916 that is aligned in a substantially vertical or perpendicular orientation such that markers 918A - 918D have a first configuration. Referring now to FIG. 23B, the guide tube 914 is pivoted, rotated, or moved such that the longitudinal axis 916 is angled relative to the perpendicular orientation such that markers 918A - 918D have a second configuration that is different from the first configuration.

[0120] In contrast to the embodiments described with respect to FIGS. 23A - 23D, with all four markers 918A - 918D remaining firmly attached to the guide tube 914, if there is a swivel between the guide tube 914 and the arm 104 (e.g., a wrist fixture) and this swivel is adjusted by the user, the robotic systems 100, 300, 600 cannot automatically detect that the orientation of the guide tube 914 has changed. The robotic systems 100, 300, 600 track the position of the marker array 908 and calculate inaccurate robot axis movements assuming that the guide tube 914 was attached to the wrist (robotic arm 104) in its previous orientation. By firmly holding one or more of the markers 918A - 918D (e.g., two markers 918C, 918D) on the tube 914 and holding one or more of the markers 918A - 918D (e.g., two markers 918A, 918B) across the swivel, automatic detection of the new position becomes possible and correct robot movements are calculated based on the detection of the new tool or end effector 112, 912 on the end of the robotic arm 104.

[0121] One or more of the markers 918A - 918D are configured to be moved, pivoted, rotated, etc. according to any suitable means. For example, the markers 918A - 918D can be moved by hinges 920 such as clamps, springs, levers, slides, toggles, or by any other suitable mechanism for moving the markers 918A - 918D individually or in combination, moving the arrays 908A, 908B individually or in combination, moving any part of the end effector 912 relative to another part, or moving any part of the tool 608 relative to another part.

[0122] As shown in FIGS. 23A and 23B, the array 908 and the guide tube 914 can be reconfigured by simply loosening the clamp or hinge 920, moving a portion of the arrays 908A, 908B relative to the other portions 908A, 908B, and then retightening the hinge 920 so that the guide tube 914 is oriented in a different position. For example, two markers 918C, 918D may be rigidly interconnected with the tube 914, and two markers 918A, 918B may be rigidly interconnected across the hinge 920 to the base 906 of the end effector 912 attached to the robotic arm 104. The hinge 920 may be in the form of a clamp, such as a wing nut, that can be loosened and retightened to allow the user to quickly switch between a first configuration (FIG. 23A) and a second configuration (FIG. 23B).

[0123] The cameras 200, 326 detect the markers 918A-918D, for example, in one of the templates identified in FIGS. 23C and 23D. When the array 908 is in the first configuration (FIG. 23A) and the tracking cameras 200, 326 detect the markers 918A-918D, the tracked markers match the array template 1 as shown in FIG. 23C. When the array 908 is in the second configuration (FIG. 23B) and the tracking cameras 200, 326 detect the same markers 918A-918D, the tracked markers match the array template 2 as shown in FIG. 23D. The array template 1 and the array template 2 are recognized by the systems 100, 300, 600 as two separate tools, each having a uniquely defined spatial relationship between the guide tube 914, the markers 918A-918D, and the robot attachment. Thus, the user can adjust the position of the end effector 912 between the first and second configurations without notifying the systems 100, 300, 600, and the systems 100, 300, 600 appropriately adjust the movement of the robot 102 to stay on track.

[0124] In this embodiment, there are two assembly positions where the marker arrays match a unique template that enables systems 100, 300, 600 to recognize the assembly as two different tools or two different end effectors. At any position of rotation between or outside these two positions (i.e., array template 1 and array template 2 shown in FIGS. 23C and 23D respectively), markers 918A - 918D do not match any template, and systems 100, 300, 600 do not detect any existing array even though the individual markers 918A - 918D are detected by cameras 200, 326. As a result, it is the same as when markers 918A - 918D are temporarily blocked from the field of view of cameras 200, 326. It will be understood that other array templates may exist for other configurations, for example, to identify different instruments 608 or other end effectors 112, 912, etc.

[0125] In the described embodiment, two separate assembly positions are shown in FIGS. 23A and 23B. However, there may be multiple separate positions on a swivel joint, a linear joint, a combination of a swivel joint and a linear joint, a pegboard, or other assemblies where a unique marker template can be created by adjusting the position of one or more of the markers 918A - 918D of the array relative to other markers, and each separate position defines a unique tool 608 or end effector 112, 912 that aligns with a specific template and has different known attributes. Additionally, although the end effector 912 is illustrated, it will be understood that the movable and fixed markers 918A - 918D may be used with any suitable instrument 608 or other object being tracked.

[0126] When using external 3D tracking systems 100, 300, 600 to track the perfect rigid body arrangement of three or more markers attached to the end effector 112 of a robot (for example, as depicted in FIGS. 22A and 22B), it is possible to directly track or calculate the 3D position of any section of the robot 102 in the coordinate systems of the cameras 200, 326. The geometric orientation of the joints with respect to the tracker is known by design, and the linear or angular position of the joints is known from the encoders of each motor of the robot 102, completely defining the 3D positions of all the moving parts from the end effector 112 to the base 116. Similarly, when the tracker is mounted on the base 106 (not shown) of the robot 102, it is similarly possible to track or calculate the 3D position of any section of the robot 102 from the base 106 to the end effector 112 based on the known joint shapes and joint positions from the encoders of each motor.

[0127] In some situations, it may be desirable to track the positions of all segments of the robot 102 from fewer than three markers 118 rigidly attached to the end effector 112. Specifically, when the tool 608 is introduced into the guide tube 114, it may be desirable to track the perfect rigid body motion of the robot 902 with only one additional marker 118 being tracked.

[0128] Referring now to FIGS. 24A - 24E, an alternative version of the end effector 1012 having only a single tracking marker 1018 is shown. The end effector 1012 may be similar to other end effectors described herein and may include a guide tube 1014 extending along the longitudinal axis 1016. A single tracking marker 1018 similar to the other tracking markers described herein may be rigidly attached to the guide tube 1014. This single marker 1018 can serve the purpose of adding degrees of freedom lost to enable perfect rigid body tracking and / or can serve the purpose of functioning as a monitoring marker to ensure that the assumptions regarding the positioning of the robot and the camera are valid.

[0129] A single tracking marker 1018 may be attached to the robotic end effector 1012 as a rigid extension to the end effector 1012 that protrudes in any convenient direction and does not obstruct the surgeon's view. The tracking marker 1018 may be attached to the guide tube 1014 or any other suitable location on the end effector 1012. When attached to the guide tube 1014, the tracking marker 1018 may be positioned at a location between the first and second ends of the guide tube 1014. For example, in FIG. 24A, a single tracking marker 1018 is shown as a reflective sphere mounted at the end of a thin shaft 1017 that extends forward from the guide tube 1014 and is longitudinally disposed below the inlet of the guide tube 1014 above the midpoint of the guide tube 1014. This position allows the marker 1018 to be generally visible by the cameras 200, 326, but does not obstruct the surgeon's view or collide with other tools or objects near the surgery. Further, the guide tube 1014 having the marker 1018 in this position is designed such that the marker array on any tool 608 introduced into the guide tube 1014 is visible at the same time as the single marker 1018 on the guide tube 1014 is visible.

[0130] As shown in FIG. 24B, when a closely fitting tool or instrument 608 is disposed within the guide tube 1014, the instrument 608 becomes mechanically constrained in four of six degrees of freedom. That is, the instrument 608 cannot be rotated in any direction except about the longitudinal axis 1016 of the guide tube 1014, and the instrument 608 cannot be translated in any direction except in a direction along the longitudinal axis 1016 of the guide tube 1014. In other words, the instrument 608 can only translate along the centerline of the guide tube 1014 and rotate about its centerline. If two further parameters are known, such as (1) the angle of rotation about the longitudinal axis 1016 of the guide tube 1014, and (2) the position along the guide tube 1014, the position of the end effector 1012 in the camera coordinate system is fully defined.

[0131] Referring now to FIG. 24C, the systems 100, 300, 600 should be able to know when the tool 608 is actually positioned inside the guide tube 1014 and is somewhere within the field of view of the cameras 200, 326, rather than outside the guide tube 1014. The tool 608 has a longitudinal axis or centerline 616 and an array 612 having a plurality of tracked markers 804. Rigid body calculations can be used to determine where the centerline 616 of the tool 608 is located in the camera coordinate system based on the tracked positions of the array 612 on the tool 608.

[0132] The fixed normal (perpendicular) distance DF from a single marker 1018 to the centerline or longitudinal axis 1016 of the guide tube 1014 is fixed and geometrically known, and the position of the single marker 1018 can be tracked. Thus, when the detected distance DD from the tool centerline 616 to the single marker 1018 matches the known fixed distance DF from the guide tube centerline 1016 to the single marker 1018, it can be determined whether the tool 608 is within the guide tube 1014 (the centerlines 616, 1016 of the tool 608 and the guide tube 1014 coincide), or whether the tool 608 happens to be at a point within the locus of positions where this distance DD could match the fixed distance DF. For example, in FIG. 24C, the normal detected distance DD from the tool centerline 616 to the single marker 1018 matches the fixed distance DF from the guide tube centerline 1016 to the single marker 1018 in both frames (tracked marker coordinates) of the data represented by the transparent tool 608 at two positions, and thus additional considerations may be required to determine when the tool 608 is positioned within the guide tube 1014.

[0133] Next, referring to FIG. 24D, using the programmed logic, in order to satisfy the condition that the tool 608 is moving within the guide tube 1014, even though the tool 608 is moving in space beyond a minimum distance with respect to the single ball 1018, a frame of tracking data can be searched for in which the detection distance DD from the tool centerline 616 to the single marker 1018 remains fixed at the correct length. For example, the first frame F1 can be detected with the tool 608 in the first position, and the second frame F2 can be detected with the tool 608 in the second position (i.e., linearly moved with respect to the first position). The marker 804 on the tool array 612 can move more than a given amount (e.g., more than a total of 5 mm) from the first frame F1 to the second frame F2. Even with this movement, the detection distance DD from the tool centerline vector C' to the single marker 1018 is substantially the same in both the first frame F1 and the second frame F2.

[0134] Logically, the surgeon 120 or the user can place the tool 608 within the guide tube 1014, rotate it slightly, or slide it into the guide tube 1014, and the systems 100, 300, 600 can detect that the tool 608 is within the guide tube 1014 from the tracking of five markers (a single marker 1018 on the guide tube 1014 in addition to the four markers 804 on the tool 608). Knowing that the tool 608 is within the guide tube 1014, all six degrees of freedom defining the position and orientation of the robotic end effector 1012 in space can be calculated. Without the single marker 1018, even if it is known for sure that the tool 608 is within the guide tube 1014, it is unknown where the guide tube 1014 is located along the centerline vector C' of the tool and how the guide tube 1014 is rotated with respect to the centerline vector C'.

[0135] Looking at FIG. 24E, the presence of a single marker 1018 that is tracked in the same way as the four markers 804 on the tool 608 makes it possible to construct the centerline vector C' of the guide tube 1014 and the tool 608, and a normal vector passing through the single marker 1018 and the centerline vector C'. This normal vector has a known orientation with respect to the robot's forearm distal to the wrist (in this example, it is oriented parallel to that segment) and has an orientation that intersects the centerline vector C' at a specific fixed position. For convenience, as shown in FIG. 24E, three mutually orthogonal vectors k', j', i' that define the rigid body position and orientation of the guide tube 1014 can be constructed. One of the three mutually orthogonal vectors k' is constructed from the centerline vector C', the second vector j' is constructed from the normal vector passing through the single marker 1018, and the third vector i' is the vector cross product of the first and second vectors k', j'. The joint positions of the robot with respect to these vectors k', j', i' are known and are fixed when all joints are at 0, and thus, using rigid body calculations, the location of any section of the robot with respect to these vectors k', j', i' when the robot is in the home position can be determined. During the movement of the robot, if the positions of the tool markers 804 (while the tool 608 is within the guide tube 1014) and the position of the single marker 1018 are detected by the tracking system and the angular / linear position of each joint is known from the encoder, the position and orientation of any part of the robot can be determined.

[0136] In some embodiments, it may be useful to fix the orientation of tool 608 relative to guide tube 1014. For example, end effector guide tube 1014 may be oriented in a particular position about its axis 1016 to enable machining or implant positioning. The orientation of what is attached to tool 608 inserted into guide tube 1014 is known from the tracked marker 804 on tool 608, but the rotational orientation of guide tube 1014 itself in the camera coordinate system is not known without additional tracked marker 1018 (or multiple tracked markers in other embodiments) on guide tube 1014. This marker 1018 provides an essentially “clock position” of -180 degrees to +180 degrees based on the orientation of marker 1018 relative to centerline vector C’. Thus, a single marker 1018 can provide an additional degree of freedom that enables full rigid body tracking and / or can function as a monitoring marker to ensure that assumptions regarding the positioning of the robot and camera are valid.

[0137] FIG. 25 is a block diagram of a method 1100 for navigating and moving a robot 102 end effector 1012 (or any other end effector described herein) along a desired target trajectory. Another use of single marker 1018 on robot end effector 1012 or guide tube 1014 is to enable automated and safe movement of robot 102 without a full tracking array attached to robot 102 as part of method 1100. This method 1100 functions when tracking cameras 200, 326 do not move relative to robot 102 (i.e., they are in fixed positions), the coordinate system of the tracking system and the coordinate system of the robot are aligned with each other, and robot 102 is calibrated such that the position and orientation of guide tube 1014 can be accurately determined in the robot's Cartesian coordinate system based only on the encoded positions of each robot axis.

[0138] In this method 1100, the coordinate systems of the tracker and the robot must be aligned with each other, which means that a coordinate transformation from the Cartesian coordinate system of the tracking system to the Cartesian coordinate system of the robot is required. For convenience, this coordinate transformation can be a 4×4 matrix of translation and rotation, which is well known in the field of robotics. This transformation is called Tcr to refer to the "transformation from camera to robot". When this transformation is known, any new frame of tracking data is received as x, y, and z coordinates in vector form for each tracked marker and can be multiplied by the 4×4 matrix, and the resulting x, y, and z coordinates are in the coordinate system of the robot. To obtain Tcr, a complete tracking array on the robot is tracked while firmly attached to the robot at a known location in the robot's coordinate system, and then the coordinate transformation is calculated using known rigid body methods. It is obvious that any tool 608 inserted into the guide tube 1014 of the robot 102 can provide the same rigid body information as the firmly attached array when an additional marker 1018 is also read. That is, the tool 608 only needs to be inserted at any position within the guide tube 1014 and at any rotation within the guide tube 1014, rather than at a fixed position and orientation. Therefore, it is possible to determine Tcr by inserting any tool 608 having a tracking array 612 into the guide tube 1014, reading the array 612 of the tool and the single marker 1018 of the guide tube 1014, and at the same time determining the current location of the guide tube 1014 in the coordinate system of the robot from the encoders on each axis.

[0139] Logic for navigating and moving the robot 102 along a target trajectory is provided in method 1100 of FIG. 25. Assume that the transformation Tcr was previously memorized before entering loop 1102. Thus, before entering loop 1102, at step 1104, after the robot base 106 is fixed, at least one frame of tracking data of the tool inserted into the guide tube is memorized while the robot is stationary. At step 1106, the transformation of the robot guide tube position from camera coordinates to robot coordinates Tcr is calculated from this stationary data and previous calibration data. Tcr should remain valid as long as cameras 200, 326 do not move relative to robot 102. If cameras 200, 326 move relative to robot 102 and Tcr needs to be reacquired, systems 100, 300, 600 can prompt the user to insert tool 608 into guide tube 1014 and then automatically perform the necessary calculations.

[0140] In the flowchart of method 1100, each frame of the collected data consists of the tracked position of DRB 1404 on patient 210, the tracked position of a single marker 1018 on end effector 1014, and a snapshot of the position of each robot axis. From the position of the robot's axes, the location of a single marker 1018 on end effector 1012 is calculated. This calculated position is compared with the actual position of marker 1018 recorded by the tracking system. If the values match, it can be ensured that robot 102 is in a known location. The transformation Tcr is applied to the tracked position of DRB 1404 so that the goal of robot 102 can be provided with respect to the robot's coordinate system. Then, robot 102 can be commanded to move to reach the goal.

[0141] After steps 1104 and 1106, loop 1102 includes step 1108 of receiving the rigid body information of DRB1404 from the tracking system, step 1110 of converting the target tip and trajectory from image coordinates to tracking system coordinates, and step 1112 of converting the target tip and trajectory from camera coordinates to robot coordinates (applying Tcr). Loop 1102 further includes step 1114 of receiving the single fiducial marker position of the robot from the tracking system and step 1116 of converting the single fiducial marker from tracking system coordinates to robot coordinates (applying the stored Tcr). Loop 1102 also includes step 1118 of determining the current location of the single robot marker 1018 in the robot coordinate system from forward kinematics. The information from steps 1116 and 1118 is used in step 1120 to determine whether the fiducial marker coordinates from the transformed tracking position match the calculated coordinates that are less than a given tolerance. If yes, proceed to step 1122 to calculate the robot movement and apply it to the target x, y, z, and trajectory. If no, proceed to step 1124 to stop and request a full array insertion into the guide tube 1014 before proceeding. In step 1126, after the array is inserted, recalculate Tcr and then proceed to repeat steps 1108, 1114, and 1118.

[0142] This method 1100 is advantageous over methods where continuous monitoring of a single marker 1018 for location confirmation is omitted. In the absence of a single marker 1018, it is still possible to use Tcr to determine the position of the end effector 1012 and send the end effector 1012 to the target location, but it is not possible to verify that the robot 102 was actually at the expected location. For example, if cameras 200, 326 collide and Tcr becomes no longer valid, the robot 102 will move to the wrong location. Therefore, the single marker 1018 provides value in terms of safety.

[0143] For a given fixed position of the robot 102, it is theoretically possible to move the tracking cameras 200, 326 to a new location, and the single tracked marker 1018 remains unmoved because it is a single point rather than an array. In such a case, the systems 100, 300, 600 do not detect any errors because the calculated and tracked locations of the single marker 1018 match. However, when the axis of the robot moves the guide tube 1012 to a new location, the calculated and tracked locations do not match, and the safety check becomes effective.

[0144] The term "monitoring marker" can be used, for example, with respect to a single marker at a fixed location relative to the DRB1404. In this example, if the DRB1404 collides or otherwise disengages, the relative location of the monitoring marker changes, and the surgeon 120 can be warned that there may be a problem with navigation. Similarly, in the embodiments described herein, using the single marker 1018 on the guide tube 1014 of the robot, the systems 100, 300, 600 can continuously check whether the cameras 200, 326 have moved relative to the robot 102. If the alignment of the coordinate system of the tracking system with respect to the coordinate system of the robot is lost, for example, due to the cameras 200, 326 colliding or malfunctioning, or due to the robot malfunctioning, the systems 100, 300, 600 can warn the user and perform corrections. Therefore, this single marker 1018 can also be considered a monitoring marker for the robot 102.

[0145] For a complete array (e.g., the plurality of tracking markers 702 on the end effector 602 shown in FIGS. 16A - 16C) permanently mounted on the robot 102, it is not necessary for the cameras 200, 326 to be in fixed positions relative to the robot 102, and since Tcr is updated in each frame based on the tracked position of the robot 102, it will be clear that such a function of a single marker 1018 as a robot monitoring marker is not necessary. The reason for using a single marker 1018 instead of a complete array is that the complete array is bulkier and obstructive, thereby blocking the surgeon's view and access to the surgical field 208 more than the single marker 1018, and the line of sight to the complete array is more easily blocked than the line of sight to the single marker 1018.

[0146] Referring now to FIGS. 26A - 26B and 27A - 27B, instruments 608 such as implant holders 608B, 608C are depicted that include both fixed and movable tracking markers 804, 806. The implant holders 608B, 608C may have a handle 620 and an outer shaft 622 extending from the handle 620. The shaft 622 may be positioned substantially perpendicular to the handle 620, as shown, or in any other suitable orientation. The inner shaft 626 may extend through the outer shaft 622 and have a knob 628 at one end. The implants 10, 12 are connected to the distal end 624 of the implant holders 608B, 608C and to the shaft 622 at the other end using typical connection mechanisms known to those skilled in the art. The knob 628 may be rotated, for example, to expand or articulate the implants 10, 12. U.S. Patent Nos. 8,709,086 and 8,491,659, which are incorporated herein by reference, describe expandable fixation devices and installation methods.

[0147] When tracking tools 608 such as implant holders 608B, 608C, the tracking array 612 can include a combination of fixed markers 804 and one or more movable markers 806 that either constitute the array 612 or are otherwise attached to the implant holders 608B, 608C. The navigation array 612 may include at least one or more (e.g., at least two) fixed position markers 804 that are positioned at known locations relative to the implant holder instruments 608B, 608C. These fixed markers 804 cannot move in any orientation with respect to the geometric shape of the instrument and are useful for defining where the instrument 608 is in space. In addition, there is at least one marker 806, which can be attached to the array 612 or to the instrument itself and can move (e.g., slide, rotate, etc.) within a predetermined boundary relative to the fixed marker 804. Systems 100, 300, 600 (e.g., software) correlate the position of the movable marker 806 to a specific position, orientation, or other attribute of the implant 10 (such as the height of the expandable intervertebral spacer shown in FIGS. 26A-26B, or the angle of the contiguous intervertebral spacer shown in FIGS. 27A-27B). Thus, the system and / or user can determine the height or angle of the implants 10, 12 based on the location of the movable marker 806.

[0148] In the embodiment shown in FIGS. 26A-26B, four fixed markers 804 are used to define the implant holder 608B, and a fifth movable marker 806 is slidable within a predetermined path to provide feedback regarding the implant height (e.g., a contracted or expanded position). FIG. 26A shows the expandable spacer 10 at its initial height, and FIG. 26B shows the spacer 10 in an expanded state with the movable marker 806 translated to a different position. In this case, the movable marker 806 approaches the fixed markers 804 as the implant 10 expands, although it is contemplated that this movement may be reversed or otherwise different. The linear translation amount of the marker 806 corresponds to the height of the implant 10. Although only two positions are shown, it is possible to have this as a continuous function, thereby correlating any given expanded height to a particular position of the movable marker 806.

[0149] Referring now to FIGS. 27A-27B, four fixed markers 804 are used to define the implant holder 608C, and a fifth movable marker 806 is configured to slide within a predetermined path to provide feedback regarding the implant connection angle. FIG. 27A shows the connection spacer 12 in its initial linear state, and FIG. 27B shows the spacer 12 in a connected state at an offset angle with the movable marker 806 translated to a different position. The linear translation amount of the marker 806 corresponds to the connection angle of the implant 12. Although only two positions are shown, it is possible to have this as a continuous function, thereby correlating any given connection angle to a particular position of the movable marker 806.

[0150] In these embodiments, the movable marker 806 slides continuously to provide feedback regarding the attributes of the implants 10, 12 based on position. It is also contemplated that there may be discrete positions where a separate movable marker 806 must exist to provide additional information about the implant attributes. In this case, each discrete configuration of all the markers 804, 806 correlates to the specific geometry of the implant holders 608B, 608C and the implants 10, 12 at a particular orientation or particular height. Additionally, any movement of the movable marker 806 can be used for other variable attributes of any other type of navigated implant.

[0151] Although linear movement of the movable marker 806 is illustrated and described, the movable marker 806 should not be limited to sliding only, as there may be uses where rotation or other movement of the marker 806 can be useful in providing information regarding the implants 10, 12. Any relative change in position between a set of fixed markers 804 and the movable marker 806 can be relevant information regarding the implants 10, 12 or other devices. Additionally, although expandable linked implants 10, 12 are illustrated, the instrument 608 can function with other medical devices and materials such as spacers, cages, plates, fasteners, nails, screws, rods, pins, wire structures, sutures, anchor clips, staples, stents, bone grafts, biologics, etc.

[0152] According to the principles of the present invention, a system 420 is provided for aligning a patient image from an imaged device 10 navigated within an imaging space to an actual patient anatomical structure within a camera space without using radiopaque references embedded in the image. General embodiments of the present invention include an intraoperative imaging system 10, a navigation and / or robotic guidance system 100, and imaging transfer and data tracking devices (I / O) 42, 52.

[0153] The intraoperative imaging system 10 can include (1) capturing three-dimensional (3D) images (e.g., CT, CBCT, MCT, PET, angiogram, MRI, ultrasound, etc.), (2) capturing two-dimensional (2D) images (e.g., fluoroscopy, digital radiography, ultrasound, etc.), and (3) an integrated or detachable navigation array 68 having tracking markers (e.g., NIR retroreflection, NIR LED, visible, etc.) calibrated to the image space of 2D and 3D images.

[0154] The navigation and / or robotic guidance system 100 can include (1) using 2D and / or 3D images aligned for surgical planning, navigation, and guidance in various workflows (e.g., intraoperative 3D, intraoperative 2D, preoperative 3D-2D, and intraoperative 3D-2D, etc.), (2) an integrated or detachable stereotracking camera 200 capable of tracking markers 116, 68, 702. 612, 804 (e.g., NIR retroreflection, NIR LED, visible light, etc.).

[0155] The patient reference array (DRB116) can (1) be firmly attached to the patient's anatomical structure and (2) include an array of tracking markers (e.g., NIR retroreflection, NIR LED, visible light, etc.).

[0156] The imaging transfer and data tracking devices (I / O) 42, 52 (e.g., wired data transfer, wireless data transfer, data transfer via a portable memory device, etc.) connected between the imaging system 10 and the navigation system 100 can (1) transfer complete patient registration information and related images and (2) identify and establish a unique peer-to-peer connection based on system type compatibility between the image transmission side (intraoperative imaging system) and the image reception side (navigation and / or robotic guidance system).

[0157] Motors 66, 67, 102 for moving the gantry have encoders that track the positions of the inner and outer C-arms 70, 72, and thus the position of the detector panel for detecting X-ray transmission. Specifically, the system tracks pitch (rainbow), roll (tilt), and further yaw (using the encoder within the motor that rotates the vertical shaft 59). Alternatively, a separate tracking array can be attached at a known location on either the X-ray transmitter or the detector panel to track the detector position.

[0158] In the embodiment shown in FIG. 28, alignment is performed using a transformation matrix, specifically the homogeneous transformation matrix T described below, although any other processing method can also be used.

[0159] As used herein, the homogeneous transformation matrix T has four rows and four columns representing the three-dimensional geometric arrangement between two rigid bodies, and an example of which is shown below.

[0160] [Table 1] Encoded within this matrix T are both relative rotation / orientation and position / translation. The orientation is represented as a rotation matrix consisting of three rows and three columns, where each column defines a rotation about the x-axis, y-axis, and z-axis, and consists of three-dimensional mutually orthogonal unit vectors with x-component, y-component, and z-component. The relative position / translation is represented as a three-dimensional vector consisting of the x, y, and z components (denoted as "px", "py", and "pz" above) of these two objects. The last row of [0, 0, 0, 1] is an optional row and is used to simplify the equations to enable the entire transformation (both rotation and translation) in a single matrix operation. This transformation matrix T can then be used to fully define the shift in the three-dimensional pose (position and orientation) between two objects in three-dimensional space.

[0161] Assume that there are two similar objects (Object A and Object B) in space, or two positions (Position A and Position B) of the same object. Then, the homogeneous transformation T that defines the shift from the pose of B to the pose of A can be denoted as A_X_B. If there is a third object or position C, and its pose relative to B is defined as B_X_C, then the pose of C relative to A can be calculated by multiplying these two transformations.

[0162] In the context of alignment in a navigated surgery, one objective can be defined as calculating the homogeneous transformation T between the patient and the image IM captured by the imaging device 10. In the case of the patient pose, the DRB116 can be used as it is firmly attached to the patient. Thus, the transformation between the patient and the image IM can be represented as "DRB_X_IM", that is, the transformation that defines the relative position and orientation of the DRB116 at the time of imaging with respect to the position and orientation of the X-ray image IM.

[0163] As described above, the DRB116 includes an array of image tracking markers and functions as an optical tracker indicating the patient's coordinate system. The image IM is in the coordinate system of an image captured by an imaging system 10 such as Excelsius3D (hereinafter "E3D") available from Globus Medical, Inc. in Audubon, PA. The image IM may be a 3D scan (e.g., a CT scan or an MRI) or a 2D fluoroscopy / radiography shot.

[0164] With the above introduction of the homogeneous transformation T between rigid bodies, a method for aligning the patient image IM in the imaging space with the physical patient in the physical space will be described next, preferably without using radiopaque fiducials embedded in the patient image.

[0165] DRB_X_Image is the transformation between the origin of the DRB coordinate system and the origin of the image coordinate system. There are many different standards regarding defining the location and orientation of the coordinate system of medical images. For example, the LPS (Left, Posterior, Superior) standard that places the origin at the front, bottom, and the right corner of the patient in the image space can be used. The axes of this 3D coordinate system point towards the posterior, superior, and the patient's left from the origin (thus, abbreviated as LPS).

[0166] Therefore, this DRB_X_Image transformation (transformation C) results in the relative pose between the patient's physical coordinate system via the patient reference array DRB116 in the physical space and the patient's virtual coordinate system embedded in the image IM in the image space. This is defined as "patient alignment". This transformation process enables the surgical robot system 100 to virtually track the tool on the 3D scan / 2D shot and provides guidance navigated to the robot arm 104 and the end effector 112, knowing where to move relative to the patient based on the trajectory plan in the virtual image IM space. This alignment is automatically calculated by the processor of the imaging system 10 when the image IM is captured.

[0167] To calculate DRB_X_Image (transformation C), several intermediate transformations (transformations A and B) are calculated when the image IM is taken. The tracking device 200 (camera system) includes a computing system similar to the system 40 in FIG. 2, and can track the optical tracking markers of both the DRB116 and the E3D array 68 (shown as "E3D" for transformation notation) mounted on the gantry mount 58, and collect their relative poses with respect to the cameras: DRB_X_Camera and E3D_X_Camera. E3D_X_Camera can be inverted to generate Camera_X_E3D (a mathematical operation of matrix inversion). Multiplying these two matrices gives DRB_X_Camera * Camera_X_E3D = DRB_X_E3D. Therefore, this transformation (transformation A), DRB_X_E3D, indicates the pose of the E3D marker 68 with respect to the DRB116.

[0168] A second transformation (transformation B), E3D_X_Image, is also calculated. E3D_X_Image is derived from a dual calculation of E3D_X_Detector76 and Detector_X_Image (where "Detector" refers to the coordinate system of the x-ray detector panel 76 on E3D).

[0169] E3D_X_Detector is calculated from the kinematic configuration of E3D10. This is essentially mechanical and represents the geometric configuration between the E3D array 68 and the detector panel 76. The position and orientation of the detector panel 76 relative to the array 68 are derived from information from the encoders in motors 66, 67, 102, and other motors within the gantry mount 58 and gantry 56. Alternatively, a separate optical array fixture mounted on either the detector panel 76 or the x-ray transmitter 74 may be used.

[0170] Detector_X_Image is collected from the image construction process and is a characteristic of the x-ray detector / transmitter layout and how the imaging device 10 constructs the resulting image. Multiplying these two transformations gives E3D_X_Detector * Detector_X_Image = E3D_X_Image (transformation B).

[0171] To calculate the overall patient alignment, the two previously calculated transformations, DRB_X_E3D (transformation A) and E3D_X_Image (transformation B), are multiplied together to give DRB_X_E3D * E3D_X_Image = DRB_X_Image (transformation C, representing the overall patient alignment transformation).

[0172] When determining the E3D_X_Detector as part of deriving transformation B, the imaging device 10 may need to be calibrated because the C-arm bends in various amounts depending on their orientation. When the E3D marker 10 tilts the gantry 70 at a non-zero angle, there is a certain amount of mechanical bending due to the gravity pulling down the gantry. This bending can cause a slight mismatch in the E3D_X_Detector transformation because the kinematic configurations of these two mechanical components change (by about a few millimeters in some cases). The amount of bending depends on the system itself and the degree of gantry tilt used. Therefore, this calibration process is completed for each individual system 10 over a grid of gantry tilt angles and rotor angles. This calibration process requires a radiation-transparent calibration fixture 416 with embedded radiopaque fiducials 418 (e.g., BBs) that are visible under X-rays. The fixture 416 also has an optical marker 420 on one side that is visible to the camera 200, resulting in a Fixture_X_Camera transformation. The location of the fiducial 418 relative to the marker 420 on the fixture 416 is a known transformation based on the manufacture of the fixture itself. This transformation is essentially mechanical and results in BBs_X_Fixture. Multiplying these two transformations gives BBs_X_Fixture * Fixture_X_Camera = BBs_X_Camera. Since the position E3D_X_Camera of the E3D (or its array 68) relative to the camera 200 is known, the BBs_X_Camera transformation can be inverted, and the following can be calculated: E3D_X_Camera * Camera_X_BBs = E3D_X_BBs. This E3D_X_BBs transformation represents the true bend-compensated physical pose of the fixture BBs 416 relative to the E3D array 68.

[0173] When an X-ray image / CT scan of the calibration fixture 416 is taken at E3D10, the radiopaque BBs 416 appear in the image. Their positions within the image can be detected using an image processing algorithm. Combining this positioning with the known E3D_X_Image results in E3D_X_Detector * Detector_X_Image * Image_X_DetectedBBs = E3D_X_DetectedBBs. This E3D_X_DetectedBBs transformation represents the detected physical pose of the BBs 418 relative to the E3D array 68 along the computational alignment path and thus includes potential errors due to the E3D_X_Detector transformation.

[0174] Here, there are two transformations, namely E3D_X_BBs and E3D_X_DetectedBBs, that represent the same physical pose between the BBs 418 and the E3D array 68. E3D_X_BBs represents the true pose (control value), and E3D_X_DetectedBBs represents the detected potentially error-prone pose. Analyze the differences in both orientation and translation between E3D_X_BBs and E3D_X_DetectedBBs. This difference is stored in the memory of the imaging device 10 as an offset applied to the E3D_X_Detector that accounts for mechanical flexure caused by gravity. This offset ensures that E3D_X_BBs and E3D_X_DetectedBBs have the same resulting transformation and guarantees sufficient accuracy when utilizing the E3D_X_Detector transformation in the patient alignment process.

[0175] This calibration process is completed over a large combination of gantry 56 tilt angles and rotor angles and is preferably completed separately for each individual system 10. The offset calibration data is stored in the memory of the system 10 and is utilized when completing the alignment process to guarantee the accuracy in the calculated DRB_X_Image patient alignment transformation.

[0176] As an example, during the calibration procedure, the system 10 takes an image of the calibration frame / fixture 416 every 10-degree rotation at a specific pitch angle, which means 36 images are taken. The same procedure is repeated at a pitch 10 degrees higher. Thus, 1,296 images are taken and stored together to calibrate the C-arm. When imaging an actual patient at angles other than 10-degree increments using the imaging system 10, the calibration data is fine-tuned in situ using interpolation between two surrounding calibration data 10 degrees apart, and this is used to determine a patient alignment transformation that compensates for the mechanical flexion of the gantry 56 at a specific position and orientation.

[0177] In one embodiment, for 2D calibration, E3D_X_FocalSpot (the focal spot of the transmitter / imaging device 10) and E3D_X_Detector (the detector panel of the imaging device) are stored in memory. For a given acquired image, a camera model is used to calculate the location of the focal spot and detector panel with respect to the E3D10 imaging marker 68. These calibration locations are determined for a series of tilt angles and rainbow angles and stored in memory / file for future use. During use, the desired alignment location is captured and the boundary calibration points are determined. There are four corners. The focal spot and detector panel transformations are spherically bilinearly interpolated to the resulting focal spot and detector positions, which are used to calculate the 4×4 projection matrix used by the navigation / robot system 100.

[0178] In one embodiment, for 3D calibration, E3D_X_image3D (3D image) is transformed at a series of tilt angles and stored in memory / file for future use. During use, the desired alignment location is captured and the boundary calibration points are determined. There are two calibration transformations. These transformations are spherically interpolated to generate the desired E3D_X_image3D transformation at the current location. This is the alignment information (calculated by the imaging device 10) transmitted from the imaging device 10 to the navigation / robot system 100.

[0179] FIG. 29 is a flowchart of a method for aligning an intraoperative 3D image of a patient in an imaging space to a physical patient in a physical space, preferably without using radiopaque fiducials embedded in the image. This method does not require a preoperative image for patient alignment purposes.

[0180] The steps of FIG. 29 and all other flowcharts in other drawings are, unless otherwise specified, mainly executed by the imaging control module 54 of the imaging device 10.

[0181] In step 422, the patient 210 is placed on the operating table, and a dynamic reference base (``DRB'') 116 having an array of patient tracking markers 118 is firmly attached to a patient anatomical structure (e.g., the patient's pelvic bone).

[0182] In step 424, the imaging system 10 is positioned within the surgical area such that the camera 200 has good visibility of an array of imaging tracking markers 68 on both sides of the DRB 116 and the gantry mount 58.

[0183] In step 426, the tracking system / processor 532 of the robotic system 100 begins to continuously track the calibrated pose (i.e., position and orientation) of the imaging device 10 from the imaging tracking markers 68 included in the optical image coming from the camera 200.

[0184] In step 428, the tracking system 532 of the robotic system 100 also begins to continuously track the patient pose via the DRB 116.

[0185] Once the connection from the robot system 100 to the imaging device 10 is established via the I / O of the robot system, the processor 408 of the robot system begins to transmit camera frame data to the imaging device 10 via the communication link 52. The camera frame data includes the relative location from the imaging tracking marker 68 to the DRB 116 within the camera space (e.g., continuously tracked data of the imaging device 10 and the DRB 116). The camera 200 uses any visible reference array 68 (left or right array on both sides of the gantry mount 58).

[0186] In step 430, while the camera 200 is tracking the DRB 116 and imaging the tracking marker 68, a 3D image of the patient anatomical structure of interest is captured by the imaging device 10. As discussed above, the imaging device 10 can rotate the inner and outer C-arms 70, 72 to image the patient with a full 360-degree rotation. For example, the imaging device performs a full 3D X-ray CT scan, which is then converted into a 3D image or image volume based on a well-known CT conversion process. In this process, a series of X-ray images taken from different angles around the patient anatomical structure are processed by the processor 46 to create a series of cross-sectional slices of the patient anatomical structure. The series of cross-sectional slices is stored in memory as a DICOM (Digital Imaging and Communications in Medicine) file that includes a header file and a series of image data, each typically representing a corresponding cross-sectional slice.

[0187] In step 432, the 3D image obtained from step 430 is aligned as described above. Specifically, the DRB_X_Image represented by transformation C is derived by multiplying transformation A and transformation B. The tracking data required by the various transformations is received from the robot system 100 and stored in the memory 44 for use by the alignment algorithm.

[0188] As described above, transformation C represents the relative pose between the patient's physical coordinate system via the patient reference array DRB116 in the physical space as seen by camera 200 and the patient's virtual coordinate system embedded in the image IM in the image space. The transformation C data is then embedded in the private tag / header section of the DICOM file representing the 3D image of the patient anatomical structure.

[0189] In step 434, the registered 3D image (transformation C) in the form of a DICOM file containing registration information in the header is transferred from the imaging device 10 to the robotic system 100.

[0190] The software within the robotic system 100 then extracts the patient reference - scan transformation (transformation C) from the 3D image, stores it in memory, and uses it for navigation when other surgical instrument arrays are tracked relative to the DRB marker 116, as the robotic system knows where the instrument is in the image space using transformation C.

[0191] In step 436, the imaging device 10 can now be removed from the surgical area. In step 438, the robotic system 100 uses the 3D image that has been aligned for navigation and robotic surgery. For example, the robotic system 100 may move the end effector (which is also tracked by the camera 200 through the marker 702) to place the end effector along a selected trajectory for inserting a bone screw. The robotic system 100 may also display the aligned 3D image on a display and manipulate the displayed image as the end effector is moved. The system 100 also displays the planned instrument trajectory, the dynamically updated trajectory of the optically tracked surgical instrument 608 and end effector 112 / 114, and the virtual representation of the tracked instrument, all of which are overlaid on the displayed 3D image and other displayed 2D images (A / P, lateral, and axial). Additionally, the system 100 can display any number of 2D views of 3D medical images, such as A / P, lateral, and axial views, which are simulated fluoroscopic shots (e.g., DRR images) or planar slices generated from the 3D image by image processing.

[0192] The robotic system 100 may also display the tracked surgical instrument overlaid on the displayed 3D image to assist the physician's visualization.

[0193] The robotic system can also display various images on the display to assist the physician. For example, four separate images can be displayed simultaneously, including Ap, lateral, axial, and probe-eye (3D) images. The axial view (either a DRR image or a slice image) representing a horizontal slice of bone at a point on the planned trajectory is particularly useful and can be generated by synthesizing the correct slice or DRR image in the correct orientation from the 3D image.

[0194] In step 439, the robotic system 100 detects that the alignment is impaired. This can be detected, for example, when the distance from the monitoring marker to the DRB 116 changes beyond a threshold amount. Then, steps 424-438 are repeated to realign the patient.

[0195] FIG. 30 is a flowchart of a method of aligning an intraoperative 3D image of a patient in an imaging space to a physical patient in a physical space and restoring the alignment of the 3D image with an intraoperative 2D image, preferably without using radiopaque fiducials embedded in the image.

[0196] Steps 422-439 of FIG. 30 when aligning the intraoperative 3D image are the same as the steps of FIG. 29. When a loss of alignment is detected in step 439, step 440 is executed by the processor 46 of the imaging device 10.

[0197] In step 440, the imaging system 10 is removed and positioned within the surgical area such that the camera 200 has good visibility with respect to an array of imaging tracking markers 68 on either side of the DRB 116 and the gantry mount 58.

[0198] In step 442, the tracking system / processor 532 of the robotic system 100 begins to continuously track the calibrated pose (i.e., position and orientation) of the imaging device 10 from the imaging tracking markers 68 included in the optical image coming from the camera 200.

[0199] In step 444, the tracking system 532 of the robotic system 100 also begins to continuously track the patient pose via the DRB 116.

[0200] Once the connection from the robot system 100 to the imaging device 10 is established via the I / O of the robot system, the processor 408 of the robot system begins to transmit camera frame data to the imaging device 10 via the communication link 52. The camera frame data includes the relative positions from the imaging tracking marker 68 to the DRB 116 within the camera space (e.g., the continuously tracked data of the imaging device 10 and the DRB 116). The camera 200 uses any visible reference array 68 (the left or right array on both sides of the gantry mount 58).

[0201] In step 446, a set of 2D images in two different orientations of the relevant patient anatomical structure, e.g., A / P and transverse, are captured by the imaging device 10. These images may be taken manually by the user or automatically by the imaging device 10, and the imaging device 10 rotates the second image by a preselected angle, e.g., 90 degrees, from the first image.

[0202] For each image, the above-described transformations (transformations A, B, and C) are performed such that each image in the image space is aligned to the physical patient in the patient coordinate system in the form of the DRB marker 116.

[0203] The alignment of each image can be done in several different ways. In the first method, the imaging device 10 is tracked using a fluorescence fixture temporarily mounted on a fluorescence collector (image intensifier or flat panel). An example of the fluoroscopy fixture 2100 is disclosed in U.S. Patent Application No. 15 / 267,950, filed on September 16, 2016 (Patent No. 10,799,298), which is incorporated herein by reference. This fixture has two planes of BBs and enables the fluorescence image to be processed such that the tracked position of the fluorescence X-ray emitter of the imaging device 10 can be accurately extrapolated from the tracked position of the fluorescence collector and thus from the image IM position within the imaging space.

[0204] In another variant of the method, preferably, the imaging device is optically tracked, for example, using the imaging tracking marker 68, and the locations of the emitter and the collector, and thus the image IM, are accurately extrapolated using the transformations (transformations A, B, and C) as described above.

[0205] Using each image, the position of the DRB116 attached to the patient is simultaneously tracked by the same optical tracking system 200 while tracking the position of the imaging tracking marker 68, and as a result, the image position in the imaging space is correlated to the DRB116 coordinate system representing the patient's physical space. In other words, for each image, transformation C is calculated as described above, and the image is stored as a DICOM file having a header containing the relevant alignment information.

[0206] In step 446, the DICOM file is transmitted via the communication link 52 to the surgical robotic system 100 for further processing. As can be understood, the DICOM file of the 2D image contains interval information including the angle and offset (displacement) of one 2D image with respect to the other 2D image, and this information can be used later for the merging operation.

[0207] In step 450, the imaging device 10 can now be removed from the surgical area.

[0208] Step 452 is executed by a processor in the computer 408 of the robotic system 100. In step 452, the aligned 2D images are merged with the 3D image acquired in step 430.

[0209] In one merging method, a simulated fluoroscopic shot called a digital reconstructed radiograph or DRR is created from the 3D image. Each DRR image represents an artificially or synthetically created fluoroscopic image based on a particular orientation and position of a theoretical fluoroscopic device with respect to the scan volume through image processing. In the case of a 3D image derived from a CT scan volume, the DRR image is generated by calculating the pixel intensities that are intersected and accumulated by rays passing through the CT scan volume from a point source (emitter) on one side of the CT volume through a diverging path to a plane (collector) on the other side. Optionally, the DRR image, 2D image, or 3D image volume may be processed with a high-pass filter to enhance the contrast around the bone boundaries.

[0210] In each comparison, two DRR images are generated at selected angles and offset from each other. The selected angles and offsets are based on the known (tracked) angles and offsets used to capture the two 2D images (A / P and lateral) in step 446. In this way, the two generated DRR images can be compared simultaneously with the two 2D images via an image processing algorithm. If the image content (contrast gradient of the bone edges) on the DRR image and the 2D image does not match anywhere within a given threshold, the theoretical position and orientation of the 3D image with respect to the fixed locations of the X-ray collector and emitter from the 2D image are slightly modified and two new DRR images are generated.

[0211] When generating a new DRR image, the direction and angle by which the 3D image volume is nudged (incremental changes in position and orientation) with respect to the fixed positions of the theoretical X-ray collector and emitter are based on whether the previous nudge (incremental change) caused an improvement or deterioration in the offset of the image content of the DRR image with respect to the actual 2D fluoroscopic image.

[0212] The iteration stops when further fine-tuning of the orientation and position of the 3D image volume with respect to the fixed positions of the theoretical X-ray collector and emitter no longer provides a better match of the 2D image content to the DRR image.

[0213] In an alternative embodiment, the matching is performed using a two-step process, whereby one of the two 2D images is matched to a particular DRR image using the merging method described above, and then the other of the two 2D images is matched using the merging method described above.

[0214] Once the theoretical emitter and collector positions are known with respect to the image volume (3D image), alignment is achieved because the theoretical emitter and collector positions provide corresponding locations in the imaging space (e.g., CT space) relative to the known tracked positions of the actual emitter and collector within the DRB116 space. Thereby, the alignment of the 3D image is restored and the alignment information is stored in the memory of the robotic system 100.

[0215] In an alternative merging method, in step 446, when an ultrasound transducer having an imaging tracking marker is used to capture a 2D image instead of the imaging device 10, a simulated 2D ultrasound image is created from the 3D image volume. Each 2D ultrasound image represents an ultrasound image artificially or synthetically created from a theoretical ultrasound imaging handpiece held at a particular position with respect to the 3D volume. In the case of a 3D image derived from a CT scan volume, the synthetic 2D ultrasound image is generated by calculating the pixel intensities that cross and accumulate due to sound waves that pass through the CT scan volume from an ultrasound source (emitter), reflect from different types of tissue (bone, soft tissue, blood), and are monitored by an ultrasound receiver. Optionally, the simulated 2D ultrasound image, the 3D image volume, or both may be processed with a high-pass filter to enhance the contrast around bone boundaries.

[0216] In each comparison, two simulated ultrasound images are generated at the selected angle and offset from each other. In step 446, the selected angle is based on the angles and offsets used to capture two 2D tracking ultrasound images (A / P and lateral). In this way, the two simulated 2D ultrasound images can be compared simultaneously with two actual 2D ultrasound images via an image processing algorithm. If the image content (gradient in the contrast of the bone margins) on the simulated image and the actual image does not match anywhere within a given threshold, the theoretical position and orientation of the 3D image volume relative to the fixed location of the theoretical ultrasound handpiece from the 2D image are slightly modified and two new simulated 2D ultrasound images are generated.

[0217] When generating a new 2D simulated ultrasound image, the direction and angle by which the image volume is nudged (incremental change in position and orientation) relative to the theoretical position of the ultrasound handpiece are based on whether the previous nudge (incremental change) caused an improvement or deterioration in the offset of the image content of the 2D simulated ultrasound image relative to the 2D actual ultrasound image.

[0218] The iteration stops when further fine-tuning of the orientation and position of the image volume no longer provides a better match of the simulated 2D ultrasound image to the actual 2D ultrasound image content. The use of the ultrasound transducer to generate the 2D image as described above may be used as an alternative in any alignment or alignment restoration procedure using the imaging device 10.

[0219] In step 454, the robotic system 100 uses the aligned 3D image (3D image with restored alignment) for navigation and robotic surgery, such as for controlling its arm 104 and end effector 112, and the content of the display 110. Similar to step 438, the processor / computer 408 displays the aligned 3D image and the two 2D images (A / P and lateral) on the display 304 together with the planned instrument trajectory, and the dynamically updated trajectories of the optically tracked surgical instrument 608 and end effector 112 / 114, and the virtual representation of the tracked instrument, all of which are overlaid on the displayed 3D and 2D images.

[0220] In step 456, the robotic system 100 detects that the alignment has been compromised again. In that case, control returns to step 440, and instead of requiring another full 3D scan of the patient, only the 2D images are required to realign the 3D image.

[0221] FIG. 31 is a flowchart of a method for aligning a preoperative 3D image of a patient in an imaging space to the physical patient in a physical space and restoring the alignment of the 3D image with intraoperative 2D images, preferably without using radiopaque fiducials embedded in the image.

[0222] In step 458, a 3D image of the relevant patient anatomical structure is captured by the imaging device 10 preoperatively. For example, the imaging device 10 can perform a full 3D X-ray CT scan, which is then converted to a 3D image or image volume based on a well-known CT conversion process. The 3D image is then stored in memory as a DICOM file.

[0223] Descriptions of steps with the same reference numbers as the previous figures are omitted for brevity.

[0224] In step 456, the robotic system 100 detects that the alignment is impaired. If so, control transfers to step 424, where alignment can be restored using only two 2D images (e.g., A / P image and lateral image) and merged them into the preoperative 3D image without performing another full 3D scan of the patient, thereby shortening the overall treatment time and reducing the patient's harmful X-ray exposure.

[0225] FIG. 32 is a flowchart of a method of aligning an intraoperative 2D image of a patient in an imaging space to a physical patient in a physical space (as represented by a camera / optical space as shown in FIG. 28) and restoring the alignment of the 2D image with another set of 2D images, preferably without using radiopaque fiducials embedded in the image.

[0226] This method may be useful when no 3D imaging system is available. Steps 422-428, 446-454 of FIG. 32 are the same as the steps in other figures.

[0227] As part of step 454, the processor / computer 408 displays the aligned 2D images (A / P and lateral) on the display 304 along with the dynamically updated trajectories of the optically tracked surgical instruments 608 and end effectors 112 / 114 and the virtual representations of the tracked instruments.

[0228] In step 456, the robotic system 100 detects that the alignment is impaired. In that case, control transfers to step 424, where two additional 2D images are acquired from the imaging device 10 and aligned by performing transforms A, B, and C, where transform C ultimately represents the alignment information between the 2D image in the imaging space and the DRB 116 in the camera 200 space (physical patient coordinate system).

[0229] However, as can be understood by those skilled in the art, it may not be easy to navigate based on only two 2D images displayed on the display 304. As an alternative to the method of FIG. 32 without access to a 3D imaging device, 3D images can be synthetically created from a general model such as an atlas model.

[0230] FIG. 33 is a flowchart of a method for aligning a 3D image synthetically created based on intraoperative 2D images of a patient, preferably without using radiopaque references embedded in the image. Image processing techniques such as statistical shape modeling and artificial intelligence or machine learning model techniques can be used to customize a general model using 2D images from the patient (e.g., A / P and lateral). Thus, after step 448, in step 460, the robotic system 100 generates a 3D image synthetically created based on the general model and the 2D images. One way to do this is to stretch and compress a general 3D model based on the 2D images. Before doing so, the 2D images can be processed for segmentation to identify each vertebral level and their well-known points.

[0231] In step 452, the aligned 2D images are merged with the customized 3D image obtained in step 460, and then the aligned synthetic 3D image is displayed on the display 304 in addition to the two 2D images. In this way, navigation using the tracking instrument, its trajectory, and the end effector, which is displayed and overlaid on the displayed 3D and 2D images, can be easier and more accurate to visualize.

[0232] Alternatively, when the general 3D model is customized, since the pose of the customized model is already synchronized with the 2D images, steps 460 and 452 are executed simultaneously or in reverse order. In that case, the alignment step is simply to convert the 3D image pose information (matching the orientation of the 2D image) into a form that can be used by the surgical robotic system 100.

[0233] These and other advantages of the present invention will be apparent to those skilled in the art from the foregoing specification. Accordingly, it will be recognized by those skilled in the art that changes or modifications can be made to the above-described embodiments without departing from the broad inventive concept of the invention. Therefore, it is to be understood that the present invention is not limited to the specific embodiments described herein, but is intended to cover all changes and modifications within the scope and spirit of the invention as defined by the claims.

Claims

Claim 1 A system for restoring alignment between a patient in a physical space and the intraoperative 3D image volume of the patient in an intraoperative three-dimensional image generated by imaging the patient during surgery, comprising: A portable imaging system having an imaging tracking marker array (ITMA); A dynamic reference base (DRB) including patient tracking markers and attachable to the patient; A tracking device configured to simultaneously track the positions and orientations of both the ITMA and the DRB; A surgical robot configured to assist in the surgery of the patient and communicate with the imaging system, the surgical robot including a base, a robot arm coupled to the base, and an end effector coupled to the robot arm; Operable to control the movement of the robot arm and the end effector, and when the distance between the position of the DRB in the physical space and a monitoring marker attached to the end effector exceeds a threshold, detecting that the alignment of the intraoperative 3D image volume in the intraoperative three-dimensional image with respect to the patient in the physical space is impaired, and if the detection is performed, after the detection: Receiving first and second 2D images of the patient at different orientations captured during surgery by the imaging system; Receiving, from the tracking device at the time of image capture, an optical image of the patient corresponding to the optical image, the optical image including the ITMA and the DRB; Identifying a simulated 2D image among a plurality of simulated 2D images that simulates an image obtained by photographing the patient in the physical space preoperatively by image processing a preoperative 3D image volume in a preoperative three-dimensional image captured preoperatively, wherein the position of the patient shown in each of the plurality of simulated 2D images matches the position of the patient shown in the first 2D image, and a simulated 2D image in which the position of the patient shown in the second 2D image matches; A system comprising a processor adapted to restore the alignment between the patient in the physical space and the intraoperative 3D image volume in the intraoperative three-dimensional image based on the simulated 2D image in which the position of the patient coincides with the position of the patient shown in the first 2D image and the simulated 2D image in which the position of the patient coincides with the position of the patient shown in the second 2D image. **Claim 2** The processor compares each of the first and second 2D images with a plurality of 2D digital reconstructed radiograph (DRR) images that are fluoroscopic images generated through calculation of the intensity of light rays passing through the patient along different orientations with respect to each other based on the preoperative 3D image volume, to identify a plurality of the DRR images in which the position of the patient shown coincides with the position of the patient shown in each of the first and second 2D images. The system according to claim 1, configured to repeat generation of the DRR images and comparison of the DRR images with the first and second 2D images while changing the orientation of the light rays until a plurality of the DRR images in which the position of the patient shown coincides with the position of the patient shown in each of the first and second 2D images are identified. **Claim 3** The processor generates a digital reconstructed radiograph (DRR) image of the preoperative 3D image volume, which is a fluoroscopic image generated through calculation of the intensity of light rays passing through the patient along a selected orientation based on the preoperative 3D image volume, compares each of the first and second 2D images with the DRR image, and is configured to identify the DRR image in which the position of the patient shown in each of the first and second 2D images coincides with the position of the patient shown in any of the DRR images by repeating generation of the DRR image and the comparison while changing the orientation of the light rays until a match is found between the position of the patient shown in each of the first and second 2D images and the position of the patient shown in any of the DRR images. The system according to claim 1. **Claim 4** The processor receives intraoperative ultrasound images as the first and second 2D images, and generates an ultrasound image through calculation of the intensity of ultrasound reflected at each part of the patient. The system according to claim 3. **Claim 5** The system according to claim 3, wherein the processor is configured to compare the contrast gradients of the bone edges of each of the first and second 2D images with the contrast gradient of the bone edge in any of the DRR images.

6. The system according to claim 5, wherein the processor is configured to select a new orientation of the ray for generating a new DRR image based on the magnitude of the correction of the orientation of the ray in the generation of the DRR image and the comparison before the generation of the new DRR image.

7. The second 2D image is an image captured in an orientation rotated by an angle selected from the orientation when the first 2D image was captured. The specifying by the processor includes: generating a first DRR image using the ray along a predetermined orientation; generating a second DRR image using the ray along an orientation obtained by rotating the orientation of the ray used for generating the first DRR image by the selected angle; simultaneously comparing the first and second 2D images with the first and second DRR images; repeating the generating step and the comparing step while changing the predetermined orientation of the ray until a match is found between the positions of the patient shown in each of the first and second 2D images and the positions of the patient shown in each of the first and second DRR images; The system according to claim 2, comprising:

8. The processor is configured to: calculate a transformation A representing the posture of the imaging system in the intraoperative three-dimensional image with respect to the DRB in the physical space when the first 2D image is captured, based on the received optical image, and restore the alignment between the patient in the physical space and the intraoperative 3D image volume in the intraoperative three-dimensional image based on the value of the transformation A. The system according to claim 1.

9. The processor is configured to: calculate a transformation B representing the position of the patient shown in the first 2D image with respect to the position of the imaging system in the physical space, based on the received optical image, and restore the alignment between the patient in the physical space and the intraoperative 3D image volume in the intraoperative three-dimensional image based on the value of the transformation B. The system according to claim 1.

10. The processor is configured to calculate the transformation B by calculating the relationship between the position of the patient shown in the first 2D image and the position of the ITMA in the physical space calibrated using calibration data that varies according to the amount of flexion of the imaging arm of the imaging system. The system according to claim 9.

11. The processor is configured to calculate the transformation B by calculating the position of the detector panel that detects the light rays used in the imaging of each of the first and second 2D images with respect to the position of the ITMA. The system according to claim 9.

12. The processor Based on the received optical image, calculating a transformation A representing the pose of the imaging system in the intraoperative three-dimensional image with respect to the DRB in the physical space when the first 2D image is captured; Based on the received optical image, calculating a transformation B representing the position of the patient shown in the first 2D image with respect to the position of the imaging system in the physical space; By multiplying transformation A by transformation B, calculating a transformation C representing the position of the patient shown in the first 2D image with respect to the position of the DRB in the physical space, thereby restoring the alignment between the patient in the physical space and the intraoperative 3D image volume in the intraoperative three-dimensional image. The system according to claim 1.

13. The processor is configured to display the intraoperative 3D image volume on a display, overlay the planned trajectory of the surgical instrument, and dynamically adjust the displayed trajectory on the displayed intraoperative 3D image volume as the trajectory is changed by the user based on the restored position of the patient's intraoperative 3D image volume in the intraoperative three-dimensional image with respect to the position of the patient in the physical space. The system according to claim 1.

14. The processor includes an imaging system processor resident in the imaging system and a robot processor resident in the surgical robot. The imaging system processor Add alignment information between the patient in the physical space and the patient's intraoperative 3D image volume in the intraoperative three-dimensional image to the image files of the first and second 2D images. configured to transfer the image file including the 2D image and the alignment information from the imaging system to the surgical robot, the robot processor, The system according to claim 1, wherein the robot processor is configured to restore alignment between the patient in the physical space and the intraoperative 3D image volume of the patient in the intraoperative three-dimensional image based on the image file transferred to the surgical robot.

15. The imaging system processor is configured to align the patient shown in each of the first and second 2D images with the patient in the physical space using the captured image from the imaging system and the optical image including the ITMA and the DRB. The robot processor is configured to restore alignment between the patient in the physical space and the intraoperative 3D image volume of the patient in the intraoperative three-dimensional image based on the first and second 2D images and the alignment information of the first and second 2D images. The system according to claim 14.

16. A system for restoring alignment between a patient in a physical space and an intraoperative 3D image volume of the patient in an intraoperative three-dimensional image generated by imaging the patient during surgery, a portable imaging system having an imaging tracking marker array (ITMA); a dynamic reference base (DRB) including a patient tracking marker and attachable to the patient; a tracking device having a plurality of cameras configured to simultaneously track the positions and orientations of both the ITMA and the DRB; a surgical robot configured to assist in the surgical operation of the patient and communicate with the imaging system, the surgical robot including a base, a robot arm coupled to the base, and an end effector coupled to the robot arm; operable to control the movement of the robot arm and the end effector, and when the distance between the position of the DRB in the physical space and a monitoring marker attached to the end effector exceeds a threshold, detecting that the alignment of the intraoperative 3D image volume in the intraoperative three-dimensional image with respect to the patient in the physical space is impaired, and when the detection is performed, after the detection, Receive the first and second intraoperative 2D images of the patient in different orientations captured during the operation by the imaging system, Receive an optical image corresponding to the patient from the camera at the time of image capture, the optical image including the DRB and the ITMA attached to the patient, Based on comparing a plurality of simulated 2D images that simulate an image obtained by photographing the patient in the physical space before the operation, which is generated by image processing of the preoperative 3D image volume in the preoperative three-dimensional image captured before the operation, identify the simulated 2D images in which the position of the patient shown in each of the plurality of simulated 2D images matches the position of the patient shown in the first 2D image, and the simulated 2D images in which the position of the patient shown in the second 2D image matches the position of the patient shown in the second 2D image, Restore the alignment between the patient in the physical space and the intraoperative 3D image volume in the intraoperative three-dimensional image based on the simulated 2D image in which the position of the patient matches the position of the patient shown in the first 2D image, the simulated 2D image in which the position of the patient matches the position of the patient shown in the second 2D image, and the alignment information between the patient in the physical space and the intraoperative 3D image volume of the patient in the intraoperative three-dimensional image added to the image files of the first and second 2D images, a system comprising a processor adapted to:

17. The processor is Calculate a transformation A representing the posture of the imaging system in the intraoperative three-dimensional image with respect to the DRB in the physical space when the first 2D image is captured based on the received optical image, and restore the alignment between the patient in the physical space and the intraoperative 3D image volume in the intraoperative three-dimensional image based on the value of the transformation A. The system according to claim 16, configured as such.

18. The processor is Based on the received optical image, calculate a transformation B representing the position of the patient shown in the first 2D image with respect to the position of the imaging system in the physical space, and based on the value of the transformation B, restore the alignment between the patient in the physical space and the intraoperative 3D image volume in the intraoperative three-dimensional image. The system according to claim 16, which is configured to do so.

19. The processor is configured to calculate a relationship between the position of the patient shown in the first 2D image and the position of the ITMA in the physical space calibrated using calibration data that changes according to the amount of bending of the imaging arm of the imaging system, and calculate the transformation B. The system according to claim 18, which is configured to do so.

20. The processor is Based on the received optical image, calculate a transformation A representing the posture of the imaging system in the intraoperative three-dimensional image with respect to the DRB in the physical space when the first 2D image is captured; Based on the received optical image, calculate a transformation B representing the position of the patient shown in the first 2D image with respect to the position of the imaging system in the physical space; By multiplying transformation A by transformation B, calculate a transformation C representing the position of the patient shown in the first 2D image with respect to the position of the DRB in the physical space, thereby restoring the alignment between the patient in the physical space and the intraoperative 3D image volume in the intraoperative three-dimensional image. The system according to claim 16, which is configured to do so.

Images