Method and system for displaying an image of an anatomical structure generated during a surgical procedure

Abstract

The present disclosure relates to a method and system for displaying an image of an anatomical structure generated during a surgical procedure. The method includes: segmenting at least one vertebral body from at least one image in a first three-dimensional image dataset. The method includes: receiving at least one image in a second three-dimensional image dataset. The method includes the steps of: registering the segmented at least one vertebral body from the at least one image in the first three-dimensional image dataset with the at least one image in the second three-dimensional image dataset. The method includes: determining the position of the at least one surgical implant based on the at least one image in the second three-dimensional image dataset and a three-dimensional geometric model of the at least one surgical implant. The method includes: overlaying a virtual representation of the at least one surgical implant on the registered and segmented at least one vertebral body from the at least one image in the first three-dimensional image dataset.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 846,319, filed on May 10, 2019, the entire disclosure of which is incorporated herein by reference. Background Art

[0003] Many surgical procedures require obtaining images of a patient's internal body structures such as organs and bones. In some procedures, the surgical procedure is completed with the aid of periodic images of the surgical site. A surgical procedure can broadly represent any invasive examination or intervention performed by medical personnel such as surgeons, interventional radiologists, cardiologists, pain management physicians, etc. In surgeries, procedures, and interventions that are effectively guided by continuous imaging (referred to herein as image - guided), frequent patient images are necessary for a doctor to correctly place surgical instruments whether they are catheters, needles, instruments, or implants, or to perform certain medical procedures. Fluoroscopy or fluoro is a form of intraoperative X - ray and is taken by a fluoroscopy unit (also called a C - arm). The C - arm sends an X - ray beam through the patient and takes pictures of anatomical structures such as bone and vascular structures in the area. Like any picture, the picture is a two - dimensional (2D) image of a three - dimensional (3D) space. However, like any picture taken with a camera, key 3D information can exist in the 2D image based on what is in front of what and how big one thing is relative to another.

[0004] Digital Reconstructed Radiographs (DRRs) are digital representations of X - rays made by CT - scanning a patient and simulating taking X - rays from different angles and distances. The result is that any possible X - ray that could be taken of the patient by a C - arm fluoroscope, for example, can be simulated, which is unique and specific for how the patient's anatomical features look relative to each other. Since the "scene" is controlled, i.e., by controlling the virtual position of the C - arm relative to the patient and the angles relative to each other, pictures can be generated that look like any X - ray taken by a C - arm in an operating room (OR).

[0005] Many imaging methods, such as taking fluoroscopic images, involve exposing the patient to radiation, albeit in small doses. However, during these image-guided procedures, the small doses add up such that the total radiation exposure is detrimental not only to the patient but also to the surgeon or radiologist and other personnel involved in the surgical procedure. When taking images, there are various known methods to reduce the amount of radiation exposure to the patient / surgeon, but these methods come at the cost of reducing the resolution of the images being obtained. For example, in contrast to standard imaging, some methods use pulsed imaging while other methods involve manually changing the exposure time or intensity. Narrowing the field of view can also potentially reduce the area and amount of radiation exposure (and alter the amount of radiation "scatter"), but this again comes at the cost of reducing the information available to the surgeon when making medical decisions. Collimators are available which can specifically reduce the area of exposure of a selectable region. However, since the collimator specifically excludes certain regions of the patient from exposure to X-rays, no images are available in those regions. Thus, medical personnel have an incomplete view of the patient, being restricted to a specifically selected region. Additionally, images taken during surgical intervention are often blocked by irrelevant OR equipment or by the actual instruments / implants being used to perform the intervention. Summary of the Invention

[0006] In one embodiment, a method includes the steps of: segmenting at least one vertebral body from at least one image in a first three-dimensional image dataset. The first three-dimensional image dataset is based on an initial scan of a surgical site that includes spinal anatomy. The method further includes the steps of: receiving at least one image in a second three-dimensional image dataset. The second three-dimensional image dataset is based on a second scan of the surgical site after the initial scan of the surgical site. The second scan of the surgical site includes spinal anatomy and at least one surgical implant. The method further includes the steps of: registering the segmented at least one vertebral body from the at least one image in the first three-dimensional image dataset with the at least one image in the second three-dimensional image dataset. The method further includes the steps of: determining the position of the at least one surgical implant based on the at least one image in the second three-dimensional image dataset and a three-dimensional geometric model of the at least one surgical implant. The method further includes the steps of: overlaying a virtual representation of the at least one surgical implant on the registered and segmented at least one vertebral body from the at least one image in the first three-dimensional image dataset based on the determined position of the at least one surgical implant.

[0007] In another embodiment, a method includes the steps of: segmenting an image in a first three-dimensional image dataset, where the first three-dimensional image dataset is based on an initial scan of a surgical site including a spinal anatomy. The method further includes the steps of: receiving an image in a second three-dimensional image dataset. The second three-dimensional image dataset is based on a second scan of the surgical site after the initial scan of the surgical site. The second scan of the surgical site includes the spinal anatomy and at least one surgical implant. The method further includes the steps of: registering the segmented image in the first three-dimensional image dataset with the segmented image in the second three-dimensional image dataset. The method further includes the steps of: determining the position of the at least one surgical implant based on the at least one image in the second three-dimensional image dataset and a three-dimensional geometric model of the at least one surgical implant. The method further includes the steps of: overlaying a virtual representation of the at least one surgical implant on the registered and segmented image in the first three-dimensional image dataset based on the determined position of the at least one surgical implant.

[0008] In another embodiment, a system for generating a display of an image of a patient's internal anatomy during a surgical procedure includes a display and a processor communicatively coupled to the display. The processor is configured to segment an image in a first three-dimensional image dataset, where the first three-dimensional image dataset is based on an initial scan of a surgical site including a spinal anatomy. The processor is further configured to receive an image in a second three-dimensional image dataset, where the second three-dimensional image dataset is based on a second scan of the surgical site after the initial scan of the surgical site. The second scan of the surgical site includes the spinal anatomy and at least one surgical implant. The processor is further configured to register the segmented image in the first three-dimensional image dataset with the image in the second three-dimensional image dataset. The processor is further configured to determine the position of the at least one surgical implant based on the image in the second three-dimensional image dataset and a three-dimensional geometric model of the at least one surgical implant. The processor is further configured to overlay a virtual representation of the at least one surgical implant on the registered and segmented image in the first three-dimensional image dataset based on the determined position of the at least one surgical implant. The processor is further configured to provide instructions to display, via the display, the virtual representation of the at least one surgical implant as an overlay on the registered and segmented image in the first three-dimensional image dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A diagram illustrating an example system for performing a surgical procedure according to an example embodiment.

[0010] Figure 2Illustrates an example robotic device that can be used during a surgical procedure according to an example embodiment.

[0011] Figure 3 Illustrates a block diagram of a computing device according to an example embodiment.

[0012] Figure 4 Illustrates an example diagram of an image from a three-dimensional image dataset according to an example embodiment.

[0013] Figure 5 Illustrates an example diagram of another image according to an example embodiment.

[0014] Figure 6 Illustrates an example diagram of another image from a second three-dimensional image dataset according to an example embodiment.

[0015] Figure 7 Illustrates an example diagram of another image according to an example embodiment.

[0016] Figure 8 Illustrates an example diagram of another image according to an example embodiment.

[0017] Figure 9 Illustrates a flowchart of an example method for three-dimensional visualization during a surgical procedure according to an example embodiment.

[0018] Figure 10 Illustrates a flowchart of another example method for three-dimensional visualization during a surgical procedure according to an example embodiment.

[0019] Figure 11 Illustrates an example computer-readable medium according to an example embodiment. Detailed Description

[0020] For the purpose of facilitating an understanding of the principles of the present invention, reference will now be made to the embodiments illustrated in the drawings and described in the following written specification. It should be understood that the scope of the present invention is not thereby limited. It should also be understood that the present invention includes any changes and modifications to the illustrated embodiments and includes further applications of the principles of the present invention that would typically occur to those skilled in the art to which the present invention pertains.

[0021] In one example, a system for displaying an image of a patient's internal anatomy during a surgical procedure includes a display and a processor communicatively coupled to the display. In one example, the processor is configured to receive a three-dimensional image data set of a surgical site of the patient captured via an imaging device prior to the start of the surgical procedure. The surgical site includes the patient's spinal anatomy relevant to the surgical procedure. In one scenario, the initial three-dimensional scan of the patient is performed at a higher radiation level than subsequent scans of the patient performed during the surgery to ascertain the progress of the surgical procedure. In one example, the three-dimensional scan is captured via an imaging device such as a C-arm imaging device.

[0022] The processor is further configured to segment an image in a first three-dimensional image data set based on an initial scan of the surgical site including the spinal anatomy. In one example, the processor may be configured to run one or more instructions for segmenting one or more vertebral bodies according to a deep neural network. In one example, the processor is configured to segment an image in the first three-dimensional image data set based on user input via the display. In one example, the user input is received via a touchscreen display.

[0023] In one scenario, depending on the progress of the surgical procedure, a second scan of the surgical site may be performed to determine whether a surgical implant has been inserted according to the surgical plan. In such a scenario, the processor is configured to receive an image in a second three-dimensional image data set. The second three-dimensional image data set is based on a second scan of the surgical site after the initial scan of the surgical site. At the time of the second scan, the surgical site includes the spinal anatomy and at least one surgical implant. In one example, the radiation level corresponding to the second scan is lower than the radiation level corresponding to the initial scan. The lower radiation level will be sufficient to determine the placement of the surgical implant. However, due to the lower radiation level, the resolution of the second three-dimensional image data set may not be sufficient for a user (e.g., a surgeon) to determine whether the surgical implant has been inserted according to the surgical plan. To overcome the resolution of the second three-dimensional image data set, the processor is configured to register the segmented image in the first three-dimensional image data set with the image in the second three-dimensional image data set. The processor is further configured to determine the position of the surgical implant based on one or more imaging algorithms used with the image in the second three-dimensional image data set and the three-dimensional geometric model of the at least one surgical implant. The processor is further configured to overlay a virtual representation of the at least one surgical implant on the registered and segmented image in the first three-dimensional image data set based on the determined position of the at least one surgical implant. The processor is further configured to provide instructions to display the virtual representation of the at least one surgical implant as an overlay on the registered and segmented image in the first three-dimensional image data set via the display.

[0024] Referring now to the drawings, Figure 1 is a diagram of an example system 100 for performing a surgical procedure and for displaying an image of a patient's internal anatomy during the surgical procedure. The example system 100 includes a base unit 102 that supports a C-Arm imaging device 103. The C-Arm includes a radiation source 104 that is located below the patient P and directs a radiation beam upward to a receiver 105. The receiver 105 of the C-Arm 103 sends image data to a processing device 122. The processing device 122 can communicate with a tracking device 130 to obtain position information of various instruments T used during the surgical procedure. The tracking device 130 can communicate with a robotic device 140 to provide position information of various tracking components, such as markers 150. The robotic device 140 and the processing device 122 can communicate via one or more communication channels.

[0025] The base unit 102 includes a control panel 110 through which a user can control the position of the C-Arm 103, as well as radiation exposure. Thus, the control panel 110 enables a radiologic technologist to "take a picture" of the surgical site in the direction of the surgeon, control the radiation dose, and initiate a radiation pulse image.

[0026] For different perspectives of the surgical site, the C-Arm 103 can rotate around the patient P in the direction of arrow 108. In one example, the C-Arm 103 is used to capture a three-dimensional scan of the patient. In one example, an initial three-dimensional scan is used to provide a first three-dimensional image dataset, and a second three-dimensional scan is used to provide a second three-dimensional image dataset. In one example, the first three-dimensional image dataset and the second three-dimensional image dataset include one or more of fluoroscopic images or computed tomography images. In another example, the first three-dimensional image dataset and the second three-dimensional image dataset include fluoroscopic images, and the initial scan of the surgical site is associated with a higher radiation level than the second scan of the surgical site. In another example, the second scan includes a plurality of surgical implants associated with one vertebral body from the at least one image in the first three-dimensional image dataset.

[0027] In some cases, the implant or instrument T may be located at the surgical site and it may be necessary to change the perspective to obtain an unobstructed view of the site. Thus, the position of the receiver relative to the patient and more particularly relative to the surgical site of interest may change during the surgery according to the needs of the surgeon or radiologist. Accordingly, the receiver 105 may include a tracking target 106 mounted thereto, which enables the use of a tracking device 130 to track the position of the C-arm 103. By way of example only, the tracking target 106 may include a plurality of infrared reflectors or emitters spaced around the target, and the tracking device 130 may be configured to triangulate the position of the receiver 105 based on the infrared signals reflected or emitted by the tracking target 106.

[0028] The processing device 122 may include a digital memory associated therewith and a processor for executing digital and software instructions. The processing device 122 may also incorporate a frame grabber using frame grabber technology to create digital images for projection as display portions 123 and 124 on the display device 126. The display portions 123 and 124 are positioned for interactive viewing by the surgeon during the surgery. The two display portions 123 and 124 may be used to show images from two fields of view (such as lateral and A / P), or may show a baseline scan and a current scan of the surgical site. An input device 125 (such as a keyboard or touch screen) may allow the surgeon to select and manipulate the images on the screen. It should be understood that the input device may incorporate an array of keys or touch screen icons corresponding to the various tasks and features implemented by the processing device 122. The processing device 122 includes a processor that converts the image data obtained from the receiver 105 into a digital format. In some cases, the C-arm 103 may operate in a cine exposure mode and generate many images per second. In these cases, multiple images may be averaged together into a single image over a short period of time to reduce motion artifacts and noise.

[0029] In one example, once an initial three-dimensional scan is acquired, a three-dimensional image dataset is generated, where the three-dimensional image is digitally rotated, translated, and resized to create thousands of permutations of the three-dimensional image. For example, a typical 128×128 pixel two-dimensional (2D) image can be translated in the x and y directions by + / −15 pixels at 1 pixel intervals, rotated by + / −9 degrees at 3-degree intervals, and scaled from 92.5% to 107.5% at 2.5% intervals (4 degrees of freedom, 4D), resulting in 47,089 images in the image dataset. Due to the addition of two additional rotations orthogonal to the x and y axes, a three-dimensional (3D) image would imply a 6D solution space. The original computed tomography image dataset can be used to form thousands of digital reconstructed radiographs in a similar manner. Thus, the original three-dimensional image yields thousands of new image representations as if the initial three-dimensional image was acquired in each of the various permutations of different movements. Depending on the number of images in the "solution space" and the speed at which the graphics processing unit (GPU) of the processing device 122 can generate these images, the solution space can be stored in the graphics card memory (such as stored in the GPU), or formed into new images that are then sent to the GPU.

[0030] During surgery, new three-dimensional images are acquired at a lower radiation dose and stored in the memory associated with the processing device 122. Since the new images are acquired at a lower radiation dose, the noise may be significant. In one example, the processing device 122 is configured to "merge" the new images with the segmented images from the initial three-dimensional image dataset to generate a clearer image for display to convey more useful information to the user (e.g., a surgeon). The new images are compared with the images in the initial three-dimensional image dataset to find statistically significant matches.

[0031] In one example, image registration occurs in less than one second so that there is no significant delay between when an image is taken by the C-arm and when the merged image is shown on the display device 126. Various algorithms can be employed, which can depend on various factors such as the number of images in the initial three-dimensional image dataset, the size and speed of the computer processor or graphics processor performing the algorithm calculations, the time allotted to perform the calculations, and the size of the images being compared (e.g., 128×128 pixels, 1024×1024 pixels, etc.). In one method, comparisons are made between pixels at the aforementioned predetermined positions in a grid pattern throughout the 4D or 6D space. In another heuristic method, pixel comparisons can be concentrated in regions of the image that are considered to offer a greater likelihood of a relevant match. These regions can be "pre-seeded" based on knowledge from the grid or PCA search (defined below), data from a tracking system such as an optical surgical navigation device, or position data from a DICOM file or equivalent file. Alternatively, the user can specify one or more regions of the image for comparison by marking anatomical features on the initial three-dimensional image that are considered relevant to the surgery. Using this input, each pixel in the region can be assigned a correlation score between 0 and 1, which can measure the contribution of the pixel to the image similarity function when comparing the new image to the initial three-dimensional image. The correlation score can be calibrated to identify regions to focus on or regions to ignore. In one example, the step of registering a segmented image in a first three-dimensional image dataset with an image in a second three-dimensional image dataset includes performing one or more of the following operations on the image information contained within one vertebral body of the at least one image from the first three-dimensional image dataset: horizontal translation, vertical translation, rotation, and scaling.

[0032] Tracking device 130 includes sensors 131 and 132 that are used to determine position data associated with various components used in a surgical procedure (e.g., infrared reflectors or transmitters). In one example, sensors 131 and 132 can be charge-coupled device (CCD) image sensors. In another example, sensors 131 and 132 can be complementary metal-oxide semiconductor (CMOS) image sensors. It is also contemplated that other image sensors can be used to achieve the described functionality.

[0033] In one aspect of the present invention, robotic device 140 can assist in maintaining an instrument T relative to a patient P during a surgical procedure. In one scenario, robotic device 140 can be configured to maintain the relative position of instrument T with respect to patient P as patient P moves (e.g., due to breathing) or is moved during the surgical procedure (e.g., due to manipulation of the patient's body).

[0034] The robotic device 140 may include: a robotic arm 141, a foot pedal 142, and a mobile housing 143. The robotic device 140 may also communicate with a display 126. The robotic device 140 may also be configured to be coupled to an operating table by a fixing device.

[0035] The robotic arm 141 may be configured to receive one or more end effectors depending on the surgical procedure. In one example, the robotic arm 141 may be a six-joint arm. In this example, each joint includes an encoder that measures the angular value of the joint. The movement data provided by the one or more encoders combined with the known geometry of the six joints may enable the determination of the position of the robotic arm 141 and the position of the instrument T coupled to the robotic arm 141. It is also contemplated that a different number of joints may be used to achieve the functions described herein.

[0036] The mobile housing 143 ensures easy maneuverability of the robotic device 140 by using wheels or a handle or both. In one embodiment, the mobile base may include a fixing pad or an equivalent device. The mobile housing 143 may also include a control unit that provides one or more commands to the robotic arm 141 and enables a surgeon to manually input data through an interface such as a touch screen, a mouse, a joystick, a keyboard, or a similar device.

[0037] Figure 2 An example robotic device 200 that may be used during a surgical procedure is illustrated. The robotic device 200 may include hardware such as a processor, a memory, or a storage device, and sensors that enable the robotic device 200 to operate the robotic device for use in a surgical procedure. The robotic device 200 may be powered by various devices such as an electric motor, a pneumatic motor, a hydraulic motor, etc. The robotic device 200 includes: a base 202, linkages 206, 210, 214, 218, 222, and 226, joints 204, 208, 212, 216, 220, 224, and 230, and a manipulator 228.

[0038] The base 202 may provide a platform to support the robotic device 200. The base 202 may be stationary or coupled to wheels to provide movement of the robotic device 200. The base may include any number of materials such as aluminum, steel, stainless steel, etc., that may be suitable for a given environment associated with the robotic device 200.

[0039] The linkages 206, 210, 214, 218, 222, and 226 can be configured to move according to a programmable instruction set. For example, the linkages can be configured to follow a predetermined set of movements to perform a task under the supervision of a user. By way of example, the linkages 206, 210, 214, 218, 222, and 226 can form a kinematic chain that defines the relative movement of a given linkage among the linkages 206, 210, 214, 218, 222, and 226 at a given joint among the joints 204, 208, 212, 216, 220, 224, and 230.

[0040] The joints 204, 208, 212, 216, 220, 224, and 230 can be configured to rotate by using a mechanical gear system. In one example, the mechanical gear system can be driven by a strain wave gearing, a cycloidal drive, etc. The mechanical gear system selected will depend on a number of factors related to the operation of the robotic device 200, such as the length of a given linkage among the linkages 206, 210, 214, 218, 222, and 226, the rotational speed, the desired gear reduction, etc. Powering the joints 204, 208, 212, 216, 220, 224, and 230 will enable the linkages 206, 210, 214, 218, 222, and 226 to move in a manner that allows the manipulator 228 to interact with the environment.

[0041] The manipulator 228 can be configured to enable the robotic device 200 to interact with the environment. In one example, the manipulator 228 can perform proper placement of components by various operations (such as grasping a surgical instrument). By way of example, the manipulator can be replaced with another end effector that will provide different functions for the robotic device 200.

[0042] The robotic device 200 can be configured to operate according to a robotic operating system (e.g., an operating system designed for a specific function of the robot). The robotic operating system can provide libraries and tools (e.g., hardware abstraction, device drivers, visualization tools, messaging, package management, etc.) to enable the implementation of robotic applications.

[0043] Figure 3 is a block diagram of a computing device 300 according to an example embodiment. In some examples, Figure 3 some of the components shown may be distributed across multiple computing devices (e.g., a desktop computer, a server, a handheld device, etc.). However, for the sake of example, these components are shown and described as part of one example device. The computing device 300 can include: an interface 302, a mobile unit 304, a control unit 306, a communication system 308, a data storage 310, and a processor 314. Figure 3The illustrated components may be linked together via communication link 316. In some examples, computing device 300 may include hardware enabling communication within computing device 300 and another computing device (not shown). In one embodiment, robotic device 140 or robotic device 200 may include computing device 300.

[0044] Interface 302 may be configured to enable computing device 300 to communicate with another computing device (not shown). Thus, interface 302 may be configured to receive input data from one or more devices. In some examples, interface 302 may also store and manage data received and transmitted by computing device 300. In other examples, the data may be stored and managed by other components of computing device 300. Interface 302 may also include a receiver and a transmitter for receiving and transmitting data. In some examples, interface 302 may also include a user interface for receiving input as well, such as a keyboard, a microphone, a touch screen, etc. Additionally, in some examples, interface 302 may also be connected to output devices such as a display, a speaker, etc.

[0045] For example, interface 302 may receive input indicating such location information that corresponds to one or more elements of the environment where the robotic device (e.g., robotic device 140, robotic device 200) is located. In this example, the environment may be an operating room in a hospital, which includes a robotic device configured to operate during a surgical procedure. Interface 302 may also be configured to receive information associated with the robotic device. For example, the information associated with the robotic device may include the operating characteristics of the robotic device and the range of motion of the components of the robotic device.

[0046] The control unit 306 of the computing device 300 can be configured to run control software that exchanges data with components of a robotic device (e.g., robotic device 140, robotic device 200) (such as robotic arm 141, robotic pedal 142, joints 204, 208, 212, 216, 220, 224, and 230, robotic hand 228, etc.) and one or more other devices (e.g., processing device 122, tracking device 130, etc.). The control software can communicate with a user via a display monitor (e.g., display 126) and a user interface that communicates with the robotic device. The control software can also communicate with the tracking device 130 and the processing device 122 via a wired communication interface (e.g., parallel port, USB, etc.) and / or a wireless communication interface (e.g., antenna, transceiver, etc.). The control software can communicate with one or more sensors to measure the effort applied by a user at an instrument T mounted to a robotic arm (e.g., robotic arm 141, link 226). The control software can communicate with the robotic arm to control the position of the robotic arm relative to a marker (e.g., marker 150).

[0047] As described above, the control software can communicate with the tracking device 130. In one scenario, the tracking device 130 can be configured to track a marker 150 attached to a patient P. For example, the marker 150 can be attached to the spinous process of a vertebra of the patient P. In this example, the marker 150 can include one or more infrared reflectors that are visible to the tracking device 130 to determine the position of the marker 150. In another example, multiple markers can be attached to one or more vertebrae and used to determine the position of the instrument T.

[0048] In one example, the tracking device 130 can provide updates to the position information of the marker 150 to the control software of the robotic device 140 in near real-time. The robotic device 140 can be configured to receive updates to the position information of the marker 150 from the tracking device 130 via a wired and / or wireless interface. Based on the received updates to the position information of the marker 150, the robotic device 140 can be configured to determine one or more adjustments to a first position of the instrument T to maintain a desired position of the instrument T relative to the patient P.

[0049] The control software may include independent modules. In an exemplary embodiment, these independent modules run simultaneously in a real-time environment and use a shared memory to ensure the management of various tasks of the control software. These modules may have different priorities. For example, there may be a safety module with the highest priority. The safety module may monitor the state of the robotic device 140. In one scenario, when an emergency is detected (for example, an emergency stop, a software failure, or a collision with an obstacle), the safety module may send an instruction to the control unit 306 to stop the robotic arm 141.

[0050] The control unit 306 may be configured to manage functions associated with various components of the robotic device 140 (for example, the robotic arm 141, the pedal 142, etc.). For example, the control unit 306 may send one or more commands to maintain a desired position of the robotic arm 141 relative to the marker 150. The control unit 306 may be configured to receive movement data from the mobile unit 304.

[0051] The mobile unit 304 may be configured to determine movements associated with one or more components of the robotic arm 141 to perform a given surgery. In one embodiment, the mobile unit 304 may be configured to use forward and inverse kinematics to determine the trajectory of the robotic arm 141. In one scenario, the mobile unit 304 may access one or more software libraries to determine the trajectory of the robotic arm 141.

[0052] The mobile unit 304 may be configured to simulate the operation of moving the instrument T along a given path by the robotic device 140. In one example, based on the simulated operation, the mobile unit 304 may determine metrics associated with the instrument T. Additionally, the mobile unit 304 may be configured to determine forces associated with the metrics based on the simulated operation. In one example, the mobile unit 304 may include instructions for determining forces based on an open kinematic chain.

[0053] The mobile unit 304 may include a force module that monitors forces and torques measured by one or more sensors coupled to the robotic arm 141. In one scenario, the force module is capable of detecting a collision with an obstacle and warning the safety module.

[0054] The interface 302 may be configured to allow the robotic device 140 to communicate with other devices (for example, the processing device 122, the tracking device 130). Thus, the interface 302 may be configured to receive input data from one or more devices. In some examples, the interface 302 may also maintain and manage data records received and sent by other devices. In other examples, the interface 302 may use a receiver and a transmitter for receiving and sending data.

[0055] The interface 302 can be configured to manage the communication between the user and the control software through a user interface and a display screen (e.g., via display units 123 and 124). The display screen can display a graphical interface that guides the user through different modes associated with the robotic device 140. The user interface can enable the user to control, for example, the movement of the robotic arm 141 associated with the start of a surgical procedure, enable a tracking mode to be used during the surgical procedure, and stop the robotic arm 141 if necessary.

[0056] In one example, the user can control the actuation of the robotic arm 141 by using the robotic pedal 142. In one embodiment, the user can press the robotic pedal 142 to enable one or more modes of the robotic device 140. In one scenario, the user can press the robotic pedal 142 to enable the user to manually position the robotic arm 141 according to a desired position. In another scenario, the user can press the robotic pedal 142 to enable a tracking mode that enables the robotic arm 141 to hold the instrument T in a relative position with respect to the patient P. In another scenario, the user can press the robotic pedal 142 to stop the robotic arm 141 from making any further movement.

[0057] In one scenario, the control unit 306 can instruct the robotic arm 141 to operate according to a collaborative mode. In the collaborative mode, the user can manually move the robotic arm 141 by holding the tool T coupled to the robotic arm 141 and moving the instrument T to a desired position. In one example, the robotic device 140 can include one or more force sensors coupled to the end effector of the robotic arm 141. For example, when the user grabs the instrument T and starts moving the instrument in one direction, the control unit 306 receives the forces measured by the force sensors and combines these forces with the position of the robotic arm 141 to generate the movement desired by the user.

[0058] In one scenario, the control unit 306 can instruct the robotic arm 141 to operate according to a tracking mode. In the tracking mode, the robotic device 140 will maintain the position of the instrument T relative to a given IR reflector or emitter (e.g., marker 150). In one example, the robotic device 140 can receive updated position information of the marker 150 from the tracking device 130. In this example, the movement unit 304 can determine which joint(s) of the robotic arm 141 need to move based on the received updated position information of the marker 150 in order to maintain the relative position of the instrument T and the marker 150.

[0059] In one embodiment, the robotic device 140 may communicate with the processing device 122. In one example, the robotic device 140 may provide the position information of the instrument T to the processing device 122. In this example, the processing device 122 may be configured to store the position information of the instrument T for further processing. In one scenario, the processing device 122 may use the received position of the instrument T to overlay a virtual representation of the instrument T on the display 126.

[0060] In another aspect of the present invention, a robotic device (e.g., robotic device 140, robotic device 200) may assist an instrument in moving along a planned or learned path. In one scenario, a surgeon may plan the trajectory of a pedicle screw or the placement of a retractor preoperatively based on a three-dimensional (3D) image or intraoperatively by holding a position outside the skin and projecting the tip of the instrument into the patient's body to see how the trajectory intersects with the anatomical structure of interest.

[0061] In one scenario, a surgeon may select an ideal pedicle screw trajectory for a particular level of the spine preoperatively, but then select a different trajectory intraoperatively to penetrate the skin and muscle. In one example, the surgeon may guide a tool coupled to the robotic device along a different trajectory until the tool intersects the ideal pedicle screw trajectory. In this example, the robotic device may provide a signal to a computing device (e.g., processing device 122, computing device 300), which in turn may notify the surgeon via an auditory or visual cue that the ideal pedicle screw trajectory has been reached.

[0062] In another scenario, once the instrument of a robotic arm (e.g., robotic arm 141, links 206, 210, 214, 218, 222, and 226) coupled to the robotic device reaches the ideal pedicle screw trajectory, the robotic device may be configured to receive an input from the surgeon to travel along the ideal pedicle screw. In one example, the surgeon may provide an input (e.g., depress the pedal 142) to the robotic device to confirm that the surgeon desires to enable the robotic device to travel along the ideal pedicle screw. In another example, the user may provide another form of input to the robotic device or the computing device to assist the instrument in moving along a predetermined path.

[0063] In one scenario, once the robotic device has received confirmation to travel along the ideal pedicle screw trajectory, the robotic device may receive instructions from the mobile unit 304 to pivot from the current trajectory to the ideal pedicle screw trajectory. The mobile unit 304 may provide the required movement data to the control unit 306 to enable the robotic device to move along the ideal pedicle screw trajectory.

[0064] In another example, the mobile unit 304 can provide one or more trajectories to a computing device (e.g., the processing device 122) for display on the display 126. In this example, the user can select from one or more predetermined movements associated with a given procedure. For example, a given predetermined movement can be associated with a particular movement direction and movement amount to be performed by depressing the pedal 142 of the robotic device 140.

[0065] In another aspect of the present invention, one or more infrared (IR) reflectors or transmitters can be coupled to the robotic arm (e.g., the robotic arm 141, linkages 206, 210, 214, 218, 222, and 226) of the robotic device (e.g., the robotic device 140, the robotic device 200). In one scenario, the tracking device 130 can be configured to determine the positions of the one or more IR reflectors or transmitters before the robotic device begins operation. In such a scenario, the tracking device 130 can provide the position information of the one or more IR reflectors or transmitters to a computing device (e.g., the processing device 122, the computing device 300).

[0066] In one example, the processing device 122 or the computing device 300 can be configured to compare the position information with data stored in a local or remote database that contains information about the robotic device (e.g., a geometric model of the robotic device) to assist in determining the positioning or location of the robotic arm. In one example, the processing device 122 can determine a first position of the robotic arm based on the information provided by the tracking device 130. In this example, the processing device 122 can provide the determined first position of the robotic arm to the robotic device or a computing device (e.g., the computing device 300). In one example, the robotic device can use the received first position data to perform calibration of one or more components (e.g., encoders, actuators) associated with the one or more joints of the robotic arm.

[0067] In one scenario, the instrument coupled to the robotic arm of the robotic device can be used to determine the difference between the expected tip position of the instrument and the actual tip position of the instrument. In such a scenario, the robotic device can continue to move the instrument to a known position via the tracking device 130 so that the tip of the tool contacts the known position. The tracking device 130 can capture the position information corresponding to the one or more IR reflectors or transmitters coupled to the robotic arm and provide this information to the robotic device or a computing device (e.g., the processing device 122, the computing device 300). Additionally, the robotic device or the computing device can be configured to adjust the coordinate system offset between the robotic device and the tracking device 130 based on the expected tip position and the actual tip position of the tool.

[0068] In one example, a surgeon may use a three-dimensional image of the spine in combination with one or more planes that the instrument is not to pass through. In this example, although a force or pressure sensor detects a force that causes the instrument to move, the robotic arm will not allow the surgeon to move the instrument beyond the one or more planes specified, in accordance with constraints associated with a predefined plan. For example, the robotic device may be configured to provide an alert to the surgeon when the instrument approaches the one or more limiting planes.

[0069] In another aspect of the invention, a robotic device (e.g., robotic device 140, robotic device 200) may be used to navigate one or more surgical instruments and implants. The robotic device may be configured to provide navigation information to a computing device (e.g., processing device 122, computing device 300) for further processing. In one example, the computing device may be configured to determine a virtual representation of a surgical instrument or implant. Additionally, the computing device may be configured to overlay the virtual representation of the surgical instrument or implant on a two-dimensional or three-dimensional image of the surgical site.

[0070] In one example, the robotic device may perform a calibration process with the tracking device 130 to eliminate the reliance on the position information of the tracking device 130 in the event that the line of sight between the robotic device and the tracking device 130 is blocked. In one example, using a robotic device (as described above) that has been registered to the navigation system and a three-dimensional image of the patient corresponding to the surgical site may allow the robotic device to become immune to a decrease in the accuracy of the distance associated with the tracking device 130.

[0071] In another example, the robotic device may be used in one or more such surgical procedures: in which the patient's anatomy is oriented in a manner that restricts the ability of the tracking device 130 to maintain a line of sight with the anatomy. For example, the tracking device 130 may have difficulty maintaining a line of sight during a single-position surgical procedure.

[0072] In one example, the robotic arm of a robotic device can be coupled to an end effector configured to attach to the spinous process of a patient's vertebra. The end effector can include one or more force or pressure sensors to detect the magnitude of the force or pressure exerted by the spinous process during surgery. This will enable the robotic device to send positioning or location information associated with the patient's movement to a computing device (e.g., processing device 122, computing device 300). In one scenario, the computing device can be configured to receive positioning or location information associated with the patient's movement and update one or more images of the surgical site based on the movement information. For example, the computing device can access a database containing a baseline image set of the surgical site and provide an image for display on display 126 that corresponds to the updated position of the patient based on the position information received from the robotic device when the end effector is attached to the spinous process. One advantage of using a robotic device to track the patient's movement is that it is not necessary to attach one or more IR reflective markers to the spinous process to track the patient's movement via tracking device 130.

[0073] In another example, the robotic device can assist in tracking an instrument coupled to the robotic arm at one or more locations during surgery. Tracking the instrument via the movement of the robotic device can enable the instrument to be placed in a location that is difficult for the surgeon to see. For example, the instrument may be placed behind a surgical drape but can be tracked via the robotic device and a computing device (e.g., processing device 122, computing device 300). In another example, the robotic device can assist in tracking the patient's movement under a sterile barrier. In this example, the robotic device can be used to reposition the operating table to keep the patient in a known orientation during surgery.

[0074] In one example, the surgeon can input the path of the surgery before starting the surgery. For example, the surgeon can use two-dimensional or three-dimensional images of the patient's anatomy and determine the path to the surgical site. In one example, a computing device (e.g., processing device 122, computing device 300) can store information corresponding to a predetermined path and provide this information to the robotic device before the surgery begins. Once the robotic device knows its position relative to the patient, the movement unit 304 can use the information corresponding to the predetermined path to determine one or more allowed trajectories.

[0075] In another example, the path restricting the movement of the robotic arm can correspond to one or more inputs corresponding to anatomic dissection. For example, the surgeon can select a specific vertebra to restrict the movement of the robotic arm to that specific vertebra. By way of example, the robotic device can be further instructed to restrict the movement of the robotic arm to a specific part of the vertebra (e.g., the spinous process, etc.).

[0076] In one embodiment, an end effector can be coupled to a robotic arm (e.g., robotic arms 141, links 206, 210, 214, 218, 222, and 226) and assist in the placement of fasteners. In one scenario, the robotic device can receive three-dimensional geometric information about the plate and fasteners being used in a surgical procedure from a computing device (e.g., processing device 122, computing device 300).

[0077] In one example, a given plate may require four fasteners to be installed during a surgical procedure. In this example, the robotic device can use the end effector to retract soft tissue corresponding to the first fastener location of the given plate based on a trajectory determined by the mobile unit 304. Additionally, the mobile unit 304 can be configured to determine an optimal trajectory for placing the fastener through the given plate. After placing the first fastener, the robotic device can be configured to move the end effector in such a way as to allow the soft tissue to return to its original position and move to retract soft tissue corresponding to the second fastener location of the given plate. In this example, the robotic device can minimize the time that the soft tissue is retracted, thereby reducing the risk of damaging the soft tissue when each of the fasteners is installed into the given plate.

[0078] In another aspect of the present invention, a robotic device (e.g., robotic device 140, robotic device 200) can use position information captured by a tracking device 130 to determine the position of an instrument and an implant coupled to a robotic arm (e.g., robotic arms 141, links 206, 210, 214, 218, 222, and 226) relative to a patient. In one embodiment, the robotic device can use movement information determined by encoders associated with one or more joints of the robotic device (e.g., joints 204, 208, 212, 216, 220, 224, and 230) to determine the position of a surgical tool after a calibration process between the robotic device and the tracking device 130. In another embodiment, the tracking device 130 can provide position information to a computing device (e.g., processing device 122, computing device 300) to assist in tracking the robotic device during a surgical procedure.

[0079] In one example, the tracking device 130 can track the position of an instrument coupled to a robotic arm based on one or more IR reflectors or transmitters. For example, the tracking device 130 can detect an IR reflector or transmitter coupled to a tool and provide position information to the processing device 122. The processing device 122 can be configured to compare the last known position information of the IR reflector or transmitter coupled to the instrument with the most recent position information and determine a change in position associated with the instrument.

[0080] In another example, a virtual representation of a path associated with an instrument coupled to a robotic arm can be overlaid on a corresponding location of a patient's anatomy. The virtual representation of the path can be displayed using a variety of visual effects to represent multiple passes of the instrument over a particular region of the spine.

[0081] In another aspect of the invention, a robotic device (e.g., robotic device 140, robotic device 200) can include more than one robotic arm (e.g., robotic arm 141, linkages 206, 210, 214, 218, 222, and 226). In one scenario, as described above, the robotic device and tracking device 130 may have completed a registration process to correct for any offsets between their respective coordinate systems. In such a scenario, in addition to completing the registration process, the processing device 122 can be configured to receive a three-dimensional scan of a patient's spine. In one embodiment, the robotic device can be configured to maintain a patient's spinal alignment according to a preoperative plan for spinal alignment.

[0082] In one example, the robotic device can use an end effector that is configured to grasp critical components of a surgical procedure. For example, the robotic device can use a first gripper coupled to a robotic arm to grasp a first pedicle screw, and a second gripper coupled to a second robotic arm to grasp a second pedicle screw. The robotic device can be configured to provide position information associated with each of the first robotic arm and the second robotic arm to a computing device (e.g., processing device 122, computing device 300). Based on the received position information, the computing device can determine the current spinal alignment. In addition, the computing device can analyze the current spinal alignment to determine the desired correction of the spine during the surgical procedure.

[0083] In another aspect of the invention, a robotic arm (e.g., robotic arm 141, linkages 206, 210, 214, 218, 222, and 226) of a robotic device (e.g., robotic device 140, robotic device 200) can be configured to receive an ultrasound probe. In one scenario, the ultrasound probe is held by the robotic arm in a known orientation to enable registration of the position of an anatomy in an image relative to the image for subsequent instrument manipulation using the robotic arm or a co-registered navigation system (e.g., registration between robotic device 140 or robotic device 200 and tracking device 130).

[0084] Figure 4 is an example view of an image 400 from a three-dimensional image data set of an initial scan of a surgical site 402 that includes spinal anatomy. In one example, the initial scan of the surgical site 402 can be obtained by Figure 1performed by the C-arm imaging device 103. The image 400 includes a first vertebral body 404 and a second vertebral body 406. In one example, Figure 1 the processing device 122 is configured to segment the first vertebral body 404 from the image 400. For example, Figure 5 is an example diagram of an image 500 including the first vertebral body 404 segmented from the second vertebral body 406.

[0085] In one example during a given surgical procedure, one or more surgical implants (e.g., pedicle screws) can be inserted into the first vertebral body 404 to achieve a given surgical outcome. In this example, after inserting the one or more surgical implants into the first vertebral body 404, a user (e.g., a surgeon) may want to check the positioning of the implant. In one example, a second scan of the surgical site 402 can be performed at a lower radiation level than the initial scan. By way of example, Figure 6 is an example diagram of an image 600 in a second three-dimensional image dataset. The second three-dimensional image dataset is based on a second scan of the surgical site 402 after an initial scan of the surgical site 402. The second scan of the surgical site 402 includes the spinal anatomy and the surgical implant 604.

[0086] In one example, the processing device 122 is configured to receive the image 600 and register the segmented vertebral body 404 from the image 500 with the image 600. By way of example, Figure 7 is an example diagram of the segmented vertebral body 404 from the image 500 to the image 600.

[0087] In one example, the processing device 122 is configured to determine the position of the surgical implant 604 based on the image 600 and a three-dimensional geometric model of the surgical implant 604. In one example, the processing device 122 is configured to overlay a virtual representation of the surgical implant 604 on the registered and segmented vertebral body 404. For example, Figure 8 is an example diagram of an image 800 that includes the registered and segmented vertebral body 404 together with the image 600, in addition to the virtual representation 804 of the surgical implant 604. In this example, the user can benefit from checking the positioning of the surgical implant 604 based on the image 400 of the initial scan of the surgical site 402 at a higher radiation level than the second scan of the surgical site, in order to have a better representation of the surgical site while using less radiation during the surgical procedure.

[0088] Figure 9It is a flowchart of an example method for three-dimensional visualization during a surgical procedure according to at least one embodiment described herein. Although the blocks in the various figures are illustrated in sequential order, in some cases, these blocks may be executed in parallel and / or in a different order than that described therein. Moreover, various blocks may be combined into fewer blocks, divided into additional blocks, and / or removed based on the desired implementation.

[0089] As shown in block 902, the method 900 includes the following steps: segmenting at least one vertebral body from at least one image in a first three-dimensional image dataset, wherein the first three-dimensional image dataset is based on an initial scan of a surgical site including a spinal anatomy. In one example, the first three-dimensional image dataset includes one or more of fluoroscopic images or computed tomography images. In one example, a C-arm 103 is used to capture an initial three-dimensional scan of the surgical site, as described above. By way of example, the surgical site includes a patient's spinal anatomy. In this example, the processing device 122 is configured to receive a three-dimensional image dataset corresponding to the surgical site based on the captured initial three-dimensional scan. Continuing with this example, the processing device 122 is configured to segment at least one vertebral body from the received three-dimensional image dataset.

[0090] In one scenario, the processing device 122 is configured to apply a fully convolutional network with a residual neural network to the received one or more images and segment one or more vertebral bodies from the one or more images of the spinal anatomy. In one example, given the segmented vertebral bodies, the vertebral segments are semi-automatically labeled. For example, the user indicates the segment as the L5 vertebra on the segmented image. In this example, a three-dimensional connected component extraction algorithm is applied to label different vertebral regions. In one example, isolated components smaller than a predetermined threshold are removed. In this example, a discrete marching cubes algorithm is applied to each component, followed by mesh smoothing using a windowed sampling function applied in the frequency domain. This is implemented as an interpolation kernel executed on each voxel. Additionally, the remaining vertebrae are labeled in sequential order based on what segment the user defines as the bottommost vertebra. In one example, the processing device 122 is configured to segment the at least one vertebral body from the at least one image in the first three-dimensional image dataset according to a deep neural network.

[0091] The three-dimensional image dataset can also be used as a basis for planning a surgical procedure using manual or automated planning software. For example, a plan for placing pedicle screws can be exported from the planning tools included in the planning software used. Such planning software provides the surgeon with an understanding of the patient's anatomical orientation, the appropriate sizing of surgical instruments and implants, and the correct trajectory of the implants. According to some implementations, the system provides a plan for pedicle screws, whereby the system identifies, in a surgical plan for a given patient's anatomy and measurements, not only the diameter and length of each pedicle screw but also the desired trajectory. In one example, one or more images from the three-dimensional image dataset can be displayed on a display (e.g., display device 126), and a representation of the plan for placing pedicle screws can be overlaid on the one or more displayed images.

[0092] In one scenario, the surgeon can proceed with inserting one or more implants based on the surgical plan. The surgeon can use a handheld tool or a robotic guidance device (e.g., robotic device 140, robotic device 200) to insert the one or more implants. In such a scenario, the tracking device 130 is configured to determine position data of the handheld tool or the robotic device as the one or more implants are inserted. The tracking device 130 is configured to provide the position data of the handheld tool or the robotic device to the processing device 122 for further processing.

[0093] As shown in block 904, method 900 further includes the step of receiving at least one image from a second three-dimensional image dataset, where the second three-dimensional image dataset is based on a second scan of the surgical site after an initial scan of the surgical site, and where the second scan of the surgical site includes spinal anatomy and at least one surgical implant. In one example, the second three-dimensional image dataset includes one or more of fluoroscopic images or computed tomography images. In one example, the second scan includes a plurality of surgical implants associated with one vertebral body from the at least one image from the second three-dimensional image dataset. In one example, once one or more of the implants are inserted, the C-arm 103 is configured to capture a subsequent three-dimensional scan of the surgical site. In one example, the processing device 122 is configured to receive a three-dimensional image dataset corresponding to the subsequent three-dimensional scan of the surgical site. In one example, the processing device 122 is configured to segment at least one vertebral body from the three-dimensional image dataset corresponding to the subsequent three-dimensional scan of the surgical site, as described above.

[0094] As shown in block 906, method 900 further includes the steps of registering at least one segmented vertebral body from the at least one image in the first three-dimensional image dataset with the at least one image in the second three-dimensional image dataset. In one scenario, processing device 122 is configured to compare one or more images corresponding to a subsequent three-dimensional scan with one or more images corresponding to the segmented vertebral bodies from an initial three-dimensional scan in order to obtain image registration. In one example, processing device 122 is configured to register at least one segmented vertebral body from the at least one image in the first three-dimensional image dataset with the at least one image in the second three-dimensional image dataset, which includes performing one or more of the following operations on the image information contained within the one vertebral body from the at least one image in the first three-dimensional image dataset: horizontal translation, vertical translation, rotation, and scaling. In another example, a subsequent three-dimensional scan is compared with the position determined from tracking device 130. For example, when both the instrument / implant and the C-arm are tracked, the position of the anatomical structure relative to the imaging source and the position of the device relative to the imaging source are known. Thus, this information can be used to quickly and interactively ascertain the position of the device or hardware relative to the anatomical structure. In one example, if the position of one or more vertebral bodies is adjusted during a surgical procedure, processing device 122 is configured to create a new virtual scan of the segmented vertebral bodies based at least on the current position of the one or more vertebral bodies.

[0095] In one example, processing device 122 is configured to find the best fit of the segmented vertebral bodies from the initial scan with one or more images from a subsequent scan according to a predetermined correlation score. A variety of algorithms can be employed, which can depend on various factors, such as the number of images in each of the respective image datasets within the image dataset, the size and speed of the computer processor or graphics processor performing the algorithm calculations, the time allotted to perform the calculations, and the size of the images being compared. In one method, comparisons are made between pixels at the aforementioned predetermined positions in a grid pattern throughout a 4D space. In one method, comparisons are made between pixels at the aforementioned predetermined positions in a grid pattern throughout a 6D space. In another heuristic method, pixel comparisons can be concentrated in image regions that are considered to offer a greater likelihood of a relevant match. These regions can be "pre-seeded" based on knowledge from a grid or PCA search, data from a tracking system (e.g., tracking device 130), or position data from a DICOM file or equivalent file. In one example, processing device 122 can be configured to use the captured navigation position (from which the final screw is placed) and the approximate position in the metal artifact scan to simplify the search criteria of the algorithm.

[0096] As shown in block 908, the method 900 further includes the following steps: determining the position of the at least one surgical implant based on the at least one image in the second three-dimensional image dataset and the three-dimensional geometric model of the at least one surgical implant. In one example, the processing device 122 is configured to use the three-dimensional geometric model of the implant to identify the best-correlated final position of each implant in one or more images corresponding to a subsequent scan. The three-dimensional geometric model may include a plurality of measurements of the implant that can be used by the processing device 122 for identification. For example, the processing device 122 may use the length and diameter of the pedicle screw to identify the best-correlated final position of each pedicle screw in one or more images corresponding to a subsequent scan.

[0097] As shown in block 910, the method 900 further includes the following steps: covering a virtual representation of the at least one surgical implant on the at least one registered and segmented vertebral body from the at least one image in the first three-dimensional image dataset based on the determined position of the at least one surgical implant. In one example, the processing device 122 is configured to overlay one or more three-dimensional geometric models on one or more images corresponding to the segmented vertebral bodies from the initial three-dimensional scan. In this example, the processing device 122 is configured to use i) the best fit of the segmented vertebral bodies from the initial scan to one or more images from the subsequent scan according to a predetermined correlation score, and ii) the best-correlated final position of each implant in the one or more images corresponding to the subsequent scan, as described above, to display an image of a given segmented vertebral body with a virtual representation of the implant overlaid. Continuing with this example, the displayed image corresponds to an image of the inserted implant according to a subsequent three-dimensional scan captured via the C-arm 103. In one example, the processing device 122 is configured to display a three-dimensional image of the inserted implant from the subsequent three-dimensional scan and a three-dimensional image of the initial three-dimensional scan that also includes the overlaid virtual representation of the implant, so that the surgeon can determine whether the final implant placement meets any remaining steps for continuing the surgery.

[0098] Figure 10 is a flowchart of an example method for three-dimensional visualization during surgery according to at least one embodiment described herein. Although the blocks in the various figures are illustrated in sequential order, in some cases, these blocks may be executed in parallel and / or in a different order than that described. Also, various blocks may be combined into fewer blocks, divided into additional blocks, and / or removed based on the desired implementation.

[0099] As shown in block 1002, the method 1000 includes the following steps: segmenting the images in the first three-dimensional image dataset, where the first three-dimensional image dataset is based on an initial scan of a surgical site including a spinal anatomy. In one example, the images in the first three-dimensional image dataset are segmented based on user input. In another example, the user input includes input received via a touchscreen display. For example, the display device 126 may include a touchscreen display as display units 123 and 124.

[0100] As shown in block 1004, the method 1000 further includes the following steps: receiving the images in the second three-dimensional image dataset, where the second three-dimensional image dataset is based on a second scan of the surgical site after the initial scan of the surgical site, and where the second scan of the surgical site includes the spinal anatomy and at least one surgical implant. In one example, the first three-dimensional image dataset and the second three-dimensional image dataset include one or more of fluoroscopic images or computed tomography images. In another example, the first three-dimensional image dataset and the second three-dimensional image dataset include fluoroscopic images, and the initial scan of the surgical site is associated with a higher radiation level than the second scan of the surgical site. In one example, the second scan includes a plurality of surgical implants associated with one vertebral body from the at least one image in the second three-dimensional image dataset.

[0101] As shown in block 1006, the method 1000 further includes the following steps: registering the segmented images in the first three-dimensional image dataset with the segmented images in the second three-dimensional image dataset. In one example, the step of registering the segmented images in the first three-dimensional image dataset with the images in the second three-dimensional image dataset includes performing one or more of the following operations on the image information contained within one vertebral body from the at least one image in the first three-dimensional image dataset: horizontal translation, vertical translation, rotation, and scaling.

[0102] As shown in block 1008, the method 1000 further includes the following steps: determining the position of the at least one surgical implant based on the at least one image in the second three-dimensional image dataset and a three-dimensional geometric model of the at least one surgical implant.

[0103] As shown in block 1010, the method 1000 further includes the following steps: overlaying a virtual representation of the at least one surgical implant on the registered and segmented images in the first three-dimensional image dataset based on the determined position of the at least one surgical implant.

[0104] Figure 9 and Figure 10The flowchart shows two possible functions and operations of this embodiment. In this regard, each box may represent a module, a section, or a part of program code, which includes one or more instructions executable by a processor to implement specific logical functions or steps in the processing. The program code can be stored on any type of computer-readable medium, for example, a storage device such as a disk or a hard disk drive. A computer-readable medium can, for example, include a non-transitory computer-readable medium (such as register memory, processor cache, or random access memory (RAM)) that stores data for a relatively short period, and / or a persistent long-term storage device (such as read-only memory (ROM), an optical disk or a magnetic disk, or a compact disc read-only memory (CD-ROM)). A computer-readable medium can or may include any other volatile or non-volatile storage system. A computer-readable medium can, for example, be considered a computer-readable storage medium, a tangible storage device, or other manufactured article.

[0105] Alternatively, Figure 9 and Figure 10 each box in can represent a circuit wired to perform a specific logical function in the processing. Exemplary methods (such as the methods shown in Figure 9 and Figure 10 ) can be performed in whole or in part by one or more components in the cloud. However, it should be understood that, without departing from the scope of the present invention, the example methods can instead be performed by other entities or combinations of entities (i.e., by other combinations of computing devices and / or computer devices). For example, Figure 9 and Figure 10 the functions of the method can be performed entirely by a computing device (or components of a computing device, such as one or more processors), or can be distributed across multiple components of a computing device, across multiple computing devices, and / or across servers.

[0106] Figure 11 Depicts an example computer-readable medium configured according to an example embodiment. In the example embodiment, the example system can include: one or more processors, one or more forms of memory, one or more input devices / interfaces, one or more output devices / interfaces, and machine-readable instructions that, when executed by the one or more processors, cause the system to perform the various functions, tasks, capabilities, etc. described above.

[0107] As mentioned above, in some embodiments, the disclosed technologies (e.g., the functions of the robot device 140, the robot device 200, the processing device 122, the computing device 300, etc.) can be implemented by computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other media or articles. Figure 11FIG. is a schematic diagram conceptually showing a partial view of an example computer program product configured according to at least some embodiments disclosed herein. The example computer program product includes a computer program for performing computer processing on a computing device.

[0108] In one embodiment, an example computer program product 1100 is provided using a signal-bearing medium 1102. The signal-bearing medium 1102 may include one or more programming instructions 1104 that, when executed by one or more processors, may provide the functionality or a portion of the functionality described above with reference to Figures 1 to 10 In some examples, the signal-bearing medium 1102 may be a computer-readable medium 1106, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video disc (DVD), a digital tape, a memory, etc. In some implementations, the signal-bearing medium 1102 may be a computer-recordable medium 1108, such as, but not limited to, a memory, a read / write (R / W) CD, an R / W DVD, etc. In some implementations, the signal-bearing medium 1102 may be a communication medium 1110 (e.g., an optical fiber cable, a waveguide, a wired communication link, etc.). Thus, for example, the signal-bearing medium 1102 may be embodied by a communication medium 1110 in a wireless form.

[0109] The one or more programming instructions 1104 may be, for example, computer-executable and / or logic-implemented instructions. In some examples, a computing device may be configured to provide various operations, functions, or actions in response to the programming instructions 1104 conveyed to the computing device by one or more of the computer-readable medium 1106, the computer-recordable medium 1108, and / or the communication medium 1110.

[0110] The computer-readable medium 1106 may also be distributed among a plurality of data storage elements that may be remotely located from one another. Some or all of the computing devices that execute the stored instructions may be external computers, or mobile computing platforms (such as smartphones, tablet devices, personal computers, wearable devices, etc.). Alternatively, some or all of the computing devices that execute the stored instructions may be remotely located computer systems (such as servers).

[0111] It should be understood that the arrangements described herein are for illustrative purposes only. Thus, those skilled in the art will recognize that other arrangements and other elements (e.g., machines, interfaces, functions, orders, and groupings of functions, etc.) may be used instead, and some elements may be omitted altogether according to the desired results. In addition, many of the elements described may be implemented as functional entities that may be combined in any suitable combination and location as discrete or distributed components or in combination with other components, or other structural elements described as separate structures may be combined.

Claims

1. A method for generating a display of an image of a patient's internal anatomy during a surgical procedure, the method comprising the steps of: Segmenting at least one vertebral body from at least one image in a first three-dimensional image dataset, wherein the first three-dimensional image dataset is based on an initial scan of a surgical site including spinal anatomy; Receiving at least one image in a second three-dimensional image dataset, wherein the second three-dimensional image dataset is based on a second scan of the surgical site after the initial scan of the surgical site, and wherein the second scan of the surgical site includes the spinal anatomy and at least one surgical implant; Registering the segmented at least one vertebral body from the at least one image in the first three-dimensional image dataset with the at least one image in the second three-dimensional image dataset; Determining the position of the at least one surgical implant based on the at least one image in the second three-dimensional image dataset and a three-dimensional geometric model of the at least one surgical implant; and Overlaying a virtual representation of the at least one surgical implant on the registered and segmented at least one vertebral body from the at least one image in the first three-dimensional image dataset based on the determined position of the at least one surgical implant.

2. The method according to claim 1, wherein, The first three-dimensional image dataset and the second three-dimensional image dataset include one or more of fluoroscopic images or computed tomography images.

3. The method according to claim 2, wherein, The first three-dimensional image dataset and the second three-dimensional image dataset include fluoroscopic images, wherein the initial scan of the surgical site is associated with a higher radiation level than the second scan of the surgical site.

4. The method according to claim 1, wherein, The second scan includes a plurality of surgical implants associated with a vertebral body from the at least one image in the first three-dimensional image dataset.

5. The method according to claim 1, wherein The step of segmenting at least one vertebral body from at least one image in the first three-dimensional image dataset is determined according to a deep neural network.

6. The method according to claim 1, wherein The step of registering the segmented at least one vertebral body from the at least one image in the first three-dimensional image dataset with the at least one image in the second three-dimensional image dataset includes performing one or more of the following operations on the image information contained within a vertebral body from the at least one image in the first three-dimensional image dataset: horizontal translation, vertical translation, rotation, and scaling.

7. A method for generating a display of an image of a patient's internal anatomy during a surgical procedure, the method comprising the steps of: Segmenting an image in a first three-dimensional image dataset, wherein the first three-dimensional image dataset is based on an initial scan of a surgical site including spinal anatomy; Receiving an image in a second three-dimensional image dataset, wherein the second three-dimensional image dataset is based on a second scan of the surgical site after the initial scan of the surgical site, and wherein the second scan of the surgical site includes the spinal anatomy and at least one surgical implant; Registering the segmented image in the first three-dimensional image dataset with the segmented image in the second three-dimensional image dataset; Determine the position of the at least one surgical implant based on at least one image in the second three-dimensional image dataset and the three-dimensional geometric model of the at least one surgical implant; and Overlay a virtual representation of the at least one surgical implant on the registered and segmented images in the first three-dimensional image dataset based on the determined position of the at least one surgical implant.

8. The method according to claim 7, wherein The first three-dimensional image dataset and the second three-dimensional image dataset include one or more of fluoroscopic images or computed tomography images.

9. The method according to claim 8, wherein The first three-dimensional image dataset and the second three-dimensional image dataset include fluoroscopic images, wherein the initial scan of the surgical site is associated with a higher radiation level than the second scan of the surgical site.

10. The method according to claim 7, wherein The second scan includes a plurality of surgical implants associated with a vertebral body from the at least one image in the first three-dimensional image dataset.

11. The method according to claim 7, wherein, The step of segmenting the images in the first three-dimensional image dataset is performed based on user input.

12. The method according to claim 11, wherein, The user input includes an input received via a touch screen display.

13. The method according to claim 7, wherein The step of registering the segmented images in the first three-dimensional image dataset with the images in the second three-dimensional image dataset includes performing one or more of the following operations on the image information contained within a vertebral body from the at least one image in the first three-dimensional image dataset: horizontal translation, vertical translation, rotation, and scaling.

14. A system for generating a display of an image of a patient's internal anatomy during a surgical procedure, the system comprising: A display; And A processor communicatively coupled to the display, the processor configured to: Segment images in a first three-dimensional image dataset, wherein the first three-dimensional image dataset is based on an initial scan of a surgical site including spinal anatomy; Receive images in a second three-dimensional image dataset, wherein the second three-dimensional image dataset is based on a second scan of the surgical site after the initial scan of the surgical site, wherein the second scan of the surgical site includes the spinal anatomy and at least one surgical implant; Register the segmented images in the first three-dimensional image dataset with the images in the second three-dimensional image dataset; Determine the position of the at least one surgical implant based on the images in the second three-dimensional image dataset and the three-dimensional geometric model of the at least one surgical implant; Overlay a virtual representation of the at least one surgical implant on the registered and segmented images in the first three-dimensional image dataset based on the determined position of the at least one surgical implant; and Provide instructions to display the virtual representation of the at least one surgical implant as an overlay on the registered and segmented images in the first three-dimensional image dataset via the display.

15. The system according to claim 14, wherein The first three-dimensional image dataset and the second three-dimensional image dataset include one or more of fluoroscopic images or computed tomography images.

16. The system according to claim 15, wherein, The first three-dimensional image dataset and the second three-dimensional image dataset include fluoroscopic images, wherein an initial scan of the surgical site is associated with a higher radiation level than a second scan of the surgical site.

17. The system according to claim 14, wherein The second scan includes a plurality of surgical implants associated with a vertebral body from at least one image in the first three-dimensional image dataset.

18. The system according to claim 14, wherein The step of segmenting an image in the first three-dimensional image dataset is performed based on user input via the display.

19. The system according to claim 14, wherein, The step of segmenting an image in the first three-dimensional image dataset is performed based on input received via a touchscreen display.

20. The system according to claim 14, wherein The step of registering the segmented image in the first three-dimensional image dataset with the image in the second three-dimensional image dataset includes performing one or more of the following operations on the image information contained within a vertebral body from the at least one image in the first three-dimensional image dataset: horizontal translation, vertical translation, rotation, and scaling.

Images