Motion programming of robotic devices

Abstract

A method includes acquiring an image of a surgical site of a patient. The method includes the steps of capturing motion data of an instrument based on a user-defined path to the surgical site. The method includes the steps of determining a motion path based on the captured motion data. The motion path corresponds to actuation of one or more components of a robotic device. The method includes the steps of displaying a portion of the motion path on the image. The method includes the steps of determining one or more instructions for actuating the one or more components along the determined motion path. The method includes the steps of providing the one or more instructions to a robotic device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. Provisional Application No. 62 / 794,545, filed on January 18, 2019, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] This disclosure describes programming the motion of a robotic device based on tracked surgical instruments. Background Art

[0004] Minimally invasive surgery usually limits the size of the incision in the human body to allow for quick recovery from surgery and reduce the chance of infection. However, during minimally invasive surgery, only a few tools may be used by the same surgeon at the same time. Tool changes may sometimes occur to place the appropriate tools in the surgical kit. Tool changes may exceed the time of minimally invasive surgery. Moreover, minimally invasive surgery may be burdensome to the surgeon, especially when manually operating the surgical controller for a long time. Summary of the invention

[0005] In one embodiment, a method is provided, the method comprising the steps of acquiring an image of a surgical site of a patient. The method further comprises the steps of capturing motion data of an instrument based on a user-defined path to the surgical site. The method further comprises the steps of determining a motion path based on the captured motion data. The motion path corresponds to actuation of one or more components of a robotic device. The method further comprises the steps of displaying a portion of the motion path onto the image. The method further comprises the steps of determining one or more instructions for actuating the one or more components along the determined motion path. The method further comprises the steps of providing the one or more instructions to the robotic device.

[0006] In another embodiment, a method is provided, the method comprising the steps of acquiring an image of a surgical site of a patient. The method further comprises the steps of capturing a posture of an instrument based on a user-defined arrangement of the instrument at the surgical site of the patient. The method further comprises the steps of determining a motion path corresponding to the captured posture of the instrument. The motion path is associated with actuation of one or more components of a robotic device. The method further comprises the steps of displaying a portion of the motion path onto a merged image. The method further comprises the steps of determining one or more instructions for actuating the one or more components along the determined motion path. The method further comprises the steps of providing the one or more instructions to a robotic device.

[0007] In another embodiment, a system is provided, comprising: a tracking device, a robotic device, and a processing device. The processing device comprises a processor and a non-transitory computer-readable medium. The non-transitory computer-readable medium comprises stored instructions, which, when executed by the processor, cause the system to acquire an image of a surgical site of a patient. The stored instructions also cause the system to capture a posture of an instrument based on a user-defined arrangement of the instrument at the surgical site of the patient via the tracking device. The stored instructions also cause the system to determine, by the processor, a motion path corresponding to the captured posture of the instrument. The motion path is associated with actuation of one or more components of the robotic device. The stored instructions also cause the system to provide instructions for displaying a portion of the motion path on the image. The stored instructions also cause the system to determine, by the processor, one or more instructions for actuating the one or more components along the determined motion path. The stored instructions also cause the system to provide the one or more instructions to the robotic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Numerous advantages of the present invention will become apparent to those skilled in the art upon reading this specification in conjunction with the accompanying drawings, wherein like reference numerals are applied to like elements, and wherein:

[0009] Figure 1 An example system for performing a surgical procedure according to an embodiment of the present disclosure is illustrated;

[0010] Figure 2 illustrates an example robotic device that may be used during a surgical procedure in accordance with an embodiment of the present disclosure;

[0011] Figure 3 An example block diagram illustrating a computing device according to an embodiment of the present disclosure is illustrated;

[0012] Figure 4 An example computer medium according to an embodiment of the present disclosure is illustrated;

[0013] Figure 5 A flow chart illustrating an example method during a surgical procedure according to an embodiment of the present disclosure;

[0014] Figure 6 Another flow chart illustrating an example method during a surgical procedure according to an embodiment of the present disclosure;

[0015] Figure 7 illustrates example images that may be displayed during a surgical procedure in accordance with an embodiment of the present disclosure;

[0016] Figure 8illustrates another example image that may be displayed during a surgical procedure in accordance with an embodiment of the present disclosure;

[0017] Fig. 9 Another flow chart illustrating an example method during a surgical procedure according to an embodiment of the present disclosure;

[0018] Fig.10 Another flow chart illustrating an example method during a surgical procedure according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] Below, exemplary embodiments of the present invention are described. For the sake of clarity, not all features of the actual implementation are described in this specification. Of course, it should be appreciated that in the development of any such actual implementation, many specific implementation decisions must be made to achieve the developer's specific goals (such as compliance with system-related constraints and business-related constraints), which will change from one implementation to another. In addition, it should be appreciated that such development work may be complex and time-consuming, but for those of ordinary skill in the art who benefit from this disclosure, it is only a routine task. Moreover, it should be easily understood that, although discussed primarily in the context of spinal surgery below, the systems and methods of the present invention can be used in any number of anatomical environments to provide access to any number of different surgical target sites throughout the body.

[0020] Examples described herein include systems and methods for performing surgical procedures. In one example, a method is provided, the method comprising the steps of acquiring an image of a surgical site of a patient. In another example, a method is provided, the method comprising the steps of acquiring a high-resolution baseline image of a surgical site of a patient. In one example, the high-resolution baseline image is an X-ray image captured via a mobile fluoroscopy machine. Continuing with this example, the processing device is configured to digitally manipulate the high-resolution baseline image to generate a baseline image set based on adjusting one or more aspects of the high-resolution baseline image. The method also includes the steps of acquiring a new image of the surgical site at a lower resolution than the high-resolution baseline image.

[0021] In one example, the lower resolution image is an X-ray image captured at a radiation level lower than the radiation level associated with the high resolution baseline image. Due to the image quality of the lower resolution image, the processing device is configured to find a match between the lower resolution image and an image from the high resolution baseline image set by comparing the lower resolution image to one or more images of the baseline image set. The processing device is configured to select a representative image of the baseline image set that has an acceptable correlation with the lower resolution image. The processing device is also configured to merge the selected representative image with the lower resolution image to produce a merged image. In one embodiment, the merged image is displayed on a display for a user (e.g., a surgeon) to evaluate one or more aspects of the surgical procedure.

[0022] The method further includes the steps of capturing motion data of an instrument used by a surgeon based on a user-defined path to a surgical site. The processing device is configured to determine the motion path based on the captured motion data. The motion path corresponds to actuation of one or more components of a robotic device that will assist the surgeon in performing the surgical procedure. The method further includes the steps of displaying a portion of the motion path onto a merged image for review by the surgeon.

[0023] For example, in a surgical procedure involving the use of one or more pedicle screws, a surgeon may position an instrument at a pedicle target site and view a motion path corresponding to the insertion of the pedicle screw based on the captured motion data of the instrument. In this example, the surgeon may select a point on the pedicle target site and pivot the instrument to understand one or more different tracks superimposed on the merged image. Based on the instrument pivoted around the point, one or more tracking devices are configured to capture motion data associated with the movement of the instrument. Then, as described herein, the motion data is processed accordingly and at least a portion of the motion path is superimposed on the merged image. The surgeon may provide an input (e.g., voice command, access interface, etc.) to the processing device regarding whether the motion path is aligned with the preoperative plan. Based on the approval of the motion path, the method comprises the following steps: determining one or more instructions for actuating the one or more components of the robotic device along the determined motion path. The method further comprises the following steps: providing the one or more instructions to the robotic device.

[0024] Referring now to the accompanying drawings, Figure 1is a diagram of an example system 100 for performing a surgical procedure. The example system 100 includes a base unit 102 supporting a C-arm (C-Arm) imaging device 103. The C-arm 103 includes a radiation source 104 that is located below a patient P and directs a radiation beam upward to a receiver 105. The receiver 105 of the C-arm 103 sends image data to a processing device 122. The processing device 122 can communicate with a tracking device 130 to obtain position and orientation information of various instruments T and H used during the surgical procedure. The tracking device 130 can communicate with a robotic device 140 to provide position information of various tracking components (such as markers 150). The robotic device 140 and the processing device 122 can communicate via one or more communication channels.

[0025] The base unit 102 includes a control panel 110 through which a user can control the position of the C-arm 103, as well as radiation exposure. Thus, the control panel 110 enables the radiographer to acquire images of the surgical site at the direction of the surgeon, control the radiation dose, and initiate radiation pulse images.

[0026] For different viewing angles of the surgical site, the C-arm 103 can be rotated around the patient P in the direction of arrow 108. In some cases, implants or instruments T and H may be located at the surgical site, and the viewing angle needs to be changed to obtain an unobstructed view of the site. Therefore, the position of the receiver relative to the patient P and more particularly the surgical site of interest may change during surgery according to the needs of the surgeon or radiologist. Thus, the receiver 105 may include a tracking target 106 mounted to the receiver 105, which enables the position of the C-arm 103 to be tracked using the tracking device 130. By way of example only, the tracking target 106 may include a plurality of infrared (IR) reflectors or emitters spaced around the target, and the tracking device 130 is configured to triangulate the position of the receiver 105 based on IR signals reflected or emitted by the tracking target 106.

[0027] The processing device 122 may include a digital memory associated with the processing device 122 and a processor for executing digital and software instructions. The processing device 122 may also incorporate a frame grabber using a frame grabber technique to create digital images for projection as displays 123 and 124 on a display device 126. Displays 123 and 124 are positioned for interactive viewing by surgeons during surgery. The two displays 123 and 124 may be used to show images from two fields of view (such as lateral and A / P), or may show a baseline scan and a current scan of the surgical site, or a current scan and a "merged" scan based on a previous baseline scan and a low radiation current scan, as described herein. An input device 125 (such as a keyboard or touch screen) may allow the surgeon to select and manipulate images on the screen. It should be understood that the input device may incorporate an array of keys or touch screen icons corresponding to various tasks and features implemented by the processing device 122. The processing device 122 includes a processor that converts image data obtained from the receiver 105 into a digital format. In some cases, the C-arm 103 may be operated in a movie exposure mode and generate many images per second. In these cases, multiple images may be averaged together into a single image over a short period of time to reduce motion artifacts and noise.

[0028] Tracking device 130 includes sensors 131 and 132 for determining position data associated with various components (e.g., infrared reflectors or emitters) used in a surgical procedure. In one example, sensors 131 and 132 may be charge coupled device (CCD) image sensors. In another example, sensors 131 and 132 may be complementary metal oxide semiconductor (CMOS) image sensors. It is also contemplated that a different number of other image sensors may be used to implement the described functionality.

[0029] In one aspect of the invention, the robotic device 140 can help maintain the instrument T relative to the patient P during a surgical procedure. In one scenario, the robotic device 140 can be configured to maintain the instrument T relative to the patient P as the patient P moves (e.g., due to breathing) or is moved during a surgical procedure (e.g., due to manipulation of the patient's body).

[0030] The robotic device 140 may include a robotic arm 141, a pedal 142, and a moving housing 143. The robotic device 140 may also communicate with a display such as the display 126. The robotic device 140 may also include a fixing device for fixing the robotic device 140 to the operating table.

[0031] In one example, in a first tracking mode, the tracking device 130 is configured to capture motion data of a handheld instrument based on a user-defined path at a surgical target site of a patient. In this example, the processing device 122 determines a motion path corresponding to the captured motion data. In addition, the processing device 122 determines one or more instructions for actuating the one or more components of the robotic device 140 along the determined motion path and provides the one or more instructions to the robotic device 140. In a second tracking mode, the tracking device 130 is configured to capture a pose (i.e., orientation and position) of a handheld instrument based on a user-defined arrangement of the instrument at a surgical site of a patient. In this example, the processing device 122 determines a motion path corresponding to the captured pose of the instrument. As described with the first tracking mode, the processing device 122 determines one or more instructions for actuating the one or more components of the robotic device 140 along the determined motion path and provides the one or more instructions to the robotic device 140.

[0032] In one example, the user may control the actuation of the robotic arm 141 by using the robotic pedal 142. In one embodiment, the user may depress the robotic pedal 142 to enable one or more modes of the robotic device 140. In one scenario, the user may depress the robotic pedal 142 to enable the user to manually position the robotic arm 141 according to a desired position. In another scenario, the user may depress the robotic pedal 142 to enable a mode that enables the robotic arm 141 to place the instrument T into a position according to the motion path determined as above. In another scenario, the user may depress the robotic pedal 142 to stop the robotic arm 141 from making any further movement.

[0033] The robotic arm 141 can be configured to receive one or more end effectors depending on the number of surgical procedures and associated joints. In one example, the robotic arm 141 can be a six-joint arm. In this example, each joint includes an encoder that measures the angle value of the joint. The movement data provided by the one or more encoders combined with the known geometry of the six joints can enable the position of the robotic arm 141 and the position of the instrument T connected to the robotic arm 141 to be determined. It is also contemplated that different numbers of joints can be used to implement the functions described herein.

[0034] The mobile housing 143 ensures easy maneuverability of the robotic device 140 by using wheels or handles or both. In one embodiment, the mobile housing 143 may include a fixed pad or equivalent device. The mobile housing 143 may also include a control unit that provides one or more commands to the robotic arm 141 and may enable the surgeon to manually input data by using an interface such as a touch screen, a mouse, a joystick, a keyboard or the like.

[0035] In one example, the processing device 122 is configured to capture the pose of an instrument H (e.g., a portable instrument) via the tracking device 130. The captured pose of the instrument includes a combination of position information and orientation information. In this example, the pose of the instrument H is based on a user-defined arrangement at a surgical site of a patient P. The user-defined arrangement is based on the surgeon's movement of the instrument H. In one scenario, the portable instrument includes one or more infrared reflectors or emitters. Continuing with this example, the processing device 122 is configured to determine a motion path corresponding to the captured pose of the instrument H. The motion path is associated with the actuation of one or more components (e.g., one or more links and joints) of the robotic device 140. The processing device 122 is configured to determine one or more instructions for actuating the one or more components of the robotic device 140 along the determined motion path. In addition, the processing device 122 is configured to provide the one or more instructions to the robotic device 140.

[0036] In another example, the processing device 122 is configured to compare the position of the one or more infrared reflectors of the portable instrument with the position of the one or more infrared reflectors coupled to the patient. Based on the comparison, the processor 122 is configured to determine whether the distance between the one or more infrared reflectors of the portable instrument and the one or more infrared reflectors coupled to the patient is within a safety threshold. Based on the determination of the distance between the portable instrument and the patient, the processing device 122 is configured to determine one or more adjustments to the actuation of the one or more components of the robotic device.

[0037] In another example, the processing device 122 is configured to provide an image of the surgical site for display on the display device 126. In this example, the processing device 122 is configured to superimpose at least a portion of the determined motion path onto the image of the surgical site. Superimposing at least a portion of the determined path will enable the surgeon to recheck the path and ensure that the path is consistent with the surgical plan. In one example, the processing device 122 can also be configured to receive input from the surgeon, which confirms that the superimposed portion of the determined motion path is aligned with the surgical plan. In one example, the input is received via the input device 125.

[0038] In another example, the processing device 122 is configured to determine an angle between the preoperative trajectory and the trajectory corresponding to the portion of the motion path. Based on the determined angle, the processing device 122 is configured to determine one or more movements to pivot the instrument based on the captured position of the instrument H so as to align the trajectory corresponding to the portion of the motion path with the preoperative trajectory. In this example, the processing device 122 is configured to provide the one or more movements to pivot the instrument to the robotic device 140. As described herein, the robotic device 140 is configured to convert the one or more movements for pivoting into instructions for enabling the robotic device 140 to move along the determined trajectory.

[0039] In another example, the determined angle between the preoperative trajectory and the trajectory corresponding to the portion of the motion path can be compared to one or more ranges associated with one or more scores. Based on the scores, a changing visual effect (e.g., a flashing color) can be displayed with the trajectory corresponding to the portion of the motion path. For example, if the angle is within a range associated with a higher likelihood of damaging the periphery of the pedicle, the trajectory corresponding to the portion of the motion path is displayed in flashing red. In another example, if the angle is within a range corresponding to a high degree of correlation with the preoperative trajectory, the trajectory corresponding to the portion of the motion path is displayed in green.

[0040] In another example, the processing device 122 is configured to determine that the superimposed portion of the determined motion path intersects one or more predetermined boundaries within the surgical site. In this example, the processing device 122 is configured to provide a visual effect of a change of at least a portion of the superimposed portion of the determined motion path displayed via the display device 126.

[0041] Figure 2 An example robotic device 200 that can be used during a surgical procedure is illustrated. The robotic device 200 can include hardware, such as a processor, memory or storage, and sensors that enable the robotic device 200 to be used in a surgical procedure. The robotic device 200 can be powered by various devices, such as electric motors, pneumatic motors, hydraulic motors, etc. The robotic device 200 includes a base 202, links 206, 210, 214, 218, 222, and 226, joints 204, 208, 212, 216, 220, 224, and 230, and a manipulator 228.

[0042] The base 202 may provide a platform to provide support for the robotic device 200. The base 202 may be stationary or coupled to wheels to provide movement of the robotic device 200. The base 202 may include any number of materials, such as aluminum, steel, stainless steel, etc., that may be suitable for a given environment associated with the robotic device 200.

[0043] The links 206, 210, 214, 218, 222, and 226 may be configured to move according to a programmable set of instructions. For example, the links may be configured to follow a predetermined set of movements (e.g., a motion path corresponding to the captured posture of the instrument) to complete a task under the supervision of a user. For example, the links 206, 210, 214, 218, 222, and 226 may form a kinematic chain that defines the relative movement of a given link in the links 206, 210, 214, 218, 222, and 226 at a given joint in the joints 204, 208, 212, 216, 220, 224, and 230.

[0044] The joints 204, 208, 212, 216, 220, 224, and 230 may be configured to rotate using a mechanical gear system. In one example, the mechanical gear system is driven by a strain wave gear drive, a cycloidal drive, or the like. The mechanical gear system selected will depend on many factors related to the operation of the robotic device 200, such as the length of a given link of the links 206, 210, 214, 218, 222, and 226, the rotational speed, the desired gear reduction, and the like. Providing power to the joints 204, 208, 212, 216, 220, 224, and 230 will enable the links 206, 210, 214, 218, 222, and 226 to move in a manner that allows the manipulator 228 to interact with the environment.

[0045] In one example, the manipulator 228 is configured to enable the robotic device 200 to interact with the environment according to one or more constraints. In one example, the manipulator 228 performs various operations (such as grasping surgical instruments) to perform appropriate placement of parts. For example, the manipulator 228 can be replaced with another end effector that will provide different functions for the robotic device 200.

[0046] In one example, the robotic device 200 is configured to operate according to a robotic operating system (e.g., an operating system designed for the specific functions of a robot). The robotic operating system may provide libraries and tools (e.g., hardware abstraction, device drivers, visualization tools, messaging, package management, etc.) to enable robotic applications.

[0047] Figure 3is a block diagram of a computing device 300 according to an example implementation. In some examples, Figure 3 Some of the components shown may be distributed across multiple computing devices (e.g., desktop computers, servers, handheld devices, etc.). However, for the sake of example, these components are shown and described as part of one example device. Computing device 300 includes: interface 302, mobile unit 304, control unit 306, communication system 308, data storage device 310, and processor 314. Figure 3 The components shown may be linked together by communication link 316. In some examples, computing device 300 may include hardware to enable communication within computing device 300 and another computing device (not shown). In one embodiment, robotic device 140 or robotic device 200 may include computing device 300.

[0048] The interface 302 can be configured to enable the computing device 300 to communicate with another computing device (not shown). Therefore, the interface 302 can be configured to receive input data from one or more devices. In some examples, the interface 302 can also maintain and manage data records received and sent by the computing device 300. In other examples, the data records can be maintained and managed by other components of the computing device 300. The interface 302 can also include a receiver and a transmitter for receiving and sending data. In some examples, the interface 302 can also include a user interface for receiving input, such as a keyboard, a microphone, a touch screen, etc. In addition, in some examples, the interface 302 can also be connected to an output device such as a display, a speaker, etc. via an interface.

[0049] In one example, the interface 302 may receive input indicating position information corresponding to one or more components of an environment in which a robotic device (e.g., the robotic device 140, the robotic device 200) is located. In this example, the environment may be an operating room in a hospital that includes a robotic device configured to function during a surgical procedure. The interface 302 may also be configured to receive information associated with the robotic device. For example, the information associated with the robotic device may include operating characteristics of the robotic device and ranges of motion of one or more components (e.g., joints 204, 208, 212, 216, 220, 224, and 230) of the robotic device (e.g., the robotic device 140, the robotic device 200).

[0050] The control unit 306 of the computing device 300 may be configured to run control software that exchanges data with components (e.g., robot arm 141, robot pedal 142, joints 204, 208, 212, 216, 220, 224, and 230) of a robotic device (e.g., robotic device 140, robotic device 200) and one or more other devices (e.g., processing device 122, tracking device 130, etc.). The control software may communicate with a user through a user interface and a display monitor (e.g., display 126) that communicate with the robotic device. The control software may also communicate with the tracking device 130 and the processing device 122 through a wired communication interface (e.g., parallel port, USB, etc.) and / or a wireless communication interface (e.g., antenna, transceiver, etc.). The control software may communicate with one or more sensors to measure the effort applied by the user at an instrument T mounted to a robotic arm (e.g., robotic arm 141, link 226). The control software can communicate with the robotic arm to control the position of the robotic arm relative to the marker 150.

[0051] As described above, the control software can communicate with the tracking device 130. In one scenario, the tracking device 130 can be configured to track a marker 150 attached to the patient P. For example, the marker 150 can be attached to the spinous process of the vertebra of the patient P. In this example, the marker 150 can include one or more infrared reflectors that are visible to the tracking device 130 to determine the position of the marker 150. In another example, multiple markers can be attached to one or more vertebrae and used to determine the position of the instrument T.

[0052] In one example, the tracking device 130 may provide updates of the position information of the marker 150 to the control software of the robotic device 140 in near real time. The robotic device 140 may be configured to receive updates of the position information of the marker 150 from the tracking device 130 via a wired and / or wireless interface. Based on the received updates of the position information of the marker 150, the robotic device 140 may be configured to determine one or more adjustments to the first position of the instrument T in order to maintain the desired position of the instrument T relative to the patient P.

[0053] In one embodiment, the control software may include independent modules. In an exemplary embodiment, these independent modules run simultaneously in a real-time environment and use shared memory to ensure management of the various tasks of the control software. These modules may have different priorities, such as, for example, a safety module having the highest priority. The safety module may monitor the status of the robotic device 140. In one scenario, when an emergency situation is detected (such as, for example, an emergency stop, software failure, or collision with an obstacle), the safety module may send a command to the control unit 306 to stop the robotic arm 141.

[0054] In one example, the interface 302 is configured to allow the robotic device 140 to communicate with other devices (e.g., the processing device 122, the tracking device 130). Thus, the interface 302 is configured to receive input data from one or more devices. In some examples, the interface 302 may also maintain and manage data records received and sent by other devices. In other examples, the interface 302 may use a receiver and a transmitter to receive and send data.

[0055] The interface 302 may be configured to manage communications between a user and control software through a user interface and a display screen (e.g., via displays 123 and 124). The display screen may display a graphical interface that guides a user through different modes associated with the robotic device 140. The user interface may enable a user to control movement of the robotic arm 141 associated with the start of a surgical procedure, enable a tracking mode to be used during a surgical procedure, and, for example, stop the robotic arm 141 if desired.

[0056] The mobile unit 304 may be configured to determine the movements associated with one or more components of the robotic arm 141 to perform a given procedure. In one embodiment, the mobile unit 304 may be configured to determine the trajectory of the robotic arm 141 using forward and inverse kinematics. In one scenario, the mobile unit 304 may access one or more software libraries to determine the trajectory of the robotic arm 141. In another example, the mobile unit 304 is configured to receive one or more instructions from the processing device 122 for actuating the one or more components of the robotic device 140 based on the captured motion data of the instrument based on the user-defined path.

[0057] The movement unit 304 may be configured to simulate the operation of the robotic device 140 to move the instrument T along a given path. In one example, based on the simulated operation, the movement unit 304 may determine a metric associated with the instrument T. In addition, the movement unit 304 may be configured to determine a force associated with the metric based on the simulated operation. In one example, the movement unit 304 may include instructions for determining a force based on an open kinematic chain.

[0058] The mobile unit 304 may include a force module for monitoring forces and torques measured by one or more sensors coupled to the robotic arm 141. In one scenario, the force module can detect a collision with an obstacle and alert the safety module.

[0059] The control unit 306 may be configured to manage functions associated with various components of the robotic device 140 (e.g., the robotic arm 141, the pedal 142, etc.). For example, the control unit 306 may send one or more commands to maintain a desired position of the robotic arm 141 relative to the marker 150. The control unit 306 may be configured to receive movement data from the movement unit 304.

[0060] In one scenario, the control unit 306 may instruct the robotic arm 141 to function according to the collaborative mode. In the collaborative mode, the user can manually move the robotic arm 141 by holding a tool T coupled to the robotic arm 141 and moving the tool T to a desired position. In one example, the robotic device 140 may include one or more force sensors coupled to the end effector of the robotic arm 141. For example, when the user grabs the tool T and begins to move the tool in one direction, the control unit 306 receives the forces measured by the force sensors and combines these forces with the position of the robotic arm 141 to produce the movement desired by the user.

[0061] In one scenario, the control unit 306 can instruct the robotic arm 141 to function according to a given mode, which will cause the robotic device 140 to maintain the relative position of the instrument T and a given IR reflector or emitter (e.g., marker 150). In one example, the robotic device 140 can receive updated position information of the marker 150 from the tracking device 130 and make adjustments as needed. In this example, the movement unit 304 can determine which joint(s) of the robotic arm 141 needs to be moved based on the received updated position information of the marker 150 in order to maintain the relative position of the instrument T and the marker 150.

[0062] In another scenario, a restrictive collaboration mode may be defined by the user to restrict the movement of the robotic device 140. For example, the control unit 306 may restrict the movement of the robotic arm 141 to a plane or axis based on user preferences. In another example, the robotic device 140 may receive information about one or more predetermined boundaries within the surgical site that should not intersect a portion of the determined motion path.

[0063] In one embodiment, the robotic device 140 can communicate with the processing device 122. In one example, the robotic device 140 can provide the position and orientation data of the instrument T to the processing device 122. In this example, the processing device 122 can be configured to store the position and orientation data of the instrument T for further processing. In one scenario, the image processing device 122 can use the received position and orientation data of the instrument T to superimpose a virtual representation of the instrument T on the display 126.

[0064] In one embodiment, a sensor configured to detect pressure or force may be coupled to the last joint of the robotic arm (e.g., link 226). Based on a given movement of the robotic arm, the sensor may provide a reading of the pressure applied to the last joint of the robotic arm to a computing device (e.g., a control unit of the robotic device). In one example, the robotic device may be configured to transmit force or pressure data to a computing device (e.g., processing device 122). In another embodiment, the sensor may be coupled to an instrument such as a retractor. In this embodiment, the force or pressure applied to the retractor and detected by the sensor may be provided to a robotic device (e.g., robotic device 140, robotic device 200) or a computing device (e.g., processing device 122), or both for further analysis.

[0065] In one scenario, the robotic device can access movement data stored in a memory of the robotic device to trace the movement along the determined motion path. In one example, the robotic device can be configured to move a surgical tool along the determined motion path to reach or move away from a surgical site.

[0066] On the other hand, a robotic device (e.g., robotic device 140, robotic device 200) can help the instrument move along the determined motion path. In one scenario, the surgeon can plan the trajectory by keeping a portable instrument (e.g., instrument H) in a position outside the skin during surgery and by superimposing at least a portion of the determined motion path onto an image of the surgical site to understand how the trajectory of the pedicle screw intersects with relevant anatomical structures. In one example, the robotic device can be configured to move along different trajectories until the tool intersects with the pedicle screw trajectory captured via the portable instrument. In this example, the robotic device can provide a signal to a computing device (e.g., processing device 122, computing device 300), which in turn can notify the surgeon via an audible or visual warning that the ideal pedicle screw trajectory has been reached.

[0067] In another scenario, once the instrument coupled to the robotic arm of the robotic device (e.g., robotic arm 141, links 206, 210, 214, 218, 222, and 226) reaches the desired pedicle screw trajectory, the robotic device can be configured to receive input from the surgeon to proceed along the desired pedicle screw trajectory. In one example, the surgeon can provide input to the robotic device (e.g., depressing pedal 142) to confirm the surgeon's decision to enable the robotic device to proceed along the desired pedicle screw trajectory. In another example, the user can provide another form of input to the robotic device or computing device to assist the instrument in moving along the determined motion path.

[0068] In one scenario, once the robotic device has received confirmation to proceed along the desired pedicle screw trajectory, the robotic device may receive instructions from the movement unit 304 to pivot from the current trajectory to the desired pedicle screw trajectory. The movement unit 304 may provide the control unit 306 with the required movement data to enable the robotic device to move along the desired pedicle screw trajectory.

[0069] In another aspect of the present invention, the robotic device (e.g., the robotic device 140, the robotic device 200) can be configured to pivot about an important area based on a captured pose of a portable instrument (e.g., the instrument H). For example, the robotic device can be configured to pivot the retractor about its tip so that all steps associated with retracting soft tissue do not need to be repeated. In one example, the movement unit 304 can determine a trajectory required to pivot the retractor.

[0070] In one example, the robotic device may be coupled to a retractor that holds soft tissue away from the surgical site. In this example, the surgeon may need to slightly reposition the retractor due to patient movement. To this end, the surgeon may enable a mode for the robotic device that pivots the retractor by moving the robotic arm (e.g., robotic arm 141, links 206, 210, 214, 218, 222, and 226) according to a trajectory determined by the mobile unit 304. In one example, a user may input a desired direction and amount of movement via a computing device (e.g., processing device 122, computing device 300). After inputting the direction and amount of movement, the user (e.g., surgeon) may interact with the robotic device (e.g., depressing pedal 142) to initiate movement of the instrument coupled to the robotic arm. In one example, the robotic device may allow the user to view different aspects of the anatomical structure without disengaging from a docking point.

[0071] In another example, the movement unit 304 can provide one or more trajectories based on the captured pose of the portable instrument (e.g., instrument H) to the computing device (e.g., processing device 122) for display on the display 126. In this example, the user can select from one or more predetermined movements associated with a given procedure. For example, a given predetermined movement can be associated with a specific direction and amount of movement to be performed by using the pedal 142 of the depressing robotic device 140.

[0072] In another aspect of the present invention, one or more infrared (IR) reflectors or emitters may be coupled to a robotic arm (e.g., robotic arm 141, links 206, 210, 214, 218, 222, and 226) of a robotic device (e.g., robotic device 140, robotic device 200). In one scenario, the tracking device 130 may be configured to determine the location of the one or more IR reflectors or emitters before the robotic device begins operation. In such a scenario, the tracking device 130 may provide the location information of the one or more IR reflectors or emitters to a computing device (e.g., processing device 122, computing device 300) for further processing.

[0073] In one example, the processing device 122 or the computing device 300 can be configured to compare the position information of the one or more IR reflectors or emitters coupled to the robotic arm with data stored on a local or remote database (the data containing information about the robotic device (e.g., a geometric model of the robotic device)) to help determine the location or position of the robotic arm. In one example, the processing device 122 can determine a first position of the robotic arm based on information provided by the tracking device 130. In this example, the processing device 122 can provide the determined first position of the robotic arm to the robotic device or computing device (e.g., computing device 300). In one example, the robotic device can use the received first position data to perform calibration of one or more components (e.g., encoders, actuators) associated with the one or more joints of the robotic arm.

[0074] In one scenario, an instrument coupled to a robotic arm of a robotic device may be used to determine a difference between an expected tip position of the instrument and an actual tip position of the instrument. In such a scenario, the robotic device may continue to move the instrument to a known position via the tracking device 130 to bring the tip of the tool into contact with the known position. The tracking device 130 may capture position information corresponding to the one or more IR reflectors or emitters coupled to the robotic arm and provide the information to the robotic device or a computing device (e.g., the processing device 122, the computing device 300). In addition, the robotic device or the computing device may be configured to adjust a coordinate system offset between the robotic device and the tracking device 130 based on the expected tip position of the tool and the actual tip position of the tool.

[0075] In another aspect, a force or pressure sensor can be coupled to a robotic arm (e.g., robotic arm 141, links 206, 210, 214, 218, 222, and 226) of a robotic device (e.g., robotic device 140, robotic device 200). In one example, the force or pressure sensor can be positioned on an end effector of the robotic arm. In another example, the force or pressure sensor can be coupled to a given joint of the robotic arm. The force or pressure sensor can be configured to determine when a force or pressure reading is above a static threshold. The static threshold can be based on the force or pressure experienced at the sensor when the end effector is holding an instrument without applying any additional force or pressure to the instrument (e.g., a user attempting to move the instrument). In one example, if the force or pressure reading is equal to or below the static threshold, the robotic arm can stop moving.

[0076] In one example, the movement of the robotic arm 141 can be controlled by depression of the pedal 142. For example, when the pedal 142 is depressed, the control unit 306 and the movement unit 304 can be configured to receive any measurement results of force or pressure from the one or more force sensors and use the received information to determine the trajectory of the robotic arm 141.

[0077] In another example, the movement of the robotic arm 141 may be adjusted based on how much the pedal 142 is depressed. For example, if the user depresses the pedal 142 a full amount, the robotic arm 141 may move at a higher speed than if the pedal 142 is depressed halfway. In another example, the movement of the robotic arm 141 may be controlled by a user interface located on the robotic device.

[0078] In one example, the robotic device (e.g., the robotic device 140, the robotic device 200) can be configured to store movement data corresponding to a user-defined motion path based on movement of the portable instrument in a local or remote memory. In this example, the robotic device can be configured to travel in one or more directions along a trajectory corresponding to the stored movement data. For example, the surgeon can instruct the robotic device to reverse along a trajectory corresponding to the stored movement data.

[0079] In another aspect of the present invention, a robotic device (e.g., robotic device 140, robotic device 200) can be used to navigate one or more surgical instruments and provide navigation information to a computing device (e.g., processing device 122, computing device 300) for further processing. In one example, the computing device can be configured to determine a virtual representation of the surgical instrument. In addition, the computing device can be configured to superimpose the virtual representation of the surgical instrument on a two-dimensional or three-dimensional image of the surgical site.

[0080] In one example, the robotic device may perform a calibration process between the tracking device 130 to remove reliance on the position information of the tracking device 130 in the event that the line of sight between the robotic device and the tracking device 130 is blocked. In one example, using a robotic device that has been registered with a navigation system (as described herein) and a three-dimensional image of the patient corresponding to the surgical site may allow the robotic device to become independent of the reduced accuracy of the distance associated with the tracking device 130.

[0081] The communication system 308 may include a wired communication interface (e.g., parallel port, USB, etc.) and / or a wireless communication interface (e.g., antenna, transceiver, etc.) to receive signals from an external device and / or provide signals to an external device. In some examples, the communication system 308 may receive instructions for operation of the processing device 122. Additionally or alternatively, in some examples, the communication system 308 may provide output data.

[0082] The data storage device 310 may store program logic 312 that may be accessed and executed by the processor 314. The program logic 312 may contain instructions that provide control to one or more components of the processing device 122, the robotic device 140, the robotic device 200, etc. For example, the program logic 312 may provide instructions to adjust the operation of the robotic device 200 based on one or more user-defined trajectories associated with the portable instrument. The data storage device 310 may include one or more volatile storage components and / or one or more non-volatile storage components, such as optical, magnetic, and / or organic memory, and the data storage device may be integrated in whole or in part with the processor 314.

[0083] Processor 314 may include one or more general purpose processors and / or one or more special purpose processors. To the extent processor 314 includes more than one processor, such processors may work individually or in combination. For example, a first processor may be configured to operate mobile unit 304, and a second processor in processor 314 may operate control unit 306.

[0084] Further, although various ones of these components are illustrated as being integrated into the processing device 122, the robotic device 140, or the robotic device 200, in some embodiments, one or more components may be removably mounted to or otherwise connected (e.g., mechanically or electrically) to the processing device 122, the robotic device 140, or the robotic device 200 using a wired or wireless connection.

[0085] Figure 4 An example computer-readable medium configured in accordance with an example implementation is depicted. In an example implementation, an example system may include: one or more processors, one or more forms of memory, one or more input devices / interfaces, one or more output devices / interfaces, and machine-readable instructions that, when executed by the one or more processors, cause the system to perform the various functions, tasks, capabilities, etc. described above.

[0086] As mentioned above, in some embodiments, the disclosed techniques (e.g., the functionality of the robotic device 140, the robotic device 200, the processing device 122, the computing device 300, etc.) may be implemented by computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other media or articles of manufacture. Figure 4 is a schematic diagram illustrating a conceptual partial view of an example computer program product including a computer program for executing a computer process on a computing device, arranged in accordance with at least some embodiments disclosed herein.

[0087] In one embodiment, the example computer program product 400 is provided using a signal bearing medium 402. The signal bearing medium 402 may include one or more programming instructions 404 that, when executed by one or more processors, may provide the above referenced Figures 1 to 3The functions or portions of the functions described. In some examples, the signal bearing medium 402 can be a computer readable medium 406, such as, but not limited to, a hard drive, a compact disk (CD), a digital video disk (DVD), a digital tape, a memory, etc. In some implementations, the signal bearing medium 402 can be a computer recordable medium 408, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some implementations, the signal bearing medium 402 can be a communication medium 410 (e.g., an optical cable, a waveguide, a wired communication link, etc.). Thus, for example, the signal bearing medium 402 can be transported by a wireless form of the communication medium 410.

[0088] The one or more programming instructions 404 may be, for example, computer executable and / or logic implemented instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to programming instructions 404 delivered to the computing device by one or more of the computer readable medium 406, the computer recordable medium 408, and / or the communication medium 410.

[0089] The computer-readable medium 406 may also be distributed among multiple data storage components that may be remotely located from one another. The computing device that executes some or all of the stored instructions may be an external computer, or a mobile computing platform (such as a smartphone, tablet device, personal computer, wearable device, etc.). Alternatively, the computing device that executes some or all of the stored instructions may be a remotely located computer system (such as a server).

[0090] Figure 5 , Figure 6 , Fig. 9 as well as Fig.10 is a flow chart of an example method during a surgical procedure according to at least one or more embodiments described herein. Although the blocks in the various figures are illustrated in a sequential order, in some cases, the blocks may be executed in parallel and / or in a different order than those described therein. Moreover, various blocks may be combined into fewer blocks, divided into additional blocks, and / or removed based on the desired implementation.

[0091] also, Figure 5 , Figure 6 , Fig. 9 as well as Fig.10The flowchart of the present embodiment shows the possible functions and operations of implementation. In this regard, each frame can represent a part of a module, a segment or a program code, which includes one or more instructions for implementing a specific logical function or step in the process that can be executed by a processor. The program code can be stored on any type of computer-readable medium, for example, such as a storage device including a disk or a hard disk drive. The computer-readable medium can, for example, include a non-transitory computer-readable medium (such as a register memory, a processor cache or a random access memory (RAM)) that stores data for a short period of time, and / or a persistent long-term storage device (such as a read-only memory (ROM), an optical disk or a magnetic disk, or a CD-ROM read-only memory (CD-ROM)). The computer-readable medium can or can include any other volatile or non-volatile storage system. The computer-readable medium can, for example, be considered to be a computer-readable storage medium, a tangible storage device or other manufactured article.

[0092] Alternatively, Figure 5 , Figure 6 , Fig. 9 as well as Fig.10 Each block in the figure may represent a circuit that is wired to perform a specific logical function in the process described. Figure 5 , Figure 6 , Fig. 9 as well as Fig.10 Those methods shown) can be performed in whole or in part by the cloud and / or Figure 1 However, it should be understood that the example methods may be performed by other entities or combinations of entities (i.e., by other computing devices and / or combinations of computer devices) without departing from the scope of the present invention. For example, Figure 5 , Figure 6 , Fig. 9 as well as Fig.10 The functionality of the method may be performed entirely by a computing device (or a component of a computing device, such as one or more processors), or may be performed across multiple components of a computing device, across multiple computing devices (e.g., Figure 1 control unit 118 and image processing device 122) and / or distributed across servers.

[0093] Reference Figure 5 , the example method 500 during a surgical procedure may include one or more operations, functions, or actions as shown in blocks 502 to 516. In one embodiment, the method 500 is performed in whole or in part by Figure 1 The system 100 is implemented as follows.

[0094] As indicated at block 502 , method 500 includes the steps of acquiring a high-resolution baseline image of a surgical site of a patient and digitally processing the high-resolution baseline image to generate a baseline image set.

[0095] In one example, the image processing device 122 is configured to provide high-quality real-time images on displays 123, 124, which are derived from lower-detail images obtained using low-dose radiation. Low-dose images are too "noisy" to provide sufficient information about local anatomy for accurate image-guided surgery. Although full-dose images provide a clear view of the surgical site, the higher radiation dose makes it very problematic to take multiple full-dose images during surgery.

[0096] In one example, a baseline high resolution, full dose image of the surgical site is acquired and stored in a memory associated with the image processing device 122. In some cases where the C-arm moves during surgery, multiple high resolution images can be obtained at different locations of the surgical site and then these multiple images are "stitched" together to form a composite base image. The movement of the C-arm, and more specifically "tracking" the acquired images during these movements are explained in other steps described in more detail herein. For the present discussion, it is assumed that the imaging system is relatively fixed, which means that only very limited movement of the C-arm and / or the patient is considered, such as may occur in epidural pain surgery, spinal K-wire placement, or stone extraction. In one scenario, the baseline image is projected on the display 123 to verify that the surgical site is properly centered within the image.

[0097] In some cases, new full dose images may be obtained until a suitable baseline image is obtained. In surgeries where the C-arm is moved, a new baseline image is obtained at the new position of the imaging device, as discussed below. If the displayed image is acceptable as a baseline image, a button may be pressed on a user interface (such as on display device 126 or interface 125). In surgeries performed on anatomical regions where a large amount of motion is expected due to physiological processes (such as breathing), multiple baseline images may be acquired for the same region over multiple phases of the cycle. These images may be tagged with time data from other medical instruments (such as an ECG or pulse oximeter).

[0098] Once the baseline image is acquired, a baseline image set is generated where the original baseline image is digitally rotated, translated, and resized to create thousands of permutations of the original baseline image. For example, a typical 128×128 pixel two-dimensional (2D) image can be translated by .+-.15 pixels in 1 pixel intervals in the x and y directions, rotated by .+-.9 degrees in 3 degree intervals, and scaled from 92.5% to 107.5% in 2.5% intervals (4 degrees of freedom, 4D), resulting in 47,089 images in the baseline image set. Since two additional rotations orthogonal to the x and y axes are added, a three-dimensional (3D) image would imply a 6D solution space. The original computed tomography (CT) image data set can be used to form thousands of digitally reconstructed radiographs (DRRs) in a similar manner. Therefore, in this step, the original baseline image generates thousands of new image representations as if the original baseline image was acquired with each of the different shift permutations. The solution space may be stored in graphics card memory (such as in the GPU) or formed into new images that are then sent to the GPU, depending on the number of images in the "solution space" and the speed at which the graphics processing unit (GPU) of the image processing device 122 can generate these images. With current computing power, a baseline image set of nearly 850,000 images can be generated in a GPU in less than a second in a stand-alone medical-grade computer because multiple processors of the GPU can each process an image simultaneously.

[0099] As shown in box 504, method 500 includes the following steps: acquiring a new image of the surgical site at a lower resolution and comparing the new image to one or more images of the baseline image set. In one example, the low-dose image is acquired via the C-arm 103 and stored in a memory associated with the image processing device 122, and is projected on the display 123. Because the new image is obtained at a lower radiation dose, it is very noisy. Therefore, the new image can be merged with the image from the baseline image set to generate a clearer image on the second display 124 that conveys more useful information to the surgeon. In one example, the merged image can be accomplished by an image recognition or registration step, in which the new image is compared with the image in the baseline image set to find a statistically significant match. Continuing with this example, a new "merged" image is generated and can be displayed on the display 124 adjacent to the view of the original new image. At different times throughout the surgical procedure, new baseline images can be obtained for generating a new baseline image set.

[0100] As shown in box 506, method 500 includes the steps of selecting a representative image in the baseline image set that has an acceptable correlation with the new image, and merging the selected representative image with the new image to generate a merged image. In one example, the step of selecting a representative image in the baseline image set includes comparing the current new image with the image in the baseline image set. Since this step occurs during the surgical procedure, time and accuracy are critical. Preferably, this step can obtain image registration in less than one second so that there is no meaningful delay between when the image is captured by the C-arm and when the merged image is displayed on the device 126. Various algorithms can be used that may depend on various factors, such as the number of images in the baseline image set, the size and speed of the computer processor or graphics processor performing the algorithm calculations, the time allocated to perform the calculations, and the size of the images being compared (e.g., 128×128 pixels, 1024×1024 pixels, etc.). In one method, comparisons are made between pixels at the above-mentioned predetermined positions in a grid pattern throughout the 4D space. In another heuristic method, pixel comparisons can be concentrated in image areas that are considered to provide a greater likelihood of a relevant match. These regions can be "pre-seeded" based on knowledge from a grid or PCA search (defined below), data from a tracking system (such as an optical surgical navigation device), or position data from a DICOM file or equivalent. Alternatively, the user can specify one or more regions of the image for comparison by marking anatomical features on the baseline image that are considered relevant for surgery. With this input, each pixel in the region can be assigned a relevance score between 0 and 1 that measures the pixel's contribution to the image similarity function when the new image is compared to the baseline image. The relevance scores can be calibrated to identify regions to focus on or to ignore.

[0101] In another approach, principal component analysis (PCA) is performed, which allows for comparison with more large images in the amount of time allotted compared to the full-resolution grid approach. In the PCA approach, it is determined how the individual pixels of the image set co-vary with each other. Only a small portion of the total solution set (e.g., a randomly selected 10% of the baseline image set) may be used to generate the covariance matrix. Each image from the baseline image set is converted to a column vector.

[0102] In one example, a 70×40 pixel image is turned into a 2800×1 vector. These column vectors are normalized to mean 0 and variance 1 and combined into a larger matrix. The covariance matrix is ​​determined from this larger matrix and the largest eigenvectors are selected. For this particular example, it has been found that 30 PCA vectors can explain approximately 80% of the variance of the corresponding image. Therefore, each 2800×1 image vector can be multiplied by the 2800×30 PCA vector to generate a 1×30 vector. The same steps are applied to the new image—the new image is converted into a 2800×1 image vector and multiplied with the 2800×30 PCA vector to generate a 1×30 vector corresponding to the new image. The solution set (baseline image) vector and the new image vector are normalized, and the dot product of the new image vector and each vector in the solution space is calculated. The solution space baseline image vector that generates the largest dot product (i.e., closest to 1) is determined to be the closest image to the new image. It should be understood that this example can be varied using different image sizes and / or different principal components for analysis. It should also be understood that other known techniques that can utilize, for example, eigenvectors, singular value determination, mean square error, mean absolute error, and edge detection can be implemented. It is also contemplated that various image recognition methods can be applied to selected areas of an image, or that various statistical measures can be applied to find matches that fall within a suitable confidence threshold. A confidence value or correlation value can be assigned that quantifies the degree of correlation between the new image and a selected baseline image or selected images in a baseline image set, and the confidence value can be displayed for re-examination by the surgeon. The surgeon can decide whether the confidence value is acceptable for a particular display and whether another image should be acquired.

[0103] In image-guided surgery, tools, implants, and instruments will inevitably be present in the image field. These objects are typically radiation intensive and will obstruct relevant patient anatomy from view. Therefore, the new low-dose image obtained will include artifacts of the tool T that are unrelated to any baseline image set. Therefore, the presence of the tool in the image ensures that the above comparison technique will not produce a high degree of registration between the new image and any baseline image set. However, if the end result of each of the above processes is to find the highest degree of correlation (which is statistically relevant or exceeds a certain threshold), then the entire new low-dose image, tool artifacts and all, can be used for image registration.

[0104] Alternatively, the image registration step can be modified to account for tool artifacts on the new image. In one approach, the new low-dose image can be evaluated to determine the number of image pixels that are "blocked" by the tool. Such an evaluation can involve comparing the grayscale value of each pixel to a threshold and excluding pixels that fall outside of the threshold. For example, if the pixel grayscale value ranges from 0 (completely blocked) to 10 (completely transparent), then a threshold of 3 can be applied to eliminate certain pixels from the evaluation. In addition, when position data is available for the various tracked tools, the algorithm areas that are blocked can be mathematically avoided.

[0105] In another approach, the image recognition or registration step may include the steps of measuring the similarity of the low-dose image to a transformed version of a baseline image (i.e., a baseline image that has been transformed to account for movement of the C-arm) or the patient. In image-guided surgery, a C-arm system acquires multiple X-ray images of the same anatomical structure. During the acquisition of this series of images, the system may move in small increments and surgical tools may be added or removed from the field of view, even though the anatomical features may remain relatively stable. The method described below exploits this consistency of anatomical features by using anatomical features present in one image to fill in missing details in another later image. This method further allows the high quality of the full-dose image to be transferred to subsequent low-dose images.

[0106] In one embodiment, a similarity function in the form of a scalar function of the images is used to determine the registration between the current low-dose image and the baseline image. In order to determine this registration, it is first necessary to determine the incremental motion that has occurred between the images. This motion can be described by four numbers corresponding to the four degrees of freedom - scaling, rotation, and vertical and horizontal translation. For a given pair of images to be compared, knowledge of these four numbers enables manipulation of one of the images in the pair so that the same anatomical features appear in the same position between the two images. The scalar function is a measure of this registration and can be obtained using a correlation coefficient, a dot product, or a mean square error. For example, the dot product scalar function corresponds to the sum of the products of the intensity values ​​at each pixel pair in the two images. For example, the intensity values ​​of the pixels at 1234, 1234 in each of the low-dose image and the baseline image are multiplied. Similar calculations are performed for every other pixel position, and all these multiplied values ​​are added to the scalar function. It can be appreciated that when the two images are accurately registered, the dot product will have the maximum possible magnitude. In other words, when the best combination is found, the corresponding dot product is generally higher than the other dot products, which can be reported as a Z-score (i.e., the number of standard deviations above the mean). A Z-score greater than 7.5 indicates a 99.9999999% certainty that the alignment was not found by chance. It should be remembered that the alignment sought using this dot product is between a baseline image of the patient's anatomy and a real-time low-dose image of the same anatomy taken at a later time after the field of view and imaging device may have moved or a non-anatomical object was introduced into the field of view.

[0107] This method is particularly suitable for execution using a parallel computing architecture (such as a GPU), which consists of multiple processors capable of performing the same calculation in parallel. Therefore, each processor of the GPU can be used to calculate a similarity function of a transformed version of the low-dose image and the baseline image. In this way, multiple transformed versions of the baseline image can be compared with the low-dose image at the same time. The transformed baseline image can be pre-generated when the baseline image is acquired, and then the transformed baseline image is stored in the GPU memory. Alternatively, a single baseline image can be stored and transformed in the air by reading a texture fetching from the transformed coordinates during the comparison. In the case where the number of processors of the GPU greatly exceeds the number of transformations to be considered, the baseline image and the low-dose image can be divided into different segments, and the similarity functions of each segment can be calculated by different processors and then merged.

[0108] To further speed up the determination of the best transformation to align the two images, a similarity function may first be calculated using a downsampled image containing fewer pixels. The downsampling may be performed in advance by averaging together groups of neighboring pixels. A similarity function may first be calculated for the downsampled image for many transformations over a wide range of possible motions. Once the best transformation from the set is determined, that transformation may be used as the center of a finer grid of possible transformations applied to images with more pixels. In this way, multiple steps are used to determine the best transformation with high accuracy while considering a wide range of transformations in a short amount of time.

[0109] In order to reduce the bias of the similarity function caused by the difference in the overall intensity level in different images, and to preferentially align the anatomical features of interest to the user in the image, the image can be filtered before calculating the similarity function. Such filtering will ideally suppress the very high spatial frequency noise associated with low-dose images, while also suppressing the low spatial frequency information associated with large flat areas lacking important anatomical details. Such image filtering can be accomplished, for example, using convolution, multiplication in the Fourier domain, or Butterworth filters. It is therefore envisioned that both the low-dose image and the baseline image will be filtered accordingly before generating the similarity function.

[0110] As previously explained, non-anatomical features (such as surgical tools) may be present in the image, in which case it may be necessary to modify the similarity function calculation process to ensure that only anatomical features are used to determine the alignment between the low-dose image and the baseline image. A mask image can be generated that identifies whether a pixel is part of an anatomical feature. In one aspect, anatomical pixels can be assigned a value of 1, while non-anatomical pixels can be assigned a value of 0. This assignment enables both the baseline image and the low-dose image to be multiplied by the corresponding mask image before calculating the similarity function as described above. In other words, the mask image can eliminate non-anatomical pixels to avoid any impact on the similarity function calculation.

[0111] In order to determine whether a pixel is anatomical, a variety of functions can be calculated in the neighborhood around each pixel. These functions of the neighborhood can include the standard deviation, gradient magnitude, and / or corresponding value of the pixel in the original grayscale image and the filtered image. The "neighborhood" around the pixel includes a predetermined number of adjacent pixels, such as a 5×5 grid or a 3×3 grid. In addition, these functions can be compounded, for example, by finding the standard deviation of the neighborhood of the standard deviation, or by calculating a quadratic function of the standard deviation and the gradient magnitude. An example of a suitable function of the neighborhood is to use edge detection techniques to distinguish between bones and metal instruments. Metal presents a "clearer" edge than bone, and the standard deviation or gradient calculation of the neighborhood of the "edge" pixel can be used to determine this difference. Therefore, the neighborhood function can determine whether the pixel is anatomical or non-anatomical based on the edge detection method, and assign a value of 1 or 0 to the pixel as appropriate.

[0112] Once a set of values ​​has been calculated for a particular pixel, these values ​​can be compared to a threshold determined from measurements of previously acquired images, and the pixel can be assigned a binary value based on the number of thresholds exceeded. Alternatively, a fractional value between 0 and 1 can be assigned to the pixel, reflecting the degree of certainty of the pixel's identity as part of an anatomical feature or a non-anatomical feature. These steps can be accelerated using the GPU by assigning the calculations at one pixel in the image to a processor on the GPU, enabling the calculation of values ​​for multiple pixels simultaneously. A combination of morphological image operations, such as erosion and dilation, can be used to manipulate the mask to fill and expand areas corresponding to non-anatomical features.

[0113] Once that image registration is complete, the new image can be displayed in different ways together with the image selected from the baseline image set. In one method, the two images are merged. The two images can be merged by combining the digital representation of the image in a conventional manner (such as by adding or averaging the pixel data of the two images). In one embodiment, the surgeon can identify one or more specific areas of interest in the displayed image, such as through the user interface 125, and the merging operation can be configured to utilize the displayed baseline image data outside the area of ​​interest and perform the merging operation of the display within the area of ​​interest. The user interface 125 can be provided with a "slider" that controls the amount of the baseline image displayed in the merged image relative to the new image. In another method, the surgeon can alternate between the relevant baseline image and the new image or merged image. The surgeon can alternate between these views to get a clearer view of the underlying anatomical structure and a view with the current field of view of the instrument T, which actually digitally removes the instrument from the field of view by alternating images, thereby clarifying the position of the instrument relative to the anatomical structure blocked by it.

[0114] As shown in box 508, method 500 includes the following steps: capturing motion data of the instrument based on a user-defined path to the surgical site. In one example, the tracking device 130 can be configured to capture one or more images of the instrument H when the instrument H is used in a surgical procedure. The captured one or more images are processed to determine the orientation and position data associated with one or more IR markers connected to the instrument H. Then, the determined orientation and position data associated with the one or more IR markers are used to determine the three-dimensional posture data of the instrument H within a given period of time. In one example, the instrument H can be placed at a known position in the operating room to indicate a trigger for capturing motion data. Continuing with this example, the processing device 122 can be configured to determine when the instrument H has not moved within a predetermined amount of time as an indicator to end motion data capture. In another example, a button on a user interface (such as a button on a display device 126 or an interface 125) can be pressed to switch between starting and stopping the capture of motion data associated with the instrument H.

[0115] As shown in block 510, the method 500 further includes the step of determining a motion path corresponding to the captured motion data, wherein the motion path is associated with actuation of one or more components of the robotic device. In one example, the motion path may include robotic movements that place an end effector of the robotic device at a specific location in space at a specific point in time based on the captured motion data, the specific location corresponding to the location of the instrument H. In one example, the motion path may include a sequence of Cartesian coordinates and rotation angles indicating the location of the instrument H over time.

[0116] Block 510 may also or instead include comparing the position of the one or more infrared reflectors of the portable instrument with the position of the one or more infrared reflectors coupled to the patient. Block 510 may also include determining, based on the comparison, that the distance between the one or more infrared reflectors of the portable instrument and the one or more infrared reflectors coupled to the patient is within a safety threshold. Based on the determination, block 510 may also include determining one or more adjustments to the actuation of the one or more components of the robotic device.

[0117] As shown in block 512, the method 500 further includes the step of displaying a portion of the motion path on the merged image. In one example, in order to form a pilot hole for placing a pedicle screw in a vertebral pedicle by means of the robotic device 140, the surgeon advances the instrument H to the target site of the pedicle where the pilot hole is to be formed. In this example, the tracking device 130 is configured to capture the position and orientation of the instrument H and provide the position and orientation information to the processing device 122. Continuing with this example, the processing device 122 is configured to provide an image of the surgical site for display and superimpose at least a portion of the motion path on the image of the surgical site.

[0118] Reference Figure 7 , Figure 7 An example two-dimensional merged image 700 of a surgical site including a vertebra 702 and a portion of a motion path 704 is illustrated. In one example, the surgeon may position the instrument H at the pedicle target site as described above and view the merged image 700 to determine whether the trajectory of the portion of the motion path 704 is aligned with the surgical plan. In this example, based on the surgeon viewing the portion of the motion path 704 as shown and accepting the corresponding trajectory for forming a guide hole, the surgeon may provide input via the processing device 122 to determine one or more instructions using the motion path associated with the image 700 that actuate one or more components of the robotic device along the determined motion path. The processing device 122 may then process or provide the orientation and position information of the instrument H to the robotic device 140 so as to enable the robotic device 140 to perform the necessary movements along the determined portion of the motion path 704.

[0119] Alternatively, method 500 may include the following steps: displaying a portion of the motion path on the three-dimensional image. In one example, processing device 122 may include computer executable instructions configured to perform a segmentation step. As used herein, "segmentation" describes the process of identifying individual vertebrae within three-dimensional image data so that the vertebrae can be separated and processed, manipulated, and displayed differently from each other. The segmentation step may employ a segmentation algorithm that uses imaging processing and image recognition software to automate the spinal segment segmentation process. In one embodiment, computer executable instructions automatically identify and extract spinal curves, detect and identify individual vertebrae until they are segmented from each other. One or more adaptive grids may be applied to generate a segmented three-dimensional model of the spine. Individual vertebrae or other anatomical features may be individually colored to visually enhance the bone-soft tissue interface, or only the edges may be colored.

[0120] Reference Figure 8 , Figure 8 An example three-dimensional image 800 is illustrated, which includes a vertebra 802 segmented from other vertebrae and a portion of a motion path 804. In one example, a surgeon can position an instrument H at a pedicle target site as described herein and view the three-dimensional image 800 to determine whether the trajectory of the portion of the motion path 804 is aligned with the surgical plan. In this example, based on the surgeon viewing the portion of the motion path 804 as shown and accepting the corresponding trajectory for forming a guide hole, the surgeon can provide input via the processing device 122 to determine one or more instructions using the motion path associated with the image 800 that actuate one or more components of the robotic device along the determined motion path. The processing device 122 can then process or provide the orientation and position information of the instrument H to the robotic device 140 so as to enable the robotic device 140 to perform the necessary movements along the determined portion of the motion path 804.

[0121] Return to reference Figure 5 , as shown in box 514, method 500 also includes the following steps: determining one or more instructions for actuating the one or more components along the determined motion path. In one example, a sequence of robot joint parameters (including joint angles, velocities and / or accelerations) of the robot can be determined, which position and orientation of the end effector of the robot over time to position the robot tool in space. In one example, a direct mapping between the joint parameters of the robot and the position of the robot tool at a specific time stamp can be determined. In one example, instead of a direct mapping, the robot joint parameters can be determined to approximate the position of the robot tool. In one embodiment, the robot movement can be modified to smooth the motion curve of the robot and / or the robot tool to avoid jittery or disconnected movements when the robot is moving the robot tool. In one example, the position within the motion path of the robot end effector can be scaled to account for the size difference between the instrument H and the robot end effector.

[0122] In one example, the one or more instructions for actuating the one or more components along the determined motion path may be refined before providing the one or more instructions to the robot. For example, the motion path of the robot may be enlarged or reduced in size relative to the determined motion path of the instrument H.

[0123] As shown in block 516 , the method 500 further includes the step of providing the one or more instructions to the robotic device. In one example, the one or more instructions may be provided via a wireless or wired communication interface between the processing device 122 and the robotic device 140 .

[0124] Reference Figure 6 , the example method 600 during a surgical procedure may include one or more operations, functions, or actions as shown in blocks 602 to 616. In one embodiment, the method 600 is performed in whole or in part by Figure 1 The system 100 is implemented as follows.

[0125] As shown in block 602 , method 600 includes the steps of acquiring a high-resolution baseline image of a surgical site of a patient and digitally processing the high-resolution baseline image to generate a baseline image set. Block 602 may be similar in function to block 502 of method 500 .

[0126] As shown in block 604 , method 600 further includes the steps of acquiring new images of the surgical site at a lower resolution and comparing the new images to one or more images of the baseline image set. Block 604 may be similar in function to block 504 of method 500 .

[0127] As shown in block 606 , method 600 further includes the steps of selecting a representative image in the baseline image set that has an acceptable correlation with the new image and merging the selected representative image with the new image to generate a merged image. Block 606 may be similar in function to block 506 of method 500 .

[0128] As shown in block 608 , method 600 further includes the step of capturing a pose of the instrument based on the user-defined placement of the instrument at the surgical site of the patient. Block 608 may be similar in function to block 508 of method 500 .

[0129] As shown in block 610 , method 600 further includes the step of determining a motion path corresponding to the captured pose of the apparatus, wherein the motion path is associated with actuation of one or more components of the robotic device. Block 610 may be similar in function to block 510 of method 500 .

[0130] Block 610 may also or instead include comparing the position of the one or more infrared reflectors of the portable instrument with the position of the one or more infrared reflectors coupled to the patient. Block 610 may also include determining, based on the comparison, that the distance between the one or more infrared reflectors of the portable instrument and the one or more infrared reflectors coupled to the patient is within a safety threshold. Based on the determination, block 610 may also include determining one or more adjustments to the actuation of the one or more components of the robotic device.

[0131] As shown in block 612, method 600 further includes the step of displaying a portion of the motion path on the merged image. Block 612 may be similar in function to block 512 of method 500. Alternatively, the portion of the motion path may be displayed on the three-dimensional image of the vertebra.

[0132] As indicated at block 614 , the method 600 further includes the step of determining one or more instructions for actuating the one or more components along the determined motion path. Block 614 may be similar in function to block 514 of method 500 .

[0133] As shown in block 616 , method 600 further includes the step of providing the one or more instructions to the robotic device. Block 616 may be similar in function to block 516 of method 500 .

[0134] Reference Fig. 9 , the example method 900 during a surgical procedure may include one or more operations, functions, or actions as shown in blocks 902 to 912. In one embodiment, the method 900 is performed in whole or in part by Figure 1 The system 100 is implemented as follows.

[0135] As shown in box 902, method 900 includes the following steps: acquiring an image of the surgical site of the patient. In one example, the image of the surgical site can be a two-dimensional image. In another example, before the surgical procedure, a three-dimensional image dataset of the patient's anatomical structure is loaded into the system. The image dataset can be a preoperative CT scan, a preoperative MRI, or an intraoperative three-dimensional image dataset acquired from an intraoperative imager. In one scenario, the three-dimensional image dataset is uploaded to the image processing device 122 and converted into a series of DRRs to approximate all possible two-dimensional C-arm images that can be acquired, thereby serving as a baseline image for comparing and matching the intraoperative two-dimensional image.

[0136] As shown in block 904 , method 900 further includes the step of capturing motion data of the instrument based on the user-defined path to the surgical site. Block 904 may be similar in function to block 508 of method 500 .

[0137] As shown in block 906 , method 900 further includes the step of determining a motion path corresponding to the captured motion data, wherein the motion path is associated with actuation of one or more components of the robotic device. Block 906 may be similar in function to block 510 of method 500 .

[0138] As shown in block 908 , method 900 further includes the step of displaying a portion of the motion path onto the merged image. Block 908 may be similar in function to block 512 of method 500 .

[0139] As shown in block 910 , the method 900 further includes the step of determining one or more instructions for actuating the one or more components along the determined motion path. Block 910 may be similar in function to block 514 of method 500 .

[0140] As shown in block 912 , method 900 further includes the step of providing the one or more instructions to the robotic device. Block 912 may be similar in function to block 516 of method 500 .

[0141] Reference Fig.10 , the example method 1000 during a surgical procedure may include one or more operations, functions, or actions as shown in blocks 1002 to 1012. In one embodiment, the method 900 is performed in whole or in part by Figure 1 The system 100 is implemented as follows.

[0142] As shown in block 1002 , method 1000 includes the following steps: acquiring an image of a surgical site of a patient. Block 1002 may be similar in function to block 902 of method 900 .

[0143] As shown in block 1004 , method 1000 also includes the step of capturing a pose of the instrument based on a user-defined placement of the instrument at the surgical site of the patient. Block 1004 may be similar in function to block 508 of method 500 .

[0144] As shown in block 1006 , method 1000 further includes the step of determining a motion path corresponding to the captured pose of the apparatus, wherein the motion path is associated with actuation of one or more components of the robotic device. Block 1006 may be similar in function to block 510 of method 500 .

[0145] As shown in block 1008 , method 1000 further includes the step of displaying a portion of the motion path onto the merged image. Block 1008 may be similar in function to block 512 of method 500 .

[0146] As shown in block 1010 , the method 1000 further includes the step of determining one or more instructions for actuating the one or more components along the determined motion path. Block 1010 may be similar in function to block 514 of the method 500 .

[0147] As shown in block 1012 , the method 1000 further includes the step of providing the one or more instructions to the robotic device. Block 1012 may be similar in function to block 516 of the method 500 .

[0148] It should be understood that the arrangement described herein is only for illustrative purposes. Therefore, those skilled in the art will appreciate that other arrangements and other elements (e.g., machines, interfaces, functions, orders, and functional groupings, etc.) can be used instead, and some elements can be completely omitted according to the desired results. In addition, many elements in the described elements are functional entities that can be implemented as discrete or distributed components or in combination with other components in any suitable combination and positioning, or other structural elements described as independent structures can be combined.

Claims

1. A system for a surgical instrument, the system comprising: Tracking devices; a portable instrument comprising one or more infrared reflectors; Robotic devices; as well as A processing device, the processing device comprising: Processor; and A non-transitory computer-readable medium having instructions stored thereon, which, when executed by the processor, cause the system to perform the following operations: receiving images of the patient's surgical site; capturing, via the tracking device, a pose of a portable instrument based on a user-defined placement of the portable instrument at a surgical site of a patient; determining, by the processor, a motion path corresponding to the captured pose of the portable instrument, wherein the motion path is associated with actuation of one or more components of the robotic device, wherein determining the motion path comprises: comparing the position of one or more infrared reflectors of the portable instrument to the position of one or more infrared reflectors coupled to the patient; Based on the comparison, determining whether a distance between the one or more infrared reflectors of the portable instrument and the one or more infrared reflectors coupled to the patient is within a safety threshold; and determining one or more adjustments to the actuation of the one or more components of the robotic device based on the determination; providing instructions for displaying a portion of the motion path on the image; determining, by the processor, one or more instructions for actuating the one or more components along the determined motion path; and The one or more instructions are provided to the robotic device.

2. The system according to claim 1, wherein: The user defined arrangement is based on movement of the portable instrument.

3. The system according to claim 1, wherein: The image of the surgical site is a three-dimensional image.

4. The system according to claim 3, wherein: The non-transitory computer readable medium has instructions stored thereon, which when executed by the processor, cause the system to provide the one or more instructions to the robotic device, the instructions further comprising instructions for: An input is received confirming that at least a portion of the superimposed determined motion path is aligned with the surgical procedure plan.

5. The system according to claim 3, wherein: The non-transitory computer readable medium has instructions stored thereon, which when executed by the processor, further cause the system to perform the following operations: determining that at least a portion of the superimposed determined motion paths intersects one or more predetermined boundaries within the surgical site; and A visual effect of a change of at least a portion of the superimposed determined motion path is provided for display via a display.

Images