Trajectory guidance of a handheld surgical tool during a surgical procedure

Abstract

A surgical system for operating on a patient's bone is described. The surgical system can include a surgical instrument having a camera and / or an inertial measurement unit. The surgical system can further include a control system for overlaying a virtual representation onto a medical image based on an output of one of the camera and the inertial measurement unit. The virtual representation or an alternative virtual representation can be further based on an output of a depth sensor. In a particular case, the surgical instrument can have a control system configured to determine the length of an end effector based on a signal from the camera. The present disclosure can also have a control system that acquires a plurality of 2D X-ray images and selects one or more of the 2D X-ray images based on one or more criteria such as a radial distance, an image boundary, an output of a depth sensor, a relative orientation, etc. The control system can control a display for displaying the selected 2D image.

Description

Background Art

[0001] In conventional medical and surgical procedures, it is routine to use surgical tools and instruments that enable a surgeon to access and manipulate the surgical site. As a non-limiting example, in connection with orthopedic procedures, rotary instruments such as hand-held drills are commonly utilized to address various musculoskeletal disorders such as trauma, sports injuries, degenerative diseases, and joint reconstructions.

[0002] In procedures where a hand-held drill or similar surgical instrument is used, rotational torque selectively generated by an actuator (e.g., an electric motor) is used to rotate a releasably attachable end effector such as a drill bit or other surgical attachment at different speeds. The surgical handpiece assembly creates a hole for applying the end effector to the bone.

[0003] One type of orthopedic procedure is the open reduction internal fixation (ORIF) procedure. During an ORIF procedure, a surgeon realigns a broken bone and secures the bone in place with one or more surgical implants. The one or more surgical implants can include bone plates and screws. The screws hold the bone plate in place relative to the bone. After a period of time has elapsed and it is determined that the bone has healed, the bone plate and screws can be removed.

[0004] Another type of procedure involves the placement of an intramedullary nail. In this procedure, it is necessary to secure set screws in screw holes within the intramedullary nail.

[0005] A C-arm or mobile intensifier device is an example of a medical imaging device based on X-ray technology that can be used during the surgical procedures described above. A mobile intensifier device can perform fluoroscopy, a type of medical imaging that shows continuous X-ray images on a monitor. During a fluoroscopy procedure, an X-ray source or transmitter emits X-rays that pass through the patient's body. An X-ray detector or image intensifier converts the X-rays passing through the body into a visible image that is displayed on the monitor of the medical imaging device. Medical imaging devices such as C-arm devices can display high-resolution X-ray images in real time, so that a physician can monitor the progress at any time during the surgery and thus take appropriate measures based on the displayed images. However, during certain procedures, for example, during procedures where attention must be paid to the patient's anatomical structure as well as the display of the medical imaging device, it is often difficult to monitor the images. Furthermore, it can be difficult and time-consuming to adjust the C-arm for the patient in order to identify the location of a fracture site or the screw holes within a wall nail.

[0006] A solution to the above problems is desired. SUMMARY OF THE INVENTION

[0007] One general aspect includes a surgical system for operating on a patient's bone. The surgical system also includes a surgical instrument for coupling to an end effector, the surgical instrument including a handpiece for driving the end effector, a camera configured to generate a first signal corresponding to the pose of an optically detectable fiducial relative to a first coordinate system, and a sensor module configured to have a fixed pose relative to the camera and generate a second signal regarding orientation data and / or motion parameters relative to a second coordinate system. The system also includes a control system configured to establish an alignment between the first coordinate system, the second coordinate system, and an image coordinate system based on the first signal, the second signal, and an image of a reference device including a radiopaque fiducial and an optically detectable fiducial. The control system can be configured to determine a starting position of the end effector relative to a bone in the image in one of the first coordinate system, the second coordinate system, and the image coordinate system based on i) the first signal and ii) one of the second signals, and the alignment. The control system can also be configured to determine a second position of the end effector relative to the bone in the image in one of the first coordinate system, the second coordinate system, and the image coordinate system based on the second signal and the starting position, and overlay a virtual representation onto the image based on the starting position and the second position of the end effector. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform method and / or system acts.

[0008] One general aspect includes a surgical system for operating on a patient's bone. The surgical system also includes a surgical instrument for coupling to an end effector, the surgical instrument including a handpiece for driving the end effector, a camera, and a sensor module configured to have a fixed orientation relative to the camera and to generate a first signal regarding orientation data and / or motion parameters. The surgical instrument can further include a depth sensor configured to provide a second signal associated with displacement of the end effector or a second effector during a drilling, tapping, or driving process. The system also includes a control system configured to determine a starting position of the end effector relative to the bone in an x-ray image based on the second signal, to determine a second position of the end effector relative to the bone in the image based on the second signal and the starting position, and to overlay a virtual representation over the image based on the starting position, the second position of the end effector, and the second signal. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the method and / or system acts.

[0009] One general aspect includes a surgical system for operating on a patient's bone. The surgical system includes a surgical instrument for coupling to an end effector, the surgical instrument including a sensor module configured to generate a first signal regarding the orientation of the surgical instrument, the instrument having a tool axis. The system can further include a control system and a display. The control system can acquire a plurality of 2D X-ray images, each of the 2D X-ray images having an image reference axis, align the plurality of 2D X-ray images to a known coordinate system, align the instrument to the known coordinate system, select a 2D X-ray image from the plurality of 2D X-ray images based on the image reference axis and the tool axis, and be configured to overlay a virtual representation of the end effector on the selected 2D X-ray image on the display, the virtual representation being based on the first signal. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the methods and system acts.

[0010] One general aspect includes a surgical system for operating on a patient's bone. The system can include a surgical instrument for coupling to an end effector. The surgical instrument includes a handpiece for driving the surgical end effector and a sensor module configured to generate a first signal regarding the orientation of the surgical instrument, and the instrument defines a tool axis. The sensor module can include a camera or an inertial measurement unit or both. The system can further include a display and a control system. The control system acquires a plurality of 2D x-ray images, each of the 2D x-ray images having an image boundary, aligns the plurality of 2D x-ray images to a known coordinate system, aligns the instrument to the known coordinate system, determines a start position of a planned trajectory in the known coordinate system based on the first signal, selects a 2D x-ray image from the plurality of 2D x-ray images based on the image boundary and the start position of each of the plurality of 2D x-ray images, and can be configured to overlay a virtual representation of the end effector over the selected 2D x-ray image on the display, the virtual representation being based on the first signal. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the methods and system acts.

[0011] Another general aspect includes a surgical system for operating on a patient's bone. The system can include a surgical instrument for attachment to an end effector, the surgical instrument including a handpiece for driving the surgical end effector and a sensor module configured to generate a first signal regarding the orientation of the surgical instrument, such as that derived from a camera or an inertial measurement unit. The system also includes a display and a control system. The control system acquires a plurality of 2D X-ray images, each of the 2D X-ray images having an image boundary, the image coordinate system including an image reference plane and image reference axes, aligns the instrument with the image coordinate system, determines a starting position of a planned trajectory in the image reference plane of each of the plurality of 2D X-ray images based on the first signal, determines a planned end position of the planned trajectory in the image reference plane of each of the plurality of 2D X-ray images based on the starting position, determines a radial distance between the starting position and the planned end position, selects a 2D X-ray image from the plurality of 2D X-ray images based on the radial distance, and is configured to overlay a virtual representation of the end effector over the selected 2D X-ray image on the display, the virtual representation being based on the first signal. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the method and system acts.

[0012] Yet another general aspect includes a surgical system for operating on a patient's bone. The system includes a surgical instrument for coupling to an end effector, the surgical instrument including a handpiece for driving the surgical end effector and a sensor module configured to generate a first signal regarding the orientation of the surgical instrument. The system can further include a display and a control system. The control system can be configured to align an X-ray image to a known coordinate system. The control system can also align the instrument to the known coordinate system, determine a starting position of a planned trajectory in the known coordinate system based on the first signal, determine a planned end position of the end effector in the known coordinate system based on the starting position, select a field of view of the aligned X-ray image based on the starting position, the planned end position, and the first signal, and be configured to overlay a virtual representation of the end effector on the X-ray image on the display, the virtual representation being based on the first signal. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the method and system acts.

[0013] In another general aspect, a surgical system for operating on a patient's bone is contemplated. The system includes a surgical instrument for coupling to an end effector, the surgical instrument including a handpiece for driving the surgical end effector and a sensor module configured to generate a first signal regarding the orientation of the surgical instrument, the instrument having a tool axis. The sensor module can further include a depth sensor configured to provide a second signal associated with displacement of the end effector during a drilling or driving or insertion process. The system can further include a display and a control system. The control system can be configured to align an x-ray image to a known coordinate system, align the instrument to the known coordinate system, and select a field of view of the aligned x-ray image based on the second signal. The control system can also be configured to overlay a virtual representation of the end effector over the x-ray image on the display, the virtual representation being based on the first signal. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the methods and system acts.

[0014] In another general aspect, a surgical system for operating on a patient's bone can include a surgical instrument for coupling to an end effector, the surgical instrument

[0015] including a handpiece for driving the surgical end effector and a sensor module configured to generate a first signal regarding the orientation of the surgical instrument. The sensor module can include an inertial measurement unit, a camera, a gyroscope, or a combination thereof. The system can further include a display and a control system. The control system can be configured to align a plurality of x-ray images to a known coordinate system, align the instrument to the known coordinate system, and select one of the plurality of aligned x-ray images based on a duration between a time at which each of the plurality of aligned x-ray images was acquired and the current time, and display the selected x-ray image.

[0016] In yet another aspect, a surgical system for operating on a patient's bone is contemplated. The system can include a surgical instrument for coupling to an end effector, the surgical instrument including a handpiece for driving the surgical end effector and a sensor module configured to generate a first signal regarding the orientation of the surgical instrument. The system can further include a display and a control system. The control system can be configured to align a plurality of X-ray images to a known coordinate system, align the instrument to the known coordinate system, select one of the plurality of aligned X-ray images based on the time at which each image was acquired and the current time, and display the selected X-ray image on the display.

[0017] In another aspect of the present disclosure, a first surgical attachment configured to removably couple to a surgical handpiece assembly or a component thereof is contemplated. The first surgical attachment can include a measurement housing, a camera, an inertial measurement unit configured to generate a signal regarding orientation data and / or motion parameters, and a coupler.

[0018] In yet another aspect of the present disclosure, a surgical system for operating on a patient's bone is contemplated. This system can include a surgical instrument for coupling to an end effector. The surgical system can include a handpiece for driving the surgical end effector and a camera coupled to the handpiece, the camera being configured to generate a first signal. The system can further include a control system configured to determine an end effector length based on the first signal.

[0019] In another aspect of the present disclosure, a surgical attachment configured to be removably coupled to a surgical handpiece assembly or a component thereof is contemplated. The surgical attachment can include a measurement housing, a camera, an inertial measurement unit configured to generate signals regarding orientation data and / or motion parameters, and a measurement coupler for attachment to a surgical handpiece, the surgical attachment, or a second surgical attachment.

[0020] In another aspect, a method of using a reference device to align a tool axis with an X-ray image is contemplated. A reference device can be provided that includes one or more radiopaque fiducials and optically detectable fiducials. The method can include using an attachment element to secure the reference device to a patient's bone, imaging the reference device with an imager after the reference device is secured to the patient's bone or tissue, positioning an instrument having a camera and an inertial measurement unit such that the optically detectable fiducials are within the camera's field of view, aligning the inertial measurement unit with the camera based on the output signal of the camera, and displaying a virtual representation of a portion of the instrument based on the output signal of the inertial measurement unit.

[0021] In yet another aspect, a different surgical system is contemplated. In this aspect, the system includes a reference device that includes one or more radiopaque fiducials, the reference device being configured to have a fixed orientation with respect to a surgical implant or a portion of a patient's anatomical structure, the reference device including optically detectable fiducials and radiopaque fiducials. The system can further include a surgical instrument for coupling to an end effector, the surgical instrument including a camera configured to generate a first signal corresponding to the orientation of the optically detectable fiducials, a sensor module configured to have a fixed orientation with respect to the camera and configured to generate a second signal regarding orientation data and / or motion parameters, and an antenna for communicating with the imager.

[0022] In another aspect of the system, a reference device is contemplated for aligning the tool axis with the x-ray image. The reference device can include one or more radiopaque and optically detectable fiducials. The reference device can further include an attachment element for coupling to a patient.

[0023] A system of one or more computers can be configured to perform particular operations or actions by installing in the system software, firmware, hardware, or a combination thereof that causes the system to perform the operations or actions during surgery. One or more computer programs can be configured to perform particular operations or actions by including instructions that, when executed by a data processing apparatus, cause the apparatus to perform the operations or actions.

[0024] It should be understood that in certain instances, the sensor module is described as including a depth sensor, an inertial measurement unit, and / or a camera. In other instances, the sensor module is described as including only an inertial measurement unit. Whenever reference is made to the sensor module in the claims, it is contemplated that the inertial measurement unit, camera, and / or depth sensor can be included.

[0025] In many of the claims, and for many of the features, the drill bit is described as an end effector, but it is clearly contemplated that this drill bit can be replaced with other bits for other surgical uses such as a screwdriver bit. Further, it should be understood that the tool identification or length function described with respect to the drill bit can also be applicable to any other end effector, such as for determining the length of a screwdriver bit. Further, a depth detection function and a measurement cannula can be used with any end effector, such as a screwdriver bit. In such a configuration, the screwdriver bit can pass through the measurement cannula, similar to the drill bit.

[0026] Any reference to the control system should be understood to encompass any suitable arrangement of the processor and communication hardware, whether the processor is associated with a mobile computing device separate from the instrument, the processor is mounted on one or more surgical attachments, the processor is mounted on one of the removable drivers, or the processor is mounted on the surgical handpiece, or the processor is mounted on a surgical hub separate from the instrument, or any combination thereof. It is intended that any step contemplated for one of the described processors be performed by any one or combination of the other described processors.

[0027] A specific example is described in detail of selecting among multiple 2D X-ray images based on various considerations, but it should be clearly understood that the control system is also contemplated to be configured to select among multiple different types of 2D images other than X-rays. Further, it is contemplated that the control system can also be configured to select from multiple 3D images using similar techniques. Further, it is contemplated that selections can also be made from multiple digitally reconstructed radiographs using the techniques described herein. Thus, any instance of multiple 2D images can be replaced with multiple 3D images, multiple medical images, or multiple digitally reconstructed radiographs.

[0028] It should be further understood that the system may be operable without using optically detectable references. In such instances, the system can use a camera to localize the instrument relative to known features of the anatomical structure. The features of the anatomical structure can be known landmarks recognized by the camera.

[0029] This disclosure will be understood in more detail from the detailed description and the accompanying drawings.

Brief Description of the Drawings

[0030]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11A

Figure 11B

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16A

Figure 16B

Figure 16C

Figure 17

Figure 18

Figure 19A

Figure 19B

Figure 19C

Figure 19D

Figure 20A

Figure 20B

Figure 20C

Figure 20D

Figure 21A

Figure 21B

Figure 21C

Figure 22A

Figure 22B

Figure 22C

Figure 23A

Figure 23B

Figure 23C

Figure 24

Figure 25A

Figure 25B

Figure 25C

Figure 26

Figure 27A

Figure 27B

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34A

Figure 34B

Figure 35

DETAILED DESCRIPTION OF THE INVENTION

[0031] In the drawings, reference numbers can be repeatedly used to identify similar and / or identical elements.

[0032] Referring to FIGS. 1 and 2, an exemplary configuration of an operating room or surgical suite for performing a medical procedure on a patient using a surgical system 12 according to the teachings of the present disclosure is shown. The surgical system 12 can be used to perform various orthopedic procedures, such as ORIF procedures, to realign a broken bone 14 and secure the bone 14 in place with one or more surgical implants. The one or more surgical implants can include a bone plate 16 and screws that are inserted into the bone 14 until the bone 14 heals. The surgical system 12 includes a control system 18. The surgical system 12 can also be used to place screws or fasteners for other surgical procedures, such as placing set screws within intramedullary nails. The surgical system 12 can also be used to guide an end effector beyond drill bits, such as taps, screwdrivers, surgical wires, and / or pins. The surgical system 12 can also be used to guide the placement of implants other than plates and screws, such as wires and pins.

[0033] A display unit 20 is operably communicatively coupled to the control system 18. The display unit 20 can be integrated with an input device, such as when the display unit 20 is a touch screen. Alternatively, the display unit 20 can communicate with one or more input devices, such as a keyboard, a mouse, a microphone (voice activation), a gesture control device. The display unit 20 can be a tablet computer, a mobile phone, a mixed reality device, such as a headset or glasses or another suitable mobile device. The display unit 20 can also be integrated with a surgical instrument.

[0034] The surgical system 12 can also include an imaging system 22, such as a C-arm X-ray imaging device 32. It is contemplated that the system can be adapted to other suitable imaging systems, such as CT or MRI imaging devices. The imaging system 22 can include a detector 24, a light source 26, an imaging computer 28, an imaging display 30, and one or more user input devices. The detector 24 and the light source 26 are configured to generate one or more medical images. The detector 24 and the light source 26 can be disposed at opposite ends of the C-arm 32. The light source 26 can include any light source used in diagnostic medical imaging that emits or generates X-rays, such as a solid-state X-ray emission source, a rotating anode X-ray source, a stationary or fixed anode X-ray source, a standard X-ray source, a solid-state X-ray emission source, and / or a fluoroscopic X-ray source, as well as a stationary or fixed anode X-ray source. The detector 24 can include an image intensifier or any other energy receiver used in diagnostic medical imaging. In some cases, the imaging device is a C-arm capable of performing only 2D X-rays.

[0035] The C-arm 32 including the detector 24 and the light source 26 can be configured to rotate around the patient 10 to provide an image of the surgical site. The imaging computer 28 can be connected to one or more user input devices, including a keyboard, a mouse, and other suitable devices, that enable a user to provide input to the imaging computer 28. The imaging computer 28 and / or the control system 18 can include software known to those skilled in the art that is capable of acquiring the images captured by the imaging system 22 and creating one or more 2D images and / or one or more 3D models of the surgical site. The imaging display 30 can be configured to display the resulting 2D images and / or 3D models.

[0036] Images from an imaging system 22, such as a C-arm x-ray imaging device, are often distorted (i.e., warped), and thus not all objects within the image are shown at the same magnification. This is due to the fact that the x-ray beam is not perfectly linear. In many cases, objects closer to the light source 26 appear larger (contain more pixels). Objects farther from the light source 26 may appear smaller (contain fewer pixels). To make accurate measurements, it is necessary to remove the distortion of these images.

[0037] Referring to FIGS. 3 and 4, the surgical system 12 can also include a fluoro disk 34 to provide a distortion removal function. For the fluoro disk 34, the content is discussed in FluoroMap (trademark) Adaptive Positioning Technology For Gamma3 (trademark) System - User Manual, which is incorporated herein by reference in its entirety (https: / / www.strykermeded.com / media / 2325 / gamma3 - adapt - fluoromap.pdf). The fluoro disk 34 is also discussed in U.S. Patent Publication No. 2018 / 0140309A1, titled "Method And Apparatus For Treating A Joint, Including The Treatment Of Cam - Type Femoroacetabular Impingement In A Hip Joint And Pincer - Type Femoroacetabular Impingement In A Hip Joint", filed on November 18, 2016, the content of which is incorporated herein by reference in its entirety. In addition to the functions discussed herein, the imaging computer 28 and / or the control system 18 can also implement one or more of the systems, methods, and / or algorithms discussed in the aforementioned references.

[0038] The fluorodisk 34 can include a transparent lens 36, a plurality of fiducial markers 38 disposed on the transparent lens 36, and an attachment member 40 configured to attach the fluorodisk 34 to the detector 24 of the imaging system 22. The fluorodisk 34 can have any suitable dimensions (more specifically, the transparent lens 36), such as a 9-inch diameter, a 12-inch diameter, etc. The attachment member 40 can include a combination of a belt, one or more hooks, and one or more loops. The plurality of fiducial markers 38 have known dimensions. The fluorodisk 34 can be configured to provide a distortion removal function by projecting a pattern of the plurality of fiducial markers 38 onto an image. FIG. 4 shows a virtual representation of the plurality of fiducial markers 38 projected onto the bone 14. Although an example is provided where the imaging system 22 includes the fluorodisk 34, in some implementations, such as when the imaging system 22 includes a flat panel detector, the imaging system 22 can automatically provide an undistorted image and thus the fluorodisk 34 can be omitted. Based on the pattern projected onto the image, the imaging computer 28 and / or the control system 18 can calibrate an appropriate pixel size based on the undistorted image.

[0039] In some implementations, the imaging computer 28 and / or the control system 18 can use the known dimensions of the plurality of fiducial markers 38 for various other calculations. For example, the distance between the patient 10 and the light source 26 can vary between images, and thus the magnification or scale factor can be different from one image to the next. Accordingly, the imaging computer 28 and / or the control system 18 can use the known dimensions of the plurality of fiducial markers 38 and / or the radiation-opaque reference on the reference device 200 to determine the magnification or scale factor for each of the images. The imaging computer 28 and / or the control system 18 can also use the known dimensions of the plurality of fiducial markers 38 to determine the orientation of the C-arm 32 with respect to the plurality of fiducial markers 38.

[0040] The imaging computer 28 can communicate with the control system 18. The imaging computer 28 can be configured to communicate (i.e., interface) with the surgical system 12 via a wired and / or wireless connection. For example, the imaging system 22 can be configured to provide images such as the resulting 2D images and / or 3D models of the surgical site to the control system 18. In some implementations, the C-arm 32 is a “closed system” and the control system 18 communicates with the C-arm 32 only via a video / image output port. For example, the control system 18 can communicate with the C-arm 32 by receiving an image from the C-arm 32 when the C-arm 32 transmits the image to the video / image output port. In either case, the control system 18 and / or the C-arm 32 can be configured to provide the resulting 2D images and / or 3D models to the display 20, where the surgeon or other medical personnel can interact with the images to identify and / or demarcate the corresponding regions and / or intervals around the bone 14.

[0041] For example, the surgeon can select a plurality of views of the bone 14 to be displayed, can define a desired trajectory for the end effector 102 of the surgical instrument 100, and / or can select an appropriate user interface for the display unit 20. A light source 26 and a detector 24 connected to the imaging computer 28 are shown, but in some configurations, the X-ray imaging device including the detector 24 and the light source 26 can be directly connected to the control system 18, thereby eliminating the need for the imaging computer 28, the imaging display 30, and the user input device. The control system 18 can be configured to perform all the same functions as the imaging computer 28.

[0042] The surgical system 12 can include a surgical instrument 100. A surgeon can use the surgical instrument 100 to perform surgery on a patient 10, including making a hole in the bone 14 of the patient 10 and / or inserting a screw through one or more openings 17 of a bone plate 16 into the drilled hole in the bone 14. Conventional surgical systems generally use an optical tracking system or an electromagnetic tracking system or a combination thereof to track the position of the end effector 102 of the surgical instrument 100 relative to the bone 14 of the patient 10. Each such system has its own drawbacks, and either system can be time-consuming to set up and expensive. A particular drawback of optical tracking systems is the need for a line of sight between the optical tracking device and the position tracking unit (i.e., the camera), which can be difficult to maintain in an operating room. A particular drawback of electromagnetic tracking systems is that they are not as accurate as optical tracking systems and are susceptible to distortion from nearby metal objects. In either system, the surgeon must perform a manual alignment process, which can also be time-consuming.

[0043] During ORIF and other trauma procedures, a surgeon may not be able to use a conventional guidance tracking system due to the drawbacks described above. Instead, the surgeon may rely on his or her procedural skills and anatomical knowledge to align the trajectory of the end effector 102 of the surgical instrument 100 to create a pilot hole for one or more screws (or other surgical implants). Similarly, during screw placement, the surgeon inserts the screw freehand. The surgeon then typically uses fluoroscopy to verify the screw placement. If the surgeon is not satisfied with the screw placement, the surgeon may replace the screw. This can cause additional damage to the bone 14 because the surgeon must make additional holes in the bone 14. Also, if the screw is improperly placed, the screw may be damaged, and re-drilling and re-verification can be time-consuming, costly, and wasteful.

[0044] Referring further to FIGS. 5-8, one exemplary surgical instrument 100 is shown. Surgical instrument 100 obviates the need for a second device such as a depth gauge for determining the depth of a perforation. Surgical instrument 100 can comprise an attachment 104 and an end effector 102 such as a drill bit. Aspects of surgical instrument 100 are discussed in International Patent Publication No. WO2017 / 0407172, titled "Powered Surgical Drill With Integral Depth Gauge That Includes A Probe That Slides Over A Drill Bit", filed on September 1, 2016, and International Patent Publication No. WO2019 / 035096A1, titled "Surgical Handpiece For Measuring Depth Of Bore Holes And Relates Accessories", filed on August 17, 2017, which are hereby incorporated by reference in their entirety.

[0045] Surgical instrument 100 can include a housing 106, a motor 108, an instrument controller 110, a trigger 112, a battery 114, and other components, as will be described in more detail below. Trigger 112 communicates with instrument controller 110 in response to actuation by a user (e.g., a surgeon), via, for example, an electrical signal provided by a magnet and a Hall effect sensor. Thus, when a surgeon actuates trigger 112 to operate surgical instrument 100, instrument controller 110 directs power from the battery to motor 108, and motor 108 generates rotational torque that is used to rotate end effector 102 or other surgical end effector 102, as will be described in more detail below.

[0046] Instrument 100 can also measure tool drive parameters, which are characteristics of the tool portion acting on patient 10. Tool drive parameters can include the speed or torque of the motor driving the tool. A tool drive signal can be used to confirm the position of end effector 102 relative to the anatomical structure.

[0047] Figures 6 and 7 show the motor 108 positioned along the motor shaft 109 within the housing 106, although other motor positions are contemplated. The motor 108 can be electric, pneumatic, ultrasonic, or hydraulic. The motor 108 can be configured to selectively generate rotational torque in response to commands, signals, etc. received from the appliance controller 110. The motor 108 can include a rotor cannula 116 supported for rotation about the shaft 109 by a pair of bearings 118. A drive gear disposed adjacent to the gear set is coupled to the rotor cannula 116 and rotates simultaneously with the rotor cannula 116 and is used to transmit rotational torque to the gear set.

[0048] In the example shown, the attachment 104 is removably coupled to the housing 106. However, the attachment 104 can also be integrally formed with the housing 106, and thus the components of the attachment 104 described throughout this specification can be part of the housing 106 or separate from the housing 106. The attachment 104 can comprise a separate housing such as the attachment housing 120. The attachment 104 can be constructed to minimize obstruction of the surgeon's view of the surgical site. The attachment 104 can further comprise a displacement sensor and a depth sensor 122 shown as the depth measurement extension 124. The depth sensor 122 is operably connected to the depth measurement extension 124. As shown, the depth measurement extension 124 can be implemented as a cannula, i.e., a tube defining a lumen. More specifically, the depth measurement extension 124 can circumferentially surround the end effector 102 and be slidably attached to the housing 106 so as to extend forwardly and / or rearwardly between a fully distal position and a proximal position relative to the housing 106. In an alternative implementation, the depth sensor 122 can be integrated with a camera 146, i.e., the camera 146 can be capable of acquiring an infrared 3D scan of the bone 14 and determining the depth of the end effector 102 relative to the bone 14 / plate 16. As yet another alternative, the depth sensor 122 can be a laser time-of-flight depth sensor or an ultrasonic sensor.

[0049] The depth measurement extension 124 is disposed within the guide bushing 126 and supported to translate along the measurement axis 128. When the attachment 104 is attached to the surgical instrument 100, the measurement axis 128 is disposed to be coaxial with the axis 109. The depth measurement extension 124 further includes a plurality of rack teeth 130 linearly disposed along at least a portion of the length of the depth measurement extension 124, and the plurality of rack teeth 130 are disposed to mesh with a gear 132 disposed adjacent to the distal end of the guide bushing 126. As shown in FIG. 6, the window of the guide bushing 126 is disposed adjacent to the gear 132 such that the rotation of the gear 132 and the movement of the depth measurement extension 124 are directly proportional to facilitate the meshing between the rack teeth 130 and the gear 132. The depth sensor 122 can be implemented by a potentiometer, a rotary encoder, etc., in response to the rotation of the gear 132 resulting from the axial movement of the depth measurement extension 124 to generate an electrical signal representing the change in the position of the depth measurement extension 124 along the measurement axis 128.

[0050] As an example, in some configurations, the depth sensor 122 can be arranged to communicate with the instrument controller 110 or the attachment controller 134, and the instrument controller 110 or the attachment controller 134 can be configured to interrupt or adjust how the motor 108 is driven based on the movement of the depth measurement extension 124, for example, to decelerate or stop the rotation of the end effector 102 at a specific penetration depth into the tissue. The attachment 104 can include one or more power terminals as described in U.S. Patent No. 11,317,927, which is hereby incorporated by reference in its entirety. These power terminals enable the attachment 104 to be powered by the battery of the surgical instrument 100 to which the attachment 104 is coupled.

[0051] Referring further to FIG. 6, to ensure the proper functioning of the depth measurement extension 124 and the depth sensor 122, the depth measurement extension 124 can be biased towards the fully distal position. This biasing causes the distal end of the depth measurement extension 124 to always maintain contact with the proximal surface of the bone 14 to be drilled or the bone plate 16 in contact with the bone 14 to be drilled. This biasing is achieved by using a spring that biases the gear 132 to rotate in a direction to extend the depth measurement extension 124 distally from the attachment housing 120. However, other methods of biasing the depth measurement extension 124 relative to the surgical instrument 100 are contemplated. An exemplary spring arrangement can be seen in PCT / US2016 / 049899, which is hereby incorporated by reference in its entirety.

[0052] Referring to FIGS. 7 and 8, the attachment 104 can include an attachment connector 138 configured to operably connect to the instrument connector 140 of the surgical instrument 100. In one example, the surgical instrument 100 can provide a power connection to the attachment 104 only through the connection between the attachment connector 138 and the instrument connector 140. In another example, the attachment 104 and the surgical instrument 100 can also exchange data through the instrument connector 140 and the attachment connector 138. This data can include motor parameters such as the speed or torque of the motor. Further, in some cases, the attachment controller 134 can use this connection to provide speed / torque control commands to the instrument controller 110. The attachment controller 134 and the instrument controller 110 can communicate with the control system 18 or other devices via a wired connection (i.e., through the attachment connector 138 and the instrument connector 140) or a wireless connection. In cases where the attachment controller 134 communicates wirelessly with the imaging computer 28, the display unit 20, or the instrument controller 110, the attachment 104 can include an antenna 142 that communicates with the attachment controller 134. The instrument controller 110 and / or the attachment controller 134 can transmit data to a display unit 20 such as a tablet or an external server.

[0053] The attachment 104 can also include a display 144, such as a display screen, one or more light emitting diodes (LEDs), etc., to display information regarding the movement of the attachment 104 to the surgeon, such as real-time penetration depth, recorded maximum penetration depth history, screw length, breakthrough notification, visual information (e.g., graphical representation) regarding the current trajectory relative to a desired trajectory to guide the surgeon to align what is desired, etc. This same information can also be communicated to the user by a speaker to provide audio notifications such as real-time penetration depth, recorded maximum penetration depth history, breakthrough notification, etc.

[0054] The attachment controller 134 can also be configured to determine a breakthrough event based on the depth signal. The attachment controller 134 and the instrument controller 110 are shown as separate controllers disposed within the attachment housing 120 and the housing 106, respectively, but the functions of the attachment controller 134 and the instrument controller 110 may also be integrated into a single controller.

[0055] The surgical instrument 100 can also include a camera 146. In one exemplary configuration, the camera 146 can be included as part of the attachment 104. The attachment controller 134 can be operably connected to the camera 146. In such a configuration, the attachment 104 can be a disposable item that is discarded after completion of each procedure.

[0056] In another alternative implementation, the camera 146 can be removably attached to the attachment 104. Further, the camera 146 can be removably attached to the housing of the instrument 100 rather than being included as part of the attachment 104. In these cases, the camera 146 can include a dedicated camera controller that can communicate camera information to the instrument controller 110 and / or the attachment controller 134.

[0057] Although one camera 146 is shown, it is also contemplated that the surgical instrument 100 can include a plurality of cameras 146. This can improve the surgical instrument 100's ability to maintain a line of sight during perforation. The plurality of cameras 146 can be arranged around the surgical instrument 100 and / or the attachment 104 such that the collective observation area of the cameras 146 extends around all or substantially all of the surgical instrument 100. In this regard, as the surgical instrument 100 is brought closer to the patient 10, the camera 146 can more easily detect the position and orientation of the optically detectable reference 204.

[0058] It is contemplated that the instrument 100 can utilize a reference device 200 that includes optically detectable references, but it is also contemplated that alignment can be achieved by recognition of anatomical features visible from outside the patient 10, a specific shape of the reference device, a pattern, etc.

[0059] The surgical instrument 100 can also include a sensor module 148 configured to have a fixed orientation with respect to the camera 146. The sensor module 148 can be an inertial measurement unit (IMU). The IMU can include a gyroscope, an accelerometer, and / or a magnetometer. The IMU can be configured to measure with respect to any number of degrees of freedom. In one example, the IMU includes a gyroscope and an accelerometer, whereby the IMU can measure the orientation of the surgical instrument 100 in six degrees of freedom. The sensor module 148 includes a sensor coordinate system, and the IMU may be capable of determining the orientation of the sensor module 148 with respect to the sensor coordinate system. The IMU can utilize motion parameters such as acceleration data and / or angular velocity data to detect changes in the orientation of the sensor module 148 with respect to the sensor coordinate system.

[0060] The instrument 100 can be configured to be at least partially disposable. As shown in FIG. 7, the attachment 104 can be removably coupled to the housing 106 of the instrument 100, and the attachment 104 can be discarded after use or reprocessed (e.g., heat sterilized) and reused. In the illustrated implementation, the attachment 104 includes a depth sensor 122, a depth measurement extension 124, a camera 146, a sensor module 148, and a control element 150, and the attachment 104 is disposable. In some implementations, the attachment 104 can be heat sterilized and reused.

[0061] Apparatus 100 can be configured to incorporate at least some of the components 122, 124, 146, 148, 150 included in attachment 104. For example, in the illustrated implementation, apparatus 100 includes an attachment interface 107 removably coupled to housing 106. In this implementation, attachment interface 107 includes an apparatus connector 140, and attachment 104 is configured to be received such that apparatus connector 140 contacts attachment connector 138 to establish electrical communication between attachment 104 and attachment interface 107. Accordingly, attachment 104 can communicate with apparatus 100 and the rest of system 12 via attachment interface 107. Attachment interface 107 can include sensor module 148 (i.e., an IMU) and other more expensive components, and attachment 104 can include less expensive components such as depth sensor 122, depth measurement extension 124, camera 146, and / or control element 150. Attachment interface 107 can be heat sterilized and / or reprocessed according to any suitable method, and attachment 104 is generally disposable. Thus, a user can perform a procedure using apparatus 100, then disconnect attachment interface 107 from housing 106, disconnect attachment 104 from interface 107, wash and reuse attachment interface 107, discard attachment 104, couple a new attachment 104 to interface 107, and re-couple interface 107 to housing 106. Although attachment 104 and attachment interface 107 have been described as including particular components 122, 124, 146, 148, 150, it will be understood that attachment 104 and attachment interface 107 can each include any of the components associated with apparatus 100.

[0062] Referring to FIGS. 9 and 10, the surgical system 12 can further include a reference device 200. The reference device 200 can include one or more radiopaque fiducials 202 and one or more optically detectable fiducials 204. The reference device 200 can be attached to an implant or a portion of the patient's anatomy, or can be attached to a component attached to an implant or a portion of the patient's anatomy, such as a pin used to temporarily attach the bone plate 16 to the bone 14, or can be attached using an adhesive. In procedures where it may be necessary to place multiple plates / implants / holes, the surgical system 12 can include multiple reference devices 200, with one reference device 200 coupled adjacent to each surgical site. In this situation, each reference device 200 can be uniquely identified in the x-ray image, and each optically detectable fiducial 204 can be uniquely identified by the camera 146.

[0063] In addition, in certain cases, such as when a longer plate is being implanted, multiple reference devices 200 can be used for a single surgical site. These different reference devices 200 can have different radiopaque elements and different optically detectable fiducials 204 in different spatial arrangements, such as different April Tags.

[0064] The reference device 200 defines a reference axis. The reference axis defines a reference coordinate system. Further, the reference axis is used herein to refer to the reference coordinate system. The reference device 200 is configured to be attached to the bone 14 and / or, in some configurations, the bone plate 16 can be interposed between the bone 14 and the reference device 200.

[0065] The reference device 200 can have one or more radiopaque fiducials 202 suitably arranged to determine the orientation of the reference device 200 (and thus the reference axis) when imaged by the imaging system 22. The radiopaque fiducials 202 create shadows when the reference device 200 is imaged. The radiopaque fiducials 202 can be, for example, radiopaque balls or other types of radiopaque elements. The one or more radiopaque fiducials 202 and their shadows enable the control system 18 to calculate the orientation (i.e., position and / or orientation) of the reference device 200 relative to the image coordinate system of the imaging system 22 (e.g., the image coordinate system of the C-arm 32). More specifically, the radiopaque fiducials 202 are fixed relative to the remainder of the reference device 200, and thus the axes of the reference device 200 can be determined by the imaging system 22 that images the radiopaque fiducials 202. The one or more radiopaque fiducials 202 can also enable the control system 18 to calculate the rotation angle between two different images captured by the imaging system 22. For example, when the imaging system 22 is the C-arm 32, images are captured as the C-arm 32 rotates relative to the bone 14. The control system 18 can calculate the difference in the orientation of the C-arm 32 when one image is captured and the orientation of the C-arm 32 when another image is captured based on the differences in how the radiopaque fiducials 202 appear in the two images.

[0066] When the reference device 200 is imaged by the imaging system 22, the imaging data includes artifacts of the radiopaque fiducials 202. Since the position of the radiopaque fiducials 202 relative to the optically detectable fiducials 204 is known, the control system 18 can provide guidance to the user regarding the orientation of the surgical instrument 100 relative to the imaging data. This can enable the user to determine the orientation and / or trajectory of the surgical instrument 100 relative to internal or invisible anatomical features shown in the imaging data.

[0067] The optically detectable fiducial 204 can be an April Tag, an ArUco Tag, or other suitable pattern observable to detect the position and / or orientation of the camera 146 relative to the reference device 200. The optically detectable fiducial 204 is positioned on the outer surface of the reference device 200 and can thus be detected by the camera 146, for example when the surgical instrument 100 is aligned with the puncture site. Each reference device 200 can include multiple optically detectable fiducials 204, such as one optically detectable fiducial 204 on two or more surfaces of the reference device 200. The position / orientation of each optically detectable fiducial 204 relative to the radiopaque fiducial 202 is indicated in a memory unit accessible by the control system 18. Each reference device 200 can have two or more optically detectable fiducials 204, but each optically detectable fiducial 204 is unique because it has a unique spatial arrangement relative to the radiopaque fiducial 202.

[0068] Different types of optically detectable fiducials 204 are contemplated. For example, different April Tag types may exist, each tag type being uniquely identifiable and having a unique ID. When two or more tags are included in a single reference device, each tag has a different ID. Thus, when the camera 146 views any of the tags, the camera 146 can determine its placement relative to the radiopaque element, such as from its ID, using pre-stored information regarding the spatial arrangement of the radiopaque element relative to the tag ID.

[0069] The reference device 200 can include a coupling portion 206. The coupling portion 206 can be configured to facilitate releasable attachment to the bone plate 16 or the bone 14 of the patient 10 via a fixing member 208, such as a Kirschner wire (K-wire), a pin, an adhesive, or another suitable fixing member.

[0070] Referring further to FIGS. 9 and 10, an exemplary bone plate 16 is shown. The bone plate 16 can be made of metal or another suitable material and is configured to immobilize a broken bone such as bone 14. As shown, the bone plate 16 can include a plurality of larger openings and a pair of smaller openings. The surgeon can first attach the bone plate 16 to the bone 14 by means of a fixing member 208 so that the bone plate 16 does not move before inserting the screw into the bone 14.

[0071] The fixing member 208 can be inserted into the bone 14 through either one of the pair of smaller openings. Another portion of the fixing member (i.e., the side that is not inserted into the bone 14) can serve as a fixture for the reference device 200. After the fixing member 208 is fed through one of the pair of smaller openings and inserted into the bone 14, the reference device 200 can be coupled to the fixing member 208. The coupling portion 206 can include a finger support collar 210, a screw portion 212, a collet 214, and a fixing knob 216. The finger support collar 210 can be attached to or integrally formed with the bottom of the reference device 200. The screw portion 212 can include a male screw and can be attached to or integrally formed with the collar 210.

[0072] The collet 214 can be attached to or integrally formed with the screw portion 212. The collet 214 is configured to form a collar around the fixing member 192 to fix the reference device 200 to the fixing member 192. When the collar forms a collar around the fixing member 192, rotation and axial movement of the reference device 200 are prevented. Although an example is provided where the coupling portion includes the collet 214 to fix the reference device 200 to the fixing member 192, the reference device 200 can be fixed to the fixing member 192 using any suitable member, element, device so that the reference device 200 cannot move independently of the fixing member 192.

[0073] The fixed knob 216 can be configured to be coupled to the threaded portion 212. For example, the fixed knob 216 can be shaped to define a passage configured to receive the threaded portion 212 and the collet 214. The fixed knob 216 can include an internal thread on its inner surface. The threaded portion 212 and the fixed knob 216 can form a mating pair. In some implementations, the sum of the dimensions (e.g., height) of the threaded portion 212 and the dimensions (e.g., height) of the collet 214 can be made substantially the same as the dimensions (e.g., height) of the fixed knob 216. The fixed knob 216 can serve to cover the collet 214 so that the connection between the reference device 200 and the fixed member 208 remains fixed and is not inadvertently severed throughout the procedure.

[0074] In other configurations, even when the bone plate 16 is used in a surgical procedure, the reference device 200 can still be directly connected to the bone 14 without the bone plate 16 intervening. In such a configuration, the fixing member 192 can be inserted into the bone 14 near the bone plate 16 rather than passing through either of the pair of openings. Alternatively, an adhesive, pin, or other coupling mechanism can be used to fix the reference device 200 to the bone 14.

[0075] After the reference device 200 is temporarily attached to the bone 14, a surgeon or other healthcare provider can image the patient 10 with the imaging system 22. The imaging system 22 then generates images and transmits those images to the control system 18.

[0076] The optically detectable reference 204 enables the camera 146 to determine the position and / or orientation of the reference device 200 relative to the camera 146 when imaged by the camera 146. The pose (i.e., position and orientation) of the optically detectable reference 204 is fixed relative to the remainder of the reference device 200, and this relationship is stored in a memory accessible by the control system 18. The camera 146 is configured to generate a first signal. The first signal can correspond to the pose of the optically detectable reference 204 relative to a first coordinate system. The first signal corresponds to the drill tip position at the start of the drilling. The camera 146 determines the rotation and translation of the camera 146 relative to the optically detectable reference. Determining the drilling start position using the camera 146 helps reduce the number of inputs from the surgeon and automates the process. The sensor module 148 can generate a second signal regarding orientation data and / or position data relative to a second coordinate system.

[0077] The sensor module 148 generates a second signal, and the control system 18 can use the second signal to derive the orientation and / or position of the surgical instrument 100, more specifically the orientation and / or position of the attachment 104 and the measurement axis 128. The sensor module 148 is configured to have a pose (i.e., position and orientation) fixed relative to the surgical instrument 100, particularly the end effector 102. In some examples, the sensor module 148 is disposed within the attachment 104 or the housing 106 of the surgical instrument 100.

[0078] In one potential implementation, the camera 146 needs to be directed only once at the start of the drilling towards the optically detectable reference 204 of a particular reference device 200 in order to determine the signal from the sensor module 148 that belongs to the starting point determined by the camera 146.

[0079] The attachment 104 can include one or more control elements 150, such as a first button and a second button. The sensor module 148 can receive power from the surgical instrument 100 and does not require a separate power source when disposed within the attachment 104. In other configurations, the attachment 104 can include a power source, such as a battery.

[0080] The position and / or orientation of the end effector 102 can also be fixed with respect to the camera 146 and the sensor module 148 in one or more degrees of freedom. More particularly, the axis (e.g., tool axis) of the end effector 102 with respect to the camera 146 and the sensor module 148 can be known. Thus, after the position / orientation of the camera 146 with respect to the image is known, the axis of the end effector 102 with respect to the image can also be known. Further, after the position / orientation of the sensor module 148 with respect to the image is known, the axis of the end effector 102 with respect to the image can also be known. After the axis of the end effector 102 with respect to the image is known, the position / orientation of the sensor module 148 with respect to the image can also be known. The relationship between the axis of the end effector 102, the camera 146, and the sensor module 148 can be stored in a memory unit accessible by the control system 18.

[0081] In certain configurations, the length of the end effector 102 can be selected by the user by using a user input device and input to the control system 18. The control system 18 can utilize this input length of the end effector 102 to perform calculations, obtain the depth end point of the end effector 102, and display the depth end point of the end effector 102 superimposed on the patient image. This can function to convert a point in the camera coordinate system to the drill tip location. In certain configurations, the length of the end effector 102 can be referred to as the drill bit length.

[0082] Alternatively, when the drill bit or end effector 102 is coupled to the instrument 100, the surgical instrument 100 can include an identification sensor 152 for generating an identification signal in response to an identification feature 154 of the drill bit or other end effector 102. The identification sensor 152 can provide the identification signal to an attachment controller 134. A control system 18, such as the attachment controller 134, can determine the length of the end effector 102 based on this identification signal. In certain implementations, the attachment 104 can include the identification sensor 152. The identification feature 154 and the identification sensor 152 can be as described in WO2020 / 232413, which is hereby incorporated by reference in its entirety. Alternatively, the identification sensor 152 may be included at other locations within the instrument 100, such as within the handpiece body.

[0083] The identification sensor 152 can be a magnetoresistive sensor, an optical sensor, or the like. The identification feature 154 can be a magnetic identification feature, such as an array of magnetic materials, or an optical identification feature, such as a laser marking.

[0084] The end effector 102 can also be identified and / or measured by the camera 146 using feature matching or the like. For example, the end effector 102 can be coupled to the instrument 100 such that at least a portion of the end effector 102 comes within the field of view of the camera 146 and is thus visible to the camera 146. The control system 18 can then be configured to determine the identification and / or length of the end effector 102 based on the portion of the end effector 102 that is within the field of view of the camera. If the control system 18 uses feature matching to identify and / or determine the length of the end effector 102, the system 18 can have access to pre-stored image data associated with known end effector types / identifications. The control system 18 then compares the visible portion of the end effector 102 to the pre-stored image data to determine if a match exists. For example, if the visible portion of the end effector 102 matches pre-stored image data associated with a drill bit having a length of 99 millimeters, the end effector 102 is likewise determined to have a length of 99 millimeters. Other sizes such as 129 mm and 169 mm, as well as any other length or identification of the end effector 102, are contemplated. This can be used in conjunction with the image processing techniques described hereinafter.

[0085] Referring to FIGS. 11A and 11B, the instrument 100 is shown near the bone 14 and the reference device 200. The end effector 102 is positioned within a very close range to the bone 14, and the camera 146 can see the reference device 200 and thus the optically detectable reference 204. Although not shown, the imaging system 22 has imaged the bone 14 and the reference device 200 prior to the introduction of the instrument 100. Thus, as described above, the imaging system 22 has already determined the orientation of the reference axis of the reference device 200 via the radiopaque reference 202.

[0086] FIG. 11B is a schematic diagram of the transformation utilized by system 12. Starting from the C-arm 32 (or alternative imaging system 22), when an X-ray image is generated, an imaging system-reference device transformation is calculated by the control system 18. The imaging system-reference device transformation is based on the orientation of the reference device 200 in the X-ray image, which is derived from the orientation of the radiopaque fiducial 202 and the known spatial relationship between the radiopaque fiducial 202 and the reference axes of the reference device 200. Thus, through the imaging system-reference device transformation, the control system 18 can know the orientation of the reference axes with respect to the image coordinate system of the C-arm 32 based on the orientation of the radiopaque fiducial 202 in the X-ray image.

[0087] The camera-reference device transformation is also shown in FIG. 11B. The camera-reference device transformation is calculated by the control system 18 when an optical image is generated by the camera 146. The camera-reference device transformation is based on the orientation of the reference device 200 in the optical image, which is derived from the orientation of the optical fiducial 204 and the known spatial relationship between the optical fiducial 204 and the reference axes of the reference device 200. Thus, through the camera-reference device transformation, the control system 18 can know the orientation of the reference axes with respect to the camera coordinate system of the camera 146 based on the orientation of the optical fiducial 204 in the optical image.

[0088] Finally, the IMU-camera transformation is shown in FIG. 11B. The IMU-camera transformation is calculated by the control system 18 when the control element 150 is actuated by the user. The IMU-camera transformation is based on the orientation of the sensor module 148 when the control element 150 is actuated. More specifically, the actuation of the control element 150 aligns the sensor coordinate system with the global reference frame (i.e., gravity) by setting the starting orientation of the sensor coordinate system relative to the orientation of the sensor module 148. In other words, the IMU-camera transformation enables the control system 18 to define the sensor coordinate system with respect to the camera 146 and the image coordinate system. As a result, the control system 18 can utilize the IMU and the sensor module 148 to track changes in the orientation of the optical coordinate system via the IMU-camera transformation.

[0089] With access to / knowledge of the imaging system-reference device transformation, the camera-reference device transformation, and the IMU-camera transformation, the control system 18 can determine the orientation of the instrument 100 with respect to the imaging system 22. Accordingly, the orientation of the instrument 100 with respect to the image coordinate system can be known. That being said, it is important to understand that when the imaging system 22 is the C-arm 32, the orientation of the image coordinate system depends on the orientation of the light source 26 and the detector 24 when the X-ray image is generated. The control system 18 utilizes these transformations along with it to determine the orientation of the instrument 100 with respect to the image coordinate system for all X-ray images generated by the imaging system 22. If the orientation of the image coordinate system changes due to the rotation of the C-arm 32 between images, the control system 18 can similarly determine how the orientation of the instrument 100 in the image coordinate system changes.

[0090] As a result of the above-described transformation, the control system 18 can align the optical coordinate system of the camera 146 with the image coordinate system of the imaging system 22. Further, the control system 18 can continuously change this alignment as the attitude of the IMU changes. Since the IMU-camera transformation aligns the sensor coordinate system and the optical coordinate system, any rotation / translation of the sensor module 148 correlates to an equal rotation / translation of the optical coordinate system. Finally, the control system 18 can know this relationship as the light-image transformation. The light-image transformation enables the control system 18 to transform the attitude of an object known with respect to the optical coordinate system into the attitude of the object with respect to the image coordinate system. Thereby, the control system 18 can depict the objects seen by the camera 146, such as the end effector 102 and the reference device 200, in the X-ray image.

[0091] Referring to FIG. 12, since only the reference reference axis is known in the image coordinate system, the control system 18 is configured to determine the perforation start point 300 with respect to the reference reference axis of the reference device 200. When the X-ray image of the bone 14 to be processed is acquired by the reference device 200, the conversion between the radiation impermeability reference 202 of the X-ray image and the image coordinate system can be determined. This is the imaging system-reference device conversion described above. As described above, the reference reference axis is defined by the arrangement of the radiation impermeability reference 202 within the reference device 200. Since the control system 18 does not rely on conventional tracking means to track the position of the surgical instrument 100, it must determine the actual position of the surgical instrument 100 with respect to the image.

[0092] The control system 18 is configured to determine the perforation start point 300 based on the attitude of the surgical instrument 100 and / or the attitude of the attachment 104 with respect to the reference device 200 derived from the camera 146 at the time when the input signal is received.

[0093] The input signal can be defined as a user input signal obtained from the actuation of the control element 150 on the attachment 104. Alternatively, the input signal can be based on a signal output by a depth sensor 122 that provides a signal associated with the displacement of the drill bit 102 relative to the bone 14 during the drilling process.

[0094] The input signal can be generated when the depth measurement extension 124 is depressed by a threshold amount such as 5 mm. This slight depression of the depth measurement extension 124, and the associated depth signal resulting from such movement, can serve as a basis for concluding that the user is ready to start drilling. Whether the input signal is a user input signal derived from the activation of the control element 150 or the input signal is determined based on the depth sensor 122, in either implementation, the controller can use the input signal to determine the drilling start point 300. In a particular implementation, the control system 18 can utilize both the control element 150 and the depth sensor 122 to initiate the measurement of the drilling start point 300.

[0095] When the control system 18 receives the input signal, the control system 18 can activate the camera 146 to start searching for the optically detectable reference 204. When the camera 146 detects the optically detectable reference 204, the control system 18 determines the camera-reference device transformation matrix. The camera-reference device transformation matrix is a matrix that indicates the alignment of the reference axes of the reference device 200 with respect to the optical coordinate system of the camera 146.

[0096] This is advantageous because it does not require the user to identify the position of the tip of the end effector 102 with respect to the image, which is a step prone to error. Further, in implementations where the input signal is derived from the depth sensor 122 and / or the IMU, this step can be performed without a separate action by the user outside of a typical drilling or driving workflow. This saves the surgeon's time and allows the surgeon to focus on the trajectory of the surgical instrument 100 and / or the implant.

[0097] The control system 18 can also calculate various transformations using the known position and / or orientation of the end effector 102, and / or the length of the end effector 102. For example, the control system 18 can transform the coordinates of the distal end of the end effector 102 from a known end effector axis to a known axis of the camera 146 based on a pre-defined stored transformation matrix from the end effector to the camera. Since the end effector 102, such as a drill bit, can have a fixed orientation with respect to the sensor module 148 and the camera 146, this can be used.

[0098] The sensor module 148 may include a gyroscope and / or an accelerometer that are susceptible to bias instability that causes signal drift over time. Thus, the surgeon can periodically align the sensor module 148 with the reference device 200 during the surgical procedure to minimize tracking errors caused by this drift. One way to address this is to place several different reference devices 200 near the patient 10, with at least one reference device 200 placed near each hole to be drilled. Additionally, an alignment step can be performed for each hole to be drilled by the instrument 100, or for each trajectory along which the implant can be advanced. The control system 18 can use the input signal to indicate to the camera 146 that alignment is desired. In this step, the control system 18 realigns the images of the sensor module 148, the camera 146, and the reference device 200. The instrument 100 can automatically align the sensor module 148 with the camera 146 when the camera 146 sees the optically detectable reference 204 during determination of the drilling start point 300. As a result, at the start of opening each hole and / or advancing along each trajectory, the sensor module 148 is automatically aligned with the coordinate system of the camera 146. This process creates / sets an IMU-camera transformation that indicates the alignment between the optical coordinate system and the sensor coordinate system.

[0099] After the control system 18 aligns the camera coordinate system with respect to the sensor module coordinate system, the control system 18 can start determining the handpiece orientation signal. This determination of the handpiece orientation signal can be performed by the attachment controller 134.

[0100] As already discussed, the optically detectable fiducial 204 is fixedly attached to the reference device 200, and since the reference device 200 has a fixed orientation with respect to the bone 14, the optically detectable fiducial 204 can serve as a patient tracking device for each trajectory. Each time an image is acquired, the relationship between the optically detectable fiducial 204 and the bone 14 is recorded and associated with the particular image acquired.

[0101] The control system 18 can further transform the coordinates of the distal end of the end effector 102 from the camera coordinate system to the reference axes of the reference device 200 using the measured camera-reference device transformation matrix. This transformation matrix from the camera to the reference device can be calculated based on signals generated by the camera 146 when detecting an optically detectable fiducial 204 such as an April Tag.

[0102] The control system 18 can transform the coordinates of the distal end of the end effector 102 from those with respect to the axes of the optically detectable fiducial 204 to those with respect to the axes of the reference device 200 known with respect to the radiation-impermeable fiducial 202. This calculation can be performed using the identification of each optically detectable fiducial 204 and the known transformation of each identification with respect to the reference axes of the particular reference device 200. This can be referred to as the transformation matrix from the optically detectable fiducial to the radiation-impermeable fiducial. It will be understood that these transformations can be understood by the system 12 as the spatial relationships between the optical fiducial 204, the radiation-impermeable fiducial 202, and the remainder of the reference device 200.

[0103] Referring to FIGS. 12-14, the control system 18 can overlay virtual representations 306, 308 on the image based on the starting position of the end effector and the determined orientation of the surgical instrument 100. The virtual representations 306, 308 can include a model of the surgical instrument 100, such as a model of the attachment 104 and / or the end effector 102. The virtual representations 306, 308 can represent the current pose and the planned end pose of the end effector 102, respectively. The virtual representation can also include a virtual trajectory that penetrates the bone 14. The surgeon can then determine the angle of the surgical instrument 100 until the surgeon is satisfied with the position of the virtual trajectory, i.e., the position of the virtual representations 306, 308 relative to the bone 14 or tissue in the image. For example, if the surgical system 12 includes a bone plate 16, a model of the end effector 102 can be overlaid on the selected aperture. In another example, a model of the depth measurement attachment or the tip of the end effector 102 can be overlaid on the selected anatomical feature in the image, such as when the surgical system 12 does not include a bone plate 16. The control system 18 can control the display to indicate the drilling depth or screw length based on a signal from the depth sensor 122. Such an implementation is described in PCT Publication No. WO2019035096, which is hereby incorporated by reference in its entirety.

[0104] The features of the system can include any of the features described in PCT / US2021 / 044429, which is hereby incorporated by reference in its entirety.

[0105] The transformed distal end of the end effector 102 relative to the axis of the reference device 200 is referenced as the drilling start point 300.

[0106] The control system 18 can further operate to establish a relationship between the axis of the sensor module 148 and the reference axis of the reference device 200. This relationship can be based on a combination of IMU-camera transformation and camera-reference device transformation. This relationship can be established using the transformation of the camera 146 - optically detectable reference 204 described above. Additionally, this calculation can be based on the transformation of the coordinates of the distal end of the end effector 102 from the known axis of the end effector to the known axis of the sensor module 148, based on a pre-defined stored transformation matrix from the pre-defined stored end effector to the sensor module. The net effect is the establishment of the transformation from the axis of the sensor module 148 to the axis of the reference device 200.

[0107] The control system 18 is then used to determine the trajectory end point 302 with respect to the axis of the reference device 200. This is calculated by using the transformation from the sensor module 148 to the reference axis of the reference device 200 described above, to transform the perforation start point 300 from that with respect to the reference axis of the reference device 200 to that with respect to the axis of the end effector 102. The trajectory end point 302 can be the planned end position of the end effector 102. The planned end position can be the planned position in the bone 14 where the user desires the end effector 102 to stop and / or reach upon completion of the perforation.

[0108] The control system 18 can further determine the remaining cannula travel length. This remaining cannula travel length is the potential travel distance remaining for the depth measurement extension 124 to retract relative to the distal end of the end effector 102. The maximum cannula travel length can be stored by the control system 18 in an accessible memory unit. The control system 18 can calculate the remaining travel length based on the measured distance of the depth sensor 122 and the maximum cannula travel length. The maximum cannula travel length is defined between the fully distal position and the proximal position. The remaining cannula travel length value can be useful for the user to understand whether the measuring cannula can continue to penetrate without reaching the bottom within the instrument 100. This remaining cannula travel length can represent the range of the movement limit for the depth measurement extension 124 of the instrument 100. An exemplary maximum cannula travel length can be 110 mm.

[0109] With respect to FIGS. 13 and 14, the trajectory end point 302 can be calculated based on the piercing start point 300 and the remaining cannula travel length. The trajectory end point 302 can be offset from the piercing start point 300 by the remaining cannula travel length. The trajectory end point 302 is positioned along the drill reference axis. The trajectory end point 302 with respect to the drill reference axis is converted to the trajectory end point 302 with respect to the axis of the reference device 200 using the transformation matrix from the sensor module 148 to the reference axis. The trajectory end point 302 can be displayed, and thus the user can visualize how deep the drill bit can progress relative to the real-time trajectory, which is determined using the sensor module 148. FIG. 14 shows that the user can drill all the way to the fibula without the measuring cannula or other depth measurement extension reaching the bottom, and thus the user may want to carefully complete the hole without allowing the drill bit to penetrate into the fibula.

[0110] Continuing to refer to FIGS. 13 and 14, control system 18 can also determine a depth end point 304 relative to a reference axis of reference device 200. Control system 18 can further transform the drilling start point 300 from the axis of the reference device to the axis of end effector 102 using a transformation matrix from the sensor module 148 to the axis of the reference device (i.e., a combination of IMU-camera and camera-reference device transformations). The depth end point 304 represents the end of the drill bit or other end effector 102. Control system 18 can display the depth end point 304 to enable the user to understand how deep the end effector 102 is with respect to the image data, i.e., the bone 14 to be drilled. In the figures shown, since the depth end point 304 is within the tibia, it can be determined that the distal end of the drill bit has not yet penetrated the tibia, and thus the user needs to continue the drilling. Since the drilling has not yet started, the depth end point 304 should overlap the drilling start point 300, so the depth end point 304 is not shown on the display 20.

[0111] Control system 18 can further determine the depth of end effector 102 relative to bone 14. Throughout the description, a method for determining this depth using a measurement cannula will be described, but other depth sensing configurations are also contemplated.

[0112] Based on the measured depth, control system 18 determines a depth end point 304 relative to the drill reference axis. The depth end point 304 is offset from the drilling start point 300 based on the measured depth, and the depth end point 304 is located along the motor axis 109. The depth end point 304 relative to the reference axis of end effector 102 is transformed to the depth end point 304 relative to the axis of reference device 200 using a transformation matrix from the sensor module 148 to the axis of the reference device. This depth end point 304 can serve as the depth end point 304 of the virtual representation 306.

[0113] In some examples, the control system 18 can be configured to depict an actual depth profile of the drilling procedure as the end effector 102 drills through the bone 14 by displaying the depth end point 304. As already discussed, since the magnification / zoom factor can vary from one image to the next, the control system 18 can use the magnification / zoom factor to depict the actual depth profile, and the depth end point 304 is overlaid on the image with respect to the bone 14. For example, the control system 18 can use the magnification / zoom factor when determining the conversion from appropriate pixels to millimeters for each of the images.

[0114] The control system 18 then uses the imaging system-reference device transformation to convert the drilling start point 300, the trajectory end point 302, and the depth end point 304 into the image coordinate system. The control system 18 then operates to overlay the representations 306, 308 based on this information. For example, the control system 18 can be configured to overlay two lines, the first virtual representation 306 and the second virtual representation 308, on the X-ray image using these points. In some implementations, these lines can be representations of the end effector 102 or the holes to be created by the end effector 102. The first and second virtual representations can have different colors, shapes, transparencies, etc. Since both the first virtual representation 306 and the second virtual representation 308 are projected based on the orientation determined by the sensor module 148, they can be located on the same line as each other.

[0115] The first virtual representation ends at the depth end point 304, which represents the end of the tip of the drill bit 102. The second virtual representation 308 can extend from the depth end point 304 and terminate at the trajectory end point 302, which represents the maximum drilling depth.

[0116] When the surgeon is satisfied with the virtual trajectory depicted in the image, the control system 18 can be configured to enable the surgeon to make the trajectory stationary or memorize it as the desired trajectory. When the desired trajectory is memorized, the display unit 20 can also include a graphical representation for providing a visual indication to the surgeon when the current trajectory matches the memorized desired trajectory. For example, a movement notification that moves with the orientation of the surgical instrument 100 can be displayed on the display unit 20. The movement notification can be shown on the display unit 20 relative to a target (e.g., a representation of a aiming line or a circle). The target represents the desired trajectory. In addition, visual information (e.g., a graphical representation) can be displayed on the display 144 of the surgical instrument 100 so that the current trajectory can be aligned with the memorized desired trajectory without the surgeon having to look at the display unit 20.

[0117] The surgical system 12 can be configured to generate an audible, visual, or tactile alert when the desired trajectory is achieved. Although this disclosure contemplates that the images and virtual representations will be displayed on the display unit 20, the control system 18 can also transmit the virtual representations to a display attached to the surgical instrument 100, a tablet, an operating room display unit, a display unit attached to an imaging device, a laptop computer, a mixed reality device, or any other computing device and / or display within the operating room. In instances where the display unit 20 is provided in a form other than being attached to the surgical instrument 100, a processor can be provided to the display unit 20 along with a suitable communication device such as an antenna.

[0118] The attachment controller 134 can be configured to directly transmit the first signal, the second signal, and / or the third signal to the instrument controller 110 or some other controller, or can perform the calculations described above.

[0119] In the case where the attachment controller 134 performs one or more of the calculations described throughout this specification, the attachment controller 134 can transmit all of the drilling start point 300, the trajectory end point 302, and the depth end point 304 with respect to the reference axis to the display unit 20 such as a tablet. As a result, since less data needs to be transmitted from the surgical instrument 100 to the display unit 20, the transmission time can be simplified.

[0120] The control system 18 can further determine the conversion from the axis of the reference device 200 to the X-ray image using the radiopacity standard 202 for each captured image. The radiopacity standards 202 included in each of the reference devices 200 may be spatially different from each other, and thus it should be understood that this calculation can be performed for any reference device 200 and for any acquired image.

[0121] The output on the display 20 can include a graphical or pictorial representation of the position of the instrument 100 relative to the patient 10. This output can also be presented in an augmented or virtual reality environment, such as a wearable display that the user views to observe the surgical site.

[0122] In the case where the display unit 20 is a tablet or other mobile computing device, the instrument 100 can communicate wirelessly with the display unit 20 to provide points related to the display 20. The display unit can also have a wired or wireless connection to the X-ray machine to acquire X-ray images. The application on the display unit 20 acquires the X-ray image from the X-ray machine, processes the X-ray image to remove distortion, determines the conversion matrix from the reference to the image using the detected radiopacity standard 202 on the reference device, and provides trajectory and drilling position guidance using the information provided by the instrument 100 such as the attachment 104.

[0123] The display unit can project the drilling start point 300, the trajectory end point 302, and the depth end point 304 with respect to the reference axis onto the X-ray image by using the above-described conversion and conversion matrix. The display unit can provide a simulated guide on all the captured X-ray images by superimposing the trajectory and drill tip guidance information on the captured X-ray image.

[0124] The system 12 can determine the attitude of the imaging system 22 with respect to the reference device 200 by utilizing the imaging system-reference device conversion. More specifically, when the imaging system 22 includes the detector 24 and the light source 26, the system 12 can know the attitude of the light source 26 (and / or the detector 24) with respect to the reference device 200 when the X-ray image is generated by the imaging system 22 based on the attitude of the reference device 200 in the X-ray image.

[0125] Referring to FIG. 15, both the light source 26 and the instrument 100 are shown in a plurality of attitudes with respect to the reference device 200 and the bone 14. For simplicity of the figure, FIG. 15 is a two-dimensional abstraction of the above-described attitudes, the reference device 200, and the bone 14. Each light source 26 shown in FIG. 15 is an individual position where the X-ray image was generated, and the image boundaries of each respective image are shown as dotted lines extending from each light source 26. These image boundaries represent the field of view of the imaging system 22, and when the bone 14 is the target of the imaging system 22, each image boundary includes the bone 14 and the reference device 200. The image boundary is meant to depict one dimension of the resulting X-ray image (e.g., the "height" of the resulting X-ray image), and the second dimension of the resulting X-ray image (e.g., "width") extends in the paper plane direction and is thus not shown in these figures. Each attitude of the light source 26 is depicted by a pair of arrows extending from each respective light source 26. Each attitude of the instrument 100 is depicted by a pair of arrows extending from each respective instrument 100.

[0126] As described above, the postures of the light source 26 and the instrument 100 can be calculated using a transformation matrix. In summary, the control system 18 knows and / or can determine the posture of the reference device 200 with respect to the image coordinate system based on the posture of the radiopaque reference 202 in the X-ray image. The control system 18 knows and / or can determine the posture of the instrument 100 with respect to the reference device 200 based on the posture of the optical reference 204 in the image captured by the camera 146. The control system 18 then tracks the posture of the instrument 100 by means of the IMU as described above. For example, the IMU included in the sensor module 148 can be reset / calibrated while the reference device 200 is within the field of view of the camera 146 to determine the transformation between the IMU and the reference device 200 via the IMU-camera and camera-reference device transformations. Since the sensor module 148 is fixed with respect to the instrument 100, the transformation between the instrument 100 and the reference device 200 can be determined via the camera-reference device transformation. Thereafter, the control system 18 can determine the transformation between the instrument 100 and the image coordinate system and determine the posture of the instrument 100 in the image coordinate system. As a result, the control system 18 can know and / or determine the posture of the instrument 100 in the image coordinate system. The above-described transformation is used by the control system 18 to determine this posture of the instrument 100 in the image coordinate system.

[0127] It will be appreciated that the image coordinate system varies with respect to the global coordinate system according to the postures of the light source 26 and the detector 24. The posture of the reference device 200 is used to determine the posture of the image coordinate system with respect to the global coordinate system. Since the reference device 200 remains fixed in the global coordinate system, the reference device 200 enables the control system 18 to determine the posture of the image coordinate system.

[0128] Referring further to FIG. 15, the posture of the instrument 100 can be used to determine which X-ray images to display on the display 20. In many cases, the user prefers the display 20 to show two orthogonal X-ray images, one image representing the X-ray of the patient 10 aligned with the user's viewing point, and the other image representing the X-ray of the patient 10 acquired orthogonally thereto. In other words, one image should match the way the patient 10 looks to the user. For example, if the user plans to puncture the leg of the patient 10 and the user stands in front of the patient 10 and the patient 10 is facing the user, one image shown on the display 20 should be the anteroposterior (AP) X-ray of the patient 10, and the other image shown on the display 20 should be the mediolateral (ML) X-ray of the patient 10. Theoretically, the puncture start point 300, the trajectory end point 302, and the depth end point 304 should overlap within the AP X-ray, and the points 300, 302, 304 should appear as a straight line within the ML X-ray. As used herein, "AP image" and "AP X-ray" refer to an X-ray image substantially aligned with the points 300, 302, 304 and / or the tool axis of the instrument 100. This means that the image coordinate system of the X-ray image should be substantially aligned with the coordinate system of the camera 146 on the drill, and thus the AP image shown on the display 20 should have a field of view substantially similar to the image acquired by the camera 146. Similarly, as used herein, "ML image" and "ML X-ray" refer to an X-ray image that is substantially orthogonal to and aligned with the AP image and substantially orthogonal to the points 300, 302, 304 and / or the tool axis of the instrument 100.

[0129] When an X-ray image is generated for patient 10, particularly when generated by an imaging system 22 such as a C-arm 32, a plurality of X-ray images are generated while the imaging system 22 is rotated with respect to patient 10 (or vice versa). As described above, the control system 18 can determine which of these X-ray images to display on the display 20 based on the posture of the instrument 100. These images are the AP image and the ML image described above. The system can determine, as will be described later, which of the X-ray images generated by the imaging system 22 should best represent the AP image and the ML image with respect to the posture of the instrument 100.

[0130] Referring to FIGS. 16A - 16C, the instrument 100 and the light source 26 are shown in a specific posture with respect to the reference device 200. In this specification, the posture of the light source 26 is referred to as the light source position. Each individual light source position is indicated by a numbered light source position P1, P2, P3, P4, etc. Each light source 26 includes arrows that overlap the respective light source 26, and these arrows correspond to the posture of the light source 26 and thus the posture of the image coordinate system. The extension of one dimension of the image coordinate system is shown as a dotted line extending from the light source 26, which is referred to as the image reference axis in this specification. The image reference axis is orthogonal to the resulting X-ray image and is best considered as a line extending inside and outside the two-dimensional X-ray image. In other words, this two-dimensional X-ray image can have a height and a width corresponding to, for example, the x-axis and y-axis of the image coordinate system. In this example, the image reference axis should be the z-axis of the image coordinate system. The instrument 100 is shown by a dashed line extending from the instrument 100, and this dashed line is the tool axis T of the instrument 100. The tool axis T corresponds to a straight line extending from the distal end of the end effector 102 and is parallel to the end effector 102.

[0131] The system can select appropriate AP and ML images based on the relationship between the posture of the instrument 100 and the image reference axes such as the instrument offset angle IA. The instrument offset angle IA is the angle between the tool axis T and the image reference axis. FIG. 16A helps to illustrate this function for the AP image, and FIG. 16B helps to illustrate this function for the ML image.

[0132] Referring to FIG. 16A, for the AP image, the control system 18 can be configured to automatically select the image that minimizes the instrument offset angle IA. For example, in FIG. 16A, the light source 26 is shown at the first light source position P1 and the second light source position P2. The first image generated when the light source 26 is at the first light source position P1 results in the first instrument offset angle IA1. The second image generated when the light source 26 is at the second light source position P2 results in the second instrument offset angle IA2. As can be seen in FIG. 16A, the first instrument offset angle IA1 is smaller than the second instrument offset angle IA2. Thus, for the AP image, the control system 18 selects and displays the image generated by the imaging system 22 when the light source 26 is at the first light source position P1. The control system 18 can also update which image is selected as the AP image as the posture of the instrument 100 changes. For example, if the instrument 100 is rotated counterclockwise such that the first instrument offset angle IA1 becomes larger than the second instrument offset angle IA2, the control system 18 should change the AP image to the image generated by the imaging system 22 when the light source 26 is at the second light source position P2. This function eliminates the need for medical personnel to manually select which image should be displayed, and instead, the image most suitable for visualization of the trajectory is selected.

[0133] Referring to FIG. 16B, in another proposed implementation for the ML image, the control system 18 selects an image in which the instrument offset angle IA approaches 90 degrees as closely as possible. In other words, the control system 18 selects an image having an image reference axis that is most orthogonal to the tool axis T of the available images. In FIG. 16B, for example, the light source 26 is shown at a third light source position P3 and a fourth light source position P4. The first image generated when the light source 26 is at the third light source position P3 results in a third instrument offset angle IA3. The fourth image generated when the light source 26 is at the fourth light source position P4 results in a fourth instrument offset angle IA4. As can be seen in FIG. 16B, the third instrument offset angle IA3 is closer to 90 degrees than the fourth instrument offset angle IA4. Thus, in the case of the ML image, the control system 18 selects and indicates the image generated by the imaging system 22 when the light source 26 is at the third light source position P3. The control system 18 can also update which image is selected as the ML image as the posture of the instrument 100 changes. For example, if the instrument 100 is rotated counterclockwise such that the fourth instrument offset angle IA4 approaches 90 degrees closer than the third instrument offset angle IA3, the control system 18 should change the ML image to the image generated by the imaging system 22 when the light source 26 is at the fourth light source position P4.

[0134] FIGS. 16A and 16B show the angle IA in only two dimensions, but it will be understood that both the image reference axis and the tool axis T are three-dimensional vectors in space. As will be described further below, the control system 18 can also check whether the trajectory of the instrument 100 is visible within the X-ray image when selecting the AP and ML images.

[0135] Referring to FIG. 16C, an exemplary scenario is depicted where the orientation of the instrument 100, and thus the tool axis T, is substantially aligned with one image captured by the imaging system 22 and substantially orthogonal to another image captured by the imaging system 22. In other words, the tool axis T is parallel to the image reference axis of the X-ray image generated when the light source 26 is at the first light source position P1, and the tool axis T is substantially orthogonal to the image reference axis of the X-ray image generated when the light source 26 is at the third light source position P3. This is shown by the fact that the instrument offset angle IA with respect to the third light source position P3 is substantially 90 degrees and the instrument offset angle with respect to the first light source position P1 is substantially 0 degrees. In this scenario, the control system 18 can provide an AP image that replicates the field of view of the user holding the instrument 100 and a corresponding ML image that replicates a field of view orthogonal to the user's field of view.

[0136] Referring to FIGS. 17A and 17B, various X-ray views of the bone 14 are shown. Each X-ray has a uniquely aligned image reference axis (not shown), which is represented by the orientation of the global coordinate axes at the upper left corner of each X-ray image. FIGS. 17A and 17B also show how the length and orientation of the trajectory of the instrument 100 change based on how the X-ray images are generated (e.g., the light source 26 and detector 24 with respect to the patient 10 when the images are acquired).

[0137] To utilize this phenomenon, the control system 18 can also select appropriate AP and ML images based on a radial distance calculation. The radial distance calculation is a measurement of the length of the instrument 100's trajectory in the X-ray image. More specifically, the radial distance is the distance between the puncture start point 300 and the trajectory end point 302. Since the X-ray image is two-dimensional, the radial distance changes as the image reference axis is rotated with respect to the instrument 100's trajectory. The radial distance is minimized when the image reference axis is parallel to the instrument 100's trajectory. The radial distance is maximized when the image reference axis is orthogonal to the instrument 100's trajectory. Thereby, the control system 18 can select an appropriate AP image by determining in which X-ray image the radial distance is minimized, and can select an appropriate ML image by determining in which X-ray image the radial distance is maximized.

[0138] Another way to explain the radial distance calculation is with respect to the field of view of the X-ray image. When a two-dimensional X-ray image is captured, the image coordinate system can be considered to have x and y axes corresponding to the height and width of the resulting X-ray image, as described above. The plane formed by the x and y axes can be called the image reference plane. In this example, the image reference axis should still be the z-axis of the image coordinate system, and the image reference axis should be orthogonal to the image reference plane. When determining the X-ray image results at the minimum and maximum radial distances, the control system 18 projects the puncture start point 300 and the trajectory end point 302 onto the image reference plane. After projecting points 300, 302 onto the image reference plane, the radial distance is the distance between the projected points 300, 302, constrained to the image reference plane. The radial distance can also be normalized according to the image magnification factor and / or scale.

[0139] Referring to FIG. 17A, two X-ray images are shown side by side to show how the control system 18 determines which X-ray image to display on the display 20 as the ML image. By the change in the track length between the two images, the change in the radial distance can be seen. In the example shown, the first track length L1 is shown in one image, and the second track length L2 is shown in the other image. The first track length L1 and the second track length L2 are of different lengths, but both represent the same track represented by the line between the perforation start point 300 and the track end point 302. As described above, the control system 18 selects the ML image based on which radial distance is longer. In the example shown, the first track length L1 is longer than the second track length L2. As a result, the control system 18 shows the left image on the display 20 as the ML image.

[0140] Referring to FIG. 17B, two more X-ray images are shown side by side to show how the control system 18 determines which X-ray image to display on the display 20 as the AP image. In the example shown, the third track length L3 is shown in one image, and the fourth track length L4 is shown in the other image. The third track length L3 and the fourth track length L4 are of different lengths, but both represent the same track represented by the line between the perforation start point 300 and the track end point 302. As described above, the control system 18 selects the AP image based on which radial distance is shorter. In the example shown, the third track length L3 is shorter than the fourth track length L4. As a result, the control system 18 shows the left image on the display 20 as the AP image.

[0141] Referring to FIGS. 18A and 18B, the X-ray diagrams included in FIGS. 17A and 17B are shown together with the virtual representation 306 of the end effector 102. Both the length and orientation of the virtual representation 306 are affected by how the X-ray image was generated (e.g., the light source 26 and detector 24 are with respect to the patient 10 when the image was acquired). Although the end effector 102 was not present in the X-ray, the control system 18 can utilize the virtual representation 306 as a projection of the end effector 102 onto the image, and the length and orientation match how the end effector 102 would appear if it were present when the X-ray was acquired. The virtual representation 306 of the end effector 102 is generally aligned with the trajectory of the instrument 100 and / or the tool axis T by the control system 18 and can be given a length equal to the length of the end effector 102. However, in the illustrated implementation, the control system 18 not only aligns the virtual representation 306 with the trajectory of the instrument 100, but the control system 18 also varies the length of the virtual representation 306 to match the appearance when viewed from the field of view of the image. Similar to the trajectory itself, the end effector 102 should appear different / longer / shorter when viewed from different fields of view. In some implementations, the virtual representation 306 of the end effector 102 changes as the posture of the instrument 100 changes, and thus the X-ray image shown on the display 20 accurately shows the appearance of the end effector 102 from the field of view of the image. In any implementation, the projection of the end effector 102 may be stationary with respect to the image or move when the instrument 100 is moved.

[0142] The virtual representation 306 of the end effector 102 can appear similar to the end effector 102, as in the illustrated implementations of FIGS. 18A and 18B. Alternatively, the virtual representation 306 can have any shape with a longitudinal axis parallel to at least one of the trajectories defined between the tool axis T and the points 300, 302. The shape of the virtual representation 306 can be a simple line or a two-dimensional shape.

[0143] Referring to FIGS. 19A - 23C, the instrument 100 and the bone 14 are depicted with the light source 26 in various postures P1, P2, P3 with respect to the bone 14. The corresponding AP images are also shown. Although the instrument 100 is shown in line with the field of view of the X - ray image, it should be understood that the X - ray image is generally generated before introducing the instrument 100 into the surgical space. That being said, as described above, the posture of the instrument 100 can be used to select an appropriate image to be shown on the display 20. For this reason, the instrument 100 is shown with respect to the bone 14. The reference device 200 is omitted. FIGS. 19A - 23C each show additional factors that the control system 18 can consider when determining which X - ray image to show on the display 20.

[0144] In any implementation, regardless of the factors considered, the control system 18 can also determine that the image display can be executed as long as it is the best available image, even if it cannot satisfy all of the considered factors. Further, in a particular implementation, the control system 18 can also default to the most recently acquired image.

[0145] Referring to FIG. 19A, three light source positions P1, P2, P3 are shown with respect to the bone 14 and the instrument 100. Each light source position P1, P2, P3 is substantially parallel to the other light source positions P1, P2, P3, but is displaced in the translational direction. Therefore, the control system 18 cannot rely solely on the instrument offset angle IA (not shown) or the radial distance calculation described above. Instead, when determining which image to display on the display 20, the control system 18 can determine the presence of the drilling start point 300 and the trajectory end point 302 within the image boundary. In other words, when determining which image to display on the display 20, the control system 18 can determine whether the trajectory of the instrument 100 defined by the line segment extending between the drilling start point 300 and the trajectory end point 302 is within the field of view of the imaging system 22. This is useful because it is unlikely that the user will request a display 20 showing an X-ray image aligned with the instrument 100 in the rotational direction, and because it focuses on inappropriate areas of the bone 14 and / or the patient 10 at other points.

[0146] Referring to FIGS. 19B - 19D, corresponding AP images are shown for each of the light source positions P1, P2, P3 shown in FIG. 19A. FIG. 19B shows the corresponding AP image when the light source 26 is at the first light source position P1. FIG. 19C shows the corresponding AP image when the light source 26 is at the second light source position P2. FIG. 19D shows the corresponding AP image when the light source 26 is at the third light source position P3. As can be understood, the most desirable AP image is depicted in FIG. 19C. This is because this is the only AP image in which the trajectory of the instrument 100 is within the image boundary and is therefore visible. The other two images are similarly aligned with the posture and rotational direction of the instrument 100, but are not selected by the control system 18 because the trajectory of the instrument 100 is outside the image boundary.

[0147] Referring to FIG. 20A, three light source positions P1, P2, P3 are shown with respect to the bone 14 and the instrument 100. Similar to FIG. 19A, each light source position P1, P2, P3 is substantially parallel to the other light source positions P1, P2, P3, but is shifted in the translational direction. The light source positions P1, P2, P3 are shown at different distances from the bone 14, but this is for illustrative purposes. It should be understood that the light source positions P1, P2, P3 in FIG. 20A are at the same distance from the bone 14 (the differences in the distances from the bone 14 to the light sources will be discussed below with reference to FIGS. 21A-21C). The difference compared to FIG. 19A is that each of the light source positions P1, P2, P3 should result in an X-ray image that at least partially includes the trajectory of the instrument 100 within the image boundary. In some implementations, the control system 18 includes a threshold percentage, such as 30, 40, or 50%, to determine whether the image can be displayed according to the percentage of the trajectory present within the image boundary. Further, in some implementations, the control system 18 requires that the perforation start point 300 be within the image boundary for the image to be displayable. Additionally, in other implementations as well, the control system 18 requires that at least one of the perforation start point 300, the trajectory end point 302, and the depth end point 304 (not shown) be within the image boundary for the image to be displayable.

[0148] When the control system 18 determines that the orbit end point 302 is not within the image boundary, the control system 18 can determine the radially bounded distance within the image and the orbit end point bounded by the image. The length / distance of the radially bounded distance within the image is equal to the portion of the radial distance that is within the image boundary of the X-ray image. For example, if the image boundary is in the middle between the puncture start point 300 and the orbit end point 302, the radially bounded distance within the image is 50 percent of the radial distance. The control system 18 can utilize the radially bounded distance within the image instead of (or as a representation of) the proportion of the orbit that exists within the image boundary. In so doing, the control system 18 can also determine whether the image can be displayed according to a comparison between the radial distance and the radially bounded distance within the image. This comparison is generally a percentage value, and thus the length / distance of the radially bounded distance within the image must be greater than or equal to a threshold percentage of the radial distance. In one example, the radial distance can be equal to 5 centimeters. If the threshold percentage is 50 percent, the radially bounded distance within the image must be at least 2.5 centimeters in order to consider the display of the X-ray image on the display 20. The threshold percentage can be greater than or less than 50 percent and can be set by the user. If the control system 18 determines that the X-ray image does not include the orbit end point 302 but includes a radially bounded distance within the image that is large enough (compared to the radial distance), the system 18 can determine the orbit end point bounded by the image (not shown). The orbit end point bounded by the image represents the end point of the orbit relative to the image boundary. Since not all of the orbit is necessarily within such an image, the orbit end point bounded by the image is calculated as the point where the orbit intersects the image boundary. The orbit end point bounded by the image can be interpolated by the control system 18 as the intersection between the orbit (represented by the line extending between points 300, 302) and the image boundary.

[0149] Referring to FIGS. 20B to 20D, corresponding AP images for each of the light source positions P1, P2, P3 shown in FIG. 20A are shown. FIG. 20B shows the corresponding AP image when the light source 26 is at the first light source position P1. FIG. 20C shows the corresponding AP image when the light source 26 is at the second light source position P2. FIG. 20C shows the corresponding AP image when the light source 26 is at the third light source position P3. As can be understood, the most desirable AP image is depicted in FIG. 20D, because the trajectory of the instrument 100 is completely within the image boundary, and thus it is the only AP image in which the trajectory of the instrument 100 can be completely seen. Although the other two images are also equally aligned with the posture and rotation direction of the instrument 100, they are not selected by the control system 18 because at least a part of the trajectory of the instrument 100 is outside the image boundary. If the image generated by the imaging system 22 with the light source 26 at the third light source position P3 is not available, as an alternative, the control system 18 can select one of the images shown in FIGS. 20B and 20C. For example, if the control system 18 is configured to select an image in which the ratio of the trajectory exceeds a threshold ratio, the ratio of the trajectory within the image boundary of FIG. 20C is larger, as indicated by the presence of more line segments within the image, so the image of FIG. 20C can be used. If the control system 18 requires that the piercing start point 300 be within the image boundary, the image of FIG. 20B can be used. Further, if the control system 18 requires that the trajectory end point 302 be within the image boundary, the image of FIG. 20C can be used.

[0150] Referring to FIG. 21A, two light source positions P1, P2 with respect to the bone 14 and the instrument 100 are shown. Similar to FIGS. 19A and 20A, the first light source position P1 is substantially parallel to the second light source position P2, but the second light source position P2 is zoomed out relative to the first light source position P1, and thus the image provided by the second light source position P2 captures more of the bone 14 than the image provided by the first light source position P1. In other words, the image boundary of the image provided by the second light source position P2 encompasses more of the bone 14. Since each image reference axis is supposed to be aligned with the orientation of the instrument 100, the control system 18 can rely on the presence of at least one of the trajectory of the instrument 100, the drilling start point 300, and the trajectory end point 302 to select which image to display on the display 20. This is similar to the function described with reference to FIGS. 20A-20D, except that the control system 18 selects a more zoomed-out image to increase the visibility of the trajectory of the instrument 100 rather than selecting an image that is shifted in the translational direction. The light source positions P1, P2 are shown at different distances from the bone 14 to indicate the difference in zoom level, but it will be understood that any method of varying the zoom level of the resulting image is contemplated (including methods where the light source 26 does not change position between zoom levels).

[0151] Referring to FIGS. 21B and 21C, corresponding AP images for each of the light source positions P1, P2 shown in FIG. 21A are shown. FIG. 21B shows the corresponding AP image when the light source 26 is at the first light source position P1 and results in an image with a first zoom. FIG. 21C shows the corresponding AP image when the light source 26 is at the second light source position P2 and results in an image with a second zoom, where the second zoom provides a more zoomed-out view of the bone 14. The first light source position P1 and the second light source position P2 may be the same, and it is also contemplated that the second zoom may be provided without moving the light source 26. As will be appreciated, the most desirable AP image is depicted in FIG. 21C, as this is the only AP image in which the trajectory of the instrument 100 is entirely within the image boundaries and thus the trajectory of the instrument 100 can be seen in its entirety. The other images are also equally aligned with the orientation and rotational direction of the instrument 100, but are not selected by the control system 18 because at least a portion of the trajectory of the instrument 100 is outside the image boundaries. If an image generated by the imaging system 22 with the light source 26 in the second light source position P3 (and / or the second zoom) is not available, the control system 18 may alternatively select the image shown in FIG. 21B.

[0152] Referring to FIG. 22A, two light source positions P1, P2 with respect to the bone 14 and the instrument 100 are shown. Unlike the previous figures, the light source positions P1, P2 provide images having different alignments with respect to the orientation of the instrument 100. The first light source position P1 is aligned to provide an image having an image reference axis (not shown) that is substantially parallel to the tool axis T, while the second light source position P2 is aligned to provide an image having an image reference axis (not shown) that is not substantially parallel to the tool axis T. As described above, when selecting the image to be shown on the display 20, the control system 18 can consider whether at least one of the trajectory of the instrument 100, the perforation start point 300, and the trajectory end point 302 is within the image boundary. In some implementations, the control system 18 prioritizes the presence of the trajectory and points 300, 302 within the image boundary over the alignment between the image reference axis and the trajectory. In such implementations, the selection of the image by the control system 18 is based on the fact that, even if not aligned with the surgeon's field of view like other images, the user should see an image that includes the target perforation area (i.e., the trajectory and points 300, 302).

[0153] Referring to FIGS. 22B and 22C, corresponding AP images for each of the light source positions P1, P2 shown in FIG. 22A are shown. Since the images shown in FIG. 22B do not include the target perforation area and are thus likely to be of little use to the user, these figures emphasize the value of the priority. The images shown in FIG. 22C may not be aligned as desired by the user, but the images are still better than the images in FIG. 22B for showing the user the orientation of the instrument 100 with respect to the bone 14.

[0154] Referring to FIG. 23A, two light source positions P1, P2 are shown with respect to the bone 14 and the instrument 100. Similar to FIG. 21A, the first light source position P1 is substantially parallel to the second light source position P2, and the second light source position P2 is zoomed out relative to the first light source position P1. Thus, the image provided by the second light source position P2 captures more of the bone 14 than the image provided by the first light source position P1. In other words, the image boundary of the image provided by the second light source position P2 encompasses more of the bone 14. Also in this case, since each image reference axis should be equally aligned with the posture and position of the instrument 100, the control system 18 must rely on separate factors when determining which image to display on the display 20. Further, since the image boundary of either image should include the entire trajectory and points 300, 302, the control system 18 cannot rely on these to determine which image to display on the display 20. Having said that, FIGS. 23A-23C illustrate another function of the control system 18. Instead of or in addition to choosing which image to show based on the considerations described above, the control system 18 can determine which image to show based on the distance between the distal end of the end effector 102 and the drilling start point 300.

[0155] Referring to FIGS. 23B and 23C, corresponding AP images for each of the light source positions P1, P2 shown in FIG. 23A are shown. FIG. 23B shows the corresponding AP image when the light source 26 is at the first light source position P1 and results in an image with a first zoom. FIG. 23C shows the corresponding AP image when the light source 26 is at the second light source position P2 and results in an image with a second zoom, the second zoom providing a more zoomed-in view of the bone 14 and the trajectory. The first light source position P1 and the second light source position P2 may be the same, and it is also contemplated that the second zoom may be provided without moving the light source 26. As described above, the control system 18 can determine which image to display based on the distance between the distal end of the end effector 102 and the perforation start point 300. When the distance is greater (i.e., the instrument 100 is further away from the patient 10), the control system 18 can display the image shown in FIG. 23B to provide the user with a wider and more comprehensive view of the bone 14. When the distance is smaller (e.g., below a threshold distance), the control system 18 can display the image shown in FIG. 23C to provide the user with a closer and more focused view of the bone 14. Thereby, the user can better see the position of the end effector 102 with respect to the points 300, 302, making it possible to use the instrument 100 more precisely. If different images (e.g., the image of FIG. 23C) are not available, as an alternative, the control system 18 can increase the zoom level of the image of FIG. 23B and crop the zoomed-in image to make the position of the end effector 102 more visible.

[0156] In certain implementations, the control system 18 can determine which of the two images shown in FIGS. 23B and 23C to display on the display 20 based on the displacement of the end effector 102 when the user is perforating the bone 14 with the instrument 100. More specifically, the control system 18 can receive a signal from the depth sensor 122 indicating that the end effector 102 is moving between the perforation start point 300 and the trajectory end point 302. In other words, it indicates that the depth end point 304 is moving from the perforation start point 300 towards the trajectory end point 302. The control system 18 can switch from the X-ray shown in FIG. 23B to the X-ray shown in FIG. 23C when the depth end point 304 moves within a threshold distance of the trajectory end point 302.

[0157] Except for what is shown in FIGS. 18A and 18B, the projected end effector 102 is not shown in any of the images, but it will be understood that this feature can be included in any of the implementations discussed.

[0158] Furthermore, it will be understood that the ideal AP and ML images shown on the display 20 (in any implementation) should be substantially aligned with the tool axis T and substantially orthogonal to the tool axis T, as described above. To avoid losing the details of these figures, the selected images for these figures are slightly intentionally shifted. For example, if the ideal AP image were shown in the figure, the perforation start point 300 and the trajectory end point 302 would overlap, making it difficult to distinguish the trajectory. In practice, the ideal AP and ML images can be shown on the display 20.

[0159] In some implementations, the control system 18 is configured to reduce the computational cost of determining which X-ray images to display on the display 20 as AP and ML images. Therefore, the AP image can be selected from a plurality of AP X-ray images created when the light source 26 and the detector 24 are substantially disposed on the front side and the rear side of the patient 10 (and / or the bone 14), respectively. For example, all of the plurality of AP X-ray images may have been captured by the C-arm 32 when the light source 26 was disposed within a specific range of rotational degrees of the light source position correlated with the ideal AP image. Thus, the control system 18 can be optimized by restricting the selection of the AP image to the plurality of AP X-ray images. The control system 18 can also optimize the selection of the ML image in a similar manner. For example, all of the plurality of ML X-ray images may have been captured by the C-arm 32 when the light source 26 was disposed within a specific range of rotational degrees of the light source position correlated with the ideal ML image.

[0160] Referring to FIG. 24, the instrument 100 is depicted in various poses with respect to the bone 14, and the display 20 shows the images selected by the control system 18 according to the corresponding pose of the instrument 100. As shown in this figure, the X-ray images shown on the display 20 are visible to the user and change as the instrument 100 is moved with respect to the bone 14 (i.e., with respect to the reference device 200). This enables the user to operate on the bone 14 without having to interpret where the instrument 100 is going with respect to non-aligned X-ray images. Instead, as described herein, X-rays aligned with the pose of the instrument 100 are provided to the user. As a result, the images of the bone 14 provided to the user are aligned with the user's field of view of the bone 14, making the X-rays much easier for the user to interpret.

[0161] Referring to FIGS. 25-33, as described above, a flowchart is provided that details how the control system 18 assists the user during the procedure (such as by determining and displaying AP and ML images). It will be understood that the steps shown in these figures can be performed in any suitable order. Some steps may be omitted and other steps may be added.

[0162] Referring to FIGS. 25A-25C, a method 500 for guiding a user during a procedure is provided. Method 500 begins with a series of steps that mean setting up various devices associated with the procedure. In the method 500 shown, the control system 18 and the display 20 are implemented as elements of a tablet computer. It will be understood that the steps described with reference to the tablet can also be performed by alternative forms of the control system 18 and the display 20. For example, system 12 can include a control system 18 and a display 20 implemented by any alternative means identified herein. In the method 500 shown, the user is a surgeon.

[0163] At 502, a fluoro disk is attached to the light source 26 of the C-arm 32. At 504, the tablet (and / or alternative display 20 and control system 18) is powered on. At 506, communication is established between the C-arm 32 and the tablet. Thereby, the control system 18 communicates with the display 20 and / or the C-arm 32. The communication can be wired or wireless. At 508, the attachment 104 is connected to the battery 114 to power on the attachment 104. The attachment 104 can be automatically powered on after electrical communication with the battery 114 is established, or the control element 150 can include a power button. At 510, wireless communication is established between the tablet and the attachment 104. Thereby, the control system 18 communicates with the attachment 104 and / or the instrument controller 110.

[0164] At 512, patient 10 is placed on the operating table (shown in FIG. 1 but without reference numerals). At 514, the surgeon incises patient 10 to expose bone 14. At 516, optionally, the surgeon attaches bone plate 16 to bone 14 via temporary fixation means such as pins or K-wires. At 518, reference device 200 is fixed to the table and / or patient 10. Reference device 200 is fixed at a location visible to the imaging system 22 (in this case, C-arm 32) as well as to a location visible to the surgeon. At 520, C-arm 32 is positioned relative to patient 10 and / or bone 14 within the light source position such that it can acquire an x-ray image. At 522, C-arm 32 captures an x-ray image of patient 10 and / or bone 14. At 524, the tablet automatically detects that an x-ray image has been captured at 522. Further at 526, the tablet displays the x-ray image captured at 522. More specifically, the control system can be configured to control-select the x-ray image based on the time the image was acquired and the current time. For example, the control system can be configured to select the x-ray image for which the duration between the time the image was acquired and the current time is minimized.

[0165] At 530, an x-ray image process is executed. The x-ray image process is shown in detail in FIG. 26 and is described in detail below with reference to FIG. 26.

[0166] At 532, the control system 18 determines whether it can calculate the light source position based on the visibility of the reference device 200 in the X-ray image. If the light source position can be calculated, the position of the light source 26 (with respect to the reference device 200) is calculated and stored as metadata associated with the X-ray image. Otherwise, the method 500 proceeds to 534, and the tablet instructs the surgeon to change the position of at least one of the C-arm 32 and the reference device 200 to ensure that the light source position can be calculated based on the orientation of the reference device 200 in the X-ray image. If necessary, the method 500 returns to 520 and ignores that X-ray image. Otherwise, the method 500 proceeds to 536. At 536, the surgeon determines whether the X-ray image is desirable for guidance. If the X-ray image is desirable for guidance, the method 500 proceeds to 538. Otherwise, the X-ray image is ignored and the method 500 returns to 520. At 538, the surgeon uses the control element 150 of the attachment 104 to select the X-ray image for guidance. At 540, the attachment 104 communicates that selection to the tablet / control system 18.

[0167] At 542, the tablet stores the X-ray image along with its transformation matrix as metadata associated with the image. The transformation matrix is the imaging system-reference device transformation shown in and described above with respect to FIG. 11B. The imaging system-reference device transformation is different for each X-ray image and is calculated based on the orientation of the reference device 200 in the appropriate X-ray image. The X-ray image can be stored on the tablet or any device that communicates with the control system 18, such as a separate server and / or cloud device.

[0168] At 544, the field of view of the image slices of all the stored X-ray images is shown to the user on the display 20. At 546, the surgeon determines whether additional / new images are desired. If additional / new images are desired, at 548, the surgeon can use the control element 150 to initiate the capture of a new X-ray image. If new images are desired, the method returns to 520. However, if not, the method proceeds to 550. At 550, the surgeon places the instrument 100 (i.e., the end effector 102) on the bone 14 and uses the control element 150 to initiate the guiding process. As described herein, the guiding process can also be initiated by depressing the cannula instead of using the control element 150. The guiding process is shown in more detail in FIGS. 27A and 27B. At 560, the guiding process is continuously performed until the procedure is completed or until the surgeon desires a new image to be generated. At 580, if the procedure is completed, the method 500 ends. Otherwise, the method 500 returns to 544.

[0169] Referring to FIG. 26, the X-ray image process is shown in more detail. The X-ray image process is performed at 530 during method 500. At 530A, the X-ray image is processed to remove distortion. The distortion is detected and removed using the fluoroscopic disk 34. At 530B and 530C, the intrinsic and extrinsic parameters of the C-arm 32 are determined by the control system 18. The intrinsic parameters indicate the parameters of the C-arm 32, such as the focal length, field of view, aperture, resolution, and other parameters associated with how the C-arm 32 captures the X-ray image (excluding the extrinsic parameters). The intrinsic parameters can be determined using the posture of the fluoroscopic disk 34 and / or the reference device 200. The extrinsic parameters, on the other hand, indicate the position and orientation (e.g., posture) of the C-arm 32 and are calculated using the posture of the reference device 200 within the image. Both the intrinsic and extrinsic parameters are calculated for each image, and the combination of these parameters forms the imaging system-reference device transformation. Thus, at 530D, the imaging system-reference device transformation is determined for the X-ray image. Finally, at 530E, artifacts present in the image for the fluoroscopic disk 34 and the reference device 200 are removed from the X-ray image. The X-ray image process then ends.

[0170] Referring to FIGS. 27A and 27B, the guidance process is shown in more detail. At 560A and 560B, the control system 18 determines whether the control element 150 has been utilized by the surgeon and / or whether the cannula 116 has been compressed. When each element 150, 124 has been utilized and compressed respectively, the guidance process proceeds to 560C. Otherwise, the control guidance process returns to 560A. As an alternative, the control system 18 can require only a signal from one of the control element 150 and the cannula 116 to proceed to 560C. At 560C, an optical image is captured using the camera 146. At 560D, the control system 18 checks whether an image has been captured. If no image has been captured, the process returns to 560C. If an image has been captured by the camera 146, the process proceeds to 560E. At 560E, the control system 18 determines whether the optically detectable reference 204 is visible within the optical image. If the optically detectable reference 204 is not visible within the optical image, the process returns to 560C. If the optically detectable reference 204 is visible within the optical image, the process proceeds to 560F.

[0171] At 560F, the control system 18 determines the camera-reference device transformation matrix as described above. After calculating the camera-reference device transformation matrix, the process proceeds to 560G. At 560G, the IMU is reset to align the sensor coordinate system with the coordinate system of the camera 146 (i.e., the camera coordinate system). For that purpose, the process proceeds to 560H where the IMU-camera transformation matrix is determined. The IMU-camera transformation matrix is determined by a combination of the camera-reference device transformation, the known spatial relationship between the optical reference 204 and the rest of the reference device 200, and the known spatial relationship between the IMU and the camera 146 as described above. The IMU-camera transformation and the camera-reference device transformation can be combined for the IMU-reference device transformation matrix. The IMU-reference device transformation matrix (or the combination of the IMU-camera transformation and the camera-reference device transformation) enables the control system 18 to transform signals from the IMU into changes in the pose of the instrument 100 relative to the reference device 200.

[0172] At 560I / 562, the guidance process determines the drilling start point 300 using the drilling start point determination process. The drilling start point determination process is shown in more detail in FIG. 28. At 560J / 564, the guidance process determines the track end point 302 using the track end point determination process. The track end point determination process is shown in more detail in FIG. 29. At 560K / 566, the guidance process determines the depth end point 304 using the depth end point determination process. The depth end point determination process is shown in more detail in FIG. 30. At 560L, points 300, 302, and 304 are sent to the tablet (i.e., sent to the control system 18). At 560M / 568, the guidance process calls the image display process to determine which image to show on the tablet (i.e., the display 20). At 560N, the control system 18 determines whether the surgeon is using the instrument 100 to drill the bone 14 yet. In other words, it determines whether the drilling is completed. If the drilling is not completed, the guidance process returns to 560L. If the drilling is completed, the guidance process ends.

[0173] Referring to FIG. 28, the determination of the drilling start point is shown in more detail. At 562A, the drilling start point determination process determines the transformation matrix between the drill reference axis and the camera coordinate system. At 562B, based on alternative means such as the identification feature 154 of the drill bit or user input indicating the type and / or length of the end effector, or based on the detection of the camera of the end effector length, a point representing the distal end of the end effector 102 on the end effector axis (and / or tool axis) is calculated or known. At 562C, the process transforms the coordinates of the distal end of the end effector 102 from the known end effector / tool axis to the known axis of the camera 146 based on a pre-defined stored end effector-camera transformation matrix. At 562D, the process transforms the coordinates of the distal end of the end effector 102 from the camera coordinate system to the reference coordinate system based on the camera-reference device transformation. At 562E, the drilling start point 300 in the reference coordinate system (i.e., relative to the reference reference axis) is determined.

[0174] Referring to FIG. 29, the determination of the trajectory end point is shown in more detail. At 564A, the drilling start point 300 with respect to the reference axis determined at 562E is converted from the reference coordinate system to the sensor coordinate system. The point 300 is converted using a combination of the IMU-camera transformation matrix and the camera-reference device transformation matrix, as well as the current attitude of the IMU. At 564B, based on the known relationship between the end effector / tool axis and the camera coordinate system and the IMU-camera transformation matrix, the coordinates of the point 300 in the sensor coordinate system are converted from the sensor coordinate system to a point on the end effector / tool axis. At 564C, based on the drilling start point 300 (with respect to the end effector / tool axis) and the cannula travel length, the trajectory end point 302 with respect to the end effector / tool axis is determined. For example, if the cannula travel length is 100 millimeters, the trajectory end point 302 (with respect to the end effector / tool axis) is displaced 100 millimeters from the drilling start point 300 along the end effector / tool axis. At 564D, the process converts the trajectory end point 302 from a point on the end effector / tool axis to a point in the sensor coordinate system based on the known relationship between the end effector / tool axis and the sensor coordinate system. At 564E, the process uses a combination of the IMU-camera transformation and the camera-reference device transformation to convert the trajectory end point 302 from the sensor coordinate system to the reference coordinate system. At this point, the trajectory end point 302 is determined in the reference coordinate system (i.e., with respect to the reference axis), and the process ends.

[0175] Referring to FIG. 30, the depth end point determination is shown in more detail. At 566A, the drilling start point 300 with respect to the reference reference axis determined at 562E is converted from the reference coordinate system to the sensor coordinate system. The point 300 is converted using a combination of the IMU-camera transformation matrix and the camera-reference device transformation matrix, as well as the current orientation of the IMU. At 566B, based on the known relationship between the end effector / tool axis and the camera coordinate system and the IMU-camera transformation matrix, the coordinates of the point 300 in the sensor coordinate system are converted from the sensor coordinate system to a point on the end effector / tool axis. At 566C, based on the signal from the drilling start point 300 (with respect to the end effector / tool axis) and the depth sensor 122, the depth end point 304 with respect to the end effector / tool axis is determined. For example, if the depth sensor 122 reports that the distal end of the end effector 102 has advanced 10 millimeters into the bone 14, the depth end point 304 (with respect to the end effector / tool axis) is displaced 10 millimeters from the drilling start point 300 along the end effector / tool axis. At 566D, the process converts the depth end point 304 from a point on the end effector / tool axis to a point in the sensor coordinate system based on the known relationship between the end effector / tool axis and the sensor coordinate system. At 566E, the process converts the depth end point 304 from the sensor coordinate system to the reference coordinate system using a combination of the IMU-camera transformation and the camera-reference device transformation. At this point, the depth end point 304 is determined in the reference coordinate system (i.e., with respect to the reference reference axis), and the process ends.

[0176] Referring to FIG. 31, the image display process is shown in more detail. At 560M / 568, the image display process is called by the guidance process. At 568A, the AP image determination process is used to determine which X-ray images to display on the display 20 as AP images. At 568B, the ML image determination process is used to determine which X-ray images to display on the display 20 as ML images. After appropriate AP and ML images are determined, the image display process proceeds to 568C. At 568C, the depth value of the end effector 102 is determined. At 568D, if the end effector 102 has advanced less than, for example, 10 millimeters within the bone 14, the AP and ML images are shown side by side on the display 20. On the other hand, if the end effector 102 has advanced more than, for example, 10 millimeters into the bone 14, only the ML image is shown on the display 20. Regardless of whether one or both of the AP and ML images are shown on the display 20, the image display process proceeds to 568E. At 568E, the points 300, 302, 304 are each transformed into the image coordinate system of the selected AP and / or ML image. At 568F, a trajectory line segment is drawn between the perforation start point 300 and the trajectory end point 302. At 568G, the current pose of the end effector 102 is drawn as a line between the perforation start point 300 and the depth end point 304.

[0177] At 568, the image display process can change the zoom level and / or crop the image shown on the display 20. Alternatively, the image display process can determine new AP and / or new ML images that provide the surgeon with a more zoomed-in / less zoomed-in view of the bone 14. For example, the image display process can utilize the process described with reference to FIGS. 23A - 23C. After 568H, the image display process ends.

[0178] Referring to FIG. 32, the AP image determination process is shown in more detail. The AP image determination process is performed on each of the X-ray images 568A, 570, for example, on all the stored X-ray images shown at 544. At 570A, the piercing start point 300 is converted from the reference coordinate system to the image coordinate system of the X-ray image. At 570B, the control system 18 determines whether the piercing start point 300 is within the image boundary of the X-ray image. If the piercing start point 300 is not within the image boundary of the X-ray image, the AP image determination process proceeds to the next X-ray image and returns to 570A. If the piercing start point 300 is within the image boundary of the X-ray image, the process proceeds to 570C. At 570C, the trajectory end point 302 is converted from the reference coordinate system to the image coordinate system of the X-ray image. It is assumed that for the remaining part of the AP image determination process, the AP image is selected using the radial distance measurement described above with reference to FIGS. 17A - 18D. It is further contemplated to modify the AP image determination process to use instead the instrument offset angle IA described above with reference to FIGS. 16A and 16B.

[0179] At 570D, as described above, the radial distance between the piercing start point 300 and the trajectory end point 302 is determined. At 570E, as described above, the image-bounded radial distance between the piercing start point 300 and the image-bounded trajectory end point is determined. At 570F, the AP image determination process determines whether all the X-ray images have been processed. If not all the X-ray images have been processed, the process returns to 570A and is repeated for each X-ray image stored at 542 / 544. After all the images have been processed, the process proceeds to 570G. At 570G, the AP image is selected as the image having the minimum radial distance, and optionally, the length of the image-bounded radial distance is equal to at least 50 percent of the length of the radial distance calculated at 570D. After the AP image is selected, the process ends.

[0180] Referring to FIG. 33, the ML image determination process is shown in more detail. The ML image determination process is performed on each of the X-ray images 568B, 572, for example, on all the stored X-ray images shown at 544. At 572A, the piercing start point 300 is converted from the reference coordinate system to the image coordinate system of the X-ray image. At 572B, the control system 18 determines whether the piercing start point 300 is within the image boundary of the X-ray image. If the piercing start point 300 is not within the image boundary of the X-ray image, the ML image determination process proceeds to the next X-ray image and returns to 572A. If the piercing start point 300 is within the image boundary of the X-ray image, the process proceeds to 572C. At 572C, the trajectory end point 302 is converted from the reference coordinate system to the image coordinate system of the X-ray image. Assume that the remaining part of the ML image determination process uses the radial distance measurement described above with reference to FIGS. 17A-18D to select the ML image. It is further contemplated to modify the ML image determination process to use the instrument offset angle IA described above with reference to FIGS. 16A and 16B instead.

[0181] At 572D, as described above, the radial distance between the piercing start point 300 and the trajectory end point 302 is determined. At 572E, as described above, the image-bounded radial distance between the piercing start point 300 and the image-bounded trajectory end point is determined. At 572F, the ML image determination process determines whether all the X-ray images have been processed. If not all the X-ray images have been processed, the process returns to 572A and is repeated for each X-ray image stored at 542 / 544. After all the images have been processed, the process proceeds to 572G. At 572G, the ML image is selected as the image having the maximum radial distance, and the length of the image-bounded radial distance is equal to at least 50 percent of the length of the radial distance calculated at 572D. After the ML image is selected, the process ends.

[0182] In certain implementations, it is not necessary to include a depth sensor within attachment 600. In such cases, the depth sensor may be completely omitted or may be included within a separate attachment 602 that is separable from attachment 60 which includes a camera and / or an inertial measurement unit, or separable from the surgical handpiece 604 (see FIG. 35). The surgical attachment 600 can include a camera, a gyroscope, and one or more accelerometers. Additionally, the surgical attachment 602 can include the depth sensor and the measurement cannula described throughout this specification. One or both of these attachments 600, 602 can include an antenna, a battery, or a terminal for receiving or sensing data and / or transmitting or receiving power.

[0183] Furthermore, a surgical system as described herein and as shown in FIG. 5 or FIG. 35 can be adapted to a wide range of surgical end effectors. Thus, while a specific example is described with respect to a drill bit, such an example is also contemplated to have any of the end effectors such as a drill bit, a screwdriver bit, a surgical pin, a surgical wire, a reamer, or an excavating tool.

[0184] In a particular example, referring to FIG. 34A, the surgical attachment 702 can be removably coupled directly to the surgical drivers 700, 700', 700'', 700''', 700'''', 700''''' including those without a depth sensor. The surgical driver can include a housing having a first end for coupling to a surgical handpiece 704 and a second end opposite the first end. The surgical driver 700 can be configured to drive an end effector. Thus, the surgical driver 700 and the attachment 702 can each have complementary mating features, such as suitable slots and protrusions, magnets and ferromagnetic features, latches, etc., to facilitate their coupling to each other. The driver 700 can include a transmission for performing a speed reduction, a speed increase, a torque increase, or a torque reduction. The designs of the various drivers and / or the surgical handpiece are described in U.S. Pat. Nos. 9,192,394, 5,993,454, 10,925,657, 10,025,657, 10,406,606, which are hereby incorporated by reference in their entirety and are contemplated for use in the surgical system described herein. The attachment 702 can be configured to be coupled to the surgical driver 700 or the surgical handpiece 704. The attachment 702, in such a configuration, can include a battery and can include an antenna as described separately. Further, the attachment 702 can include an inertial measurement unit and / or a camera.

[0185] Referring to FIG. 34B, a wide range of different surgical drivers 800 are contemplated, including but not limited to reamer 800, pin and / or wire driver 800', drill bit driver 800'', and saw driver 800'''. These can be connected to a surgical attachment 800 that includes a camera, gyroscope, and one or more accelerometers. Each driver 800 can be configured to drive a respective end effector that includes a reamer 804, pin / wire 806, drill bit 808, and saw blade 810. Thus, the attachment and associated control system can enable the user to visualize the trajectory of these end effectors relative to one or more X-ray images.

[0186] In one example, the systems described throughout this specification can be made operable to allow switching of end effectors during a procedure while using the same attachment. This can enable visualization of the drill bit trajectory history obtained in the same trajectory as visualized for the real-time trajectory of the screwing trajectory. The drill bit should need to be removed from the surgical handpiece and replaced with a surgical screwdriver. The trajectory of the first end effector can also have a different virtual representation from the trajectory of the second end effector.

[0187] In another example, in a case where there is an interchangeable surgical driver, a surgical attachment including a camera and an inertial measurement unit can be removably attached to a first surgical driver. Then, using the first surgical driver, a trajectory such as a trajectory set while placing a surgical pin or a surgical wire can be set. This trajectory history can be viewed when later positioning a second surgical driver, such as a driver that drives a drill bit or a screw. The surgical attachment can be removed from the first surgical driver and attached to the second surgical driver. The trajectory of the end effector coupled to the handpiece via the second surgical driver can be visualized simultaneously when visualizing the trajectory of the end effector of the first surgical driver. This can enable a user to determine that they are following the trajectory set using the first surgical driver. The trajectory of the first surgical driver can also have a different virtual representation from the trajectory of the second surgical driver.

[0188] In another aspect of the present disclosure, the surgical instrument can include a camera coupled to the handpiece. The camera can be included as an integrated component of the surgical instrument or as an attachment removably coupled to the surgical handpiece. The surgical handpiece can be configured to drive the surgical end effector directly or via the separable surgical drive described above. The camera can generate a camera signal. The control system can be configured to determine the end effector length based on the camera signal using various image processing techniques while the end effector is coupled to the surgical handpiece. Alternatively, the control system can be configured to determine the end effector type based on the camera signal using various image processing techniques while the end effector is coupled to the surgical handpiece. This can be advantageous because the system can detect the end effector length without requiring separate workflow steps such as pulling a trigger, holding the end effector in front of the camera, or interacting with a user interface or other control element to indicate the length or type of the end effector. The surgical system can be operable to make this determination based on the camera signal and an input signal. More specifically, the surgical system can perform image processing when an input signal indicating that the end effector is fully coupled to the surgical handpiece is received. In these instances, the surgical attachment or instrument can communicate the end effector length to various other connected devices such as the control system, display, imager, etc.

[0189] In one example, the input signal is further defined as a user input signal, the surgical instrument includes a control element, and the control element is operable to generate the user input signal. The control system can be configured to determine the end effector length when the user input signal is received. The control element can be a button on an attachment, display, or surgical handpiece, which in some examples includes a virtual icon on the display.

[0190] In another example, where the surgical instrument includes a depth sensor configured to provide a depth signal associated with the displacement of the end effector, the input signal can be derived from the depth sensor. In this implementation, the control system is configured to determine the end effector length when the depth signal indicates a depth that exceeds a predetermined threshold. For example, an image processing algorithm can process an image from a camera when a measurement cannula is depressed by a specific distance, such as 10 mm. This ensures that the instrument is ready to operate and thus that the end effector is fully inserted and coupled to the surgical handpiece.

[0191] The control system can include a processor and memory programmed by an image processing algorithm, and the image processing algorithm includes steps configured to determine whether the coupled end effector is a first length or a second length. More specifically, the surgical system can include only end effectors of several different sizes, and thus the image processing algorithm need only distinguish end effectors of significantly different lengths, i.e., end effectors that differ in length by at least 5 mm.

[0192] As described, in certain instances, the sensor module can further include a measurement cannula movably coupled to the depth sensor, and the control system is configured to determine the distal end of the end effector and the distal end of the measurement cannula, and the control system is configured to determine the length of the end effector based on the positions of the distal end of the end effector and the distal end of the measurement cannula. In some instances, the length of the end effector is such that the end effector does not protrude from the measurement cannula when the measurement cannula is in the fully distal position. In other instances, the length of the end effector is such that the end effector protrudes from the measurement cannula when the measurement cannula is in the fully distal position. The control system can utilize this relationship to discriminate the length of the end effector coupled to the surgical instrument.

[0193] The image processing algorithm can include steps for extracting features from a portion of the image of the end effector. The extracted features of that portion of the image preferably enable correlation (or pairing) of that portion of the image with respect to the time at which the input signal was received. The features to be extracted can be values related to the intensity, lightness, hue, saturation, luminance, gloss, or other color in at least one component space such as red, blue, green, cyan, magenta, yellow, color tone, and / or the Lab component space of that portion of the image. Further, the features to be extracted can be histograms of various color values in a set of pixels within that portion of the image. Additionally, or alternatively, the features to be extracted can be the estimated surface area of the end effector within the image, the number of pixels of the end effector, the number of pixels of the overall feature, the color intensity value of the end effector, or any other relevant feature specific to that portion of the image of the end effector or available for extraction from that portion of the image of the end effector.

[0194] The image processing algorithm can also include the step of segmenting the image, which includes separating a first segment of the sample image that represents the end of the measurement cannula and further segmenting the first region to define a portion of the image that corresponds to a particular portion of the end effector. Segmenting the sample image into a plurality of image segments preferably increases the resolution and / or accuracy of the length determination. The size and shape of each image segment can be fixed, and each segment can include a predefined number of pixels in the image and / or predefined dimensions in physical space. However, that portion of the image can include the entire end effector, or the image can be segmented or partitioned according to any other scheme. That portion of the sample image is preferably a single segment or region that includes a plurality of pixels of the image, but that portion of the image can alternatively be a plurality of image segments or regions of the sample image, can have any other size, and / or can have any other form.

[0195] It should be understood that the display may be omitted from the surgical system. In such an implementation, the surgical system can transmit data to a display on the imaging device. This data includes suitable virtual representations described throughout this specification. Alternatively, the display can be any suitable mobile device or monitor. The display can include a suitable antenna for wireless communication with any other component of the system.

[0196] The image processing algorithm can also or alternatively be used to estimate the physical dimensions of the sample by localizing, segmenting any object (e.g., using algorithms based on edge detection, background subtraction, graph cuts, etc.), measuring, clustering, pattern recognition, template matching (using any one of various metrics), feature extraction, descriptor extraction (e.g., extraction of texton maps, color histograms, HOG, SIFT, etc.), feature dimensionality reduction (e.g., PCA, K-Means, linear discriminant analysis, etc.), feature selection, thresholding, positioning, color analysis, parametric regression, non-parametric regression, unsupervised or semi-supervised parametric or non-parametric regression, or implementing any other type of machine learning or machine vision. Such methods preferably compensate for varying lighting conditions or any other discrepancies or variations commonly seen in any usage scenario.

[0197] The above description is merely illustrative in nature and is not intended to limit the present disclosure, its applications, or uses in any way. The broad teachings of the present disclosure can be implemented in various forms. Thus, while the present disclosure includes specific examples, other modifications will become apparent upon review of the drawings, this specification, and the following claims, and the true scope of the present disclosure should not be so limited. It should be understood that one or more steps within the scope of one method can be performed in a different order (or simultaneously) without changing the principles of the present disclosure. Further, for each example, while described above as having certain features, any one or more of those features described in connection with any example of the present disclosure can be implemented with any of the features of any other example and / or can be combined with such features even if their combination is not explicitly described. In other words, the described examples are not mutually exclusive, and permutations of one or more examples with respect to each other remain within the scope of the present disclosure.

[0198] For the spatial and functional relationships between elements (e.g., a controller, a circuit element, a semiconductor layer, etc.), various terms including "connected", "engaged", "coupled", "adjacent", "next to", "on", "above", "below", and "disposed" are used for description. Unless explicitly stated to be "direct", when describing the relationship between a first element and a second element in the above disclosure, the relationship may be a direct relationship without other intervening elements between the first element and the second element, or an indirect relationship with one or more intervening elements (spatially or functionally) between the first element and the second element.

[0199] In this specification, the phrase "at least one of A, B, and C" should be interpreted to mean a logical OR (A or B or C) using non-exclusive disjunction, and should not be interpreted to mean "at least one of A, at least one of B, and at least one of C". The term subset does not necessarily require a proper subset. In other words, a first subset of a first set can have (be equal to) the same extent as the first set.

[0200] In these figures, the direction of the arrow indicated by the tip of the arrow generally demonstrates the flow of information (such as data or instructions) related to that figure. For example, if element A and element B exchange various information, and the information transmitted from element A to element B is relevant to that figure, the arrow can point from element A to element B. This one-way arrow does not imply that no other information is transmitted from element B to element A. Further, with respect to the information transmitted from element A to element B, element B can transmit a request for information or an acknowledgment of its receipt to element A.

[0201] In this application example, which includes the following definitions, the term "controller" can be replaced with the term "circuit". The term "controller" can refer to one or more of the following: an application-specific integrated circuit (ASIC), a digital, analog, or analog / digital mixed discrete circuit, a digital, analog, or analog / digital mixed integrated circuit, a combinatorial logic circuit, a field programmable gate array (FPGA), a processor circuit (shared, dedicated, or a group) that executes code, a memory circuit (shared, dedicated, or a group) that stores code executed by the processor circuit, other suitable hardware components that provide the described functionality, or some or all combinations thereof, including as part of, or including, a system-on-chip.

[0202] The controller can include one or more interface circuits. In some examples, the interface circuit can implement a wired or wireless interface to connect to a local area network (LAN) or a wireless personal area network (WPAN). Examples of LANs include Institute of Electrical and Electronics Engineers (IEEE) Standard 802.11-2016 (also known as the WIFI wireless networking standard), and IEEE Standard 802.3-2015 (also known as the ETHERNET wired networking standard). Examples of WPANs include the BLUETOOTH wireless networking standard from the Bluetooth (R) Special Interest Group, and IEEE Standard 802.15.4.

[0203] The controller can communicate with other controllers using an interface circuit. While the controller may be depicted in this disclosure as communicating directly and logically with other controllers, in various configurations, the controller can actually communicate via a communication system. The communication system includes physical and / or virtual networking devices such as hubs, switches, routers, and gateways. In some configurations, the communication system connects to or traverses a wide area network (WAN) such as the Internet. For example, the communication system can include multiple LANs connected to each other via the Internet or a point-to-point dedicated line using technologies including Multiprotocol Label Switching (MPLS) and virtual private networks (VPNs).

[0204] In various configurations, the functions of the controller can be distributed among multiple controllers connected via a communication system. For example, multiple controllers can implement the same function distributed by a load balancing system. In a further example, the functions of the controller can be split between a server (also known as a remote or cloud) controller and a client (or user) controller.

[0205] Some or all of the hardware features of the controller can be defined using languages for hardware description such as IEEE Standard 1364-2005 (commonly referred to as "Verilog") and IEEE Standard 10182-2008 (commonly referred to as "VHDL"). Hardware description languages can be used to manufacture and / or program hardware circuits. In some configurations, some or all of the features of the controller can be defined by languages such as IEEE 1666-2005 (commonly referred to as "SystemC") that incorporate both code and hardware description as described later.

[0206] The term "code" as used above can include software, firmware, and / or microcode and can refer to a program, routine, function, class, data structure, and / or object. The term "shared processor circuit" encompasses a single processor circuit that executes some or all of the code from multiple controllers. The term "group processor circuit" encompasses a processor circuit that, in combination with additional processor circuits, executes some or all of the code from one or more controllers. References to multiple processor circuits include multiple processor circuits on separate dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or combinations of the above. The term "shared memory circuit" encompasses a single memory circuit that stores some or all of the code from multiple controllers. The term "group memory circuit" encompasses a memory circuit that, in combination with additional memory, stores some or all of the code from one or more controllers.

[0207] The term "memory circuit" is a subset of the term "computer-readable medium". As used herein, the term "computer-readable medium" does not include transient electrical or electromagnetic signals propagated through a medium (such as a carrier wave), and thus the term "computer-readable medium" can be considered tangible and non-transitory. Non-limiting examples of non-transitory computer-readable media include non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape, or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray Discs).

[0208] The apparatus and method described in this application can be implemented partially or fully by a special-purpose computer created by configuring a general-purpose computer to execute one or more specific functions implemented by a computer program. The functional blocks and flowchart elements described above serve as software specifications, and such software specifications can be converted into a computer program by the typical work of a skilled technician or programmer.

[0209] The computer program includes processor-executable instructions stored in at least one non-transitory computer-readable medium. The computer program can also include stored data or can rely on stored data. The computer program can include a basic input / output system (BIOS) that interacts with the hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, and the like.

[0210] A computer program can include (i) parsed descriptive text such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation), (ii) assembly code, (iii) object code generated from source code by a compiler, (iv) source code for execution by an interpreter, (v) source code for editing and execution by a just-in-time compiler, and the like. For illustrative purposes only, source code can be written using syntax from languages including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, JavaScript®, HTML5 (Hypertext Markup Language Revision 5), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB®, SENSORLINK, and Python®. Provisions for Additional Protection I. A reference device for aligning a tool axis with an X-ray image, the reference device comprising one or more radiopaque references and optically detectable references and attachment elements for attachment to a patient. II. The reference device according to clause I, wherein the optically detectable reference is an April tag. III. The reference device according to clause I or II, wherein the one or more radiopaque references include a plurality of radiopaque elements. IV. The reference device according to clause III, wherein the plurality of radiopaque elements include a plurality of radiopaque balls. V. The reference device according to any one of Clauses I to IV, wherein the attachment element comprises a pin for attachment to the patient's bone or an adhesive for attachment to a part of the patient's anatomical structure. VI. A surgical attachment configured to be removably coupled to a surgical handpiece assembly, comprising a measurement housing including a proximal region and a distal region, the proximal region including a proximal surface, a depth sensor, a camera, and optionally an inertial measurement unit configured to generate signals regarding orientation data and / or motion parameters, and a measurement coupler extending from the proximal surface of the proximal region of the measurement housing. VII. The surgical attachment according to Clause VI, further comprising a depth cannula movably coupled to the measurement housing and configured to move along a measurement axis relative to the measurement housing through the proximal and distal regions, the depth cannula defining a bore configured to receive a drill bit, and the depth cannula being coupled to the depth sensor. VIII. The surgical attachment according to Clause VI or VII, further comprising an antenna for wireless communication. IX. The surgical attachment according to any one of Clauses VI to VIII, wherein the inertial measurement unit is included and the inertial measurement unit includes a gyroscope and an accelerometer. X. The surgical attachment according to any one of Clauses VI to IX, further comprising a battery or a terminal for receiving power from another component of the surgical handpiece or the surgical system. XI. A surgical system for operating on a patient's bone, comprising a reference device including one or more radiopaque references and configured to have a fixed orientation with respect to a surgical implant or a part of the patient's anatomical structure, the reference device including optically detectable references and radiopaque references, and a surgical instrument for coupling to an end effector, the surgical instrument being A camera configured to generate a first signal corresponding to the pose of an optically detectable reference, A sensor module configured to have a pose fixed with respect to the camera and configured to generate a second signal regarding orientation data and / or motion parameters, A surgical system including an imager, a mobile computing device, and / or an antenna for communicating with a display. XII. A method of using a reference device for aligning a tool axis with an X-ray image, the reference device including one or more radiopaque references and optically detectable references, the method comprising: Fixing the reference device to the patient's bone using an attachment element; After the reference device is fixed to the patient's bone or tissue, imaging the reference device with an imager; Positioning an instrument having a camera and an inertial measurement unit such that the optically detectable reference is within the field of view of the camera; Aligning the inertial measurement unit with the camera based on the output signal of the camera; And displaying a virtual representation of a portion of the instrument based on the output signal of the inertial measurement unit. XIII. A surgical attachment configured to be removably coupled to a surgical handpiece assembly or a component thereof, A measurement housing, A camera, An inertial measurement unit configured to generate a signal regarding orientation data and / or motion parameters, And a measurement coupler for attaching to a surgical handpiece, the surgical attachment, or a second surgical attachment. XIV. The surgical attachment according to clause XIII, further comprising an antenna for wireless communication. XV. The surgical attachment according to clause XIII or XIV, wherein the inertial measurement unit includes a gyroscope and an accelerometer. XVI. The surgical attachment according to any one of clauses XIII to XV, further comprising a terminal for receiving power from a battery, or from another component of a surgical handpiece or a surgical system. XVII. A surgical system for operating on a patient's bone, a surgical instrument for coupling to an end effector, a handpiece for driving the surgical end effector, a surgical instrument including a camera coupled to the handpiece, the camera being configured to generate a first signal, and a control system configured to determine an end effector length based on the first signal. XVIII. The surgical system according to clause XVII, wherein the surgical instrument is configured to have a fixed orientation with respect to the camera and includes a sensor module configured to generate a second signal regarding orientation data and / or motion parameters, and further includes an antenna for communicating with an imager, a mobile computing device, and / or a display. XIX. The surgical system according to any one of clauses XVII to XVIII, wherein the control system is configured to determine an end effector length based on the first signal and an input signal. XX. The input signal is further defined as a user input signal, the surgical instrument includes a control element, the control element is operable to generate the user input signal, and the control system is configured to determine the end effector length at the time when the user input signal is received. XXI. The surgical system according to any one of clauses XVII to XX, wherein the surgical instrument includes a depth sensor configured to provide a depth signal associated with displacement of the end effector during a drilling process, and the control system is configured to determine the end effector length at the time when the depth signal indicates a depth exceeding a predetermined threshold. XXII. The surgical system according to any one of clauses XVII to XXI, wherein the control system includes a processor and a memory programmed by an image processing algorithm, and the image processing algorithm includes steps configured to determine whether the combined end effector is of a first length or a second length. XXIII. The surgical system according to clause XXII, wherein the sensor module further includes a measurement cannula movably coupled to a depth sensor, the control system is configured to determine the distal end of the end effector and the distal end of the measurement cannula, and the control system is configured to determine the length of the end effector based on the positions of the distal end of the end effector and the distal end of the measurement cannula. XXIV. A first surgical attachment configured to be removably coupled to a surgical handpiece assembly or a component thereof, a measurement housing, a camera, an inertial measurement unit configured to generate signals regarding orientation data and / or motion parameters, and a coupler. XXV. The surgical system including the first surgical attachment according to clause XXIV, further comprising an end effector selected from the group consisting of a surgical pin, a surgical wire, a reamer, a drill, or a chisel blade. XXVI. The surgical system according to clause XXV, further comprising a surgical attachment, the surgical attachment including a housing having a first end for coupling to the surgical handpiece and a second end opposite the first end, and the surgical attachment being configured to drive the end effector. XXVII. The surgical system according to clause XXVI, wherein the first surgical attachment is configured to be coupled to the surgical attachment or the surgical handpiece. XXVIII. The surgical system according to clause XXIV, further comprising a second surgical attachment, wherein the second measurement includes a depth sensor, and the second surgical attachment is configured to be coupled to the surgical handpiece and the first surgical attachment. XXIX. A surgical system for operating on a patient's bone, a surgical instrument for coupling to an end effector, a handpiece for driving the surgical end effector, a surgical instrument including a sensor module configured to generate a first signal regarding the orientation of the instrument, the instrument having a tool axis, and a display, and a control system, wherein the control system is configured to align a plurality of X-ray images to a known coordinate system, align the instrument to the known coordinate system, select one of the plurality of aligned X-ray images based on the time at which each image was acquired and the current time, and configured to display the selected X-ray image. XXX. A surgical system for operating on a patient's bone, a surgical instrument for coupling to an end effector, a handpiece for driving the surgical end effector, a surgical instrument including a sensor module configured to generate a first signal regarding the orientation of the instrument, the instrument having a tool axis, and a display, and a control system, wherein the control system is configured to align a plurality of X-ray images to a known coordinate system, align the instrument to the known coordinate system, select one of the plurality of aligned X-ray images based on the duration between the time at which each of the aligned plurality of X-ray images was acquired and the current time, and configured to display the selected X-ray image. XXXI. A surgical system for operating on a patient's bone, A surgical instrument for attachment to an end effector, a handpiece for driving a surgical end effector, a camera configured to generate a first signal corresponding to the pose of optically detectable fiducials relative to a first coordinate system, a surgical instrument including a sensor module configured to have a fixed pose relative to the camera and to generate a second signal regarding orientation data and / or motion parameters relative to a second coordinate system, and a control system, the control system being configured to, (i) receive the first signal, (ii) the second signal, and (iii) an image of a reference device and a patient's bone, receive an input signal related to the starting position of the end effector, in response to the input signal, establish an alignment between the first coordinate system, the second coordinate system, and an image coordinate system based on the first signal, the second signal, and the image, determine a starting position of the end effector relative to a bone in the image in one of the first coordinate system, the second coordinate system, and the image coordinate system based on one of (i) the first signal and (ii) the second signal and based on the alignment, determine a second position of the end effector relative to a bone in the image in one of the first coordinate system, the second coordinate system, and the image coordinate system based on the second signal and the starting position, A surgical system configured to overlay a virtual representation onto the image based on the starting position and the second position of the end effector. XXXII. A surgical system for operating on a patient's bone, comprising a surgical instrument for attachment to an end effector, a handpiece for driving a surgical end effector, a camera configured to generate a first signal corresponding to the pose of the patient relative to a first coordinate system, A surgical instrument including a sensor module configured to have a fixed attitude with respect to a camera and configured to generate a second signal regarding orientation data and / or motion parameters with respect to a second coordinate system, and a control system, the control system (i) receives a first signal, (ii) a second signal, and (iii) an image of a reference device and a patient's bone, receives an input signal related to the starting position of the end effector, in response to the input signal, based on the first signal, the second signal, and the image, establishes an alignment between the first coordinate system, the second coordinate system, and the image coordinate system, determines, in one of the first coordinate system, the second coordinate system, and the image coordinate system, the starting position of the end effector with respect to the bone in the image, based on one of (i) the first signal and (ii) the second signal and the alignment, determines, in one of the first coordinate system, the second coordinate system, and the image coordinate system, a second position of the end effector with respect to the bone in the image, based on the second signal and the starting position, A surgical system configured to overlay a virtual representation on the image based on the starting position and the second position of the end effector.

Claims

1. A surgical system for operating on a patient's bone, comprising: a reference device including one or more radiopaque references, configured to have a fixed orientation with respect to a surgical implant or a part of the patient's anatomical structure, the reference device including optically detectable references and radiopaque references; a surgical instrument for coupling to an end effector; a handpiece for driving the surgical end effector; a camera configured to generate a first signal corresponding to the orientation of the optically detectable reference with respect to a first coordinate system; a surgical instrument including a sensor module configured to have a fixed orientation with respect to the camera and configured to generate a second signal regarding orientation data and / or motion parameters with respect to a second coordinate system; and a control system, wherein the control system: (i) receives the first signal, (ii) the second signal, and (iii) an image of the reference device and the patient's bone; receives an input signal related to the starting position of the end effector; in response to the input signal, based on the first signal, the second signal, and the image, establishes an alignment between the first coordinate system, the second coordinate system, and an image coordinate system; determines, in one of the first coordinate system, the second coordinate system, and the image coordinate system, the starting position of the end effector with respect to the bone in the image based on one of (i) the first signal and (ii) the second signal and based on the alignment; determines, in one of the first coordinate system, the second coordinate system, and the image coordinate system, a second position of the end effector with respect to the bone in the image based on the second signal and the starting position; A surgical system configured to overlay a virtual representation on the image based on the starting position and the second position of the end effector.

2. The surgical system according to claim 1, wherein the sensor module is an inertial measurement unit.

3. The surgical system according to claim 1 or 2, wherein the inertial measurement unit includes at least one of a gyroscope, an accelerometer, and a magnetometer.

4. The surgical system according to any one of claims 1 to 3, wherein the inertial measurement unit is configured to measure in six degrees of freedom.

5. The surgical system according to any one of claims 1 to 4, wherein the camera is further defined as a plurality of cameras.

6. The image includes a shadow of the radiopacity standard, The control system is configured to align the posture of the image coordinate system based on the posture of the shadow of the radiopacity standard. The surgical system according to any one of claims 1 to 5.

7. The surgical system according to any one of claims 1 to 6, wherein the control system is configured to determine the rotation and translation of the camera with respect to the axis of the reference device using the optically detectable standard and to determine a camera-reference device transformation matrix.

8. The surgical system according to claim 7, wherein the control system is configured to determine a starting position of the virtual representation based on the camera-reference device transformation matrix, a known relationship between the optically detectable standard and the reference device, and optionally, a known relationship between the tool axis and the camera, and the length of the end effector.

9. The surgical system according to claim 8, wherein the control system is configured to superimpose the representation based on the camera-reference device transformation matrix, a known relationship between the optically detectable standard and the reference device, and the end effector length.

10. The surgical system according to claim 9, wherein the control system is configured to determine the virtual representation based on a known IMU-camera transformation matrix, the second signal, the camera-reference device transformation matrix, and the known relationship between the optically detectable standard and the reference device.

11. The input signal is further defined as a user input signal, The surgical instrument includes a control element, The control element is operable to generate the user input signal, The control system is configured to determine the starting position based on the posture of the surgical instrument with respect to the reference device at the time when the user input signal is received. The surgical system according to any one of claims 1 to 10.

12. The end effector is a drill bit or a screwdriver bit, The surgical instrument includes a depth sensor configured to provide a third signal associated with displacement of the drill bit or screwdriver bit during a drilling or driving process, the control system is configured to receive the third signal, and the virtual representation is further based on the third signal, The surgical system according to any one of claims 1 to 11.

13. The input signal is based on the third signal, and thus the control system is configured to determine the starting position based on the posture of the surgical instrument when the control system receives the third signal from the depth sensor. The surgical system according to claim 12.

14. The control system is configured to identify the starting position when the depth sensor indicates that the drill bit or screwdriver bit is within the range of the opening of the plate or within the range of the bone. The surgical system according to claim 13.

15. The drill bit or screwdriver bit is coupled to the surgical instrument in a fixed orientation. The surgical system according to claim 14.

16. The surgical instrument further comprises an attachment configured to be attached to the surgical instrument, the attachment including the camera and the sensor module. The surgical system according to any one of claims 1 to 15.

17. The surgical instrument further comprises an attachment configured to be attached to the surgical instrument, the attachment including a depth sensor configured to provide a third signal associated with displacement of the end effector during a drilling or driving process. The surgical system according to claim 13.

18. The control system is configured to control the virtual representation based on the third signal, The surgical system according to claim 17.

19. The control system is configured to determine the depth end point of the virtual representation based on the third signal. The surgical system according to claim 18.

20. The attachment includes a housing, When one of the bits is coupled to the surgical instrument, it includes a measurement cannula configured to circumferentially surround the drill bit or screwdriver bit, and the measurement cannula is slidably attached to the housing such that the measurement cannula extends forward or backward between a fully distal position and a proximal position relative to the housing, and the measurement cannula has a distal end adapted to be positioned against an object or bone. The surgical system according to claim 19, wherein the depth sensor is a displacement sensor for generating the third signal based on the position of the distal end of the measurement cannula relative to the housing.

21. The surgical system according to claim 20, wherein the surgical instrument includes a sensor for generating an identification signal in response to an identification feature of the bit when the bit is coupled to the surgical instrument.

22. The surgical system according to claim 21, wherein the attachment includes the sensor for generating an identification signal.

23. The surgical system according to claim 21, wherein the control system is configured to determine a bit length based on the identification signal.

24. The surgical system according to claim 21, wherein the control system is configured to determine an orbit end point based on the bit length and the third signal.

25. The surgical system according to claim 24, wherein the control system is configured to determine the depth end point based on the third signal.

26. The surgical system according to claim 25, wherein the virtual representation is defined as a first virtual representation, the first virtual representation terminates at a depth end point, the control system is configured to display a second virtual representation, and the second virtual representation terminates at an orbit end point.

27. The surgical system according to claim 26, wherein the orbit end point is based on a range of motion limits for the measurement cannula, and the depth end point is based on the third signal.

28. The surgical system according to any one of claims 24 to 27, wherein the attachment includes an antenna for communicating with the control system.

29. The surgical system according to any one of claims 1 to 28, further comprising a display unit, and the control system communicates with the display unit.

30. The surgical system according to any one of claims 1 to 29, wherein the control system communicates with the imaging system.

31. The surgical system according to any one of claims 1 to 30, further comprising a surgical implant, wherein the surgical implant is a bone plate including one or more holes.

32. Further comprising a fixing member, wherein the reference device includes a coupling portion, wherein a first portion of the fixing member is configured to attach the surgical implant to the bone, and another portion of the fixing member is configured to be coupled to the coupling portion of the reference device. The surgical system according to claim 31.

33. Further comprising a device configured to be attached to the imaging system, wherein the control system is configured to determine the magnification of the image based on at least one of the device and the radiation impermeability standard. The surgical system according to any one of claims 1 to 32.

34. The device is a fluorodisk, and the fluorodisk includes a transparent lens, a plurality of reference markers disposed on the transparent lens, and an attachment member configured to attach the fluorodisk to the imaging system. The surgical system according to claim 33.

35. Further comprising an imaging system, wherein the imaging system is a C-arm capable of performing only 2D X-rays. The surgical system according to claim 33.

36. The surgical system according to any one of claims 1 to 35, wherein the optically detectable reference is an April Tag or an ArUco Tag.

37. The surgical system according to any one of claims 1 to 36, wherein the optically detectable reference is configured to enable calculation of the orientation of the reference device with respect to the camera.

38. The surgical system according to claim 37, wherein the optically detectable reference is positioned on an outer surface of the reference device.

39. The surgical system according to any one of claims 1 to 38, wherein the radiation impermeability standard is defined as a plurality of radiation impermeable elements.

40. The surgical system according to claim 25, wherein the control system is configured to determine a second starting position and display a third virtual representation based on the second starting position.

41. The surgical system according to claim 38, further comprising a second optically detectable reference, wherein the second optically detectable reference is positioned on an outer surface of the reference device.

42. The surgical system according to claim 40, wherein the control system is further configured to control a display to indicate a drilling depth or a screw length based on the third signal.

43. The end effector is coupled to the surgical instrument such that at least a portion of the end effector is within a field of view of the camera, the control system is further configured to determine an identification and / or a length of the end effector based on the portion of the end effector that is within the field of view of the camera, The surgical system according to any one of claims 1 to 42.

44. The surgical system according to claim 43, wherein the control system is configured to determine the identification and / or the length of the end effector using feature matching.

45. A surgical system for operating on a patient's bone, a surgical instrument for coupling to an end effector, a handpiece for driving the end effector, a camera configured to generate a first signal corresponding to a pose of an optically detectable reference with respect to a first coordinate system, a surgical instrument including a sensor module configured to have a fixed pose with respect to the camera and configured to generate a second signal regarding orientation data and / or motion parameters with respect to a second coordinate system, and a control system, wherein the control system establishes an alignment between the first coordinate system, the second coordinate system, and an image coordinate system based on the first signal, the second signal, and an image of a reference device including a radiation-opaque reference and an optically detectable reference, determines a starting position of the end effector with respect to the bone in the image in one of the first coordinate system, the second coordinate system, and the image coordinate system based on one of (i) the first signal and (ii) the second signal and based on the alignment, determines a second position of the end effector with respect to the bone in the image in one of the first coordinate system, the second coordinate system, and the image coordinate system based on the second signal and the starting position. A surgical system configured to superimpose a virtual representation over the image based on the starting position and the second position of the end effector. **Claim 46** A surgical system for operating on a patient's bone, A surgical instrument for coupling to an end effector, A handpiece for driving the end effector, A camera, A sensor module configured to have a fixed posture with respect to the camera and configured to generate a first signal regarding orientation data and / or motion parameters, and A surgical instrument including a depth sensor configured to provide a second signal associated with displacement of the end effector during a drilling, tapping, or driving process, A control system, wherein the control system Determines a starting position of the end effector with respect to the bone in the X-ray image based on the second signal, Determines a second position of the end effector with respect to the bone in the image based on the second signal and the starting position, A surgical system configured to superimpose a virtual representation over the image based on the starting position, the second position of the end effector, and the second signal. **Claim 47** A surgical system for operating on a patient's bone, A surgical instrument for coupling to an end effector, including a sensor module configured to generate a first signal regarding the orientation of the surgical instrument, the instrument having a tool axis, and the sensor module including at least one of an inertial measurement unit and a camera, A display, A control system, wherein the control system Acquires a plurality of 2D X-ray images, each of the 2D X-ray images having an image reference axis, Aligns the plurality of 2D X-ray images to a known coordinate system, Aligns the instrument to the known coordinate system, Selects a 2D X-ray image from the plurality of 2D X-ray images based on the image reference axis and the tool axis, A surgical system configured to superimpose a virtual representation of the end effector over the selected 2D X-ray image on the display, the virtual representation being based on the first signal. **Claim 48** Further comprising a mobile computing device, the mobile computing device including the display and a first processor, the surgical instrument including a second processor, and the control system including the first processor and / or the second processor, the surgical system according to claim 47.

49. The surgical system according to claim 47 or 48, wherein the known coordinate system is a reference coordinate system.

50. The virtual representation is selected from the group consisting of a line and a two-dimensional shape having a longitudinal axis, and the virtual representation is positioned such that the line or the longitudinal axis is aligned with the tool axis, the surgical system according to any one of claims 47 to 49.

51. The control system is configured to determine a relationship between the image reference axis and the tool axis for each of the plurality of 2D X-ray images, and the control system is configured to select a 2D X-ray image based on the relationship, the surgical system according to any one of claims 47 to 50.

52. The surgical system according to claim 51, wherein the relationship is an angle between the image reference axis and the tool axis.

53. The plurality of 2D X-ray images are further defined as a plurality of X-ray images having a first orientation, and the control system is configured to select an X-ray image of the first orientation having a minimum angle between the image reference axis and the tool axis, the surgical system according to claim 51.

54. The surgical system according to claim 53, wherein the first orientation is an AP orientation with respect to an axis of a reference device attached to the patient.

55. The plurality of 2D X-ray images are further defined as a plurality of X-ray images having a second orientation, and the control system is configured to select an X-ray image of the second orientation having the most orthogonal angle between the image reference axis and the tool axis, the surgical system according to any one of claims 47 to 54.

56. The surgical system according to claim 55, wherein the second orientation is a medial-lateral orientation with respect to an axis of a reference device attached to the patient.

57. The surgical system according to any one of claims 47 to 56, wherein the sensor module includes at least one of a camera and an inertial measurement unit.

58. The surgical system according to any one of claims 47 to 57, wherein the sensor module includes a camera and an inertial measurement unit.

59. The surgical system according to claim 58, wherein the inertial measurement unit includes at least one of a gyroscope, an accelerometer, and a magnetometer.

60. The surgical system according to claim 58, further comprising a reference device configured to have a fixed posture with respect to a surgical implant or a part of a patient's anatomical structure, the reference device including an optically detectable reference and a radiopaque reference.

61. The camera is configured to generate a first signal corresponding to the posture of the optically detectable reference with respect to a first coordinate system, the inertial measurement unit is configured to generate a second signal regarding orientation data with respect to a second coordinate system, the control system is configured to receive an input signal related to the starting position of the end effector, and the control system is configured to establish an alignment between an image coordinate system, the first coordinate system, and the second coordinate system based on the first signal, the second signal, and the starting position. The surgical system according to claim 60.

62. Each 2D X-ray image includes a shadow of the radiopaque reference, and the control system is configured to determine the posture of the image coordinate system based on the posture of the shadow of the radiopaque reference. The surgical system according to claim 61.

63. The first coordinate system is realized as an optical coordinate system, and the control system is configured to determine the posture of the optical coordinate system with respect to a reference reference axis of the reference device using the optically detectable reference and determine a camera-reference device transformation matrix. The surgical system according to claim 61.

64. The control system is configured to determine the starting position of the virtual representation based on the camera-reference device transformation matrix, and optionally, a known relationship between the tool axis and the camera, and the end effector length. The surgical system according to claim 63.

65. The surgical system according to claim 64, wherein the control system is configured to superimpose the virtual representation based on the camera-reference device transformation matrix, and a known relationship between the optically detectable reference and the radiopaque reference, and the end effector length.

66. The surgical system according to claim 65, wherein the control system is configured to determine the virtual representation based on a known IMU-camera transformation matrix, the second signal, the camera-reference device transformation matrix, and the known relationship between the optically detectable reference and the radiopaque reference.

67. The input signal is further defined as a user input signal, the surgical instrument includes a control element, the control element is operable to generate the user input signal, and the control system is configured to determine the starting position based on the posture of the surgical instrument with respect to the reference device at the time when the user input signal is received. The surgical system according to claim 61.

68. The surgical instrument includes a depth sensor configured to provide a third signal associated with the displacement of the end effector during the drilling process. The control system is configured to receive the third signal, and the virtual representation is further based on the third signal. The surgical system according to any one of claims 61 to 67.

69. The input signal is based on the third signal, and thus the control system is configured to determine the starting position based on the posture of the surgical instrument at the time when the control system receives the third signal from the depth sensor. The surgical system according to claim 68.

70. The surgical system according to claim 69, wherein the control system is configured to identify the starting position when the depth sensor indicates that the end effector is within the range of the opening of the plate or within the range of the bone.

71. The surgical system according to claim 69 or 70, wherein the end effector is coupled to the surgical instrument in a fixed orientation.

72. The surgical system according to any one of claims 55 to 71, further comprising an attachment configured to be attached to the surgical instrument, the attachment including the sensor module.

73. The surgical system according to claim 72, further comprising an attachment configured to be attached to the surgical instrument, the attachment including a depth sensor configured to provide a third signal associated with displacement of the end effector during a drilling process.

74. The control system is configured to control the virtual representation based on the third signal. The surgical system according to claim 73.

75. The surgical system according to claim 74, wherein the control system is configured to determine a depth end point of the virtual representation based on the third signal.

76. The attachment includes a housing, and a measurement cannula configured to circumferentially surround the end effector when the end effector is coupled to the surgical instrument, the measurement cannula being slidably attached to the housing so as to extend forward or backward between a fully distal position and a proximal position relative to the housing, the measurement cannula having a distal end adapted to be positioned against an object or bone. The surgical system according to claim 75, wherein the depth sensor is a displacement sensor for generating the third signal based on the position of the distal end of the measurement cannula relative to the housing.

77. The surgical system according to claim 76, wherein the surgical instrument includes a sensor for generating an identification signal in response to an identification feature of the end effector when the end effector is coupled to the surgical instrument.

78. The surgical system according to claim 77, wherein the attachment includes the sensor for generating the identification signal.

79. The surgical system according to claim 78, wherein the control system is configured to determine an end effector length based on the identification signal.

80. The surgical system according to claim 79, wherein the control system is configured to determine an orbit end point based on the end effector length and the third signal.

81. The surgical system according to claim 80, wherein the control system is configured to determine the depth end point based on the third signal.

82. The virtual representation is defined as a first virtual representation, the first virtual representation terminates at a depth end point, the control system is configured to display a second virtual representation, and the second virtual representation terminates at an orbit end point. The surgical system according to claim 81.

83. The surgical system according to claim 82, wherein the orbit end point is based on a movement limit range for the measurement cannula, and the depth end point is based on the third signal.

84. The surgical system according to any one of claims 73 to 83, wherein the attachment includes an antenna for communicating with the control system.

85. The surgical system according to claim 84, further comprising a display unit, and the control system communicates with the display unit.

86. The surgical system according to claim 85, wherein the control system communicates with an imaging system.

87. The surgical system according to any one of claims 47 to 86, further comprising an imaging system, and the imaging system is a C-arm capable of performing only 2D X-rays.

88. The surgical system according to any one of claims 60 to 86, wherein the optically detectable reference is an April Tag or an ArUco Tag.

89. The surgical system according to any one of claims 60 to 86, wherein the optically detectable reference is configured to enable calculation of the orientation of the reference device with respect to the camera.

90. The surgical system according to any one of claims 47 to 89, wherein the control system determines a second starting position and is configured to display a third virtual representation based on the second starting position.

91. The surgical system according to any one of claims 61 to 90, further comprising a second optically detectable reference, and the second optically detectable reference is positioned on an outer surface of the reference device.

92. The surgical system according to any one of claims 68 to 91, wherein the control system is further configured to control a display to indicate a drilling depth or a screw length based on the third signal.

93. The end effector is coupled to the surgical instrument such that at least a portion of the end effector is within the field of view of the camera. The control system is further configured to determine the identification and / or length of the end effector based on the portion of the end effector that is within the field of view of the camera. The surgical system according to any one of claims 47 to 92. **Claim 94** The surgical system according to claim 93, wherein the control system is configured to determine the identification and / or the length of the end effector using feature matching. **Claim 95** A surgical system for operating on a patient's bone, A surgical instrument for coupling to an end effector, A handpiece for driving a surgical end effector, A surgical instrument configured to generate a first signal regarding the orientation of the surgical instrument, the surgical instrument including at least one of an inertial measurement unit and a camera, and a sensor module in which the instrument defines a tool axis, A display, And a control system, the control system Acquires a plurality of 2D X-ray images, each of the 2D X-ray images having an image boundary, Aligns the plurality of 2D X-ray images to a known coordinate system, Aligns the instrument to the known coordinate system, Determines a starting position of a planned trajectory in the known coordinate system based on the first signal, Selects a 2D X-ray image from the plurality of 2D X-ray images based on the image boundary and the starting position of each of the plurality of 2D X-ray images, A surgical system configured to superimpose a virtual representation of the end effector on the selected 2D X-ray image on the display, the virtual representation being based on the first signal. **Claim 96** The control system is configured to determine a planned end position of the end effector in the known coordinate system, and the control system is further configured to select the 2D X-ray image from the plurality of 2D X-ray images based on the image boundary, the planned end position, and the starting position of each of the plurality of 2D X-ray images. The surgical system according to claim 95. **Claim 97** The surgical system according to claim 96, wherein the planned end position is an end point of a trajectory, and the end point of the trajectory is based on one of a stored drill bit length value or a remaining cannula travel length. **Claim 98** The surgical system according to claim 96 or 97, wherein the control system is configured to determine a depth end point of the end effector in the known coordinate system.

99. The control system determines a track segment extending between the track end point and the start position, and is configured to select the 2D X-ray image from the plurality of 2D X-ray images based on the track segment and the image boundary of each of the plurality of 2D X-ray images. The surgical system according to any one of claims 96 to 98.

100. The control system determines a track segment extending between the track end point and the start position, determines a relationship between the track segment and the image boundary of each of the plurality of 2D X-ray images, and based on the relationship, The surgical system according to claim 99, which is configured to select the 2D X-ray image from the plurality of 2D X-ray images.

101. The surgical system according to claim 100, wherein the relationship is such that the track segment is within the image boundary of each of the plurality of 2D X-ray images to some extent.

102. The surgical system according to any one of claims 95 to 101, wherein the system is configured to align the instrument with the known coordinate system based on an input signal related to the start position of the end effector.

103. The surgical system according to any one of claims 95 to 102, wherein the instrument is configured to determine the start position based on the first signal and based on the alignment of the instrument and the known coordinate system.

104. Each of the plurality of 2D X-ray images includes a respective image coordinate system, the first signal is measured with respect to a sensor coordinate system, and the system responds to an input signal to determine between the sensor coordinate system and the known coordinate system. And the surgical system according to any one of claims 95 to 103, which is configured to establish alignment between the image coordinate system and the known coordinate system.

105. The surgical system according to any one of claims 95 to 104, further comprising a mobile computing device, the mobile computing device including the display and a first processor, the surgical instrument including a second processor, and the control system including the first processor And / or the second processor.

106. The surgical system according to any one of claims 95 to 105, wherein the known coordinate system is a reference coordinate system.

107. The virtual representation is selected from the group consisting of a line and a two-dimensional shape having a longitudinal axis, and the virtual representation is positioned such that the line or the longitudinal axis is aligned with the tool axis. The surgical system according to any one of claims 95 to 106.

108. A surgical system for operating on a patient's bone, A surgical instrument for coupling to an end effector, A handpiece for driving a surgical end effector, A surgical instrument including a sensor module configured to generate a first signal regarding the orientation of the surgical instrument, including at least one of an inertial measurement unit and a camera, the instrument defining a tool axis, A display, And a control system, the control system Acquires a plurality of 2D X-ray images, each of the 2D X-ray images having an image boundary and an image coordinate system, the image coordinate system including an image reference plane and an image reference axis, Align the instrument with the image coordinate system, Based on the first signal, determine the starting position of the planned trajectory on the image reference plane of each of the plurality of 2D X-ray images, Based on the starting position, determine the planned end position of the planned trajectory on the image reference plane of each of the plurality of 2D X-ray images, Determine the radial distance between the starting position and the planned end position, Based on the radial distance, select a 2D X-ray image from the plurality of 2D X-ray images, A surgical system configured to superimpose a virtual representation of the end effector on the selected 2D X-ray image on the display, the virtual representation being based on the first signal.

109. The plurality of 2D X-ray images are further defined as a plurality of X-ray images having a first orientation, and the control system is configured to select the X-ray image having the first orientation with the minimum radial distance. The surgical system according to claim 108.

110. The surgical system according to claim 109, wherein the first orientation is an AP orientation with respect to the axis of a reference device attached to the patient.

111. The surgical system according to claim 109, wherein the planned end position is based on one of a stored drill bit length value or a remaining cannula travel length.

112. The surgical system according to any one of claims 108 to 111, further comprising determining a planned end position bounded by an image based on the start position, the planned end position, and the image boundary of each of the plurality of 2D X-ray images.

113. The surgical system according to any one of claims 108 to 112, further comprising determining a ratio of the radial distances that are within the image boundary based on a spatial relationship between the planned end position, the start position, and the image boundary.

114. The plurality of 2D X-ray images are further defined as a plurality of X-ray images having a first orientation, and the control system is configured to select an X-ray image having the largest ratio of the radial distances within the image boundary among the plurality of X-ray images having the first orientation. The surgical system according to any one of claims 108 to 113.

115. The surgical system according to claim 114, wherein the first orientation is an AP orientation with respect to an axis of a reference device attached to the patient.

116. The plurality of 2D X-ray images are further defined as a plurality of X-ray images having a second orientation, and the control system is configured to select an X-ray image having the largest ratio of the radial distances within the image boundary among the plurality of X-ray images having the second orientation. The surgical system according to claim 114.

117. The surgical system according to claim 116, wherein the second orientation is a medial-lateral orientation with respect to an axis of the reference device.

118. The plurality of 2D X-ray images are further defined as a plurality of X-ray images having a second orientation, and the control system is configured to select the X-ray image having the second orientation based on a difference between the radial distance bounded by the image and the radial distance, and based on a radial distance difference threshold. The surgical system according to any one of claims 113 to 117.

119. The surgical system according to claim 118, wherein the second orientation is a medial-lateral orientation with respect to an axis of a reference device attached to the patient.

120. The surgical system according to any one of claims 108 to 119, further comprising a mobile computing device, wherein the mobile computing device includes the display and a first processor, the surgical instrument includes a second processor, and the control system includes the first processor and / or the second processor.

121. The surgical system according to any one of claims 108 to 120, wherein the display is fixed to at least one of the surgical instrument and the patient.

122. The surgical system according to any one of claims 108 to 121, wherein the virtual representation is selected from the group consisting of a line and a two-dimensional shape having a longitudinal axis, and the virtual representation is positioned such that the line or the longitudinal axis is aligned with the tool axis.

123. A surgical system for operating on a patient's bone, A surgical instrument for coupling to an end effector, A handpiece for driving a surgical end effector, A surgical instrument including a sensor module configured to generate a first signal regarding the orientation of the surgical instrument and including at least one of an inertial measurement unit and a camera, the instrument having a tool axis, A display, And a control system, wherein the control system Aligns the X-ray image with a known coordinate system, Aligns the instrument with the known coordinate system, Determines a starting position of a planned trajectory in the known coordinate system based on the first signal, Determines a planned end position of the end effector in the known coordinate system based on the starting position, Selects a field of view of the aligned X-ray image based on the starting position, the planned end position, and the first signal, And a surgical system configured to superimpose a virtual representation of the end effector on the X-ray image on the display, the virtual representation being based on the first signal.

124. The surgical system according to claim 123, further comprising a mobile computing device, wherein the mobile computing device includes the display and a first processor, the surgical instrument includes a second processor, and the control system includes the first processor and / or the second processor.

125. The surgical system according to claim 123 or 124, wherein the known coordinate system is a reference coordinate system. **Claim 126** The surgical system according to any one of claims 123 to 125, wherein the virtual representation is selected from the group consisting of a line and a two-dimensional shape having a longitudinal axis, and the virtual representation is positioned such that the line or the longitudinal axis is aligned with the tool axis. **Claim 127** The surgical system according to any one of claims 123 to 126, wherein the planned end position is based on one of a stored drill bit length value or a remaining cannula travel length value. **Claim 128** A surgical system for operating on a patient's bone, a surgical instrument for coupling to an end effector, a handpiece for driving a surgical end effector, a sensor module configured to generate a first signal regarding the orientation of the surgical instrument, the sensor module including at least one of an inertial measurement unit and a camera for generating the first signal, the instrument having a tool axis, the sensor module including a depth sensor configured to provide a second signal associated with displacement of the end effector during a drilling process; a display; and a control system, the control system aligning an X-ray image with a known coordinate system, aligning the instrument with the known coordinate system, selecting a field of view of the aligned X-ray image based on the second signal, and configured to superimpose a virtual representation of the end effector over the X-ray image on the display, the virtual representation being based on the first signal. **Claim 129** The surgical system according to claim 128, wherein the controller is configured to select a field of view of the aligned X-ray image based on the second signal and a threshold distance. **Claim 130** The surgical system according to claim 128 or 129, wherein the control system is configured to transition between a first field of view and a second field of view, the second field of view providing a more zoomed-in view of the bone. **Claim 131** Further comprising a mobile computing device, the mobile computing device including the display and a first processor, the surgical instrument including a second processor, the control system including the first processor and / or the second processor, the surgical system according to any one of claims 128 to 130.

132. The surgical system according to any one of claims 128 to 131, wherein the known coordinate system is a reference coordinate system.

133. The virtual representation is a virtual object selected from the group consisting of a line and a two-dimensional shape having a longitudinal axis, and the virtual representation is positioned such that the line or the longitudinal axis is aligned with the tool axis, the surgical system according to any one of claims 128 to 132.

134. The surgical instrument according to claim 129, including a sensor for generating an identification signal in response to an identification feature of the end effector when the end effector is coupled to the surgical instrument.

135. The surgical system according to claim 134, wherein the control system is configured to determine the length of the end effector based on the identification signal.

136. The surgical system according to claim 135, wherein the control system is configured to determine a depth end point based on the end effector length and the second signal.

137. The virtual representation is defined as a first virtual representation, the first virtual representation terminating at a depth end point, the control system being configured to display a second virtual representation, the second virtual representation terminating at an orbit end point, the surgical system according to claim 136.

138. The surgical system according to claim 137, wherein the orbit end point is based on a range of motion limits for the measurement cannula, and the depth end point is based on the third signal.

139. The end effector is coupled to the surgical instrument such that at least a portion of the end effector is within the field of view of the camera, The control system is further configured to determine the identification and / or length of the end effector based on the portion of the end effector that is within the field of view of the camera, The surgical system according to any one of claims 128 to 139.

140. The surgical system according to claim 139, wherein the control system is configured to determine the identification and / or the length of the end effector using feature matching.

Images