Track guidance for handheld surgical tools during surgical procedures

Abstract

A surgical system for operation on a bone of a patient is described. The surgical system may include a surgical instrument having a camera and / or an inertial measurement unit. The surgical system may also include a control system to superimpose the virtual representation on the medical image based on an output of one of the camera and the inertial measurement unit. The virtual representation or alternative virtual representation may also be based on the output of the depth sensor. In some instances, the surgical instrument may have features of a control system configured to determine a length of the end effector based on a signal from the camera. The disclosure also features a control system that obtains a plurality of 2-DX-ray images and selects one or more of the 2-DX-ray images based on one or more criteria (e.g., radial distance, image boundaries, output of a depth sensor, relative orientation, etc. The null system may control a display to display the selected two-dimensional image.

Description

Background Art

[0001] Conventional medical and surgical procedures often involve the use of surgical tools and instruments to allow the surgeon to access and manipulate the surgical site. As a non-limiting example, rotary instruments such as handheld drills are often used in conjunction with orthopedic procedures to address a variety of musculoskeletal conditions, such as trauma, sports injuries, degenerative diseases, joint reconstruction, and the like.

[0002] In procedures using a handheld drill or similar surgical instrument, rotational torque selectively generated by an actuator (e.g., a motor) is used to rotate a releasably attached end effector (e.g., a drill bit) or other surgical attachment at varying speeds. The surgical handpiece assembly drills a hole in the bone against which the end effector is applied.

[0003] One type of orthopedic procedure is an open reduction internal fixation (ORIF) procedure. During an ORIF procedure, the surgeon realigns the broken bones and secures the bones in place using one or more surgical implants. The one or more surgical implants may include a bone plate and screws. The screws secure the bone plate in the correct position relative to the bone. After a period of time, when it is determined that the bone has healed, the bone plate and screws may be removed.

[0004] Another type of procedure involves the placement of an intramedullary nail. In this procedure, locking screws are secured into the screw holes within the intramedullary nail.

[0005] A C-arm or mobile enhancement device is an example of a medical imaging device based on X-ray technology that can be used during the surgical procedures described above. Mobile enhancement devices can perform fluoroscopy, which is a type of medical imaging that displays continuous X-ray images on a monitor. During fluoroscopy, an X-ray source or emitter emits X-rays that penetrate the patient's body. An X-ray detector or image intensifier converts the X-rays that pass through the body into visible images that are displayed on the monitor of the medical imaging device. Since medical imaging devices such as C-arm devices can display high-resolution X-ray images in real time, the doctor can monitor the progress at any time during the operation so that appropriate actions can be taken based on the displayed images. However, during certain procedures, monitoring images is often challenging, such as during procedures where the patient's anatomy must be kept in mind as well as the display of the medical imaging device. In addition, adjusting the C-arm relative to the patient to locate the fracture site or the screw holes in the nail can be difficult and time-consuming.

[0006] It is desirable to address the above challenges. Summary of the invention

[0007] One general aspect includes a surgical system for operating on a patient's bone. The surgical system also includes a surgical instrument for coupling to an end effector, the surgical instrument including: a handpiece for driving the end effector, a camera configured to generate a first signal corresponding to a posture of an optically detectable reference relative to a first coordinate system, and a sensor module configured to have a fixed posture relative to the camera and configured to generate a second signal related to orientation data and / or motion parameters relative to a second coordinate system. The system also includes a control system configured to: establish a registration between a first coordinate system, a second coordinate system, and an image coordinate system based on the first signal, the second signal, and an image of a reference device including a radiopaque reference and an optically detectable reference. The control system can be configured to determine a starting position of the end effector relative to the bone in the image in one of the first coordinate system, the second coordinate system, and the image coordinate system based on i) one of the first signal and ii) one of the second signal and the registration. The control system can also be configured to determine a second position of the end effector relative to the bone in the image in one of the first coordinate system, the second coordinate system, and the image coordinate system based on the second signal and the starting position; and superimpose a virtual representation on the image based on the starting position and the second position of the end effector. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, which are respectively configured to perform the actions of the method and / or system.

[0008] One general aspect includes a surgical system for operating on a patient's bone. The surgical system also includes a surgical instrument for connecting to an end effector, the surgical instrument including: a handpiece for driving the end effector; a camera; a sensor module, which is configured to have a fixed posture relative to the camera and is configured to generate a first signal related to orientation data and / or motion parameters. The surgical instrument may also include a depth sensor, which is configured to provide a second signal associated with the displacement of the end effector or a second effector during the drilling, tapping or driving process. The system also includes a control system, which is configured to determine the starting position of the end effector relative to the bone in the X-ray image based on the second signal, determine the second position of the end effector relative to the bone in the image based on the second signal and the starting position; and superimpose a virtual representation on the image based on the starting position, the second position of the end effector and the second signal. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, which are respectively configured to perform the actions of the method and / or system.

[0009] One general aspect includes a surgical system for operating on a bone of a patient. The surgical system includes a surgical instrument for coupling to an end effector, the surgical instrument including a sensor module configured to generate a first signal related to an orientation of the surgical instrument, the instrument having a tool axis. The system may also include a control system and a display. The control system may be configured to obtain a plurality of 2-DX-ray images, each 2-DX-ray image having an image reference axis; register the plurality of 2-DX-ray images to a known coordinate system; register the instrument to the known coordinate system; select a 2-DX-ray image from the plurality of 2-DX-ray images based on the image reference axis and the tool axis; and superimpose a virtual representation of the end effector on the selected 2-DX-ray image on the display, the virtual representation based on the first signal. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, which are respectively configured to perform the actions of the method and system.

[0010] One general aspect includes a surgical system for operating on a bone of a patient. The system may include a surgical instrument for coupling to an end effector. The surgical instrument includes: a handpiece for driving the surgical end effector; a sensor module configured to generate a first signal related to the orientation of the surgical instrument, wherein the surgical instrument defines a tool axis. The sensor module may include a camera or an inertial measurement unit, or both. The system may also include a display and a control system. The control system may be configured to: acquire a plurality of 2-DX ray images, each 2-DX ray image having an image boundary; register the plurality of 2-DX ray images to a known coordinate system; register the instrument to the known coordinate system; determine a starting position of a planned trajectory in the known coordinate system based on the first signal; select a 2-DX ray image from the plurality of 2-DX ray images based on the image boundary and the starting position of each of the plurality of 2-DX ray images; and superimpose a virtual representation of the end effector on the selected 2-DX ray image on the display, the virtual representation based on the first signal. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, which are respectively configured to perform the actions of the method and system.

[0011] Another general aspect includes a surgical system for operating on a patient's bone. The system may include a surgical instrument for coupling to an end effector, the surgical instrument including: a handpiece for driving the surgical end effector; a sensor module configured to generate a first signal related to the orientation of the surgical instrument, such as a signal from a camera or an inertial measurement unit. The system also includes a display and a control system. The control system is configured to: obtain a plurality of 2-DX ray images, each 2-DX ray image having an image boundary and an image coordinate system including an image reference plane and an image reference axis; align the instrument to the image coordinate system; determine a starting position of a planned trajectory in an image reference plane of each of the plurality of 2-DX ray images based on the first signal; determine a planned end position of the planned trajectory in an image reference plane of each of the plurality of 2-DX ray images based on the starting position; determine a radial distance between the starting position and the planned end position; select a 2-DX ray image from the plurality of 2-DX ray images based on the radial distance; and superimpose a virtual representation of the end effector on the selected 2-DX ray image on the display, the virtual representation based on the first signal. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, which are respectively configured to perform the actions of the method and system.

[0012] Yet another general aspect includes a surgical system for operating on a patient's bone. The system includes a surgical instrument for coupling to an end effector, the surgical instrument including: a handpiece for driving the surgical end effector; a sensor module configured to generate a first signal related to the orientation of the surgical instrument. The system may also include a display and a control system. The control system can be configured to: align an X-ray image to a known coordinate system. The control system can also be configured to align the instrument to a known coordinate system; determine a starting position of a planned trajectory in a known coordinate system based on the first signal; determine a planned end position of the end effector in a known coordinate system based on the starting position; select a view of the aligned X-ray image based on the starting position, the planned end position, and the first signal; and superimpose a virtual representation of the end effector on the X-ray image on the display, the virtual representation being based on the first signal. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, which are respectively configured to perform the actions of the method and system.

[0013] In another general aspect, a surgical system for operating on a patient's bone is contemplated. The system includes a surgical instrument for coupling to an end effector, the surgical instrument including: a handpiece for driving the surgical end effector; a sensor module configured to generate a first signal related to the orientation of the surgical instrument, the instrument having a tool axis. The sensor module may also include a depth sensor configured to provide a second signal associated with the displacement of the end effector during a drilling or driving or insertion process. The system may also include a display and a control system. The control system may be configured to register an X-ray image to a known coordinate system; register the instrument to a known coordinate system; and select a view of the registered X-ray image based on the second signal. The control system may also be configured to superimpose a virtual representation of the end effector on the X-ray image on the display, the virtual representation being based on the first signal. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, which are respectively configured to perform the actions of the method and system.

[0014] In another general aspect, a surgical system for operating on a bone of a patient may include a surgical instrument for coupling to an end effector, the surgical instrument comprising:

[0015] A handpiece for driving a surgical end effector; and a sensor module configured to generate a first signal related to the orientation of the surgical instrument. The sensor module may include an inertial measurement unit, a camera, a gyroscope, or a combination thereof. The system may also include a display and a control system. The control system may be configured to register a plurality of X-ray images to a known coordinate system; register the instrument to the known coordinate system; select one of the plurality of registered X-ray images based on a duration between a capture time of each of the plurality of registered X-ray images and a current time; and display the selected X-ray image.

[0016] In yet another aspect, a surgical system for operating on a patient's bone is contemplated. The system may include a surgical instrument for coupling to an end effector, the surgical instrument including a handpiece for driving the surgical end effector, and a sensor module configured to generate a first signal related to the orientation of the surgical instrument. The system may also include a display and a control system. The control system may be configured to register a plurality of X-ray images to a known coordinate system; register the instrument to the known coordinate system; select one of a plurality of registered X-ray images based on a capture time of each image and a current time; and display the selected X-ray image on the display.

[0017] In another aspect of the present disclosure, a first surgical attachment configured to be removably coupled to a surgical handpiece assembly or a component thereof is contemplated. The first surgical attachment may include a measurement housing, a camera, an inertial measurement unit configured to generate a signal related to orientation data and / or motion parameters; and a coupling.

[0018] In another aspect of the present disclosure, a surgical system for operating on a bone of a patient is contemplated. The system may include a surgical instrument for coupling to an end effector. The surgical system may include a handpiece for driving the surgical end effector and a camera coupled to the handpiece, the camera being configured to generate a first signal. The system may also include a control system configured to determine an end effector length based on the first signal.

[0019] In another aspect of the present disclosure, a surgical accessory configured to be removably coupled to a surgical handpiece assembly or a component thereof is contemplated. The surgical accessory may include a measurement housing, a camera, an inertial measurement unit configured to generate signals related to orientation data and / or motion parameters, and a measurement connector for attaching to the surgical handpiece, the surgical accessory, or a second surgical accessory.

[0020] In another aspect, a method of aligning a tool axis to an x-ray image using a reference device is contemplated. A reference device comprising one or more radiopaque fiducials and an optically detectable fiducial may be provided. The method may include securing the reference device to a bone of a patient using an attachment element; after securing the reference device to the bone or tissue of the patient, imaging the reference device using an imager; positioning an instrument having a camera and an inertial measurement unit such that the optically detectable fiducial is within a field of view of the camera; aligning the inertial measurement unit to the camera based on an output signal of the camera; and displaying a virtual representation of a portion of the instrument based on the output signal of the inertial measurement unit.

[0021] In yet another aspect, a different surgical system is contemplated. In this aspect, the system includes a reference device including one or more radiopaque fiducials, the reference device being configured to have a fixed pose relative to a surgical implant or a portion of a patient's anatomy, the reference device including an optically detectable fiducial and a radiopaque fiducial. The system may also include a surgical instrument for coupling to an end effector, the surgical instrument including a camera configured to generate a first signal corresponding to a pose of the optically detectable fiducial; and a sensor module configured to have a fixed pose relative to the camera and configured to generate a second signal related to orientation data and / or motion parameters; and an antenna for communicating with an imager.

[0022] In another aspect of the system, a reference device for aligning a tool axis to an X-ray image is contemplated. The reference device may include one or more radiopaque fiducials and optically detectable fiducials. The reference device may also include an attachment element for coupling to a patient.

[0023] A system of one or more computers may be configured to perform specific operations or actions through software, firmware, hardware, or a combination thereof installed on the system, which operations or actions, when operated, cause or result in the system performing the actions. One or more computer programs may be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions.

[0024] It should be understood that in some cases, the sensor module is described as including a depth sensor, an inertial measurement unit, and / or a camera. In other cases, the sensor module is described as including only an inertial measurement unit. It is contemplated that any time a sensor module is mentioned in a claim, it is contemplated that it includes an inertial measurement unit, a camera, and / or a depth sensor.

[0025] It should be understood that while a drill bit is described as an end effector in many of the claims and for many of the features, it is expressly contemplated that the drill bit may be replaced by other tool bits for other surgical applications, such as a screwdriver bit. Furthermore, it should be understood that the tool identification or length functionality described for a drill bit may also be applicable to any other end effector, such as for determining the length of a screwdriver bit. Furthermore, the depth detection functionality and the measuring cannula may be used with any end effector, such as with a screwdriver bit. In this configuration, the screwdriver bit may be passed through the measuring cannula in the same manner as the drill bit.

[0026] It should be understood that any reference to a control system should be interpreted to encompass any suitable arrangement of processors and communication hardware, whether the processor is associated with a mobile computing device separate from the instrument, a processor on one or more surgical accessories, a processor on one of the removable drives or on a surgical handpiece, a processor on a surgical hub separate from the instrument, or any combination thereof. Any steps contemplated for one of the processors are contemplated to be performed by any other of the processors or combination of processors.

[0027] It should be understood that while the specific examples are described in detail as selecting among a plurality of 2-D X-ray images based on various considerations, it is expressly contemplated that the control system may be configured to select among a plurality of different types of 2-D images other than X-rays. Furthermore, it is contemplated that the control system may also be configured to select from a plurality of 3-D images using similar techniques. Furthermore, it is contemplated that the techniques described herein may also be used to select from a plurality of digitally reconstructed radiographs. Thus, any instance of a plurality of 2-D images may be replaced with a plurality of 3-D images, a plurality of medical images, or a plurality of digitally reconstructed radiographs.

[0028] It should also be understood that the system can operate without the use of optically detectable fiducials. In this case, the system can use a camera to position the instrument based on known anatomical features. The anatomical features may be known landmarks that the camera recognizes. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present disclosure will be more fully understood through the detailed description and accompanying drawings.

[0030] Figure 1 is a perspective view of an exemplary layout of an operating room including a surgical system in accordance with the teachings of the present disclosure.

[0031] Figure 2 is a functional block diagram of a surgical system according to the teachings of the present disclosure.

[0032] Figure 3 is a perspective view of a detector of an imaging system including a FluoroDisc according to the teachings of the present disclosure.

[0033] Figure 4 is a virtual representation of a plurality of fiducial markers applied on a medical image in accordance with the teachings of the present disclosure.

[0034] Figure 5 is a perspective view of a surgical instrument with an attachment for drilling a hole in bone in accordance with the teachings of the present disclosure.

[0035] Figure 6 yes Figure 5 Cross-sectional and partial perspective views of surgical instruments.

[0036] Figure 7 is a perspective view of a surgical instrument according to the teachings of the present disclosure with an accessory separated from the housing of the surgical instrument.

[0037] Figure 8 is an exploded view of a surgical instrument according to the teachings of the present disclosure, with the attachment and drill bit separated from the surgical instrument.

[0038] Fig. 9 A perspective view of two reference devices attached to a bone is shown.

[0039] Fig.10 is a partial perspective view of the reference device attached to the skeleton.

[0040] Fig.11A is a partial perspective view of a surgical instrument positioned on a bone adjacent to a reference device, Fig. 11B is a schematic diagram of the transformations utilized by the surgical system.

[0041] Fig.12 is a view of a display unit according to the teachings of the present disclosure, showing a virtual representation of a bone when a surgical instrument is located in a drilling starting position, and is a perspective view of the surgical instrument, a reference device, and a bone.

[0042] Fig.13 is a view of a display unit showing a bone when a surgical instrument is positioned in a pre-drilling orientation, and a perspective view of the surgical instrument, a reference device, and the bone in the pre-drilling orientation.

[0043] Fig.14 is a view of a display unit according to the teachings of the present disclosure, showing a bone when a surgical instrument is positioned and oriented in a drill hole, showing a virtual representation of the drill bit, including the distal end of the drill bit, a maximum stroke length marking, and a perspective view of the surgical instrument, a reference device, and the bone oriented in the drill hole.

[0044] Fig.15 are views of the skeleton and reference device with both the imaging system and instrument in various postures relative to the skeleton and reference device.

[0045] Fig.16A -C is a schematic diagram of the instrument and source, which are in a specific posture relative to the reference device and the skeleton, respectively.

[0046] Fig.17A and 17B yes Fig.15 Various X-ray views of a skeleton with a virtual representation overlaid on the X-ray views.

[0047] Fig.18A and 18B yes Fig.17A and 17B Various X-ray views of the device with additional virtual representations overlaid on the X-ray views.

[0048] Fig.19A are perspective views of the instrument and bone and schematic illustrations of the imaging device source in different positions.

[0049] Fig.19B -D is Fig.19A The corresponding X-ray images captured when the source is at the corresponding source position.

[0050] Fig. 20A is another perspective view of the instrument and bone, and a schematic illustration of the imaging device source at different source positions.

[0051] Fig. 20B -D is Fig. 20A The corresponding X-ray images captured when the source is at the corresponding source position.

[0052] Fig.21A is yet another perspective view of the instrument and bone, and a schematic illustration of the imaging device source at different source positions.

[0053] Fig. 21B and 21C When Fig.21A The corresponding X-ray images captured when the source is at the corresponding source position.

[0054] Fig.22A is another perspective view of the instrument and bone, and a schematic illustration of the imaging device source at different source positions.

[0055] Fig. 22B and 22C When Fig.22A The corresponding X-ray images captured when the source is at the corresponding source position.

[0056] Fig.23A is another perspective view of the instrument and bone, and a schematic diagram of the imaging device source at different source positions.

[0057] Fig. 23B and 23C When Fig.23A The corresponding X-ray images captured when the source is at the corresponding source position.

[0058] Fig.24 It is a view of the instrument in different postures relative to the skeleton, and simultaneously displays the X-ray images corresponding to the instrument in different postures.

[0059] Figures 25A-25C is a flow chart detailing a method for guiding a user during a procedure.

[0060] Fig.26 is a flowchart illustrating X-ray image processing in detail.

[0061] Fig.27A and 27B is a flowchart that details the boot process.

[0062] Fig.28 is a flow chart detailing the process of determining the starting point of a borehole.

[0063] Fig.29is a flow chart detailing the process of determining the trajectory endpoint.

[0064] Fig.30 is a flow chart detailing the depth endpoint determination process.

[0065] Fig.31 is a flowchart that details the image display process.

[0066] Fig.32 is a flow chart illustrating the AP image determination process in detail.

[0067] Fig.33 is a flow chart illustrating the ML image determination process in detail.

[0068] Fig.34A is a schematic illustration of various surgical drivers that may be coupled to a surgical handpiece and a sensor module coupled thereto.

[0069] Fig.34B is a schematic illustration of various surgical actuators associated with various end effectors coupled to a surgical handpiece and sensor modules associated therewith.

[0070] Fig.35 is a schematic illustration of a first surgical accessory removably mounted to a second surgical accessory.

[0071] Among the drawings, reference numerals may be repeated to identify similar and / or identical elements. DETAILED DESCRIPTION

[0072] refer to Figure 1 and 2 , discloses an exemplary configuration of an operating room or surgical suite for performing a medical procedure (or surgery) on a patient using a surgical system 12 according to the teachings of the present disclosure. The surgical system 12 can be used to perform various orthopedic surgical procedures, such as ORIF procedures, to realign a broken bone 14 and to fix the bone 14 in place using one or more surgical implants. The one or more surgical implants may include a bone plate 16 and screws that are inserted into the bone 14 until the bone 14 heals. The surgical system 12 includes a control system 18. The surgical system 12 can also be used to place screws or fasteners for other surgical procedures, such as placing a locking screw in an intramedullary nail. The surgical system 12 can also be used to guide end effectors other than drill bits, such as taps, screwdrivers, surgical wires, and / or pins. The surgical system 12 can also be used to guide the placement of implants other than plates and screws, such as wires and pins.

[0073] The display unit 20 is in operational communication with the control system 18. The display unit 20 may be integrated with an input device, for example, if the display unit 20 is a touch screen. Alternatively, the display unit 20 may communicate with one or more input devices (e.g., a keyboard, a mouse, a microphone (voice activation), a gesture control device). The display unit 20 may be a tablet, a cell phone, a mixed reality device (e.g., a headset or glasses), or another suitable mobile device. The display unit 20 may also be integrated with a surgical instrument.

[0074] The surgical system 12 may also include an imaging system 22, such as a C-arm X-ray imaging device 32. It is contemplated that the system may be compatible with other suitable imaging systems, such as CT or MRT imaging devices. The imaging system 22 may include a detector 24, a source 26, an imaging computer 28, an imaging display 30, and one or more user input devices. The detector 24 and the source 26 are configured to generate one or more medical images. The detector 24 and the source 26 may be arranged at opposite ends of the C-arm 32. The source 26 may include any source of emission or generation of X-rays used in diagnostic medical imaging processes, such as a solid-state X-ray emission source, a rotating anode X-ray source, a stationary or fixed anode X-ray source, a standard X-ray source, a solid-state X-ray emission source, and / or a fluoroscopic X-ray source, and a stationary or fixed anode X-ray source. The detector 24 may include an image intensifier or any other energy receiver used in diagnostic medical imaging processes. In some cases, the imaging device is a C-arm that can only perform 2-DX rays.

[0075] The C-arm 32 including the detector 24 and the source 26 can be configured to rotate around the patient 10 to generate images of the surgical site. The imaging computer 28 can be connected to one or more user input devices, including a keyboard, a mouse, and other suitable devices, allowing a user to provide input to the imaging computer 28. The imaging computer 28 and / or the control system 18 can include software, as known to those skilled in the art, that can take the images captured by the imaging system 22 and generate one or more 2-D images and / or one or more 3-D models of the surgical site. The imaging display 30 can be configured to display the generated 2-D images and / or 3-D models.

[0076] Images from an imaging system 22 (e.g., a C-arm X-ray imaging device) may often be distorted (i.e., distorted) so that all objects in the image may not scale at the same rate. This is because the X-ray beam is not completely linear. Generally, objects that are closer to the source 26 appear larger (and contain more pixels). Objects that are farther from the source 26 may appear smaller (and contain fewer pixels). In order to make accurate measurements, the image needs to be dedistorted.

[0077] refer to Figure 3 and 4 The surgical system 12 may also include a Fluorodisc 34 to provide a dewarping function. The Fluorodisc 34 is TM Adaptive Positioning Technology For Gamma3 TM The Fluorodisc 34 is discussed in the System-User Manual, which is hereby incorporated by reference in its entirety (https: / / www.strykermeded.com / media / 2325 / gamma3-adapt-fluoromap.pdf). The Fluorodisc 34 is also discussed in U.S. Patent Publication No. 2018 / 0140309A1, filed on November 18, 2016, entitled "Method And Apparatus For Treating A Joint, Including The Treatment Of Cam-Type Femoroacetabular Impingement In A Hip Joint And Pince-Type Femoroacetabular Impingement In A Hip Joint," which is hereby incorporated by reference in its entirety. In addition to the capabilities discussed herein, the imaging computer 28 and / or control system 18 may implement one or more systems, methods, and / or algorithms discussed in the aforementioned references.

[0078] The Fluorodisc 34 may include a transparent lens 36, a plurality of fiducial markers 38 disposed on the transparent lens 36, and an attachment member 40 configured to attach the Fluorodisc 34 to the detector 24 of the imaging system 22. The Fluorodisc 34 may have any suitable size (more specifically the transparent lens 36), for example, a 9 inch diameter, a 12 inch diameter, etc. The attachment member 40 may include a combination of a strap, one or more hooks, and one or more loops. The plurality of fiducial markers 38 have known sizes. The Fluorodisc 34 may be configured to provide a dewarping function by projecting a pattern of the plurality of fiducial markers 38 onto an image. Figure 4, a virtual representation of the plurality of fiducial markers 38 projected onto the bone 14 is shown. Although an example is provided where the imaging system 22 includes a Fluorodisc 34, in some embodiments, such as when the imaging system 22 includes a flat panel detector, the imaging system 22 can automatically provide a dewarped image, and thus the Fluorodisc 34 can be omitted. Based on the pattern projected onto the image, the imaging computer 28 and / or the control system 18 can calibrate the appropriate pixel size based on the dewarped image.

[0079] In some embodiments, the imaging computer 28 and / or control system 18 can use the known dimensions of the plurality of fiducial markers 38 to perform various other calculations. For example, the distance between the patient 10 and the source 26 may vary from image to image, and thus the scale factor or magnification factor may vary from one image to the next. Thus, the imaging computer 28 and / or control system 18 can use the known dimensions of the plurality of fiducial markers 38 and / or the radiopaque fiducials on the reference device 200 to determine the scale factor or magnification factor for each image. The imaging computer 28 and / or control system 18 can also use the known dimensions of the plurality of fiducial markers 38 to determine the orientation of the C-arm 32 relative to the plurality of fiducial markers 38.

[0080] The imaging computer 28 can communicate with the control system 18. The imaging computer 28 can be configured to communicate (e.g., interface) with the surgical system 12 via a wired and / or wireless connection. For example, the imaging system 22 can be configured to provide images, such as a generated 2-D image and / or 3-D model of the surgical site, to the control system 18. In some embodiments, the C-arm 32 is a "closed system" and the control system 18 communicates with the C-arm 32 only through the video / image output port. For example, when the C-arm 32 sends the image to the video / image output port, the control system 18 can communicate with the C-arm 32 by receiving the image from the C-arm 32. In either case, the control system 18 and / or the C-arm 32 can then be configured to provide the generated 2-D image and / or 3-D model to the display 20, where the surgeon or other medical professional can interact with the image to identify and / or define the corresponding area and / or zone around the bone 14.

[0081] For example, the surgeon may select multiple views of the bone 14 for display, may define a desired trajectory for the end effector 102 of the surgical instrument 100, and / or select an appropriate user interface for the display unit 20. Although the source 26 and the detector 24 are shown as being connected to the imaging computer 28, in some configurations, the X-ray imaging apparatus including the detector 24 and the source 26 may be directly connected to the control system 18, thereby eliminating the need for the imaging computer 28, the imaging display 30, and the user input device. The control system 18 may be configured to perform all of the same functions as the imaging computer 28.

[0082] The surgical system 12 may include a surgical instrument 100. The surgeon may use the surgical instrument 100 to perform surgery on the patient 10, including drilling holes in the bone 14 of the patient 10 and / or inserting screws into the drilled holes in the bone 14 through one or more openings 17 of the bone plate 16. Conventional surgical systems typically use an optical tracking system or an electromagnetic tracking system or a combination thereof to track the position of the end effector 102 of the surgical instrument 100 relative to the bone 14 of the patient 10. Each of these systems has its own disadvantages, and both systems are time-consuming and costly to set up. A particular disadvantage of optical tracking systems is that they require a line of sight between the optical tracker and the position tracking unit (i.e., camera), which is difficult to maintain in the operating room. A particular disadvantage of electromagnetic tracking systems is that they are not as accurate as optical tracking systems and are easily distorted by surrounding metal objects. Both systems also require the surgeon to perform a manual alignment process, which can be very time-consuming.

[0083] During ORIF and other trauma procedures, surgeons may not use traditional guidance tracking systems due to the above-mentioned disadvantages. Instead, the surgeon may rely on his / her surgical skills and anatomical knowledge to align the trajectory of the end effector 102 of the surgical instrument 100 in order to create a pilot hole for one or more screws (or other surgical implants). Similarly, when placing a screw, the surgeon inserts the screw freehand. The surgeon then typically uses fluoroscopy to verify the placement of the screw. If the surgeon is not satisfied with the placement of the screw, the surgeon may replace the screw. This can cause additional damage to the bone 14 because the surgeon must drill an additional L on the bone 14. Improper screw placement can also cost extra money and waste time because the screw may be damaged and re-drilling and re-inspection are also time-consuming.

[0084] Additional references Figure 5-8, an exemplary surgical instrument 100 is shown. The surgical instrument 100 eliminates the need to use a second device (e.g., a depth gauge) to determine the depth of the bore. The surgical instrument 100 may include an accessory 104 and an end effector 102, such as a drill. Various aspects of the surgical instrument 100 are discussed in International Patent Publication No. WO2017 / 0407172, entitled “Powered Surgical Drill With Integral Depth Gauge That Includes A Probe That Slides Over ADrill Bir”, filed on September 1, 2016, and International Patent Publication No. WO2019 / 035096A1, entitled “Surgical Handpiece For Measuring Depth Of Bore Holes And Related Accessories”, filed on August 17, 2017, which are incorporated herein by reference in their entirety.

[0085] The surgical instrument 100 may include a housing 106, a motor 108, an instrument controller 110, a trigger 112, a battery 114, and other components, as described in more detail below. The trigger 112 responds to actuation by a user (e.g., a surgeon) and communicates with the instrument controller 110, such as through electrical signals generated by a magnet and a Hall effect sensor. Thus, when the surgeon actuates the trigger 112 to operate the surgical instrument 100, the instrument controller 110 manages the supply of power from the battery to the motor 108, which in turn generates a rotational torque for rotating the end effector 102 or other surgical end effector 102, as described in more detail below.

[0086] The instrument 100 can also measure tool drive parameters, which are characteristics of the portion of the tool acting on the patient 10. Tool drive parameters can include the speed or torque of the motor driving the tool. The tool drive signal can be used to confirm the position of the end effector 102 relative to the anatomy.

[0087] Figure 6 and 7 The motor 108 is shown positioned within the housing 106 along a motor axis 109, but other motor locations are contemplated. The motor 108 may be electric, pneumatic, ultrasonic, or hydraulic. The motor 108 may be configured to selectively generate a rotational torque in response to commands, signals, etc. received from the instrument controller 110. The motor 108 may include a rotor sleeve 116 supported by a pair of bearings 118 for rotation about the axis 109. A drive gear arranged proximate to the gear set is coupled to the rotor sleeve 116 and rotates simultaneously therewith for transmitting the rotational torque to the gear set.

[0088] In the example shown, the attachment 104 is removably connected to the housing 106. However, the attachment 104 can be integrally formed with the housing 106, so that the various components of the attachment 104 described throughout the text can be part of the housing 106 or separate from the housing 106. The attachment 104 can include different housings, such as an attachment housing 120. The attachment 104 can be constructed in a certain way to minimize the obstruction of the surgeon's view of the surgical site. The attachment 104 can also include a depth sensor 122 (illustrated as a displacement sensor) and a depth measurement extension 124. The depth sensor 122 is operably connected to the depth measurement extension 124. As shown, the depth measurement extension 124 can be implemented as a sleeve, i.e., a tube defining a lumen. More specifically, the depth measurement extension 124 can circumferentially surround the end effector 102 and can be slidably mounted to the housing 106 so as to extend forward and / or backward relative to the housing 106 between a fully distal position and a proximal position. In an alternative embodiment, the depth sensor 122 may be integrated with the camera 146, in other words, the camera 146 may be capable of acquiring an infrared 3D scan of the bone 14 to determine the depth of the end effector 102 relative to the bone 14 / plate 16. Alternatively, the depth sensor 122 may be a laser time-of-flight depth sensor or an ultrasonic sensor.

[0089] The depth measurement extension 124 is disposed within the guide bushing 126 and is supported for translational movement along a measurement axis 128. When the accessory 104 is attached to the surgical instrument 100, the measurement axis 128 is arranged coaxially with the axis 109. The depth measurement extension 124 also includes a plurality of rack teeth 130 arranged linearly along at least a portion of the length of the depth measurement extension 124, the rack teeth being arranged to meshingly engage with a gear 132 disposed adjacent to the distal end of the guide bushing 126. Figure 6 As shown, the window of the guide bushing 126 is arranged adjacent to the gear 132 to facilitate meshing engagement between the rack teeth 130 and the gear 132, so that the rotation of the gear 132 is proportional to the movement of the depth measurement extension 124. The depth sensor 122 responds to the rotation of the gear 132 caused by the axial movement of the depth measurement extension 124, and can be implemented with a potentiometer, a rotary encoder, etc. to generate an electrical signal representative of the change in the position of the depth measurement extension 124 along the measurement axis 128.

[0090] For example, in some configurations, the depth sensor 122 can be arranged to communicate with the instrument controller 110 or the accessory controller 134, which can be configured to interrupt or adjust the driving mode of the motor 108 based on the movement of the depth measurement extension 124, such as slowing down or stopping the rotation of the end effector 102 at a specific drilling depth into the tissue. The accessory 104 can include one or more power terminals, such as described in U.S. Patent No. 11,317,927, which is incorporated herein by reference in its entirety. These power terminals allow the accessory 104 to be powered by a battery of the surgical instrument 100 coupled to the accessory 104.

[0091] Still refer to Figure 6 In order to ensure that the depth measurement extension 124 and the depth sensor 122 function properly, the depth measurement extension 124 can be biased toward the fully distal position. Through this bias, the distal end of the depth measurement extension 124 always maintains contact with the proximal surface of the bone 14 to be drilled or with the bone plate 16 abutting the bone 14 to be drilled. The bias is achieved by using a spring that biases the gear 132 so that the gear 132 rotates in a direction that extends the depth measurement extension 124 distally out of the attachment housing 120. However, other ways of biasing the depth measurement extension 124 relative to the surgical instrument 100 are also contemplated. Exemplary spring arrangements can be found in PCT / US2016049899, ​​which is incorporated herein by reference in its entirety.

[0092] refer to Figure 7 and 8 , the accessory 104 may include an attachment connector 138 that is configured to be operably connected to the instrument connector 140 of the surgical instrument 100. In one example, the surgical instrument 100 may provide a power connection for the accessory 104 only through the connection between the accessory connector 138 and the instrument connector 140. In another example, the accessory 104 and the surgical instrument 100 may also exchange data through the instrument connector 140 and the accessory connector 138. The data may include motor parameters, such as the speed of the motor or the torque of the motor. In addition, in some cases, the accessory controller 134 may use the connection to provide speed / torque control commands to the instrument controller 110. The accessory controller 134 and the instrument controller 110 may communicate with the control system 18 or other devices through a wired connection (i.e., through the accessory connector 138 and the instrument connector 140) or a wireless connection. In the case where the accessory controller 134 wirelessly communicates with the imaging computer 28, the display unit 20, or the instrument controller 110, the accessory 104 may include an antenna 142 that communicates with the accessory controller 134. The instrument controller 110 and / or the accessory controller 134 may send data to a display unit 20, such as a tablet computer or an external server.

[0093] The attachment 104 may also include a display 144, such as a display screen, one or more light emitting diodes (LEDs), etc., to provide the surgeon with information related to the movement of the attachment 104, such as displaying real-time drilling depth, recorded historical maximum drilling depth, screw length, breakthrough indication, visual information (e.g., a graphical representation) related to the current trajectory relative to the desired trajectory to guide the surgeon to align the desired trajectory, etc. The same information may also be conveyed to the user via a speaker to provide an audio indication of the real-time drilling depth, recorded historical maximum drilling depth, breakthrough indication, etc.

[0094] The accessory controller 134 can also be configured to determine a breakthrough event based on the depth signal. The accessory controller 134 and the instrument controller 110 are shown as separate controllers disposed within the accessory housing 120 and the housing 106, respectively, and their functions can also be integrated into a single controller.

[0095] The surgical instrument 100 may also include a camera 146. In one exemplary configuration, the camera 146 may be included as part of the accessory 104. The accessory controller 134 may be operably connected to the camera 146. In this configuration, the accessory 104 may be a disposable item that is discarded after each procedure is completed.

[0096] In another alternative embodiment, the camera 146 can be removably mounted to the accessory 104. Additionally, the camera 146 can be removably mounted to the housing of the instrument 100 and not included as part of the accessory 104. In these cases, the camera 146 can include a dedicated camera controller that can communicate camera information to the instrument controller 110 and / or the accessory controller 134.

[0097] Although one camera 146 is shown, it is contemplated that the surgical instrument 100 may include multiple cameras 146. This may improve the ability of the surgical instrument 100 to maintain a line of sight during the drilling process. The multiple cameras 146 may be arranged around the surgical instrument 100 and / or the accessory 104 so that the total viewing field of the cameras 146 extends around the entire or substantially the entire surgical instrument 100. In this way, when the surgical instrument 100 is brought close to the patient 10, the cameras 146 may more easily detect the position and orientation of the optically detectable fiducial 204.

[0098] While it is contemplated that the apparatus 100 may utilize a reference device 200 that includes an optically detectable fiducial, it is also contemplated that registration may be performed through recognition of externally visible anatomical features of the patient 10, a particular shape, pattern, or the like of the reference device.

[0099] The surgical instrument 100 may also include a sensor module 148 configured to have a fixed attitude relative to the camera 146. The sensor module 148 may be an inertial measurement unit (IMU). The IMU may include a gyroscope, an accelerometer and / or a magnetometer. The IMU may be configured to measure in any number of degrees of freedom. In one example, the IMU includes a gyroscope and an accelerometer, allowing the IMU to measure the attitude of the surgical instrument 100 in 6 degrees of freedom. The sensor module 148 includes a sensor coordinate system, and the IMU may be able to determine the attitude of the sensor module 148 relative to the sensor coordinate system. The IMU can use motion parameters (e.g., acceleration data and / or angular velocity data) to detect changes in the attitude of the sensor module 148 relative to the sensor coordinate system.

[0100] The device 100 may be configured to be at least partially disposable. Figure 7 As shown, the accessory 104 can be detachably coupled to the housing 106 of the instrument 100, and the accessory 104 can be discarded after use or reprocessed (e.g., autoclaved) and used again. In the illustrated embodiment, the accessory 104 includes a depth sensor 122, a depth measurement extension 124, a camera 146, a sensor module 148, and a control element 150, and the accessory 104 is disposable. In some embodiments, the accessory 104 can be autoclaved and reused.

[0101] The instrument 100 can be configured to include at least some of the components 122, 124, 146, 148, 150 included in the accessory 104. For example, in the illustrated embodiment, the instrument 100 includes an attachment interface 107 that is removably coupled to the housing 106. In this embodiment, the accessory interface 107 includes an instrument connector 140 and is configured to receive the accessory 104, such that the instrument connector 140 contacts the accessory connector 138 to establish electrical communication between the accessory 104 and the accessory interface 107. In this way, the accessory 104 can communicate with the rest of the instrument 100 and the system 12 through the accessory interface 107. The accessory interface 107 can include a sensor module 148 (i.e., an IMU) and other more expensive components, while the accessory 104 can include less expensive components, such as a depth sensor 122, a depth measurement extension 124, a camera 146, and / or a control element 150. The accessory interface 107 can be autoclaved and / or reprocessed according to any suitable method, and the accessory 104 is typically disposable. In this way, a user can perform a procedure using the instrument 100, then detach the accessory interface 107 from the housing 106, detach the accessory 104 from the interface 107, clean and reuse the accessory interface 107, discard the accessory 104, couple a new accessory 104 to the interface 107, and recouple the interface 107 to the housing 106. It should be understood that while the accessory 104 and the accessory interface 107 have been referred to as including certain components 122, 124, 146, 148, 150, the accessory 104 and the accessory interface 107 can each include any component associated with the instrument 100.

[0102] refer to Fig. 9 and 10 , the surgical system 12 may also include a reference device 200. The reference device 200 may include one or more radiopaque fiducials 202 and one or more optically detectable fiducials 204. The reference device 200 may be attached to an implant or a portion of a patient's anatomy, or to a component that is attached to an implant or a portion of a patient's anatomy, such as a pin or adhesive used to temporarily attach a bone plate 16 to a bone 14. In procedures that may require placement of multiple plates / implant holes, the surgical system 12 may include multiple reference devices 200, with one reference device coupled adjacent to each surgical site. In this case, each reference device 200 is uniquely identifiable in the x-ray image, and each optically detectable fiducial 204 is uniquely identifiable by the camera 146.

[0103] Furthermore, in some cases, such as when implanting (plural) longer plates, multiple reference devices 200 may be used at a single surgical site. These different reference devices 200 may have different spatial arrangements of radiopaque elements and different optically detectable fiducials 204, such as different April Tags.

[0104] The reference device 200 defines a reference reference axis. The reference reference axis defines a reference coordinate system. The reference reference axis is used herein to refer to the reference coordinate system. The reference device 200 is configured to be attached to the bone 14 and / or in some configurations, the bone plate 16 can be interposed between the bone 14 and the reference device 200.

[0105] The reference device 200 may have one or more radiopaque fiducials 202 arranged in a manner suitable for determining the pose of the reference device 200 (and the reference reference axis) when it is imaged by the imaging system 22. The radiopaque fiducials 202 cast a shadow when the reference device 200 is imaged. For example, the radiopaque fiducials 202 may be radiopaque spheres or other types of radiopaque elements. The one or more radiopaque fiducials 202 and their shadows allow the control system 18 to calculate the pose (i.e., position and / or orientation) of the reference device 200 relative to the image coordinate system of the imaging system 22 (e.g., the image coordinate system of the C-arm 32). More specifically, the radiopaque fiducials 202 are fixed relative to the rest of the reference device 200 so that the axis of the reference device 200 can be determined by the imaging system 22 that images the radiopaque fiducials 202. The one or more radiopaque fiducials 202 may also allow the control system 18 to calculate the rotation angle between two different images captured by the imaging system 22. For example, if the imaging system 22 is a C-arm 32, the images are captured when the C-arm 32 is rotated relative to the bone 14. The control system 18 may calculate the difference in the pose of the C-arm 32 when one image is captured and the pose of the C-arm 32 when the other image is captured based on the difference in how the radiopaque fiducial 202 appears in the two images.

[0106] When reference device 200 is imaged by imaging system 22, the imaging data includes artifacts of radiopaque fiducial 202. Because the position of radiopaque fiducial 202 is known relative to optically detectable fiducial 204, control system 18 can provide guidance to the user to correlate the pose of surgical instrument 100 with the imaging data. This can allow the user to determine the pose and / or trajectory of surgical instrument 100 relative to internal or non-visible anatomical features depicted in the image data.

[0107] The optically detectable fiducial 204 may be an April Tag, ArUcoTag, or other suitable observable pattern for detecting the position and / or orientation of the camera 146 relative to the reference device 200. The optically detectable fiducial 204 is positioned on the outer surface of the reference device 200 so that it can be detected by the camera 146 when the surgical instrument 100 is aligned with the drill hole position. Each reference device 200 may include multiple optically detectable fiducials 204, such as one optically detectable fiducial 204 on two or more faces of the reference device 200. The position / orientation of each optically detectable fiducial 204 relative to the radiopaque fiducial 202 is stored in a memory unit accessible to the control system 18. Although each reference device 200 may have more than one optically detectable fiducial 204, each optically detectable fiducial 204 is unique because each optically detectable fiducial 204 has a unique spatial arrangement relative to the radiopaque fiducial 202.

[0108] Different types of optically detectable fiducials 204 are contemplated. For example, there may be different April Tag types, each of which is uniquely identifiable and has a unique ID. When multiple tags are included on a single reference device, each tag will have a different ID. Therefore, when the camera 146 sees any tag, from its ID, the camera 146 can determine its placement relative to the radiopaque element, such as pre-stored information about the spatial arrangement of the radiopaque element relative to the tag ID.

[0109] The reference device 200 may include a coupling portion 206. The coupling portion 206 may be configured to facilitate releasable attachment to the bone plate 16 or the bone 14 of the patient 10 via a fixation member 208 (eg, a K-wire, a pin, an adhesive, or another suitable fixation member).

[0110] Still refer to Fig. 9 and 10 , an exemplary bone plate 16 is shown. The bone plate 16 can be made of metal or another suitable material and is configured to fix a fractured bone, such as bone 14. As shown, the bone plate 16 can include a plurality of larger openings and a pair of smaller openings. The surgeon can first attach the bone plate 16 to the bone 14 using a fixing member 208 so that the bone plate 16 does not move, and then insert the screws into the bone 14.

[0111] The fixation member 208 can be driven through either of the pair of smaller openings and into the bone 14. The other portion of the fixation member (i.e., the side that is not driven into the bone 14) can be used as an anchor for the reference device 200. Once the fixation member 208 is fed through one of the pair of smaller openings and driven into the bone 14, the reference device 200 can be coupled to the fixation member 208. The coupling portion 206 can include a finger support collar 210, a threaded portion 212, a collet 214, and a locking knob 216. The finger support collar 210 can be mounted together with or integrally formed with the bottom portion of the reference device 200. The threaded portion 212 can include external threads and can be mounted to or integrally formed with the collar 210.

[0112] The collet 214 may be mounted to the threaded portion 212 or formed integrally with the threaded portion 212. The collet 214 is configured to form a collar around the fixing member 192 so as to fix the reference device 200 to the fixing member 192. When the collar forms a collar around the fixing member 192, rotation and axial movement of the reference device 200 are prevented. Although an example is provided in which the coupling portion includes the collet 214 to fix the reference device 200 to the fixing member 192, any suitable member, element, device may be used to fix the reference device 200 to the fixing member 192 so that the reference device 200 cannot move independently of the fixing member 192.

[0113] The locking knob 216 can be configured to be coupled to the threaded portion 212. For example, the locking knob 216 can be shaped to define a channel configured to receive the threaded portion 212 and the collet 214. The locking knob 216 may include internal threads on an inner surface. The threaded portion 212 and the locking knob 216 can form a mating pair. In some embodiments, the size (e.g., height) of the threaded portion 212 and the size (e.g., height) of the collet 214 combined together can be approximately the same as the size (e.g., height) of the locking knob 216. The locking knob 216 can be used to cover the collet 214 so that the connection between the reference device 200 and the fixed member 208 remains secure and will not be accidentally disconnected throughout the procedure.

[0114] In other configurations, even when bone plate 16 is used in a surgical procedure, reference device 200 may be directly connected to bone 14 without the intervening bone plate 16. In such a configuration, fixation member 192 may be inserted into bone 14 near bone plate 16 but not through either of the pair of openings. Alternatively, reference device 200 may be fixed to bone 14 using adhesives, pins, or other coupling mechanisms.

[0115] Once the reference device 200 is temporarily attached to the bone 14, the surgeon or other medical professional may image the patient 10 using the imaging system 22. The imaging system 22 then generates and transmits the images to the control system 18.

[0116] The optically detectable reference 204 allows the camera 146 to determine the position and / or orientation of the reference device 200 relative to the camera 146 when imaged by the camera 146. The pose (i.e., position and orientation) of the optically detectable reference 204 is fixed relative to the rest of the reference device 200, and the relationship is stored in a memory accessible to the control system 18. The camera 146 is configured to generate a first signal. The first signal may correspond to the pose of the optically detectable reference 204 relative to the first coordinate system. The first signal corresponds to the position of the drill tip when drilling begins. The camera 146 determines the rotation and translation of the camera 146 relative to the optically detectable reference. Using the camera 146 to determine the starting position of drilling helps to reduce the number of surgeon inputs and automate the process. The sensor module 148 may generate a second signal related to the orientation data and / or position data relative to the second coordinate system.

[0117] The sensor module 148 generates the second signal, which the control system 18 can use to derive the orientation and / or position of the surgical instrument 100, more specifically, the orientation and / or position of the attachment 104 and the measurement axis 128. The sensor module 148 is configured to have a fixed posture (i.e., position and orientation) relative to the surgical instrument 100 (particularly the end effector 102). In some examples, the sensor module 148 is disposed within the attachment 104 or within the housing 106 of the surgical instrument 100.

[0118] In one possible embodiment, the camera 146 need only be directed toward the optically detectable fiducial 204 of the special reference device 200 once, at the beginning of drilling, for determining a signal from the sensor module 148 related to the starting point determined by the camera 146 .

[0119] The accessory 104 may include one or more control elements 150, such as a first button and a second button. The sensor module 148 may receive power from the surgical instrument 100 and does not require a separate power source when disposed within the accessory 104. In other configurations, the accessory 104 may include a power source, such as a battery.

[0120] The position and / or orientation of the end effector 102 may also be fixed in one or more degrees of freedom relative to the camera 146 and the sensor module 148. More specifically, the axis of the end effector 102 (e.g., the tool axis) may be known relative to the camera 146 and the sensor module 148. Thus, once the position / orientation of the camera 146 is known relative to the image, the axis of the end effector 102 is also known relative to the image. Furthermore, once the position / orientation of the sensor module 148 is known relative to the image, the axis of the end effector 102 is also known relative to the image. Once the axis of the end effector 102 is known relative to the image, the position / orientation of the sensor module 148 is also known relative to the image. The relationship between the axis of the end effector 102, the camera 146, and the sensor module 148 may be stored in a memory unit accessible to the control system 18.

[0121] In some configurations, the length of the end effector 102 can be selected by the user using a user input device and input to the control system 18. The control system 18 can use the inputted length of the end effector 102 to calculate the depth endpoint of the end effector 102 and overlay the depth endpoint of the end effector 102 on the patient image for display. This can transform the point in the camera coordinate system to the drill tip location. In some configurations, the length of the end effector 102 can be referred to as the drill length.

[0122] Alternatively, the surgical instrument 100 may include an identification sensor 152 for generating an identification signal in response to an identification feature 154 of a drill or other end effector 102 when the drill or end effector 102 is coupled to the instrument 100. The identification sensor 152 may provide an identification signal to the accessory controller 134. The control system 18 (e.g., the accessory controller 134) may determine the length of the end effector 102 based on the identification signal. In certain embodiments, the accessory 104 may include an identification sensor 152. The identification feature 154 and the identification sensor 152 may be as described in WO2020232413, which is incorporated herein by reference in its entirety. Alternatively, the identification sensor 152 may be included elsewhere within the instrument 100, such as within the body of the handpiece.

[0123] The identification sensor 152 may be a magnetoresistive sensor, an optical sensor, etc. The identification feature 154 may be a magnetic identification feature, such as an array of magnetic materials, or an optical identification feature, such as a laser mark.

[0124] The end effector 102 may also be identified and / or measured by the camera 146 using feature matching or the like. For example, the end effector 102 may be coupled to the instrument 100 such that at least a portion of the end effector 102 is within the field of view of the camera 146 and is therefore visible to the camera 146. The control system 18 may then be configured to determine the identity and / or length of the end effector 102 based on the portion of the end effector 102 that is within the field of view of the camera. If the control system 18 uses feature matching to identify and / or determine the length of the end effector 102, the system 18 may access pre-stored image data associated with known end effector types / identities. The control system 18 then compares the visible portion of the end effector 102 to the pre-stored image data to determine if there is a match. For example, if the visible portion of the end effector 102 matches pre-stored image data associated with a drill bit having a length of 99 mm, then it is determined that the length of the end effector 102 is also 99 mm. Other sizes, such as 129 mm and 169 mm, as well as end effectors 102 of any other length or identity are also contemplated. This can be used in conjunction with the image processing techniques described below.

[0125] refer to Fig.11A and 11B , wherein the instrument 100 is depicted in proximity to the bone 14 and the reference device 200. The end effector 102 is placed in close proximity to the bone 14, and the camera 146 can see the reference device 200 and, thus, the optically detectable fiducial 204. Although not shown, the imaging system 22 has imaged the bone 14 and the reference device 200 prior to the introduction of the instrument 100. As such, the imaging system 22 has determined the pose of the fiducial reference axis of the reference device 200 via the radiopaque fiducial 202 as described above.

[0126] Fig. 11B is an illustration of the transformation used by the system 12. Starting with the C-arm 32 (or alternatively the imaging system 22), when an X-ray image is generated, the control system 18 calculates an imaging system-reference device transformation. The imaging system-reference device transformation is based on the pose of the reference device 200 in the X-ray image, which is derived from the pose of the radiopaque fiducial 202 and the known spatial relationship between the radiopaque fiducial 202 and the reference reference axis of the reference device 200. Thus, the imaging system-reference device transformation allows the control system 18 to understand the pose of the reference reference axis relative to the image coordinate system of the C-arm 32 based on the pose of the radiopaque fiducial 202 in the X-ray image.

[0127] Fig. 11BA camera-reference device transformation is also shown. When camera 146 generates an optical image, control system 18 calculates the camera-reference device transformation. The camera-reference device transformation is based on the pose of reference device 200 in the optical image, which is derived from the pose of optical fiducial 204 and the known spatial relationship between optical fiducial 204 and the reference reference axis of reference device 200. Therefore, the camera-reference device transformation allows control system 18 to understand the pose of the reference reference axis relative to the camera coordinate system of camera 146 based on the pose of optical fiducial 204 in the optical image.

[0128] at last, Fig. 11B The IMU-camera transform is shown. When the user actuates the control element 150, the control system 18 calculates the IMU-camera transform. The IMU-camera transform is based on the pose of the sensor module 148 when the control element 150 is actuated. More specifically, the actuation of the control element 150 aligns the sensor coordinate system to the global reference system (i.e., gravity) by setting the starting pose of the sensor coordinate system relative to the pose of the sensor module 148. In other words, the IMU-camera transform allows the control system 18 to define the sensor coordinate system relative to the camera 146 and the image coordinate system. Therefore, the control system 18 can use the IMU and sensor module 148 to track changes in the pose of the optical coordinate system through the IMU-camera transform.

[0129] By accessing / knowing the imaging system-reference device transformation, the camera-reference device transformation, and the IMU-camera transformation, the control system 18 is able to determine the pose of the instrument 100 relative to the imaging system 22. Therefore, the pose of the instrument 100 relative to the image coordinate system can be understood. That being said, it is important to recognize that if the imaging system 22 is a C-arm 32, the pose of the image coordinate system depends on the pose of the source 26 and the detector 24 when the X-ray image is generated. The control system 18 uses this fact and the above-mentioned transformations to determine the pose of the instrument 100 relative to the image coordinate system for each X-ray image generated by the imaging system 22. If the pose of the image coordinate system changes between images due to rotation of the C-arm 32, the control system 18 is also able to determine how the pose of the instrument 100 changes in the image coordinate system.

[0130] As a result of the above transformation, the control system 18 can align the optical coordinate system of the camera 146 with the image coordinate system of the imaging system 22. In addition, the control system 18 can continuously change this registration as the pose of the IMU changes. Because the IMU-camera transformation aligns the sensor coordinate system and the optical coordinate system, any rotation / translation of the sensor module 148 is associated with an equal rotation / translation of the optical coordinate system. Finally, the control system 18 can know this relationship as the optical image transformation. The optical-image transformation enables the control system 18 to transform the known pose of an object relative to the optical coordinate system into the pose of the object relative to the image coordinate system. This enables the control system 18 to depict objects seen by the camera 146, such as the end effector 102 and the reference device 200, in the x-ray image.

[0131] refer to Fig.12 , the control system 18 is configured to determine the drilling start point 300 relative to the reference reference axis of the reference device 200, because only the reference reference axis is known in the image coordinate system. When an X-ray image of the bone 14 to be treated is taken using the reference device 200, the transformation between the radiopaque fiducial 202 and the image coordinate system of the X-ray image can be determined. This is the imaging system-reference device transformation described above. As described above, the reference reference axis is defined by the arrangement of the radiopaque fiducial 202 within the reference device 200. Since the control system 18 does not rely on traditional tracking means to track the position of the surgical instrument 100, the actual position of the surgical instrument 100 relative to the image must be determined.

[0132] The control system 18 is configured to determine the drilling starting point 300 based on the pose of the surgical instrument 100 and / or the pose of the accessory 104 relative to the reference device 200 obtained from the camera 146 when receiving the input signal.

[0133] The input signal may be defined as a user input signal obtained from actuation of a control element 150 on the attachment 104. Alternatively, the input signal may be based on a signal output by a depth sensor 122 that provides a signal associated with the displacement of the drill bit 102 relative to the bone 14 during the drilling process.

[0134] This input signal may be generated when the depth measurement extension 124 is depressed beyond a threshold amount (e.g., greater than 5 millimeters). It may be concluded that the user is ready to begin drilling based on this slight depression of the depth measurement extension 124 and the associated depth signal generated by this movement. In either embodiment, whether the input signal is a user input signal resulting from activation of the control element 150 or the input signal is determined based on the depth sensor 122, the controller may use the input signal to determine the drilling start point 300. In certain embodiments, the control system 18 may utilize both the control element 150 and the depth sensor 122 to trigger the measurement of the drilling start point 300.

[0135] When control system 18 receives the input signal, control system 18 may activate camera 146 to begin searching for optically detectable fiducial 204. When camera 146 detects optically detectable fiducial 204, control system 18 determines a camera-reference device transformation matrix. The camera-reference device transformation matrix is ​​a matrix that describes the alignment of the reference reference axes of reference device 200 relative to the optical coordinate system of camera 146.

[0136] This is advantageous because it does not require the user to position the tip of the end effector 102 relative to the image, which is an error-prone step. Furthermore, in embodiments where the input signal comes from the depth sensor 122 and / or IMU, this step can be performed without any separate user cloud action in addition to the typical drilling or driving workflow. This saves time for the surgeon and allows the surgeon to focus on the trajectory of the surgical instrument 100 and / or implant.

[0137] The control system 18 may also utilize the known position and / or orientation of the end effector 102 and / or the length of the end effector 102 to calculate various transformations. For example, the control system 18 may transform the coordinates of the distal end of the end effector 102 from the known end effector axes to the known axes of the camera 146 based on a predefined and stored end effector-to-camera transformation matrix. This may be used because the end effector 102 (e.g., a drill bit) may have a fixed orientation relative to the sensor module 148 and the camera 146.

[0138] The sensor module 148 may include a gyroscope and / or an accelerometer, which are susceptible to bias instability, resulting in signal drift over time. Therefore, the surgeon may periodically align the sensor module 148 with the reference device 200 during the surgical procedure to minimize tracking inaccuracies caused by such drift. One way to address this problem is to place several different reference devices 200 near the patient 10, with at least one reference device 200 placed near each hole to be drilled. In addition, the alignment step may be performed for each hole to be drilled by the instrument 100 or each trajectory along which the implant may be driven. The control system 18 may use this input signal to indicate that the camera 146 needs to be aligned. In this step, the control system 18 will re-align the images of the sensor module 148, the camera 146, and the reference device 200. When the camera 146 sees the optically detectable reference 204 during the determination of the drilling starting point 300, the instrument 100 may automatically align the sensor module 148 to the camera 146. In this way, the sensor module 148 is automatically aligned with the coordinate system of the camera 146 at the beginning of drilling of each hole and / or while driving along each trajectory. This process creates / sets the IMU-camera transformation, which describes the alignment between the optical coordinate system and the sensor coordinate system.

[0139] After the control system 18 registers the camera coordinate system relative to the sensor module coordinate system, the control system 18 can begin determining the handpiece orientation signal. This determination of the handpiece orientation signal can be performed by the accessory controller 134.

[0140] As previously described, the optically detectable fiducial 204 can be used as a patient tracker for each trajectory because the optically detectable fiducial 204 is attached to the reference device 200 in a fixed manner and the reference device 200 has a fixed pose relative to the skeleton 14. Each time an image is taken, the relationship between the optically detectable fiducial 204 and the skeleton 14 is recorded and associated with the particular image taken.

[0141] The control system 18 may further use the measured camera-reference device transformation matrix to transform the coordinates of the distal end of the end effector 102 from the camera coordinate system to the reference reference axis of the reference device 200. The camera-to-reference device transformation matrix may be calculated based on the signal generated by the camera 146 when the camera detects the optically detectable reference 204 (e.g., AprilTag).

[0142] The control system 18 can transform the coordinates of the distal end of the end effector 102 from the axis relative to the optically detectable fiducial 204 to the axis relative to the reference device 200 known to the radiopaque fiducial 202. This calculation can be performed using the identity of each optically detectable fiducial 204 and the known transformation of each identity relative to the fiducial reference axis of each particular reference device 200. This can be referred to as the optically detectable fiducial to radiopaque fiducial transformation matrix. It should be understood that the system 12 can interpret these transformations as spatial relationships between the optical fiducial 204, the radiopaque fiducial 202, and the rest of the reference device 200.

[0143] refer to Figure 12-14 , the control system 18 can overlay virtual representations 306, 308 on the image based on the starting position of the end effector and the determined orientation of the surgical instrument 100. The virtual representations 306, 308 can include a model of the surgical instrument 100, such as a model of the attachment 104 and / or the end effector 102. The virtual representations 306, 308 can represent the current posture of the end effector 102 and the planned end posture of the end effector 102, respectively. The virtual representation can also include a virtual trajectory projected into the bone 14. The surgeon can then change the angle of the surgical instrument 100 until the surgeon is satisfied with the virtual trajectory (i.e., the position of the virtual representations 306, 308 relative to the bone 14 or tissue in the image). For example, in the case where the surgical system 12 includes a bone plate 16, the model of the end effector 102 can be superimposed at a selected aperture. In another example, for example when the surgical system 12 does not include a bone plate 16, a model of the tip of the depth measurement attachment or end effector 102 can be superimposed at a selected anatomical feature in the image. The control system 18 can control the display to display the drilling depth or screw length based on the signal from the depth sensor 122. Such an embodiment is described in PCT Publication No. WO2019035096, which is incorporated herein by reference in its entirety.

[0144] Features of the system may include any of the features described in PCT / US2021 / 044429, which is incorporated herein by reference.

[0145] The transformed distal end of the end effector 102 relative to the axis of the reference device 200 is referred to as the drilling start point 300 .

[0146] The control system 18 may also be operable to establish a relationship between the axes of the sensor module 148 and the reference reference axes of the reference device 200. This relationship may be based on a combination of an IMU-camera transformation and a camera-reference device transformation. This relationship may be established using the above-described transformation of the camera 146 to the optically detectable reference 204. In addition, the calculation may be established based on a transformation of the coordinates of the distal end of the end effector 102 from a known end effector axis to a known axis of the sensor module 148, the transformation being based on a predefined and stored end effector-to-sensor module transformation matrix. The end result is a transformation established from the axes of the sensor module 148 to the axes of the reference device 200.

[0147] The control system 18 is then used to determine the trajectory endpoint 302 relative to the axis of the reference device 200. This is calculated by converting the drilling start point 300 from the reference reference axis relative to the reference device 200 to the axis relative to the end effector 102 using the above-mentioned transformation from the sensor module 148 to the reference reference axis of the reference device 200. The trajectory endpoint 302 can be a planned end position of the end effector 102. The planned end position can be a planned position in the bone 14 where the user wants the end effector 102 to stop and / or reach when drilling is completed.

[0148] The control system 18 can also determine the remaining sleeve stroke length. The remaining sleeve stroke length is the potential stroke (or amount of movement) remaining for the distal retraction of the depth measurement extension 124 relative to the end effector 102. The maximum sleeve stroke length can be stored in a memory unit accessible to the control system 18. The control system 18 can calculate the remaining stroke length based on the measured distance of the depth sensor 122 and based on the maximum sleeve stroke length. The maximum sleeve stroke length is limited between the full distal position and the proximal position. The remaining sleeve stroke length value can help the user understand whether they can continue drilling if the measuring sleeve does not bottom out in the instrument 100. The remaining sleeve stroke length can represent the range of motion limit of the depth measurement extension 124 of the instrument 100. An exemplary maximum sleeve stroke length can be 110 millimeters.

[0149] refer to Fig.13 and 14 , a trajectory endpoint 302 may be calculated based on the drilling start point 300 and the remaining casing travel length. The trajectory endpoint 302 may be offset from the drilling start point 300 by the remaining casing travel length. The trajectory endpoint 302 is located along the drill reference axis. The trajectory endpoint 302 relative to the drill reference axis is converted to a trajectory endpoint 302 relative to the axis of the reference device 200 using a transformation matrix from the sensor module 148 to the reference axis. The trajectory endpoint 302 may be displayed so that a user can see the depth that the drill head can reach relative to the real-time trajectory determined using the sensor module 148. Fig.14 This shows that the user can drill all the way to the fibula without the measuring cannula or other depth measuring extension bottoming out, so the user may need to be careful to complete the hole without digging the drill bit into the fibula.

[0150] Continue to refer Fig.13 and 14 , the control system 18 can also determine a depth endpoint 304 relative to the reference reference axis of the reference device 200. The control system 18 can further use the transformation matrix of the sensor module 148 to the reference axis of the reference device (i.e., a combination of the IMU-camera transformation and the camera reference device transformation) to convert the drilling start point 300 relative to the axis of the reference device 200 to the drilling start point 300 relative to the axis of the end effector 102. The depth endpoint 304 represents the end of the drill bit or other end effector 102. The control system 18 can display the depth endpoint 304 to allow the user to understand the depth of the end effector 102 relative to the image data (i.e., relative to the bone 14 to be drilled). In the illustration, it can be discerned that the distal end of the drill bit has not yet penetrated the tibia because the depth endpoint 304 is located within the tibia, so the user needs to continue drilling. Because drilling has not yet begun, the depth endpoint 304 is not displayed on the display 20 because it would overlap with the drilling start point 300.

[0151] The control system 18 may also determine the depth of the end effector 102 relative to the bone 14. Although it is described throughout this specification how the depth may be determined using a measurement cannula, other depth sensing configurations are also contemplated.

[0152] The control system 18 determines a depth endpoint 304 relative to the drill reference axis based on the measured depth. The depth endpoint 304 is offset from the drilling start point 300 based on the measured depth, and the depth endpoint 304 is along the motor axis 109. Using the transformation matrix of the sensor module 148 to the reference device axis, the depth endpoint 304 relative to the reference axis of the end effector 102 can be converted to a depth endpoint 304 relative to the axis of the reference device 200. The depth endpoint 304 can be used as the depth endpoint 304 of the virtual representation 306.

[0153] In some examples, the control system 18 may be configured to depict a live depth depiction of the drilling procedure as the end effector 102 drills through the bone 14 by displaying the depth endpoint 304. As previously discussed, because the scale / magnification factor may differ from one image to the next, the control system 18 may use the scale / magnification factor to depict the live depth depiction with the depth endpoint 304 overlaid onto the image relative to the bone 14. For example, the control system 18 may use the scale / magnification factor in determining the appropriate pixel-to-millimeter conversion for each image.

[0154] The control system 18 then transforms the drilling start point 300, the trajectory end point 302, and the depth end point 304 to the image coordinate system using the imaging system-reference device transformation. The control system 18 then operates to overlay representations 306, 308 based on this information. For example, the control system 18 can be configured to overlay two lines, namely a first virtual representation 306 and a second virtual representation 308, on the x-ray image using these points. In some embodiments, these lines can represent the end effector 102, or the hole to be created by the end effector 102. The first and second virtual representations can have different colors, shapes, translucency, etc. The first and second virtual representations 306, 308 can be collinear with each other because they are both projected based on the orientation determined by the sensor module 148.

[0155] The first virtual representation ends at depth endpoint 304, which represents the end of the tip of drill bit 102. A second virtual representation 308 extends from depth endpoint 304 and may terminate at trajectory endpoint 302, where trajectory endpoint 302 represents a maximum drilling depth.

[0156] The control system 18 can be configured to allow the surgeon to freeze or store the virtual trajectory as a desired trajectory when the surgeon is satisfied with the virtual trajectory depicted in the image. When the desired trajectory is stored, the display unit 20 may also include a graphical representation to provide a visual cue to the surgeon when the current trajectory matches the stored desired trajectory. For example, a movement indication may be displayed on the display unit 20 along with the directional movement of the surgical instrument 100. The movement indication may be displayed on the display unit 20 relative to a target (e.g., a representation of a crosshair or a circle). The target represents the desired trajectory. In addition, visual information (e.g., a graphical representation) may be displayed on the display 144 of the surgical instrument 100, so that the surgeon can align the current trajectory to the stored desired trajectory without looking at the display unit 20.

[0157] The surgical system 12 may be configured to generate an audible, visual, or tactile alarm when the desired trajectory is achieved. Although the present disclosure contemplates displaying the image and virtual representation on the display unit 20, the control system 18 may transmit the virtual representation to other computing devices and / or displays, including a display attached to the surgical instrument 100, a tablet computer, an operating room display unit, a display unit attached to an imaging device, a laptop computer, a mixed reality device, or any display unit in the operating room. In the event that the display unit 20 is provided in a form other than being attached to the surgical instrument 100, a processor may be provided with the display unit 20 and a suitable communication device (e.g., an antenna).

[0158] The accessory controller 134 may be configured to transmit the first signal, the second signal, and / or the third signal directly to the instrument controller 110 or some other controller, or may perform the calculations described above.

[0159] In the case where the accessory controller 134 performs one or more of the calculations described throughout, the accessory controller 134 can send the drilling start point 300, the trajectory end point 302, and the depth end point 304 relative to the reference axis to the display unit 20 (e.g., a tablet computer). This can simplify the number or time of transmissions because less data needs to be transmitted from the surgical instrument 100 to the display unit 20.

[0160] The control system 18 may also determine the transformation from the axis of the reference device 200 to the X-ray image for each captured image using the radiopaque fiducial 202. It should be understood that the radiopaque fiducial 202 included in each reference device 200 may be spatially different from each other, and therefore, the calculation may be performed for each reference device 200 and for each captured image.

[0161] The output on display 20 may include a graphical or pictorial representation of the position of instrument 100 relative to patient 10. This output may also be presented in an augmented or virtual reality environment, such as a wearable display through which a user views the surgical site.

[0162] In the case where the display unit 20 is a tablet or other mobile computing device, the instrument 100 can communicate wirelessly with the display unit 20 to provide points associated with the display 20. The display unit can also have a wired or wireless connection to an X-ray machine to obtain an X-ray image. The application on the display unit 20 obtains the X-ray image from the X-ray machine, processes the X-ray image to remove distortion, determines the transformation matrix from the reference to the image using the detected radiopaque reference 202 on the reference device, and provides trajectory and drill position guidance using information provided by the instrument 100 (e.g., the accessory 104).

[0163] The display unit can project the drilling start point 300, the trajectory end point 302, and the depth end point 304 relative to the reference axis onto the X-ray image using the transformation and transformation matrix described above. The display unit can overlay the trajectory and drill tip guidance information on the captured X-ray image to provide simulated guidance on all captured X-ray images.

[0164] The system 12 may utilize the imaging system-reference device transformation to determine the pose of the imaging system 22 relative to the reference device 200. More specifically, when the imaging system 22 includes the detector 24 and the source 26, when the imaging system 22 generates an X-ray image, the system 12 may know the pose of the source 26 (and / or the detector 24) relative to the reference device 200 based on the pose of the reference device 200 in the X-ray image.

[0165] refer to Fig.15 , the source 26 and the instrument 100 are shown in various postures relative to the reference device 200 and the skeleton 14. For ease of illustration, Fig.15 is a two-dimensional abstraction of the above-mentioned posture, reference device 200 and skeleton 14. Fig.15 Each source 26 shown in the figure is a discrete location where an X-ray image is generated, where the image boundaries of each corresponding image are shown as dashed lines extending from each source 26. These image boundaries represent the field of view of the imaging system 22, and each image boundary includes the bone 14 and the reference device 200 because the bone 14 is the target of the imaging system 22. The image boundaries are intended to depict one dimension of the generated X-ray image (e.g., the "height" of the generated X-ray image), and a second dimension of the generated X-ray image (e.g., the "width") extends into the page and is therefore not shown in the figure. Each pose of the source 26 is depicted by a pair of arrows extending from the corresponding source 26. Each pose of the instrument 100 is depicted by a pair of arrows extending from the corresponding instrument 100.

[0166] The pose of the source 26 and the instrument 100 may be calculated using the transformation matrices described above. In summary, the control system 18 may know and / or may determine the pose of the reference device 200 relative to the image coordinate system based on the pose of the radiopaque fiducial 202 in the X-ray image. The control system 18 may know and / or may determine the pose of the instrument 100 relative to the reference device 200 based on the pose of the optical fiducial 204 in the image captured by the camera 146. The control system 18 then tracks the pose of the instrument 100 using the IMU described above. For example, when the reference device 200 is in the field of view of the camera 146, the IMU contained in the sensor module 148 may be reset / calibrated so that the transformation between the IMU and the reference device 200 is determined by the IMU-camera transformation and the camera-reference device transformation. Since the sensor module 148 is fixed relative to the instrument 100, the transformation between the instrument 100 and the reference device 200 may be determined by the camera-reference device transformation. Thereafter, the control system 18 may determine the transformation between the instrument 100 and the image coordinate system to determine the pose of the instrument 100 in the image coordinate system. Therefore, the control system 18 can know and / or determine the pose of the instrument 100 in the image coordinate system. The control system 18 uses the above transformation to determine the pose of the instrument 100 in the image coordinate system.

[0167] It will be appreciated that the image coordinate system changes relative to the global coordinate system depending on the pose of the source and detectors 26, 24. To determine the pose of the image coordinate system relative to the global coordinate system, the pose of the reference device 200 is used. Since the reference device 200 remains fixed in the global coordinate system, the reference device 200 allows the control system 18 to determine the pose of the image coordinate system.

[0168] Still refer to Fig.15 , the pose of the instrument 100 can be used to determine which X-ray image is displayed on the display 20. In many cases, the user prefers that the display 20 display two orthogonal X-ray images, one image representing an X-ray of the patient 10 that is consistent or aligned with the user's perspective, and the other image representing an X-ray of the patient 10 taken orthogonally thereto. In other words, one image should match the way the user sees the patient 10. For example, if the user plans to drill a hole in the leg of the patient 10 and the user is standing in front of the patient 10 in a manner facing the patient 10, one image displayed on the display 20 will be an anterior-posterior (AP) X-ray of the patient 10, and the other image displayed on the display 20 will be a lateral-medial (ML) X-ray of the patient 10. Ideally, the drilling start point 300, the trajectory end point 302, and the depth end point 304 will be overlaid in the AP X-ray, while the points 300, 302, 304 are displayed as straight lines in the ML X-ray. Here, "AP image" and "AP X-ray" refer to an X-ray image that is substantially aligned with the points 300, 302, 304 and / or the tool axis of the instrument 100. This means that the image coordinate system of the X-ray image will be substantially aligned with the coordinate system of the camera 146 on the drill, so that the AP image displayed on the display 20 is substantially similar in perspective to the image taken by the camera 146. Similarly, "ML image" and "ML X-ray" herein refer to an X-ray image that is substantially perpendicular to the AP image and substantially perpendicular to the aligned points 300, 302, 304 and / or the tool axis of the instrument 100.

[0169] When generating x-ray images for the patient 10, particularly if generated by an imaging system 22 such as a C-arm 32, multiple x-ray images are generated as the imaging system 22 is rotated relative to the patient 10 (or vice versa). As described above, the control system 18 can determine which one or more of these x-ray images to display on the display 20 based on the posture of the device 100. These images are the AP images and ML images described above. As described below, the system can determine which of the x-ray images generated by the imaging system 22 are most representative of the AP images and ML images relative to the posture of the device 100.

[0170] refer to Figures 16A-16C, showing the instrument 100 and source 26 in a particular posture relative to the reference device 200. The posture of the source 26 is referred to herein as the source position. Individual source positions are depicted by numbered source positions P1, P2, P3, P4, etc. Each source 26 includes an arrow overlapping the corresponding source 26, which corresponds to the posture of the source 26 and thus the posture of the image coordinate system. The extension of one dimension of the image coordinate system is shown as a dashed line extending from the source 26, referred to herein as the image reference axis. The image reference axis is perpendicular to the generated X-ray image and is best thought of as a line extending into / out of the two-dimensional X-ray image. In other words, the two-dimensional X-ray image can have a height and width corresponding to, for example, the x-axis and y-axis of the image coordinate system. In this example, the image reference axis is the z-axis of the image coordinate system. The instrument 100 is shown with a dashed line extending from the instrument 100 - the dashed line is the tool axis T of the instrument 100. The tool axis T corresponds to a straight line extending from the distal end of the end effector 102 and parallel to the end effector 102.

[0171] The system can select appropriate AP and ML images according to the relationship between the posture of the instrument 100 and the image reference axis (eg, the instrument offset angle IA). The instrument offset angle IA is the angle between the tool axis T and the image reference axis. Fig.16A Helps illustrate this feature of AP images, Fig. 16B Helps illustrate this feature of ML images.

[0172] refer to Fig.16A For AP images, the control system 18 may be configured to automatically select an image that minimizes the instrument offset angle IA. Fig.16A , source 26 is shown at first and second source positions P1, P2. A first image generated when source 26 is at first source position P1 results in a first instrument displacement angle IA1. A second image generated when source 26 is at second source position P2 results in a second instrument displacement angle IA2. Fig.16A As can be seen in FIG, the first instrument offset angle IA1 is less than the second instrument offset angle IA2. Therefore, for the AP image, the control system 18 will select and display the image generated by the imaging system 22 when the source 26 is at the first source position P1. As the posture of the instrument 100 changes, the control system 18 can also update the image selected as the AP image. For example, if the instrument 100 rotates counterclockwise so that the first instrument offset angle IA1 becomes greater than the second instrument offset angle IA2, the control system 18 will change the AP image to the image generated by the imaging system 22 when the source 26 is at the second source position P2. This function eliminates the need for medical personnel to manually select which image should be displayed, and instead selects the image that is most suitable for trajectory visualization.

[0173] refer to Fig. 16BFor the ML images, in another suggested embodiment, the control system 18 selects the image that results in the instrument offset angle IA being as close to 90 degrees as possible. In other words, the control system 18 selects the image whose image reference axis is most orthogonal to the tool axis T of the available images. For example, in Fig. 16B , source 26 is shown in third and fourth source positions P3, P4. A first image generated when source 26 is at third source position P3 results in a third instrument offset angle IA3. A fourth image generated when source 26 is at fourth source position P4 results in a fourth instrument offset angle IA4. Fig. 16B As can be seen in FIG. 1 , the third instrument offset angle IA3 is closer to 90 degrees than the fourth instrument offset angle IA4. Therefore, for the ML image, the control system 18 will select and display the image generated by the imaging system 22 when the source 26 is at the third source position P3. The control system 18 can also update the image selected as the ML image as the posture of the instrument 100 changes. For example, if the instrument 100 rotates counterclockwise so that the fourth instrument offset angle IA4 becomes closer to 90 degrees than the third instrument offset angle IA3, the control system 18 will change the ML image to the image generated by the imaging system 22 when the source 26 is at the fourth source position P4.

[0174] although Fig.16A and 16B Angle IA is shown only in two dimensions, but it is understood that both the image reference axis and the tool axis T are three-dimensional vectors in space. As further described below, control system 18 can also confirm whether the trajectory of instrument 100 is visible in the X-ray image when AP and ML images are selected.

[0175] refer to Fig. 16C , depicts an exemplary scenario in which the pose of the instrument 100 and the tool cart are approximately aligned with one image captured by the imaging system 22 and approximately perpendicular to another image captured by the imaging system 22. In other words, the tool axis T is parallel to the image reference axis of the X-ray image generated when the source 26 is in the first source position P1, and the tool axis T is approximately perpendicular to the image reference axis of the X-ray image generated when the source 26 is in the third source position P3. This is represented by the instrument offset angle IA being approximately 90 degrees relative to the third source position P3 and the instrument offset angle being approximately 0 degrees relative to the first source position P1. In this scenario, the control system 18 can provide an AP image that replicates the perspective of a user holding the instrument 100 and a corresponding ML image that replicates a perspective orthogonal to the user's perspective.

[0176] refer to Fig.17A and 17B, various X-ray views of a skeleton 14 are shown. Each X-ray has a unique aligned image reference axis (not shown), which is represented by the global coordinate axes in the upper left corner of each X-ray image. Fig.17A and 17B It also shows how the length and orientation of the trajectory of the instrument 100 may vary depending on how the X-ray image was generated (eg, the position of the source 26 and detector 24 relative to the patient 10 when the image was taken).

[0177] To take advantage of this phenomenon, the control system 18 can also select the appropriate AP and ML images based on a radial distance calculation. The radial distance calculation is a measurement of the length of the trajectory of the instrument 100 in the X-ray image. More specifically, the radial distance is the distance between the drilling start point 300 and the trajectory end point 302. Because the X-ray image is two-dimensional, the radial distance changes as the image reference axis is rotated relative to the trajectory of the instrument 100. When the image reference axis is parallel to the trajectory of the instrument 100, the radial distance is minimized. When the image reference axis is perpendicular to the trajectory of the instrument 100, the radial distance is maximized. This allows the control system 18 to select the appropriate AP image by determining in which X-ray image the radial distance is the smallest, and to select the appropriate ML image by determining in which X-ray image the radial distance is the largest.

[0178] Another way to describe the radial distance calculation is relative to the viewing angle of the x-ray image. When capturing a two-dimensional x-ray image, the image coordinate system can be considered to have an x-axis and a y-axis corresponding to the height and width of the generated x-ray image as described above. The plane formed by the x-axis and the y-axis can be referred to as the image reference plane. In this example, the image reference axis is still the z-axis of the image coordinate system, and the image reference axis is perpendicular to the image reference plane. When determining which x-ray image results in the minimum and maximum radial distances, the control system 18 projects the drilling start point 300 and the trajectory end point 302 onto the image reference plane. After projecting the points 300, 302 onto the image reference plane, the radial distance is the distance between the projected points 300, 302 under the constraints of the image reference plane. The radial distance can also be normalized based on the magnification and / or scale factor of the image.

[0179] refer to Fig.17A, two X-ray images are shown side by side to illustrate how the control system 18 determines which X-ray image to display as the ML image on the display 20. The change in radial distance can be seen by the change in trajectory length between the two images. In the example shown, a first trajectory length L1 is shown in one image, while a second trajectory length L2 is shown in the other image. Although the first and second trajectory lengths L1, L2 are different in length, both represent the same trajectory, as represented by the line between the drilling start point 300 and the trajectory end point 302. As described above, the control system 18 selects the ML image based on which radial distance is longer. In the illustrated example, the first trajectory length L1 is longer than the second trajectory length L2. Therefore, the control system 18 will display the left image on the display 20 as the ML image.

[0180] refer to Fig. 17B , two other X-ray images are displayed side by side to illustrate how the control system 18 determines which X-ray image to display as the AP image on the display 20. In the example shown, the third trajectory length L3 is displayed in one image and the fourth trajectory length L4 is displayed in the other image. The third and fourth trajectory lengths L3 and L4, although different in length, both represent the same trajectory as represented by the line between the drilling start point 300 and the trajectory end point 302. As described above, the control system 18 selects the AP image based on which radial distance is shorter. In the example shown, the third trajectory length L3 is shorter than the fourth trajectory length L4. Therefore, the control system 18 displays the left image on the display 20 as the AP image.

[0181] refer to Fig.18A and 18B , together with Fig.17A and 17B1 and a virtual representation 306 of the end effector 102. The length and orientation of the virtual representation 306 are affected by how the X-ray image was generated (e.g., the position of the source 26 and detector 24 relative to the patient 10 when the image was taken). Although the end effector 102 was not present during the X-ray, the control system 18 can use the virtual representation 306 as a projection of the end effector 102 on the image, with a length and orientation that matches what the end effector 102 would have looked like if the end effector 102 was present when the X-ray was taken. The virtual representation 306 of the end effector 102 is generally aligned with the trajectory of the instrument 100 and / or the tool axis T by the control system 18, and can be given a length equal to the length of the end effector 102. However, in the illustrated embodiment, the control system 18 not only aligns the virtual representation 306 with the trajectory of the instrument 100, but the control system 18 also changes the length of the virtual representation 306 to match what it would look like when viewed from the perspective of the image. Similar to the trajectory itself, the end effector 102 will appear different / longer / shorter when viewed from different perspectives. In some embodiments, the virtual representation 306 of the end effector 102 changes as the pose of the instrument 100 changes, so that the X-ray image displayed on the display 20 accurately shows what the end effector 102 looks like from the perspective of the image. In any embodiment, the projection of the end effector 102 can be stationary relative to the image, or can move as the instrument 100 moves.

[0182] The virtual representation 306 of the end effector 102 may appear similar to the end effector 102, for example, Fig.18A and 18B Alternatively, the virtual representation 306 may be any shape having a longitudinal axis parallel to at least one of the tool axis T and the trajectory defined between the points 300, 302. The shape of the virtual representation 306 may be a simple line, or a two-dimensional shape.

[0183] Reference Figures 19A-23C , shows the instrument 100 and the bone 14 and the source 26 in various postures P1, P2, P3 relative to the bone 14. The corresponding AP image is also shown. It should be understood that although the instrument 100 is shown as being consistent with the field of view of the X-ray image, the X-ray image is generally generated before the instrument 100 is introduced into the surgical space. That being said, the posture of the instrument 100 can be used to select an appropriate image for display on the display 20, as described above. Therefore, the instrument 100 is illustrated relative to the bone 14. Reference device 200 has been omitted. Figures 19A-23C Additional factors that the control system 18 may consider when determining which X-ray image to display on the display 20 are shown separately.

[0184] Regardless of which factors are considered in any embodiment, the control system 18 may also determine that an image is feasible for display simply because it is the best image available, even if it fails to meet all of the considered factors. In addition, in some embodiments, the control system 18 may also default to using the most recently captured image.

[0185] refer to Fig.19A , three source positions P1, P2, P3 are shown relative to the bone 14 and the instrument 100. Each source position P1, P2, P3 is approximately parallel to the other source positions P1, P2, P3, but is offset translationally. Therefore, the control system 18 may not rely solely on the instrument displacement angle IA (not shown) or the radial distance calculation, both of which are described above. Instead, the control system 18 can determine the existence of the drilling start point 300 and the trajectory end point 302 within the image boundaries when determining which image to display on the display 20. In other words, the control system 18 can determine whether the trajectory of the instrument 100 (as defined by the line segment extending between the drilling start point 300 and the trajectory end point 302) is within the field of view of the imaging system 22 when determining which image to display on the display 20. This is useful because the user may not want the display 20 to display an X-ray image that is rotationally aligned with the instrument 100 but is focused on an inappropriate area of ​​the bone 14 and or the patient 10.

[0186] refer to Figures 19B-19D , showing Fig.19A The corresponding AP images for each source position P1, P2, P3 shown in FIG. Fig.19B A corresponding AP image is shown when the source 26 is in a first source position P1 . Fig.19C A corresponding AP image is shown when the source 26 is in a second source position P2. Fig.19D FIG. 2 shows a corresponding AP image when the source 26 is at a third source position P3. It should be understood that the most ideal AP image is as follows: Fig.19C 100 is shown because this is the only AP image where the trajectory of the instrument 100 lies within the image boundaries and is therefore visible. Although the other two images are also rotationally aligned with the pose of the instrument 100, the control system 18 does not select them because the trajectory of the instrument 100 lies outside the image boundaries.

[0187] refer to Fig. 20A , showing three source positions P1, P2, P3 relative to the bone 14 and the instrument 100. Fig.19A Similarly, each source position P1, P2, P3 is generally parallel to the other source positions P1, P2, P3, but is offset translationally. Although the source positions P1, P2, P3 are shown at different distances from the skeleton 14, this is for illustrative purposes only. It should be understood that Fig. 20AThe source positions P1, P2, and P3 in the image are at the same distance from the bone 14 (hereinafter referred to as Figures 21A-21C Discuss the difference in distance between the source and the skeleton 14). Fig.19A In contrast, the difference is that each of the source positions P1, P2, P3 generates an X-ray image that at least partially includes the trajectory of the instrument 100 within the image boundaries. In some embodiments, the control system 18 includes a threshold percentage, such as 30%, 40%, or 50%, to determine whether an image is suitable for display based on the percentage of the trajectory that is presented within the image boundaries. In addition, in some embodiments, the control system 18 requires that the drilling start point 300 is located within the image boundaries in order for the image to be available for display. Even further, in other embodiments, the control system 18 requires that at least one of the drilling start point 300, the trajectory endpoint 302, and the depth endpoint 304 (not shown) be present within the image boundaries in order for the image to be available for display.

[0188] If the system 18 determines that the trajectory endpoint 302 is not within the image boundary, the control system 18 can determine the image boundary radial distance and the image boundary trajectory endpoint. The length / distance of the image boundary radial distance is equal to the portion of the radial distance that is within the image boundary of the x-ray image. For example, if the image boundary is located between the drilling start point 300 and the trajectory endpoint 302, the image boundary radial distance will be 50% of the radial distance. The control system 18 can use the image boundary radial distance instead of (or represent) the percentage of the trajectory presented within the image boundary. In this process, the control system 18 can also determine whether the image is suitable for display based on the comparison between the radial distance and the image boundary radial distance. The comparison is usually a percentage value, so that the length / distance of the image boundary radial distance must be greater than or equal to a threshold percentage of the radial distance. In one example, the radial distance can be equal to 5 centimeters. If the threshold percentage is 50%, the image boundary radial distance must be at least 2.5 centimeters before the x-ray image is considered for display on the display 20. The threshold percentage may be greater than or less than 50% and can be set by the user. If the control system 18 determines that the X-ray image does not include the trajectory endpoint 302, but includes a sufficiently large radial distance within the image bounds (compared to the radial distance), the system 18 can determine an in-image bounds trajectory endpoint (not shown). The in-image bounds trajectory endpoint represents the endpoint of the trajectory relative to the image boundary. Since not all trajectories are within such an image, the in-image bounds trajectory endpoint will be calculated as the point where the trajectory intersects the image boundary. The in-image bounds trajectory endpoint can be interpolated by the control system 18 as the intersection point between the trajectory (represented by the line extending between points 300, 302) and the image boundary.

[0189] refer to Figures 20B-20D , showing Fig. 20A The corresponding AP images for each source position P1, P2, P3 shown in FIG. Fig. 20B The corresponding AP image is shown when the source 26 is in the first source position P1. Fig. 20C A corresponding AP image is shown when the source 26 is in a second source position P2. Fig. 20C FIG. 2 shows a corresponding AP image when the source 26 is at the third source position P3. It can be understood that the most ideal AP image is as follows: Fig.20D 100 is shown because this is the only AP image in which the trajectory of instrument 100 is completely within the image boundaries and is therefore completely visible. Even though the other two images are also rotationally aligned with the pose of instrument 100, they are not selected by control system 18 because at least a portion of the trajectory of instrument 100 is outside the image boundaries. If the image generated by imaging system 22 with source 26 at third source position P3 is not available, control system 18 may instead select Fig. 20B and 20C For example, if the control system 18 is configured to select an image in which the percentage of trajectories exceeds a threshold percentage, then Fig. 20C , because it has a greater percentage of trajectories within its image boundaries, as shown by the presence of larger line segments within the image. When the control system 18 requires that the drilling start point 300 be within the image boundaries, this can be used Fig. 20B When the control system 18 requires that the trajectory endpoint 302 be within the image boundary, it can be used Fig. 20C of the image.

[0190] refer to Fig.21A , showing two source positions P1, P2 relative to the bone 14 and the instrument 100. Fig.19A and 20A Similarly, the first source position P1 is roughly parallel to the second source position P2, but the second source position P2 zooms out further than the first source position P1, so that the image produced by the second source position P2 captures more of the bone 14 than the image produced by the first source position P1. In other words, the image boundary of the image produced by the second source position P2 covers more of the bone 14. Since each image reference axis will be aligned identically with the pose of the instrument 100, the control system 18 can select which image to display on the display 20 based on the existence of at least one of the trajectory of the instrument 100, the drilling start point 300, and the trajectory end point 302. This is similar to the reference axis. Figures 20A-20D The functionality described is the same except that, rather than selecting to translationally offset the image, the control system 18 selects an image that more closely magnifies the field of view to increase visibility of the trajectory of the instrument 100. It should be understood that, although the source positions P1, P2 are shown at different distances from the bone 14 to illustrate the differences in zoom levels, any method of varying the zoom level of the generated image is contemplated (including methods where the source 26 does not change position between zoom levels).

[0191] refer to Fig. 21B and 21C , showing Fig.21A The corresponding AP images for each source position P1, P2 shown in . Fig. 21B A corresponding AP image is shown when source 26 is at a first source position P1 and produces an image having a first zoom ratio. Fig. 21C 1 and 2. The corresponding AP image is shown when the source 26 is at a second source position P2 and produces an image with a second zoom ratio, which provides a more magnified view of the bone 14. It is also contemplated that the first and second source positions P1, P2 may be the same and the second zoom ratio may be produced without moving the source 26. It should be understood that the most ideal AP image is as follows: Fig. 21C , because this is the only AP image in which the trajectory of instrument 100 is completely within the image boundaries and is therefore completely visible. Even if another image is rotationally aligned with the pose of instrument 100, control system 18 will not select it because at least a portion of the trajectory of instrument 100 is outside the image boundaries. If the image generated by imaging system 22 when source 26 is at the second source position P2 (and / or the second zoom ratio) is not available, control system 18 may instead select Fig. 21B Image shown.

[0192] refer to Fig.22A , two source positions P1, P2 are shown relative to the bone 14 and the instrument 100. Unlike the previous figures, the source positions P1, P2 provide images with different degrees of alignment relative to the pose of the instrument 100. The first source position P1 is aligned to produce an image with an image reference axis (not shown) that is substantially parallel to the tool axis T, while the second source position P2 is aligned or adjusted to provide an image with an image reference axis (not shown) that is substantially non-parallel to the tool axis T. As described above, the control system 18 may consider whether at least one of the trajectory of the instrument 100, the drilling start point 300, and the trajectory end point 302 exists within the image boundary when selecting an image to be displayed on the display 20. In some embodiments, the control system 18 prioritizes the existence of the trajectory and points 300, 302 within the image boundary rather than the degree of alignment between the image reference axis and the trajectory. In such an embodiment, the control system 18 selects an image based on the fact that the user prefers to see an image that includes the target drilling area (i.e., the trajectory and points 300, 302), even if the image is not aligned or consistent with the surgeon's field of view like other images.

[0193] refer to Fig. 22B and 22C , showing Fig.22A The corresponding AP images for each source position P1, P2 are shown in . These figures emphasize the importance of the priority, because Fig. 22B The image shown in may not be helpful to the user because it does not include the target drilling area. Fig. 22C The images shown may not be aligned as the user would like, but are consistent with Fig. 22B The image can still better show the user the posture of the instrument 100 relative to the bone 14 than the image shown in FIG.

[0194] refer to Fig.23A , showing two source positions P1, P2 relative to the bone 14 and the instrument 100. Fig.21A Similarly, the first source position P1 is approximately parallel to the second source position P2, and the second source position P2 magnifies the field of view more than the first source position P1, so that the image produced by the second source position P2 captures more of the bone 14 than the image produced by the first source position P1. In other words, the image boundary of the image produced by the second source position P2 covers more of the bone 14. Similarly, because each image reference axis will be equally aligned with the pose of the instrument 100, the control system 18 must rely on independent and different factors to determine which image to display on the display 20. In addition, because the image boundary of the two images will include the entire trajectory and points 300, 302, the control system 18 cannot rely on these to determine which image to display on the display 20. That being said, Figures 23A-23C 1 shows another function of the control system 18. Instead of or in addition to selecting which image to display based on the above considerations, the control system 18 can also determine which image to display based on the distance between the distal end of the end effector 102 and the drilling start point 300.

[0195] refer to Fig. 23B and 23C , showing Fig.23A The corresponding AP images for each source position P1, P2 shown in . Fig. 23B A corresponding AP image is shown when source 26 is at a first source position P1 and produces an image having a first zoom ratio. Fig.23C The corresponding AP image is shown when the source 26 is at a second source position P2 and produces an image with a second zoom ratio, which provides a closer and more magnified view of the bone 14 and the trajectory. It is also contemplated that the first and second source positions P1, P2 can be the same and the second zoom ratio can be produced without moving the source 26. As described above, the control system 18 can determine which image to display based on the distance between the distal end of the end effector 102 and the drilling starting point 300. When the distance is greater (i.e., the instrument 100 is farther from the patient 10), the control system 18 can display Fig. 23B The image shown in order to provide the user with a wider, more comprehensive view of the skeleton 14. When the distance is small (e.g., less than a threshold distance), the control system 18 may display Fig.23C300, 302. Without different images (e.g. Fig.23C ) is available, the control system 18 may alternatively increase Fig. 23B The zoom level of the image can be adjusted and the zoomed-in image can be cropped to more closely observe the position of the end effector 102 .

[0196] In some embodiments, when a user uses the instrument 100 to drill into the bone 14, the control system 18 can determine the position of the end effector 102 to be displayed on the display 20 based on the displacement of the end effector 102. Fig. 23B and 23C More specifically, the control system 18 may receive a signal from the depth sensor 122 indicating that the end effector 102 is moving between the drilling start point 300 and the trajectory end point 302. In other words, the depth end point 304 is moving from the drilling start point 300 toward the trajectory end point 302. When the depth end point 304 moves within a threshold distance of the trajectory end point 302, the control system 18 may receive a signal from the depth sensor 122 indicating that the end effector 102 is moving between the drilling start point 300 and the trajectory end point 302. Fig. 23B The X-ray switch shown is Fig.23C X-ray shown.

[0197] Although in addition Fig.18A and 18B The projected end effector 102 is not shown in any of the images other than those shown in FIG. 1 , but it should be understood that this functionality may be included in any of the embodiments discussed.

[0198] Furthermore, it should be understood that the ideal AP and ML images (in any embodiment) displayed on the display 20 will be approximately aligned and approximately perpendicular to the tool axis T, respectively, as described above. The images selected for these graphics are intentionally offset by a small degree (or slightly) to avoid losing detail in the graphics. For example, if the ideal AP image is displayed in the graphics, the drilling start 300 and the trajectory end 302 will overlap, making the trajectory difficult to discern. In practice, the ideal AP and ML images may be displayed on the display 20.

[0199] In some embodiments, the control system 18 is configured to reduce the computational cost of determining which X-ray images to display on the display 20 as AP and ML images. To do this, an AP image can be selected from a plurality of AP X-ray images that are created when the source 26 and the detector 24 are approximately located at the anterior and posterior sides of the patient 10 (and or the bone 14), respectively. For example, when the source 26 is set within a certain rotational degree of the source position relative to the ideal AP image, the plurality of AP X-ray images may have all been captured by the C-arm 32. Therefore, the control system 18 can be optimized by limiting the selection of AP images to the plurality of AP X-ray images. The control system 18 can also optimize the selection of ML images in a similar manner. For example, when the source 26 is set within a certain rotational degree of the source position relative to the ideal ML image, the plurality of ML X-ray images may have all been captured by the C-arm 32.

[0200] refer to Fig.24 , showing different postures of the instrument 100 relative to the bone 14, wherein the display 20 displays an image selected by the control system 18 based on the corresponding posture of the instrument 100. As shown, the X-ray image displayed on the display 20 is available for viewing by the user and changes as the instrument 100 moves relative to the bone 14 (i.e., relative to the reference device 200). This allows the user to operate on the bone 14 without having to comprehend the position of the instrument 100 relative to the unaligned X-ray image. Instead, the user is provided with an X-ray that is aligned with the posture of the instrument 100 described herein. Therefore, because the image of the bone 14 provided to the user is aligned with the user's perspective of the bone 14, the X-ray is easier for the user to interpret.

[0201] Referring to Figures 25-33, flow charts are provided that detail the method by which the air control system 18 assists a user in a procedure in the manner described above (e.g., by determining and displaying AP and ML images). It should be understood that the steps shown in the figures may be performed in any suitable order. Some steps may be omitted, and other steps may be added.

[0202] refer to Figures 25A-25C , a method 500 for guiding a user during a procedure is provided. The method 500 begins with a series of steps for setting up various devices used in the procedure. In the illustrated method 500, the control system 18 and the display 20 are implemented as elements of a tablet computer. It should be understood that the steps described with reference to the tablet computer can be performed by alternative embodiments of the control system 18 and the display 20. For example, the system 12 can include the control system 18 and the display 20 implemented according to any of the alternatives described herein. In the illustrated method 500, the user is a surgeon.

[0203] At 502, the Fluorodisc is mounted to the source 26 of the C-arm 32. At 504, the tablet computer (and or alternatively the display 20 and control system 18) is powered on. At 506, communication is established between the C-arm 32 and the tablet computer. This places the control system 18 in communication with the display 20 and or the C-arm 32. The communication can be wired or wireless. At 508, the accessory 104 is connected to the battery 114 to power the accessory 104. The accessory 104 can automatically power on after being placed in electrical communication with the battery 114, or the control element 150 can include a power button. Wireless communication is established between the tablet computer and the accessory 104 at 510. This places the control system 18 in communication with the accessory 104 and / or the instrument controller 110.

[0204] At 512, patient 10 is moved to the OR table (on Figure 1 10 and / or the bone 14). At 514, the surgeon makes an incision in the patient 10 to expose the bone 14. At 516, the surgeon optionally mounts the bone plate 16 to the bone 14 by means of a temporary fixation device (e.g., pins or K-wires). At 518, the reference device 200 is fixed relative to the table and / or the patient 10. The reference device 200 is fixed in the plan view of the imaging system 22 (in this case, the C-arm 32) and the surgeon. At 520, the C-arm 32 is positioned so that the source 26 is in a source position relative to the patient 10 and / or the bone 14 so that an X-ray image can be taken. At 522, the C-arm 32 captures an X-ray image of the patient 10 and / or the bone 14. At 524, the tablet computer automatically detects the X-ray image captured at 522. At 526, the tablet computer displays the X-ray image captured at 522. More specifically, the control system can be configured to control the selection of the X-ray image based on the time the image was taken and the current time. For example, the control system may be configured to select an X-ray image having the shortest duration between the capture time of the image and the current time.

[0205] At 530, X-ray image processing is performed. X-ray image processing is described in detail in Fig.26 and described in detail below.

[0206] At 532, the control system 18 determines whether the source position can be calculated based on the visibility of the reference device 200 in the X-ray image. If yes, the position of the source 26 (relative to the reference device 200) is calculated and stored as metadata associated with the X-ray image. If no, the method 500 moves to 534, where the tablet instructs the surgeon to reposition at least one of the C-arm 32 and the reference device 200 to ensure that the source position can be calculated based on the pose of the reference device 200 in the X-ray image. The method 500 returns to 520 and ignores the X-ray image if necessary. Otherwise, the method 500 continues to 536. At 536, the surgeon determines whether the X-ray image is suitable for guidance. If yes, the method 500 continues to 538. If not, the X-ray image is ignored and the method 500 moves back to 520. At 538, the surgeon manipulates the control element 150 of the accessory 104 to select the X-ray image for guidance. At 540, the accessory 104 communicates the selection to the tablet control system 18.

[0207] At 542, the tablet stores the X-ray image along with its transformation matrix as metadata associated with the image. The transformation matrix is ​​described above and is as follows: Fig. 11B The imaging system reference device transform is shown. The imaging system reference device transform is different for each x-ray image and is calculated based on the pose of the reference device 200 in the appropriate x-ray image. The x-ray images can be stored on the tablet or on any device that communicates with the control system 18, such as a separate server and or cloud device.

[0208] At 544, an image slice view of all stored X-ray images is displayed to the user on the display 20. At 546, the surgeon determines whether more / new images are needed. If so, at 548, the surgeon can manipulate the control element 150 to begin capturing new X-ray images. If new images are needed, the method returns to 520. However, if not, the method continues to 550. At 550, the surgeon places the instrument 100 (i.e., the end effector 102) on the bone 14 and manipulates the control element 150 to begin the guidance process. As described herein, the guidance process can be initiated by pressing the sleeve instead of the control element 150. Fig.27A and 27B The guided process is shown in more detail. At 560, the guided process continues until the procedure is complete or the surgeon wishes to generate a new image. At 580, if the procedure is complete, the method 500 ends. Otherwise, the method 500 returns to 544.

[0209] refer to Fig.26, and the X-ray image processing is also shown in more detail. X-ray image processing occurs at 530 during method 500. At 530A, the X-ray image is processed to remove distortion. Distortion is detected and removed using Fluorodisc 34. At 530B and 530C, intrinsic parameters and extrinsic parameters of the C-arm 32 are determined by the control system 18. Intrinsic parameters describe parameters of the C-arm 32, such as focal length, field of view, aperture, resolution, and other parameters (except extrinsic parameters) associated with how the C-arm 32 captures the X-ray image. Intrinsic parameters can be determined using the Fluorodisc 34 and or the pose of the reference device 200. On the other hand, extrinsic parameters describe the position and orientation (e.g., pose) of the C-arm 32, which are calculated using the pose of the reference device 200 in the image. Intrinsic parameters and extrinsic parameters are calculated for each image, and the combination of these parameters forms an imaging system reference device transform. Thus, at 530D, the imaging system reference device transform of the X-ray image is determined. Finally, at 530E, image artifacts due to the Fluorodisc 34 and the reference device 200 are removed from the X-ray image. The X-ray image processing process ends.

[0210] refer to Fig.27A and 27B , showing the guidance process in more detail. At 560A and 560B, the control system 18 determines whether the control element 150 has been manipulated by the surgeon and / or whether the sleeve 116 has been compressed. If each element 150, 124 has been manipulated and compressed, respectively, the guidance process continues to 560C. Otherwise, the guidance process returns to 560A. The control system 18 may alternatively only require a signal from one of the control element 150 and the sleeve 116 and continue to 560C. At 560C, the optical image is captured using the camera 146. At 560D, the control system 18 confirms whether the image has been captured. If not, the process returns to 560C. If the camera 146 has captured an image, the process continues to 560E. At 560E, the control system 18 determines whether the optically detectable reference 204 is visible within the optical image. If not, the process returns to 560C. If yes, the process continues to 560F.

[0211] At 560F, the control system 18 determines the camera-reference device transformation matrix as described above. After calculating the camera-reference device transformation matrix, the process continues to 560G. At 560G, the IMU is reset to align the sensor coordinate system with the coordinate system of the camera 146 (i.e., the camera coordinate system). For this reason, the process continues to 560H, where the IMU-camera transformation matrix is ​​determined. The IMU-camera transformation matrix is ​​determined by the following combination as described above: camera-reference device transformation, the known spatial relationship between the optical reference 204 and the rest of the reference device 200, and the known spatial relationship between the IMU and the camera 146. The IMU-camera transformation and the camera-reference device transformation can be combined to form the IMU-reference device transformation matrix. The IMU-reference device transformation matrix (or the combination of the IMU-camera transformation and the camera-reference device transformation) allows the control system 18 to convert the signal from the IMU into a change in the posture of the instrument 100 relative to the reference device 200.

[0212] At 560I / 562, the guidance process uses the drilling start determination process to determine the drilling start 300. The drilling start determination process Fig.28 The process at 560J / 564 uses the trajectory endpoint determination process to determine the trajectory endpoint 302. The trajectory endpoint determination process is shown in more detail in FIG. Fig.29 At 560K / 566, the boot process determines the depth endpoint 304 using the depth endpoint determination process. The depth endpoint determination process is Fig.30 . At 560L, points 300, 302, 304 are sent to the tablet computer (i.e., to the control system 18). At 560M / 568, the guidance process calls the image display process to determine which images are displayed on the tablet computer (i.e., the display 20). At 560N, the control system 18 determines whether the surgeon is still using the instrument 100 to drill into the bone 14. That is, whether the drilling is complete. If not, the guidance process returns to 560L. If it is completed, the guidance process ends.

[0213] refer to Fig.28, showing the determination of the drilling start point in more detail. At 562A, the drilling start point determination process determines a transformation matrix between the drill reference axis to the camera coordinate system. At 562B, a point on the end effector axis (and / or tool axis) representing the distal end of the end effector 102 is calculated or known based on the identification feature 154 of the drill bit or alternative means (e.g., user input indicating the end effector type and or length), or based on the camera's detection of the end effector length. At 562C, the process transforms the coordinates of the distal end of the end effector 102 from the known end effector / tool ​​axis to the known axis of the camera 146 based on a predefined and stored end effector camera transformation matrix. At 562D, the process transforms the coordinates of the distal end of the end effector 102 from the camera coordinate system to the reference coordinate system based on the camera-reference device transformation. At 562E, the drilling start point 300 is determined in the reference coordinate system (i.e., relative to the reference reference axis).

[0214] refer to Fig.29 , showing the determination of the trajectory endpoint in more detail. At 564A, the drilling start point 300 relative to the reference reference axis determined at 562E is transformed from the reference coordinate system to the sensor coordinate system. The point 300 is transformed using a combination of the IMU-camera transformation matrix and the camera-reference device transformation matrix and the current pose of the IMU. At 564B, based on the known relationship between the end effector / tool ​​axis and the camera coordinate system and the IMU-camera transformation matrix, the coordinates of the point 300 in the sensor coordinate system are transformed from the sensor coordinate system to a point on the end effector / tool ​​axis. At 564C, the trajectory endpoint 302 is determined relative to the end effector / tool ​​axis based on the drilling start point 300 (relative to the end effector / tool ​​axis) and the casing stroke length. For example, if the casing stroke length is 100 mm, the trajectory endpoint 302 (relative to the end effector / tool ​​axis) is displaced 100 mm along the end effector / tool ​​axis from the drilling start point 300. At 564D, the process transforms the trajectory endpoint 302 from a point on the end effector / tool ​​axis to a point in the sensor coordinate system based on the known relationship between the end effector / tool ​​axis and the sensor coordinate system. At 564E, the process transforms the trajectory endpoint 302 from the sensor coordinate system to the reference coordinate system using a combination of the IMU-camera transformation and the camera-reference device transformation. At this point, the trajectory endpoint 302 has been determined in the reference coordinate system (i.e., relative to the reference reference axis), and the process ends.

[0215] refer to Fig.30, showing the determination of the depth endpoint in more detail. At 566A, the drilling start point 300 relative to the reference reference axis determined at 562E is transformed from the reference coordinate system to the sensor coordinate system. The point 300 is transformed using a combination of the IMU-camera transformation matrix and the camera-reference device transformation matrix and the current pose of the IMU. At 566B, the coordinates of the point 300 in the sensor coordinate system are transformed from the sensor coordinate system to a point on the end effector / tool ​​axis based on the known relationship between the end effector / tool ​​axis and the camera coordinate system and the IMU-camera transformation matrix. At 566C, the depth endpoint 304 is determined relative to the end effector / tool ​​axis based on the drilling start point 300 (relative to the end effector / tool ​​axis) and the signal from the depth sensor 122 relative to the end effector / tool ​​axis. For example, if the depth sensor 122 reports that the distal end of the end effector 102 has entered the bone 14 by 10 millimeters, the depth endpoint 304 (relative to the end effector / tool ​​axis) is displaced 10 millimeters along the end effector / tool ​​axis from the drilling start point 300. At 566D, the process transforms the depth endpoint 304 from a point on the end effector / tool ​​axis to a point in the sensor coordinate system based on the known relationship between the end effector / tool ​​axis and the sensor coordinate system. At 566E, the process transforms the depth endpoint 304 from the sensor coordinate system to the reference coordinate system using a combination of the IMU-camera transformation and the camera-reference device transformation. At this point, the depth endpoint 304 has been determined in the reference coordinate system (i.e., relative to the reference reference axis), and the process ends.

[0216] refer to Fig.31, showing the image display process in more detail. At 560M / 568, the image display process is called by the boot process. At 568A, the AP image determination process is used to determine which X-ray image is displayed on the display 20 as the AP image. At 568B, the ML image determination process is used to determine which X-ray image is displayed on the display 20 as the ML image. Once the appropriate AP and ML images are determined, the image display process continues to 568C. At 568C, the depth value of the end effector 102 is determined. At 568D, if the end effector 102 has advanced less than, for example, 10 mm in the bone 14, the AP and ML images are displayed side by side on the display 20. On the other hand, if the end effector 102 has advanced more than 10 mm into the bone 14, for example, only the ML image is displayed on the display 20. Regardless of whether one or both of the AP and ML images are displayed on the display 20, the image display process continues to 568E. At 568E, the points 300, 302, 304 are transformed into the image coordinate system of the selected AP and or ML image, respectively. At 568F, a trajectory line segment is drawn between the drilling start point 300 and the trajectory end point 302. At 568G, the current pose of the end effector 102 is drawn as a line between the drilling start point 300 and the depth end point 304.

[0217] At 568, the image display process may change the zoom level and / or crop the image displayed on the display 20. Alternatively, the image display process may determine a new AP and / or a new ML image to provide the surgeon with a more / less magnified view of the bone 14. For example, the image display process may utilize a reference image. Figures 23A-23C After 568H, the image display process ends.

[0218] refer to Fig.32 , showing the AP image determination process in more detail. The AP image determination process is performed at 568A, 570 for each X-ray image (e.g., all stored X-ray images shown at 544). At 570A, the drilling start point 300 is transformed from the reference coordinate system to the image coordinate system of the X-ray image. At 570B, the control system 18 determines whether the drilling start point 300 exists within the image boundary of the X-ray image. If not, the AP image determination process moves to the next X-ray image and returns to 570A. If so, the process moves to 570C. At 570C, the trajectory end point 302 is transformed from the reference coordinate system to the image coordinate system of the X-ray image. The remainder of the AP image determination process is performed using the same method as described above with reference to FIG. Figure 17A-1 8D to select the AP image. It is also contemplated that the AP image determination process is changed to utilize the radial distance measurement described above with reference to Fig.16A and 16B Described instrument deviation angle TA.

[0219] At 570D, the radial distance between the drilling start point 300 and the trajectory end point 302 is determined as described above. At 570E, the in-image-bounds radial distance between the drilling start point 300 and the in-image-bounds trajectory end point is determined as described above. At 570F, the AP image determination process determines whether all X-ray images have been processed. If not, the process returns to 570A and is repeated for each X-ray image stored at 542 / 544. Once all images have been processed, the process continues to 570G. At 570G, the AP image is selected as the image having the smallest radial distance, and optionally the length of the in-image-bounds radial distance is equal to at least 50% of the length of the radial distance calculated at 570D. After the AP image is selected, the process ends.

[0220] refer to Fig.33 , showing the ML image determination process in more detail. At 568B, 572, the ML image determination process is performed for each X-ray image (e.g., all stored X-ray images displayed at 544). At 572A, the drilling start point 300 is transformed from the reference coordinate system to the image coordinate system of the X-ray image. At 572B, the control system 18 determines whether the drilling start point 300 exists within the image boundary of the X-ray image. If not, the ML image determination process moves to the next X-ray image and returns to 572A. If so, the process moves to 572C. At 572C, the trajectory end point 302 is transformed from the reference coordinate system to the image coordinate system of the X-ray image. The remainder of the ML image determination process is performed using the same method as described above with reference to FIG. Figure 17A-1 8D to select the ML image. It is also contemplated to modify the ML image determination process to utilize the radial distance measurement described above with reference to Fig.16A and 16B The instrument offset angle IA.

[0221] At 572D, the radial distance between the drilling start point 300 and the trajectory end point 302 is determined as described above. At 572E, the in-image-bounds radial distance between the drilling start point 300 and the in-image-bounds trajectory end point is determined as described above. At 572F, the ML image determination process determines whether all X-ray images have been processed. If not, the process returns to 572A and repeats the process for each X-ray image stored at 542 / 544. After all images have been processed, the process will continue to 572G. At 572G, the ML image is selected as the image with the largest radial distance and the length of the in-image-bounds radial distance is equal to at least 50% of the length of the radial distance calculated at 572D. After the ML image is selected, the process ends.

[0222] In some embodiments, the depth sensor need not be included in the accessory 600. In such cases, the depth sensor may be omitted entirely, or may be included in a separate accessory 602 that is separable from the accessory 600 that includes the camera and / or inertial measurement unit, or separable from the surgical handpiece 604 (see Fig.35 ). The surgical attachment 600 may include a camera, a gyroscope, and one or more accelerometers. In addition, the surgical attachment 602 may include a depth sensor and a measurement cannula as described herein. One or both of these attachments 600, 602 may include an antenna, a battery, or terminals for receiving or sensing data and / or sending or receiving power.

[0223] In addition, the scientific systems described herein (e.g. Figure 5 The surgical system described in 35 or 36 may be compatible with a variety of surgical end effectors. Thus, although some examples are described with respect to a drill bit, such examples also contemplate any of the following end effectors: a drill bit, a screwdriver bit (or a screw driver bit), a surgical pin, a surgical wire, a reamer, or a bone drill.

[0224] In some examples, reference Fig.34A , the surgical attachment 702 (including a surgical attachment that does not include a depth sensor) can be removably connected directly to the surgical zone actuator 700, 700', 700", 700"', 700""2, 700""'. The surgical driver may include a housing having a first end portion for connecting to the surgical handpiece 704 and a second end portion opposite the first end portion. The surgical driver 700 can be configured to drive an end effector. Therefore, the surgical driver 700 and the attachment 702 can each have complementary mating features to facilitate connection to each other, such as suitable grooves and protrusions, magnets and ferromagnetic features, latches, etc. The driver 700 may include a transmission device that slows down, accelerates, increases torque, or reduces torque. Various drive designs and / or surgical handpiece designs are described in the following patents, which are incorporated herein by reference in their entirety: 9,192,394, 5,993,454, 10,925,657, 10,025,657; 10,406,606, which designs are contemplated for use in the surgical systems described herein. The accessory 702 can be configured to be coupled to the surgical driver 700 or the surgical handpiece 704. The accessory 702 can include a battery in this configuration, and can include an antenna as described elsewhere. In addition, the accessory 702 can include an inertial measurement unit and or a camera.

[0225] refer to Fig.34B, a series of different surgical drivers 800 are envisioned, including but not limited to a reamer 800; a needle and or wire drive 800'; a drill driver 800"; and a saw driver 800"'. These can be connected to a surgical attachment 800 that includes a camera, a gyroscope, and one or more accelerometers. Each driver 800 can be configured to drive a corresponding end effector, including a reamer 804, a pin / wire 806, a drill bit 808, and a saw blade 810. Thus, the attachment and accompanying control system can allow a user to visualize the trajectory of these end effectors relative to one or more X-ray images.

[0226] In one example, the system described throughout may be operable to allow switching between end effectors during a procedure while using the same attachment. This may allow visualization of historical drill trajectories acquired simultaneously with visualization of real-time trajectories of screwdriver trajectories. The drill may need to be removed from the surgical handpiece and replaced with a surgical screwdriver. The trajectory of a first end effector may have a different virtual representation than the trajectory of a second end effector.

[0227] In other examples, where there are interchangeable surgical drivers, a surgical accessory including a camera and an inertial measurement unit can be removably attached to a first surgical driver. The first surgical driver can then be used to set a trajectory, such as a trajectory set when placing a surgical pin or surgical wire. This historical trajectory can be viewed later when a second surgical driver (such as a driver for driving a drill or screw) is positioned. The surgical accessory can be removed from the first surgical driver and attached to the second surgical driver. While visualizing the trajectory of the end effector of the first surgical driver, the trajectory of the end effector connected to the handpiece by the second surgical driver can also be visualized. This can allow the user to determine that they are following the trajectory set using the first surgical driver. The trajectory of the first surgical driver may have a different virtual representation than the trajectory of the second surgical driver.

[0228] In another aspect of the present disclosure, a surgical instrument may include a camera coupled to a handpiece. The camera may be included as an integral component of the surgical instrument or as an accessory that can be removably coupled to a surgical handpiece. The surgical handpiece may be configured to drive a surgical end effector directly or through a detachable surgical drive as described above. The camera may generate a camera signal. The control system may be configured to determine the end effector length based on the camera signal using various image processing techniques when the end effector is coupled to the surgical handpiece. Alternatively, the control system may be configured to determine the end effector type based on the camera signal using various image processing techniques when the end effector is coupled to the surgical handpiece. This may be advantageous because the system can detect the length of the end effector without requiring an obvious workflow step, such as pulling a trigger, placing the end effector in front of the camera, or interacting with a user interface or other control element to indicate the length or type of the end effector. The surgical system may be operable to make this determination based on the camera signal and the input signal. More specifically, the surgical system may perform image processing when an input signal indicating that the end effector is fully coupled to the surgical handpiece is received. In these cases, the surgical attachment or instrument can communicate the length of the end effector to various other connected devices, such as control systems, displays, imagers, etc.

[0229] In one example, the input signal is further defined as a user input signal, and the surgical instrument includes a control element that is actuatable to generate the user input signal. The control system can be configured to determine the end effector length upon receiving the user input signal. The control element can be a button on an accessory, a display, or a surgical handpiece, including a virtual icon on the display in some examples.

[0230] In another example, when the surgical instrument includes a depth sensor configured to provide a depth signal associated with the displacement of the end effector, the input signal can be obtained from the depth sensor. In this embodiment, the control system is configured to determine the end effector length when the depth signal indicates that the depth exceeds a predetermined threshold. For example, when the measuring sleeve is pressed down a certain distance (e.g., 10 mm), the image processing algorithm can process the image from the camera. This ensures that the instrument must be ready for operation and the end effector is fully inserted and connected to the surgical handpiece.

[0231] The control system may include a processor and memory programmed with an image processing algorithm, the image processing algorithm including a step configured to determine whether the coupled end effector is of a first length or a second length. More specifically, the surgical system may include only a few different sized effectors, and thus, the image processing algorithm need only distinguish between end effectors of significantly different lengths, i.e., end effectors that differ in length by at least 5 mm.

[0232] As described above, in some instances, the sensor module may further include a measuring sleeve movably coupled to the depth sensor, wherein the control system is configured to determine a distal end of the end effector and a distal end of the measuring sleeve, and wherein the control system is configured to determine a length of the end effector based on the positions of the distal end of the end effector and the distal end of the measuring sleeve. In some instances, the length of the end effector is such that the end effector may not protrude from the measuring sleeve when the measuring sleeve is in a fully distal position. In other instances, the length of the end effector is such that the end effector protrudes from the measuring sleeve when the measuring sleeve is in a fully distal position. The control system may utilize this relationship to discern the length of the end effector coupled to the surgical instrument.

[0233] The image processing algorithm may include the step of extracting features from a portion of the end effector image. The extracted features of the image portion preferably enable correlation (or pairing) of the image portion with the time when the input signal was received. The extracted features may be the intensity, luminosity, hue, saturation, brightness, glossiness, or other color-related values ​​of the image portion in at least one component space, such as red, blue, green, cyan, magenta, yellow, key color, and / or Lab component space. In addition, the extracted features may be a histogram of various color values ​​for a set of pixels in the image portion. Additionally or alternatively, the extracted features may be an estimated surface area of ​​the end effector in the image, the number of pixels of the end effector, the number of pixels of the entire feature, a color intensity value of the end effector, or any other relevant feature inherent in or extractable from the end effector image portion.

[0234] The image processing algorithm may further include the step of segmenting the image, including isolating a first segment of the sample image representing the end of the measuring cannula, and further segmenting the first region to define a portion of the image corresponding to a particular portion of the end effector. Segmenting the sample image into a plurality of image segments preferably improves the resolution and / or accuracy of the length determination. The size and shape of each image segment may be static, wherein each segment comprises a predefined number of pixels in the image and / or a predefined dimension in physical space. However, the portion of the image may include the entire end effector, or the image may be segmented or allocated according to any other scheme. The portion of the sample image is preferably a single segment or region comprising a plurality of pixels of the image, but the portion of the image may optionally be a plurality of image segments or regions of the sample image may be of any other size and / or may be of any other shape.

[0235] It should be understood that the display can be omitted from the surgical system. In such an embodiment, the surgical system can transmit data to a display on the imaging device. The data includes appropriate virtual representations as described herein. Alternatively, the display can be any suitable mobile device or monitor. The display can include a suitable antenna for wireless communication with any other component of the system.

[0236] The image processing algorithm may additionally or alternatively implement any object localization, segmentation (e.g., using edge detection, background subtraction, graph cut-based algorithms, etc.), measurement, clustering, pattern recognition, template matching (using any of a variety of metrics), feature extraction, descriptor extraction (e.g., extracting texture maps, color histograms, HOG, SIFT, etc.), feature dimension reduction (e.g., PCA, K-Means, linear discriminant analysis, etc.), feature selection, thresholding, localization, color analysis, parametric regression, non-parametric regression, unsupervised or semi-supervised parametric or non-parametric regression, or any other type of machine learning or machine vision to estimate the physical size of the sample. Such methods preferably compensate for varying lighting conditions, or any other inconsistencies or variables that are prevalent in any usage scenario.

[0237] The foregoing description is illustrative only and is not intended to limit the present disclosure, its application or use in any way. The broad teachings of the present disclosure can be implemented in various forms. Therefore, although the present disclosure includes specific examples, the true scope of the present disclosure should not be limited thereto, because other modifications will become apparent after studying the drawings, the specification and the appended claims. It should be understood that one or more steps in the method can be performed in different orders (or simultaneously) without changing the principles of the present disclosure. In addition, although each example is described above as having certain features, any one or more of those features described for any example of the present disclosure can be implemented in the features of any other example and / or combined with the features of any other example, even if the combination is not explicitly described. In other words, the examples described are not mutually exclusive, and the combination of one or more examples with each other is still within the scope of the present disclosure.

[0238] The spatial and functional relationships between elements (e.g., between controllers, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "engaged," "coupled," "adjacent," "next," "on," "over," "under," and "disposed." Unless explicitly described as "directly," when the relationship between a first and a second element is described in the above disclosure, the relationship can be a direct relationship, in which there are no intervening elements between the first and second elements, but can also be an indirect relationship, in which there are one or more intervening elements (spatially or functionally) between the first and second elements.

[0239] As used herein, the phrase 'at least one of A, B, and C' should be interpreted to mean a logical "(AORBORC)", using a non-exclusive logical "OR", and should not be interpreted as "at least one of A, at least one of B, and at least one of C". The term subset does not necessarily require a proper subset. In other words, a first subset of a first set can be of the same scope (equal) as the first set.

[0240] In a diagram, the direction of an arrow (as indicated by the arrow) generally indicates the flow of information (such as data or instructions) of interest to the diagram. For example, when component A and component B exchange various information, but the information passed from component A to component B is relevant to the diagram, the arrow may point from component A to component B. This unidirectional arrow does not mean that no other information is transmitted from component B to component A. Further, for information sent from component A to component B, component B may send a request or receipt confirmation of the information to component A.

[0241] In this application, including the definitions below, the term "controller" may be replaced by the term "circuit". The term "controller' may refer to, belong to or include: an application-specific integrated circuit (ASIC); a digital, analog or analog / digital hybrid discrete circuit; a digital, analog or analog / digital hybrid integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated or group) that executes code; a memory circuit (shared, dedicated or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip.

[0242] The controller may include one or more interface circuits. In some examples, the interface circuit may implement a wired or wireless interface connected to a local area network (LAN) or a wireless personal area network (WPAN). Examples of LANs include Institute of Electrical and Electronics Engineers (IEEE) standard 802.11-2016 (also known as the WIFI wireless network standard) and IEEE standard 802.3-2015 (also known as the ETHERNET wired network standard). Examples of WPANs include the Bluetooth Special Interest Group's BLUETOOTH wireless network standard and IEEE standard 802.15.4.

[0243] The controller may communicate with other controllers using interface circuits. Although the controller may be described in this disclosure as logically communicating directly with other controllers, in various configurations the controller may actually communicate through a communication system. The communication system includes physical and / or virtual network devices, such as hubs, switches, routers, and gateways. In some configurations, the communication system is connected to or traverses a wide area network (WAN) such as the Internet. For example, the communication system may include multiple LANs connected to each other via the Internet or point-to-point dedicated lines using technologies including multi-protocol label switching (MPLS) and virtual private networks (VPNs).

[0244] In various configurations, the functionality of a controller can be distributed among multiple controllers connected by a communication system. For example, multiple controllers can implement the same functionality distributed by a load balancing system. In yet another example, the functionality of a controller can be divided between a server (also known as a remote or cloud) controller and a client (or user) controller.

[0245] Some or all of the hardware features of the controller may be defined using a hardware description language, such as IEEE Standard 1364-2005 (commonly referred to as "Verilog") and IEEE Standard 10182-2008 (commonly referred to as "VHDL"). Hardware description languages ​​may be used to manufacture and / or program hardware circuits. In some configurations, some or all of the features of the controller may be defined by a language, such as IEEE 1666-2005 (commonly referred to as "System C"), which contains both code (described below) and hardware descriptions.

[0246] As used above, the term "code" may include software, firmware, and or microcode, and may refer to a program, routine, function, class, data structure, and / or object. The term shared processor circuit includes a single processor circuit that executes some or all of the code from multiple controllers. The term "group processor circuit" includes such a processor circuit that is combined with an additional processor circuit to execute some or all of the code from one or more controllers. References to multiple processor circuits include multiple processor circuits on discrete chips, multiple processor circuits on a single chip, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term "shared memory circuit" includes a single memory circuit that stores some or all of the code from multiple controllers. The term "group memory circuit" includes such a memory circuit that is combined with an additional memory to store some or all of the code from one or more controllers.

[0247] The term memory circuit is a collection of the term "computer-readable medium". As used herein, the term "computer-readable medium" does not include transient electrical or electromagnetic signals propagated through a medium (such as a carrier wave); the term "computer-readable medium" may therefore be considered to be tangible and non-transitory. Non-limiting examples of non-transitory computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital tapes or hard drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0248] The apparatus and methods described in this application may be implemented in part or in whole by a special-purpose computer created by configuring a general-purpose computer to perform one or more specific functions embodied in a computer program. The functional blocks and flow chart elements described above are software specifications that can be converted into computer programs by the routine work of skilled technicians or programmers.

[0249] The computer program includes processor executable instructions stored on at least one non-transitory computer readable medium. The computer program may also include or depend on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0250] A computer program may include: (i) descriptive text that needs to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation), (ii) assembly code, (iii) target code generated from source code by a compiler, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. By way of example only, the source code may be written using the syntax of a language including: C, C++, C#, ObjectiveC, Swift, Haskell, Go, SQL, R, Lisp, Fortran, Perl, Pascal, Curl, OCaml, HTML5 (Hypertext Markup Language 5th Revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Visual Lua, MATLAB, SENSORLINK, and

Claims

1. A surgical system for operating on a bone of a patient, the surgical system comprising: a reference device comprising one or more radiopaque fiducials, the reference device configured to have a fixed pose relative to a surgical implant or a portion of a patient's anatomy, the reference device including an optically detectable fiducial and a radiopaque fiducial; A surgical instrument for coupling to an end effector, the surgical instrument comprising: a handpiece for driving a surgical end effector; a camera configured to generate a first signal corresponding to a pose of the optically detectable fiducial relative to a first coordinate system; a sensor module configured to have a fixed pose relative to the camera and configured to generate a second signal associated with orientation data and / or motion parameters relative to a second coordinate system; and A control system configured to: receiving (i) a first signal, (ii) a second signal, and (iii) an image of a reference device and a bone of a patient; receiving an input signal associated with a starting position of the end effector; In response to the input signal, establishing registration between the first coordinate system, the second coordinate system, and the image coordinate system based on the first signal, the second signal, and the image; determining a starting position of the end effector relative to the bone in the image in one of the first coordinate system, the second coordinate system, and the image coordinate system based on one of i) the first signal and ii) the second signal and the registration; determining a second position of the end effector relative to the bone in the image in one of the first coordinate system, the second coordinate system, and the image coordinate system based on the second signal and the starting position; and A virtual representation is superimposed on the image based on the starting position and the second position of the end effector.

2. The surgical system of claim 1, wherein the sensor module is an inertial measurement unit.

3. The surgical system of any one of claims 1-2, wherein the inertial measurement unit comprises at least one of a gyroscope, an accelerometer, and a magnetometer.

4. The surgical system of any one of claims 1-3, wherein the inertial measurement unit is configured for measuring in six degrees of freedom.

5. The surgical system of any one of claims 1-4, wherein the camera is further defined as a plurality of cameras.

6. The surgical system of any one of claims 1-5 wherein: The image includes a shadow of a radiopaque fiducial; and The control system is configured to register a pose of the image coordinate system based on a pose of the shadow of the radiopaque fiducial.

7. A surgical system according to any one of claims 1-6, wherein the control system is configured to use an optically detectable fiducial to determine a rotation and translation of the camera relative to an axis of a reference device to determine a camera-reference device transformation matrix.

8. A surgical system according to claim 7, wherein the control system is configured to determine a starting position of the virtual representation based on a camera-reference device transformation matrix and a known relationship between the optically detectable reference and the reference device, and optionally based on a known relationship between the tool axis, the camera, and the length of the end effector.

9. The surgical system of claim 8, wherein the control system is configured to superimpose the representation based on a camera-reference device transformation matrix and a known relationship between the optically detectable fiducial and the reference device and the end effector length.

10. A surgical system according to claim 9, wherein the control system is configured to determine the virtual representation based on a known IMU-camera transformation matrix, the second signal, and a camera-reference device transformation matrix and a known relationship between the optically detectable fiducial and the reference device.

11. The surgical system of any one of claims 1-10, wherein: The input signal is further defined as a user input signal, The surgical instrument comprises a control element, The control element is actuatable to generate a user input signal, and The control system is configured to determine the starting position based on a posture of the surgical instrument relative to a reference device upon receiving a user input signal.

12. The surgical system of any one of claims 1-11, wherein: The end effector is a drill bit or a screwdriver head; The surgical instrument includes a depth sensor configured to provide a third signal associated with displacement of a drill or driver bit during a drilling or driving procedure; and The control system is configured to receive a third signal and the virtual representation is further based on the third signal.

13. The surgical system of claim 12, wherein the input signal is based on a third signal such that the control system is configured to determine a home position based on a posture of the surgical instrument when the third signal is received from the depth sensor.

14. The surgical system of claim 13, wherein the control system is configured to identify the starting position when the depth sensor indicates that the drill or screwdriver head is within the opening of the plate or within the bone.

15. The surgical system of claim 14, wherein the drill or driver head is coupled to the surgical instrument in a fixed orientation.

16. The surgical system of any one of claims 1-15, further comprising an accessory configured for attachment to a surgical instrument, the accessory comprising the camera and the sensor module.

17. The surgical system of claim 13, further comprising an accessory configured for attachment to a surgical instrument, the accessory comprising a depth sensor configured to provide a third signal associated with displacement of the end effector during a drilling or driving procedure.

18. The surgical system of claim 17, wherein the control system is configured to control the virtual representation based on a third signal.

19. The surgical system of claim 18, wherein the control system is configured to determine a depth endpoint of the virtual representation based on the third signal.

20. The surgical system of claim 19, wherein the attachment comprises: housing, and a measurement sleeve configured to circumferentially surround the drill bit or the screwdriver bit when one of the drill bit or the screwdriver bit is coupled to the surgical instrument, and the measurement sleeve is slidably mounted to the housing so as to extend forwardly or rearwardly relative to the housing between a fully distal position and a proximal position, the measurement sleeve having a distal end adapted for placement against an object or bone; and The depth sensor is a displacement sensor for generating a third signal based on a position of the distal end of the measuring sleeve relative to the housing.

21. The surgical system of claim 20, wherein the surgical instrument comprises a sensor for generating an identification signal in response to an identification feature of a cutting tip when the cutting tip is coupled to the surgical instrument.

22. The surgical system of claim 21, wherein the accessory comprises a sensor for generating an identification signal.

23. The surgical system of claim 21, wherein the control system is configured to determine a cutting tip length based on the identification signal.

24. The surgical system of claim 21, wherein the control system is configured to determine a trajectory endpoint based on a cutting tip length and a third signal.

25. The surgical system of claim 24, wherein the control system is configured to determine a depth endpoint based on a third signal.

26. The surgical system of claim 25, wherein the virtual representation is defined as a first virtual representation that terminates at a depth endpoint, and the control system is configured to display a second virtual representation that terminates at a trajectory endpoint.

27. The surgical system of claim 26, wherein the trajectory endpoint is based on measuring a limited range of motion of the cannula and the depth endpoint is based on a third signal.

28. The surgical system of any one of claims 24-27, wherein the accessory comprises an antenna for communicating with a control system.

29. The surgical system of any one of claims 1-28, further comprising a display unit, the control system being in communication with the display unit.

30. The surgical system of any one of claims 1-29, wherein the control system is in communication with an imaging system.

31. The surgical system of any one of claims 1-30, further comprising a surgical implant, wherein the surgical implant is a bone plate comprising one or more holes.

32. The surgical system of claim 31 , further comprising a fixation member, wherein: The reference device comprises a coupling portion; and A first portion of the fixation member is configured to attach a surgical implant to a bone, and another portion of the fixation member is configured to couple to a coupling portion of a reference device.

33. The surgical system of any one of claims 1-32, further comprising a device configured for attachment to an imaging system, and Wherein the control system is configured to determine a scale factor for the image based on at least one of the device and a radiopaque fiducial.

34. The surgical system of claim 33, wherein the device is a Fluorodisc comprising a transparent lens, a plurality of fiducial markers disposed on the transparent lens, and an attachment member configured to attach the Fluorodisc to an imaging system.

35. The surgical system of claim 33, further comprising an imaging system which is a C-arm capable of performing 2-DX-ray alone.

36. The surgical system of any one of claims 1-35, wherein the optically detectable fiducial is an AprilTag or an ArUco Tag.

37. A surgical system according to any one of claims 1-36, wherein the optically detectable fiducial is configured to allow calculation of a pose of a reference device relative to a camera.

38. The surgical system of claim 37, wherein the optically detectable fiducial is positioned on an outer surface of a reference device.

39. The surgical system of any one of claims 1-38, wherein the radiopaque fiducial is defined as a plurality of radiopaque elements.

40. The surgical system of claim 25, wherein the control system is configured to determine a second starting position and to display a third virtual representation based on the second starting position.

41. The surgical system of claim 38, further comprising a second optically detectable fiducial positioned on an outer surface of the reference device.

42. The surgical system of claim 40, wherein the control system is further configured to control the display to display drilling depth or screw length based on the third signal.

43. The surgical system of any one of claims 1-42, wherein: The end effector is coupled to the surgical instrument such that at least a portion of the end effector is within the field of view of the camera; and The control system is further configured to determine the identity and / or length of the end effector based on the portion of the end effector being within the field of view of the camera.

44. The surgical system of claim 43, wherein the control system is configured to use feature matching to determine the identity and / or length of the end effector.

45. A surgical system for operating on a bone of a patient, the surgical system comprising: A surgical instrument for coupling to an end effector, the surgical instrument comprising: a handpiece for driving the end effector; a camera configured to generate a first signal corresponding to a pose of the optically detectable fiducial relative to a first coordinate system; a sensor module configured to have a fixed pose relative to the camera and configured to generate a second signal relating to orientation data and / or motion parameters relative to a second coordinate system; and A control system configured to: establishing registration between the first coordinate system, the second coordinate system, and the image coordinate system based on the first signal, the second signal, and an image of a reference device including a radiopaque fiducial and an optically detectable fiducial; determining a starting position of the end effector relative to the bone in the image in one of the first coordinate system, the second coordinate system, and the image coordinate system based on one of i) the first signal and ii) the second signal and the registration; determining a second position of the end effector relative to the bone in the image in one of the first coordinate system, the second coordinate system, and the image coordinate system based on the second signal and the starting position; and Based on the starting position and the second position of the end effector, a virtual representation is superimposed on the image.

46. ​​A surgical system for operating on a bone of a patient, the surgical system comprising: A surgical instrument for coupling to an end effector, the surgical instrument comprising: a handpiece for driving the end effector; camera; a sensor module configured to have a fixed pose relative to the camera and configured to generate a first signal related to orientation data and / or motion parameters; and a depth sensor configured to provide a second signal associated with the displacement of the end effector during the drilling, tapping, or driving process; A control system configured to: determining a starting position of the end effector relative to the bone in the X-ray image based on the second signal; determining a second position of the end effector relative to the bone in the image based on the second signal and the starting position; and A virtual representation is superimposed on the image based on the starting position, the second position of the end effector, and the second signal.

47. A surgical system for operating on a bone of a patient, the surgical system comprising: a surgical instrument for coupling to an end effector, the surgical instrument comprising a sensor module configured to generate a first signal related to an orientation of the surgical instrument, the instrument having a tool axis, wherein the sensor module comprises at least one of an inertial measurement unit and a camera; and monitor; A control system configured to: obtaining a plurality of 2-DX-ray images, each of the 2-DX-ray images having an image reference axis; registering multiple 2-DX-ray images to a known coordinate system; Registering instruments to a known coordinate system; selecting a 2-DX-ray image from the plurality of 2-DX-ray images based on the image reference axis and the tool axis; as well as A virtual representation of the end effector is superimposed on the selected 2-DX-ray image on the display, the virtual representation being based on the first signal.

48. The surgical system of claim 47, further comprising a mobile computing device comprising the display and the first processor, wherein the surgical instrument comprises a second processor, and wherein the control system comprises the first processor and / or the second processor.

49. The surgical system of any one of claims 47-48, wherein the known coordinate system is a reference coordinate system.

50. The surgical system of any one of claims 47-49, wherein the virtual representation is selected from the group consisting of a line and a two-dimensional shape having a longitudinal axis, wherein the virtual representation is positioned such that the line or the longitudinal axis is aligned with the tool axis.

51. A surgical system according to any one of claims 47-50, wherein the control system is configured to determine a relationship between an image reference axis and a tool axis for each of the plurality of 2-DX-ray images, and the control system is configured to select a 2-DX-ray image based on the relationship.

52. The surgical system of claim 51, wherein the relationship is an angle between an image reference axis and a tool axis.

53. A surgical system according to claim 51 wherein the plurality of 2-D X-ray images are further defined as a plurality of X-ray images having a first orientation, wherein the control system is configured to select the X-ray image having the first orientation characterized by having a minimum angle between an image reference axis and a tool axis.

54. The surgical system of claim 53, wherein the first orientation is an AP orientation relative to an axis of a reference device attached to the patient.

55. A surgical system according to any one of claims 47-54, wherein the plurality of 2-D X-ray images are further defined as a plurality of X-ray images having a second orientation, wherein the control system is configured to select the X-ray image of the second orientation characterized by having the most orthogonal angle between the image reference axis and the tool axis.

56. The surgical system of claim 55, wherein the second orientation is an in-out orientation relative to an axis of a reference device attached to the patient.

57. The surgical system of any one of claims 47-56, wherein the sensor module comprises at least one of a camera and an inertial measurement unit.

58. The surgical system of any one of claims 47-57, wherein the sensor module comprises a camera and an inertial measurement unit.

59. The surgical system of claim 58, wherein the inertial measurement unit comprises at least one of a gyroscope, an accelerometer, and a magnetometer.

60. The surgical system of claim 58, further comprising a reference device configured to have a fixed pose relative to the surgical implant or a portion of the patient's anatomy, the reference device comprising an optically detectable fiducial and a radiopaque fiducial.

61. The surgical system of claim 60, wherein the camera is configured to generate a first signal corresponding to a posture of an optically detectable reference relative to a first coordinate system; wherein the inertial measurement unit is configured to generate a second signal related to orientation data relative to a second coordinate system, and wherein the control system is configured to receive an input signal related to a starting position of the end effector, and the control system is configured to establish alignment between the image coordinate system, the first coordinate system, and the second coordinate system based on the first signal, the second signal, and the starting position.

62. The surgical system of claim 61 , wherein each 2-D X-ray image comprises a shadow of a radiopaque fiducial; and the control system is configured to determine a pose of the image coordinate system based on a pose of the shadow of the radiopaque fiducial.

63. The surgical system of claim 61 , wherein the first coordinate system is implemented as an optical reference system, and the control system is configured to determine a pose of the optical coordinate system relative to a reference reference axis of a reference device using an optically detectable reference to determine a camera-reference device transformation matrix.

64. The surgical system of claim 63, wherein the control system is configured to determine a starting position of the virtual representation based on a camera-reference device transformation matrix and, optionally, based on a known relationship between the tool axis and the camera and an end effector length.

65. The surgical system of claim 64, wherein the control system is configured to superimpose the virtual representation based on a camera-reference device transformation matrix and a known relationship between the optically detectable fiducial and the radiopaque fiducial and an end effector length.

66. A surgical system according to claim 65, wherein the control system is configured to determine the pseudo-representation based on a known IMU-camera transformation matrix, the second signal, and a camera-reference device transformation matrix and a known relationship between the optically detectable reference and the radiopaque reference.

67. A surgical system according to claim 61, wherein the input signal is further defined as a user input signal, wherein the surgical instrument includes a control element that is actuatable to generate the user input signal, and wherein the control system is configured to determine the starting position based on the posture of the surgical instrument relative to a reference device when the user input signal is received.

68. The surgical system of any one of claims 61-67, wherein: The surgical instrument includes a depth sensor configured to provide a third signal associated with displacement of the end effector during a drilling procedure; and The control system is configured to receive a third signal and the virtual representation is further based on the third signal.

69. The surgical system of claim 68, wherein the input signal is based on a third signal such that the control system is configured to determine a starting position based on a posture of the surgical instrument when the third signal is received from the depth sensor.

70. The surgical system of claim 69, wherein the control system is configured to identify a starting position when a depth sensor indicates that the end effector is within an opening in a plate or within a bone.

71. The surgical system of any one of claims 69 and 70, wherein the end effector is coupled to the surgical instrument in a fixed orientation.

72. The surgical system of any one of claims 55-71, further comprising an accessory configured for attachment to a surgical instrument, the accessory comprising the sensor module.

73. The surgical system of claim 72, further comprising an accessory configured for attachment to the surgical instrument, the accessory comprising a depth sensor configured to provide a third signal associated with the displacement of the end effector during the drilling process.

74. The surgical system of claim 73, wherein the control system is configured to control the virtual representation based on a third signal.

75. The surgical system of claim 74, wherein the control system is configured to determine a depth endpoint of the virtual representation based on the third signal.

76. The surgical system of claim 75, wherein: The attachments include: housing, and a measurement sleeve configured to circumferentially surround the end effector when the end effector is coupled to the surgical instrument, the measurement sleeve being slidably mounted to the housing so as to extend forwardly or rearwardly relative to the housing between a fully distal position and a proximal position, the measurement sleeve having a distal end adapted for placement against an object or bone; and The depth sensor is a displacement sensor for generating a third signal based on a position of the distal end of the measuring sleeve relative to the housing.

77. The surgical system of claim 76, wherein the surgical instrument comprises a sensor for generating an identification signal in response to an identification feature of the end effector when the end effector is coupled to the surgical instrument.

78. The surgical system of claim 77, wherein the accessory comprises a sensor for generating an identification signal.

79. The surgical system of claim 78, wherein the control system is configured to determine an end effector length based on the identification signal.

80. The surgical system of claim 79, wherein the control system is configured to determine a trajectory endpoint based on an end effector length and a third signal.

81. The surgical system of claim 80, wherein the control system is configured to determine a depth endpoint based on a third signal.

82. The surgical system of claim 81 wherein the virtual representation is limited to a first virtual representation that terminates at a depth endpoint, and the control system is configured to display a second virtual representation that terminates at a trajectory endpoint.

83. The surgical system of claim 82, wherein the trajectory endpoint is based on measuring a limited range of motion of the cannula and the depth endpoint is based on a third signal.

84. The surgical system of any one of claims 73-83, wherein the accessory comprises an antenna for communicating with a control system.

85. The surgical system of claim 84, further comprising a display unit, the control system being in communication with the display unit.

86. The surgical system of claim 85, wherein the control system is in communication with an imaging system.

87. The surgical system of any one of claims 47-86, further comprising an imaging system, the imaging system being a C-arm capable of performing 2-DX-ray alone.

88. The surgical system of any one of claims 60-86, wherein the optically detectable reference is an AprilTag or an ArUco Tag.

89. The surgical system of any one of claims 60-86, wherein the optically detectable fiducial is configured to allow calculation of a pose of a reference device relative to a camera.

90. The surgical system of any one of claims 47-89, wherein the control system is configured to determine a second starting position and to display a third virtual representation based on the second starting position.

91. The surgical system of any one of claims 61-90, further comprising a second optically detectable fiducial positioned on an outer surface of the reference device.

92. The surgical system of any one of claims 68-91, wherein the control system is further configured to control the display to display drilling depth or screw length based on the third signal.

93. The surgical system of any one of claims 47-92, wherein: The end effector is coupled to the surgical instrument such that at least a portion of the end effector is within the field of view of the camera; and The control system is further configured to determine the identity and / or length of the end effector based on the portion of the end effector being within the field of view of the camera.

94. The surgical system of claim 93, wherein the control system is configured to use feature matching to determine the identity and / or length of the end effector.

95. A surgical system for operating on a bone of a patient, the surgical system comprising: A surgical instrument for coupling to an end effector, the surgical instrument comprising: a handpiece for driving a surgical end effector; a sensor module configured to generate a first signal related to an orientation of a surgical instrument, wherein the sensor module includes at least one of an inertial measurement unit and a camera, the instrument defining a tool axis; monitor; A control system configured to: obtaining a plurality of 2-DX-ray images, each of the 2-DX-ray images having an image boundary; Register multiple 2-DX-ray images to a known reference frame; Register the instrument to a known reference frame; Based on the first signal, determining a starting position of the planned trajectory in a known coordinate system; selecting a 2-DX-ray image from the plurality of 2-DX-ray images based on an image boundary and a starting position of each of the plurality of 2-DX-ray images; and A virtual representation of the end effector is superimposed on the selected 2-DX-ray image on the display, the virtual representation being based on the first signal.

96. The surgical system of claim 95 wherein the control system is configured to determine a planned end position of the end effector in a known coordinate system, and wherein the control system is further configured to select the 2-DX-ray image from the plurality of 2-DX-ray images based on an image boundary, a planned end position, and a start position of each of the plurality of 2-DX-ray images.

97. The surgical system of claim 96, wherein the planned termination position is a trajectory endpoint and the trajectory endpoint is based on one of a stored drill length value or a remaining cannula stroke length.

98. The surgical system of any one of claims 96-97, wherein the control system is configured to determine a depth endpoint of the end effector in a known coordinate system.

99. The surgical system of any one of claims 96-98, wherein the control system is configured to: Determine the trajectory segment extending between the trajectory end point and the starting position and The 2-D X-ray image is selected from the plurality of 2-D X-ray images based on the trajectory segment and an image boundary of each of the plurality of 2-D X-ray images.

100. The surgical system of claim 99, wherein the control system is configured to determine a trajectory segment extending between a trajectory end point and a starting position, determine a relationship between the trajectory segment and an image boundary of each of the plurality of 2-DX-ray images, and select the 2-DX-ray image from the plurality of 2-DX-ray images based on the relationship.

101. The surgical system of claim 100, wherein the relationship is the extent to which the trajectory segment is within an image boundary of each of the plurality of 2-DX-ray images.

102. The surgical system of any one of claims 95-101, wherein the system is configured to align the instrument to a known coordinate system based on an input signal related to a starting position of the end effector.

103. The surgical system of any one of claims 95-102, wherein the instrument is configured to determine a starting position based on the first signal and based on an alignment of the instrument with a known coordinate system.

104. A surgical system according to any one of claims 95-103, wherein each of the plurality of 2-D X-ray images comprises a respective image coordinate system, and wherein the first signal is measured relative to a sensor coordinate system, and wherein the system is configured to establish an alignment between the sensor coordinate system and a known coordinate system and between the image coordinate system and the known coordinate system in response to the input signal.

105. The surgical system of any one of claims 95-104, further comprising a mobile computing device comprising the display and the first processor, wherein the surgical instrument comprises a second processor, wherein the control system comprises the first processor and / or the second processor.

106. The surgical system of any one of claims 95-105, wherein the known coordinate system is a reference coordinate system.

107. A surgical system according to any one of claims 95-106, wherein the virtual representation is selected from the group consisting of a line and a two-dimensional shape having a longitudinal axis, wherein the virtual representation is positioned so that the line or the longitudinal axis is aligned with the tool axis.

108. A surgical system for operating on a bone of a patient, the surgical system comprising: A surgical instrument for coupling to an end effector, the surgical instrument comprising: a handpiece for driving a surgical end effector; a sensor module configured to generate a first signal related to an orientation of a surgical instrument, wherein the sensor module includes at least one of an inertial measurement unit and a camera, the instrument defining a tool axis; and monitor; A control system configured to: obtaining a plurality of 2-DX-ray images, each of the 2-DX-ray images having an image boundary and an image coordinate system including an image reference plane and an image reference axis; Registering the instrument to the image coordinate system; determining a starting position of a planned trajectory in an image reference plane of each of the plurality of 2-DX-ray images based on the first signal; determining a planned end position of the planned trajectory in an image reference plane of each of the plurality of 2-DX-ray images based on the start position; determining a radial distance between a starting position and a planned ending position; selecting a 2-D X-ray image from the plurality of 2-D X-ray images based on the radial distance; and A virtual representation of the end effector is superimposed on the selected 2-DX-ray image on the display, the virtual representation being based on the first signal.

109. The surgical system of claim 108, wherein the plurality of 2-D X-ray images are further defined as a plurality of X-ray images having a first orientation, wherein the control system is configured to select the X-ray images having the first orientation characterized by having a minimum radial distance.

110. The surgical system of claim 109, wherein the first orientation is an AP orientation relative to an axis of a reference device attached to the patient.

111. The surgical system of claim 109, wherein the planned end position is based on one of a stored drill length value or a remaining cannula stroke length.

112. The surgical system of any one of claims 108-111, further comprising determining a planned end position within an image bounds based on a starting position, a planned end position, and an image boundary of each of the plurality of 2-DX-ray images.

113. The surgical system of any one of claims 108-112 further comprising determining a percentage of the radial distance that is within an image boundary based on a spatial relationship between a planned end position, a start position, and an image boundary.

114. The surgical system of any one of claims 108-113, wherein the plurality of 2-D X-ray images are further defined as a plurality of X-ray images having a first orientation, wherein: The control system is configured to select an X-ray image of the plurality of X-ray images having a first orientation and characterized by a maximum percentage of radial distances within image boundaries.

115. The surgical system of claim 114, wherein the first orientation is an AP orientation relative to an axis of a reference device attached to the patient.

116. A surgical system according to claim 114, wherein the plurality of 2-D X-ray images are further defined as a plurality of X-ray images having a second orientation, wherein the control system is configured to select an X-ray image of the plurality of X-ray images having the second orientation and characterized by a maximum percentage of radial distance within the image boundary.

117. The surgical system of claim 116, wherein the second orientation is an in-out orientation relative to an axis of a reference device.

118. A surgical system according to any one of claims 113-117, wherein the plurality of 2-DX ray images are further defined as a plurality of X-ray images having a second orientation, wherein the control system is configured to select the X-ray image having the second orientation based on a difference between an in-image radial distance and a radial distance and based on a radial distance difference threshold.

119. The surgical system of claim 118, wherein the second orientation is an inside-out orientation relative to a line of reference device attached to the patient.

120. The surgical system of any one of claims 108-119, further comprising a mobile computing device comprising the display and the first processor, wherein the surgical instrument comprises a second processor, wherein the control system comprises the first processor and / or the second processor.

121. The surgical system of any one of claims 108-120, wherein the display is fixed relative to at least one of the surgical instrument and the patient.

122. A surgical system according to any one of claims 108-121, wherein the virtual representation is selected from the group consisting of a line and a two-dimensional shape having a longitudinal axis, wherein the virtual representation is positioned so that the line or the longitudinal axis is aligned with the tool axis.

123. A surgical system for operating on a bone of a patient, the surgical system comprising: A surgical instrument for coupling to an end effector, the surgical instrument comprising: a handpiece for driving a surgical end effector; a sensor module configured to generate a first signal related to an orientation of a surgical instrument, wherein the sensor module includes at least one of an inertial measurement unit and a camera, the instrument having a tool axis; and monitor; A control system configured to: Registering the X-ray image to a known coordinate system; Registering instruments to a known coordinate system; determining a starting position of the planned trajectory in a known coordinate system based on the first signal; Determine a planned end position of the end effector in a known coordinate system based on the start position; selecting a view of the registered X-ray image based on the starting position, the planned ending position, and the first signal; and A virtual representation of the end effector is superimposed on the x-ray image on the display, the virtual representation being based on the first signal.

124. The surgical system of claim 123, further comprising a mobile computing device comprising the display and the first processor, wherein the surgical instrument comprises a second processor, and wherein the control system comprises the first processor and / or the second processor.

125. The surgical system of any one of claims 123-124, wherein the known reference system is a reference coordinate system.

126. A surgical system according to any one of claims 123-125, wherein the virtual representation is selected from the group consisting of a line and a two-dimensional shape having a longitudinal axis, wherein the virtual representation is positioned so that the line or the longitudinal axis is aligned with the tool axis.

127. The surgical system of any one of claims 123-126, wherein the planned end position is based on one of a stored drill length value or a remaining cannula stroke length.

128. A surgical system for operating on a bone of a patient, the surgical system comprising: A surgical instrument for coupling to an end effector, the surgical instrument comprising: a handpiece for driving a surgical end effector; a sensor module configured to generate a first signal related to an orientation of a surgical instrument, the sensor module including at least one of an inertial measurement unit and a camera to generate the first signal, the instrument having a tool axis, and the sensor module including a depth sensor configured to provide a second signal associated with a displacement of an end effector during a drilling process; and monitor; A control system configured to: Registering the X-ray image to a known coordinate system; Registering instruments to a known coordinate system; selecting a view of the registered X-ray image based on the second signal; and A virtual representation of the end effector is superimposed on the x-ray image on the display, the virtual representation being based on the first signal.

129. The surgical system of claim 128, wherein the controller is configured to select a view of the registered X-ray image based on the second signal and a threshold distance.

130. The surgical system of any one of claims 128-129, wherein the control system is configured to rotate between a first view and a second view, the second view providing a more zoomed-in view of the bone.

131. The surgical system of any one of claims 128-130, further comprising a mobile computing device comprising the display and the first processor, wherein the surgical instrument comprises a second processor, wherein the control system comprises the first processor and / or the second processor.

132. The surgical system of any one of claims 128-131, wherein the known coordinate system is a reference coordinate system.

133. A surgical system according to any one of claims 128-132, wherein the virtual representation is a virtual object selected from the group consisting of a line and a two-dimensional shape having a longitudinal axis, wherein the virtual representation is positioned so that the line or the longitudinal axis is aligned with the tool axis.

134. The surgical system of claim 129, wherein the surgical instrument comprises a sensor for generating an identification signal in response to an identification feature of the end effector when the end effector is coupled to the surgical instrument.

135. The surgical system of claim 134, wherein the control system is configured to determine a length of the end effector based on the identification signal.

136. The surgical system of claim 135, wherein the control system is configured to determine a depth endpoint based on the end effector length and the second signal.

137. A surgical system according to claim 136, wherein the virtual representation is limited to a first virtual representation that terminates at a depth endpoint, and the control system is configured to display a second virtual representation that terminates at a trajectory endpoint.

138. The surgical system of claim 137, wherein the trajectory endpoint is based on measuring a limited range of motion of the cannula and the depth endpoint is based on a third signal.

139. The surgical system of any one of claims 128-139, wherein: The end effector is coupled to the surgical instrument such that at least a portion of the end effector is within the field of view of the camera; and The control system is further configured to determine the identity and / or length of the end effector based on the portion of the end effector being within the field of view of the camera.

140. The surgical system of claim 139, wherein the control system is configured to use feature matching to determine the identity and / or length of the end effector. Additional protection clauses: I. A reference device for registering a tool axis to an X-ray image, the reference device comprising one or more radiopaque fiducials and an optically detectable fiducial, the reference device comprising an attachment element for coupling to a patient. II. A reference device according to clause I, wherein the optically detectable reference is an April Tag. IIII. A reference device according to clause I or II, wherein the one or more radiopaque fiducials include a plurality of radiopaque elements. IV. A reference device according to clause III, wherein the plurality of radiopaque elements comprises a plurality of radiopaque balls. V. A reference device according to any of clause IV, wherein the attachment element comprises a pin for coupling to a bone of the patient or an adhesive for coupling to a portion of the patient's anatomy. VI. A surgical accessory configured to be removably coupled to a surgical handpiece assembly, the surgical accessory comprising: a measurement housing comprising a proximal region and a distal region, wherein the proximal region comprises a proximal surface; Depth sensor; camera; an optional inertial measurement unit configured to generate signals related to orientation data and / or motion parameters; and A measurement coupling extends from a proximal surface of the proximal region of the measurement housing. VII. The surgical accessory according to clause VI further includes a depth sleeve that is movably connected to the measurement housing and is configured to move along the measurement axis relative to the measurement housing through the proximal and distal regions, the depth sleeve defining a hole configured to receive a drill bit, the depth sleeve being connected to the depth sensor. VIII. The surgical accessory of clause VI or VII further comprising an antenna for wireless communication. IX. A surgical accessory as described in any of clauses VI-VIII, comprising an inertial measurement unit, wherein the inertial measurement unit includes a gyroscope and an accelerometer. X. A surgical accessory according to any of clauses VI-X, further comprising a battery or terminals for receiving power from an external handpiece or from another component of the surgical system. XI. A surgical system for operating on a bone of a patient, the surgical system comprising: a reference device comprising one or more radiopaque fiducials, the reference device configured to have a fixed pose relative to a surgical implant or a portion of a patient's anatomy, the reference device including an optically detectable fiducial and a radiopaque fiducial; A surgical instrument for coupling to an end effector, the surgical instrument comprising: a camera configured to generate a first signal corresponding to a pose of the optically detectable fiducial; a sensor module configured to have a fixed pose relative to the camera and configured to generate a second signal related to the orientation data and / or the motion parameter; and Antennas for communicating with imagers, mobile computing devices, and / or displays. XII. A method of registering a tool axis to an x-ray image using a reference device comprising one or more radiopaque fiducials and optically detectable fiducials, the method comprising: securing the reference device to the patient's bone using an attachment element; After securing the reference device to a bone or tissue of a patient, imaging the reference device using an imager; positioning an apparatus having a camera and an inertial measurement unit such that an optically detectable fiducial is within a field of view of the camera; registering the inertial measurement unit to the camera based on an output signal of the camera; as well as A virtual representation of a portion of the instrument is displayed based on the output signal of the inertial measurement unit. XIII. A surgical accessory configured to be removably coupled to a surgical handpiece assembly or a component thereof, the external accessory comprising: Measure the housing; camera; an inertial measurement unit configured to generate signals related to orientation data and / or motion parameters; as well as A measurement coupler for attachment to a surgical handpiece, a surgical accessory, or a second surgical accessory. XIV. The surgical accessory of clause XIII further comprising an antenna for wireless communication. XV. A surgical accessory as described in clause XIII or XIV, wherein the inertial measurement unit includes a gyroscope and an accelerometer. XVI. A surgical accessory according to any of clauses XIII-XV, further comprising a battery or terminals for receiving power from the surgical handpiece or from another component of the surgical system. XVII. A surgical system for operating on a bone of a patient, the surgical system comprising: A surgical instrument for coupling to an end effector, the surgical instrument comprising: a handpiece for driving the surgical end effector; a camera coupled to the handpiece, the camera configured to generate a first signal; A control system is configured to determine an end effector length based on the first signal. XVIII. A surgical system according to clause XVII, wherein the surgical instrument further comprises a sensor module configured to have a fixed pose relative to the camera and configured to generate a second signal related to the orientation data and / or the motion parameter; and Antennas for communicating with imagers, mobile computing devices, and / or displays. XIX. The surgical system of any of clauses XVII-XVIII, wherein the control system is configured to determine an end effector length based on the first signal and the input signal. XX. The surgical system of clause XIX, wherein: The input signal is further defined as a user input signal, and the surgical instrument includes a control element, The control element is actuatable to generate a user input signal, and The control system is configured to determine an end effector length upon receiving a user input signal. XXI. A surgical system according to any of clauses XVII-XX, wherein the surgical instrument includes a depth sensor configured to provide a depth signal associated with the displacement of the end effector during the drilling process, and wherein, The control system is configured to determine the end effector length when the depth signal indicates that the depth exceeds a predetermined threshold. XXII. A surgical system according to any of clauses XVII-XXI, wherein the control system includes a processor and a memory programmed with an image processing algorithm, the image processing algorithm including a step configured to determine whether the connected end effector is a first length or a second length. XXII. A surgical system according to clause XXII, wherein the sensor module further comprises a measuring sleeve movably connected to the depth sensor, wherein the control system is configured to determine a distal end of the end effector and a distal end of the measuring sleeve, and wherein the control system is configured to determine a length of the end effector based on the position of the distal end of the end effector and the distal end of the measuring sleeve. XXIV. A first surgical accessory configured to be removably coupled to a surgical handpiece assembly or a component thereof, the first surgical accessory comprising: Measuring the shell; camera; an inertial measurement unit configured to generate signals related to orientation data and / or motion parameters; and Connector. XXV. A surgical system comprising the first surgical accessory of clause XXIV, further comprising an end effector selected from the group consisting of a surgical pin, a surgical wire, a reamer, a bone drill, or a saw blade. XXVI. The surgical system of clause XXV further comprises a surgical accessory comprising a housing having a first end portion for coupling to a surgical handpiece and a second end portion opposite the first end portion, the surgical accessory being configured to drive an end effector. XXVII. A surgical system according to clause XXVI, wherein the first surgical accessory is configured to be connected to a surgical accessory or a surgical handpiece. XXVIII. The surgical system of clause XXIV, further comprising a second surgical accessory, the second measurement comprising a depth sensor, wherein the second surgical accessory is configured to couple to the surgical handpiece and the first surgical accessory. XXIX. A surgical system for operating on a bone of a patient, the surgical system comprising: A surgical instrument for coupling to an end effector, the surgical instrument comprising: a handpiece for driving a surgical end effector; a sensor module configured to generate a first signal related to an orientation of the surgical instrument, the instrument having a tool axis; monitor; A control system configured to: registering multiple X-ray images to a known coordinate system; registering the instrument to the known coordinate system; selecting one of the plurality of registered X-ray images based on a capture time of each image and a current time; and The selected X-ray image is displayed. XXX. A surgical system for operating on a bone of a patient, the surgical system comprising: A surgical instrument for coupling to an end effector, the surgical instrument comprising: a handpiece for driving a surgical end effector; a sensor module configured to generate a first signal related to an orientation of a surgical instrument, the instrument having a tool axis; monitor; A control system configured to: registering multiple X-ray images to a known coordinate system; Registering instruments to a known coordinate system; selecting one of the plurality of registered X-ray images based on a duration between a capture time of each of the plurality of registered X-ray images and a current time; and The selected X-ray image is displayed. XXXI. A surgical system for operating on a bone of a patient, the surgical system comprising: A surgical instrument for coupling to an end effector, the surgical instrument comprising: a handpiece for driving a surgical end effector; a camera configured to generate a first signal corresponding to a pose of the optically detectable fiducial relative to a first coordinate system; a sensor module configured to have a fixed pose relative to the camera and configured to generate a second signal related to orientation data and / or motion parameters relative to a second coordinate system; and a control system configured to: receiving (i) the first signal, (ii) the second signal, and (iii) an image of a reference device and a bone of a patient; receiving an input signal related to a starting position of the end effector; In response to the input signal, establishing registration between the first coordinate system, the second coordinate system, and the image coordinate system based on the first signal, the second signal, and the image; determining a starting position of the end effector relative to the bone in the image in one of a first coordinate system, a second coordinate system, and an image coordinate system based on one of i) the first signal and ii) the second signal and the registration; determining a second position of the end effector relative to the bone in the image in one of the first coordinate system, the second coordinate system, and the image coordinate system based on the second signal and the starting position; and A virtual representation is superimposed on the image based on the starting position and the second position of the end effector. XXXII. A surgical system for operating on a bone of a patient, the surgical system comprising: A surgical instrument for coupling to an end effector, the surgical instrument comprising: a handpiece for driving a surgical end effector; a camera configured to generate a first signal corresponding to a posture of the patient relative to a first coordinate system; a sensor module configured to have a fixed pose relative to the camera and configured to generate a second signal related to orientation data and / or motion parameters relative to a second coordinate system; and A control system configured to: receiving (i) a first signal, (ii) a second signal, and (iii) an image of a reference device and a bone of a patient; receiving an input signal associated with a starting position of the end effector; In response to the input signal, establishing registration between the first coordinate system, the second coordinate system, and the image coordinate system based on the first signal, the second signal, and the image; determining a starting position of the end effector relative to the bone in the image in one of a first coordinate system, a second coordinate system, and an image coordinate system based on one of i) the first signal and ii) the second signal and the registration; determining, based on the second signal and the starting position, a second position of the end effector relative to the bone in the image in one of the first coordinate system, the second coordinate system, and the image coordinate system; and A virtual representation is superimposed on the image based on the starting position and the second position of the end effector.

Images