X-Wing Enhanced Guidance System for Distal Targeting

Abstract

A surgical instrument assembly configured to display a representation of a surgical instrument relative to a trajectory (e.g., a desired trajectory for drilling an implant) based image data from an image sensor (e.g., a camera). For example, the surgical instrument assembly may be configured to determine the desired trajectory (e.g., a central axis of a distal hole of an intramedullary nail) based on X-ray image data representing at least one fiducial marker (e.g., an array of ArUco markers) and representative of an anatomical structure and / or an implant. The surgical instrument assembly may display a representation of the orientation and / or position of the surgical instrument (e.g., the orientation and position of a drill bit of the surgical instrument) relative to the desired trajectory in real time based on image sensor data generated based on the orientation of the at least one fiducial marker detected by the image sensor.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 298,996, filed January 12, 2022, the disclosure of which is incorporated by reference herein as if set forth in its entirety.

[0002] (Joint research agreement) This disclosure has been made by or on behalf of the parties to the Joint Research Agreement listed below. The Joint Research Agreement was in effect on or before the date this disclosure was made, and this disclosure is part of the Joint Research Agreement and was made as a result of activities conducted within the scope of the Joint Research Agreement. The parties to the Joint Research Agreement are 1) Depuy Synthes Products, Inc. and 2) HCL Technologies Limited.

[0003] FIELD OF THEINVENTION The present invention relates to a system for targeting, and more particularly to a guidance system for distal targeting of an intramedullary nail, and related methods. [Background technology]

[0004] Various medical imaging devices, such as C-arm devices, can perform fluoroscopy, a type of medical imaging that displays continuous x-ray images on a monitor. During a fluoroscopy procedure, an x-ray source or transmitter emits x-rays that penetrate the patient's body. An x-ray detector or image intensifier converts the x-rays that pass through the body into a visible image that is displayed on the monitor of the medical imaging device. Such medical imaging devices can display high resolution x-ray images in real time, allowing the physician to monitor progress at any time during the procedure and therefore take appropriate action based on the displayed images.

[0005] However, monitoring the image is often difficult during certain procedures, such as those that require attention to the patient's anatomy as well as the display of the medical imaging device. For example, if the medical professional is required to operate a drill while looking at the display of the medical imaging device, it can be difficult to align the drill bit with the distal locking hole. In addition, using an x-ray emitting medical imaging device to monitor the procedure can increase patient exposure to x-ray radiation. Summary of the Invention [Means for solving the problem]

[0006] The present application provides a surgical instrument assembly configured to display a representation of a surgical instrument relative to a trajectory (e.g., a desired trajectory for drilling an implant) based image data from an image sensor (e.g., a camera). For example, the surgical instrument assembly may be configured to determine the desired trajectory (e.g., a central axis of a distal hole of an intramedullary nail) based on X-ray image data representing at least one fiducial marker (e.g., an array of ArUco markers) and representative of an anatomical structure and / or an implant. The surgical instrument assembly may display a representation of the orientation and / or position of the surgical instrument (e.g., the orientation and position of a drill bit of the surgical instrument) relative to the desired trajectory in real time based on image sensor data generated based on the orientation of the at least one fiducial marker detected by the image sensor.

[0007] For example, a single x-ray image of a target area of ​​a patient's internal anatomical structure and at least one fiducial marker external to the patient as detectable by an image sensor may be generated. A desired trajectory relative to the target area and the at least one fiducial marker may be determined based on x-ray image data representative of the single x-ray image. The image sensor (e.g., a camera) may be configured to detect an orientation of the at least one fiducial marker relative to the image sensor. A display of the surgical instrument assembly may be configured to display a representation of an orientation and / or position of a surgical instrument (e.g., a drill bit of a surgical instrument) relative to the desired trajectory in real time based on real-time image sensor data from the image sensor.

[0008] For example, the memory of the surgical instrument assembly may determine and store an orientation and position of the drill bit relative to the image sensor. The memory may be configured to instruct the processor of the surgical instrument assembly to determine and store a desired trajectory relative to the target area and at least one fiducial marker based on x-ray image data representative of a single x-ray image. The memory may be configured to instruct the processor to determine in real time an orientation and / or position of the drill bit relative to the desired trajectory based on the image sensor data and the stored orientation of the drill bit relative to the image sensor. The memory may be configured to instruct the processor to output a representation of the orientation and / or position of the drill bit relative to the desired trajectory to a display in real time.

[0009] A single x-ray image may be sufficient to generate x-ray image data. Thus, a second x-ray image may not be necessary before the procedure is completed. Thus, the surgical instrument assembly may provide reduced patient exposure to x-rays as compared to alternatives in which the patient is continuously exposed to x-rays (e.g., during a fluoroscopy procedure).

[0010] In some embodiments, a second x-ray image may be taken to verify the orientation and / or position of the surgical instrument. In such embodiments, a third x-ray image may not be necessary.

[0011] The trajectory may define an entry point and the surgical instrument may display a representation of the position of the tip of the drill bit relative to the entry point.

[0012] In one embodiment, the display is configured to display a representation of the drill bit tip and points along the shaft of the drill bit, each point may be represented by a circle such that the circles appear concentric when the drill bit is aligned with the desired trajectory on the navigation display.

[0013] The display may show a navigation target representing the center of a perfect circle, for example showing the X and Y coordinates of the drill bit tip relative to the X and Y coordinates of the navigation target. When the drill bit tip is located at the center of the target, the circle representing the drill bit tip may be centered on the navigation target.

[0014] The navigational graphics displayed by the display may provide a visual indicator to inform the user when the drill is acceptably oriented and / or positioned. For example, the center of the target may change when the drill bit tip is located within a predetermined distance from the center of the target. An outline on the display may change color when both the drill bit tip and drill bit axis alignment are within a predetermined range.

[0015] During actual drilling activities, the drill or camera may experience significant vibrations or deflections that cause noise in the navigation output. The memory may store instructions for the processor to reduce the effects of such noise. For example, the memory may store instructions for a smoothing algorithm, such as a moving average, of the drill bit position relative to the image sensor.

[0016] In some embodiments, the navigation graphic may switch from a detailed view (in which a representation of the drill tip and its axis is displayed) to a simple view in which an indication of whether the drill bit is aligned or not is displayed. The memory may store instructions for the processor to automatically switch between the detailed view and the simple view based on whether a predetermined amount of vibrational movement of the drilling and / or the camera is detected. Additionally, the memory may store a predetermined threshold for determining whether the drill bit is aligned or not, and such predetermined threshold may be based on expected noise due to drilling.

[0017] According to one embodiment of the present disclosure, a surgical instrument assembly includes a surgical instrument configured to act on an anatomical structure. An image sensor attached to the surgical instrument, the image sensor configured to detect an orientation of at least one fiducial marker relative to the image sensor, the image sensor configured to generate image sensor data based on the orientation of the at least one fiducial marker detected by the image sensor. The surgical instrument includes a processor and a memory. The memory is in communication with the processor and stores instructions that, when executed by the processor, cause the processor to 1) receive x-ray image data representative of the at least one fiducial marker and a representation of the anatomical structure and / or implant, 2) determine a trajectory for the at least one fiducial marker and the anatomical structure and / or implant extending toward the anatomical structure and / or implant based on the x-ray image data, and 3) determine an orientation of the surgical instrument relative to the trajectory based on the image sensor data. The surgical instrument assembly includes a display coupled to the processor. The display is configured to display a representation of the orientation of the surgical instrument relative to the trajectory.

[0018] According to another embodiment of the present disclosure, a method of displaying an orientation of a surgical instrument includes receiving x-ray image data representative of at least one fiducial marker and representative of an anatomical structure and / or an implant. The method includes determining a trajectory for the at least one fiducial marker extending towards the anatomical structure and / or implant based on the x-ray image data. The method includes generating, with an image sensor, image sensor data based on an orientation of the at least one fiducial marker relative to the image sensor. The method includes determining an orientation of the surgical instrument relative to the trajectory based on the image sensor data. The method includes displaying, with a display, a representation of the orientation of the surgical instrument relative to the trajectory. [Brief description of the drawings]

[0019] The above summary, as well as the following detailed description of exemplary embodiments of the present application's surgical instrument assembly, will be better understood when read in conjunction with the accompanying drawings. For the purposes of illustrating the present application's expandable intervertebral implant, there are shown in the drawings exemplary embodiments. It should be understood, however, that the present application is not limited to the precise arrangements and instrumentalities shown. Of the drawings: [Figure 1] FIG. 1 illustrates an exemplary imaging system according to an exemplary embodiment, the exemplary imaging system including an imaging device in electrical communication with a surgical instrument assembly. [Figure 2A] FIG. 2 is a perspective view of the surgical instrument assembly of FIG. 1 including a display attached to the surgical instrument. [Figure 2B] FIG. 2 is a front perspective view of the surgical instrument assembly of FIG. [Figure 2C] FIG. 2 is a rear view of the surgical instrument assembly of FIG. 1. [Figure 2D] FIG. 2 is a side elevational view of the surgical instrument assembly of FIG. 1. [Diagram 3] 2 is a block diagram of an exemplary computing device for use in the imaging system shown in FIG. 1. [Figure 4A]2A-2D depict representations of anatomical structures and implants that may be displayed by the surgical instrument assembly depicted in FIGS. 2A-2D, the representations including a target site. [Figure 4B] 1 depicts another representation of the anatomical structure and implant showing a representation of the orientation and position of a cutting instrument of a surgical instrument assembly relative to a target site of the anatomical structure. [Figure 4C] 1 depicts yet another representation of the anatomical structures and implants with the tip of the cutting instrument positioned over the target site. [Figure 4D] 1 depicts an x-ray image of an array of fiducial markers, anatomy, and an implant, showing the world coordinate system and the image sensor coordinate system. [Figure 4E] Another x-ray image of the array of fiducial markers, the anatomical structures, and the implant is depicted. [Figure 4F] 4F depicts the x-ray image of FIG. 4E in which brightness and contrast can be adjusted to detect fiducial markers. [Figure 4G] 4F depicts a cropped portion of the X-ray image of FIG. 4F. [Figure 4H] 4F depicts the x-ray image showing the world coordinate system and the desired trajectory. [Figure 4I] 4H depicts the x-ray image of FIG. 4H without the fiducial markers. [Figure 4J] Further x-ray images of the array of fiducial markers, the anatomy, and the implant viewed at an angle to the desired drilling trajectory are depicted. [Figure 4K] 4J depicts an x-ray image in which brightness and contrast can be adjusted to detect fiducial markers. [Figure 4L] 4A depicts a cropped portion of the X-ray image of FIG. 4K. [Figure 4M] 4K depicts the x-ray image showing the world coordinate system and the desired trajectory. [Figure 4N] 4D depicts the X-ray image of FIG. 4K without fiducial markers, showing a representation of a cylinder for determining the two-dimensional coordinates of the target. [Figure 4O]4K depicts an x-ray image without fiducial markers, showing the location of the desired target. [Figure 4P] Depict the ArUco marker. [Figure 4Q] Depicts ArUco marker with AprilTag. [Figure 4R] Another array of fiducial markers is depicted. [Figure 4S] Depict alternative fiducial markers. [Figure 4T] 4E depicts a perspective view of the implant of FIG. 4D having an array of fiducial markers and a stake fixed relative to the anatomical structure. [Figure 4U] 4E depicts a perspective view of an array of fiducial markers fixed relative to the anatomical structures and implants of FIG. 4D by being affixed to the skin of a patient. [Figure 4V] 4E depicts a perspective view of an array fiducial marker fixed relative to the anatomical structure and implant of FIG. 4D via a jig. [Figure 4W] 10 depicts a perspective view of a first portion of another array of fiducial markers, an anatomical structure, and an implant. [Figure 4X] 4D depicts a perspective view of a second portion of the array of fiducial markers, the anatomical structure, and the implant of FIG. 4W. [Figure 4Y] 4D depicts a perspective view of a third portion of the array of fiducial markers, the anatomical structure, and the implant of FIG. 4W. [Figure 4Z] 4D depicts a perspective view of a fourth portion of the array of fiducial markers, the anatomical structure, and the implant of FIG. 4W. [Figure 5A] 13 depicts an exemplary screenshot of a display of a surgical instrument assembly showing a visual indication of alignment of a cutting instrument relative to a desired drilling direction and in which the cutting instrument is not aligned relative to a first direction. [Figure 5B]13 depicts another exemplary screenshot of a display of a surgical instrument assembly showing a visual indication of alignment of the cutting instrument relative to the direction of travel of the x-ray and in which the cutting instrument is not aligned to the desired drilling trajectory relative to a second direction that is substantially perpendicular to the first direction. [Figure 5C] FIG. 13 depicts another exemplary screenshot of a display of a surgical instrument assembly showing a visual indication of the alignment of the cutting instrument relative to the direction of travel of the x-ray, where the cutting instrument is aligned with the desired drilling direction such that the cutting instrument and the desired drilling direction have the same orientation. [Figure 5D] 13 depicts an exemplary screenshot of a display of a surgical instrument assembly showing a visual indication of the circles and arrows of the alignment of the cutting instrument with respect to a desired drilling trajectory and where the cutting instrument is not aligned. [Figure 5E] 13 depicts an exemplary screenshot of a display of a surgical instrument assembly showing a bubble level visual indication of the alignment of the cutting instrument with respect to a desired drilling trajectory and in which the cutting instrument is not aligned. [Figure 5F] 13 depicts an exemplary screenshot of a display of a surgical instrument assembly showing a pair of circles to visually indicate alignment of the cutting instrument with respect to a desired drilling trajectory and where the cutting instrument is not aligned. [Figure 5G] 1 depicts an exemplary screenshot of a display of a surgical instrument assembly with the cutting instrument misaligned, showing a dot and a movable dot that visually indicates alignment of the cutting instrument with respect to a desired drilling trajectory; [Figure 6A] 2 illustrates the exemplary imaging system shown in FIG. 1 showing exemplary anatomical structures and an exemplary orientation of a surgical instrument assembly. [Figure 6B] 6B illustrates another exemplary orientation of the surgical instrument assembly in the imaging system shown in FIG. 6A. [Figure 7]13 is another exemplary screenshot of a display of a surgical instrument assembly, the screenshot including options for performing various procedures. [Figure 8] 1 depicts a flowchart of an exemplary algorithm that includes determining a desired trajectory orientation relative to a fiducial marker. [Figure 9] 1A-1C are cross-sectional views of exemplary anatomical structures with a cutting instrument advancing through the anatomical structure along a perforation direction. [Figure 10A] 13 is an exemplary screenshot of a display of a surgical instrument assembly showing a visual indication of the depth of the tip of a cutting instrument relative to a portion of an anatomical structure. [Figure 10B] 13 is an exemplary screenshot of a display of a surgical instrument assembly showing a visual indication of the depth of the tip of a cutting instrument relative to a portion of an anatomical structure. [Figure 11] 1 is an exemplary split screenshot of a display of a surgical instrument assembly showing simultaneously a visual indication of the alignment of the cutting instrument and a visual indication of the depth of the tip of the cutting instrument. [Figure 12] 13 is another exemplary screenshot of a display of a surgical instrument assembly showing simultaneously a visual indication of the alignment of the cutting instrument and a visual indication of the depth of the tip of the cutting instrument, and a visual indication of the depth of the tip of the cutting instrument and the cutting instrument on an x-ray image of an anatomical structure. [Figure 13] 2 is a perspective view of the surgical instrument assembly shown in FIG. 1 showing an exemplary representation of alignment of the cutting instrument with respect to a desired trajectory and showing an exemplary representation of the cutting instrument with respect to a patient's anatomy; [Figure 14A] FIG. 1 is an exemplary screenshot of a display of a surgical instrument assembly showing a representation of an anatomical structure from a first or anteroposterior (AP) view, the representation including a cutting instrument positioned to enter the anatomical structure for a particular intramedullary (IM) nailing procedure. [Figure 14B]FIG. 14B is an exemplary screenshot of a display of a surgical instrument assembly, the screenshot including the x-ray image of FIG. 14A with the AP boundary and AP representation of the desired trajectory for a particular IM nailing procedure overlaid on a representation of the anatomical structures. [Figure 15] 14C is an exemplary screenshot of a display of a surgical instrument assembly, the screenshot including the x-ray image of FIG. 14B, but with the position of the cutting instrument adjusted according to an AP representation of the desired trajectory. [Figure 16] FIG. 16 is an exemplary screenshot of a display of a surgical instrument assembly showing an x-ray image of the anatomical structures and cutting instrument shown in FIG. 15, but from a second or lateral view instead of an AP view, the screenshot including a lateral boundary and lateral representation of a desired trajectory for a particular IM nailing procedure overlaid on a representation of the anatomical structures. [Figure 17] 17 is an exemplary screenshot of a display of a surgical instrument assembly, the screenshot including the representation of FIG. 16, but where the position of the cutting instrument is adjusted according to a lateral representation of the desired trajectory. [Figure 18] 11 is another exemplary screenshot of a display of a surgical instrument assembly, the screenshot including: 1) a representation of an anatomical structure; and 2) an axis of the anatomical structure overlaid on the representation of the anatomical structure; and a representation of another desired trajectory overlaid on the representation of the anatomical structure, where the representation of the desired trajectory is offset at an angle relative to the axis. [Figure 19] 13 is another exemplary screenshot of a display of a surgical instrument assembly showing exemplary technical information related to an IM nailing process. [Figure 20]FIG. 2 is a perspective view of the surgical instrument assembly shown in FIG. 1 showing first and second representations of the anatomical structure displayed on a display, with the first representation of a desired trajectory overlaid on the first representation of the anatomical structure and the second representation of the desired trajectory overlaid on the second representation of the anatomical structure. [Figure 21A] FIG. 13 is a rear perspective view of another embodiment of a surgical instrument assembly that does not include a measurement device. [Figure 21B] FIG. 21B is a front perspective view of the surgical instrument assembly of FIG. [Figure 21C] FIG. 21C is a rear view of the surgical instrument assembly of FIG. [Figure 21D] FIG. 21D is a side elevational view of the surgical instrument assembly of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The present disclosure may be more readily understood by reference to the following detailed description in conjunction with the accompanying drawings and examples, which form a part of this disclosure. It is to be understood that the present disclosure is not limited to the specific apparatus, methods, applications, conditions or parameters described and / or shown herein, and the terminology used herein is for the purpose of describing specific embodiments by way of example only, and is not intended to limit the scope of the present disclosure. Also, as used in the specification, including the appended claims, the singular forms "a", "an", and "the" include the plural, and reference to a particular numerical value includes at least that particular value, unless the context clearly dictates otherwise.

[0021] The term "plurality," as used herein, means more than one. When a range of values ​​is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when values ​​are expressed in an approximation form, by use of "about," it will be understood that the particular value forms another embodiment. All ranges are inclusive and combinable.

[0022] Medical professionals may use medical imaging devices, such as C-arm devices, to perform various medical procedures on patients. For example, medical professionals may use imaging devices to determine fractures, guide surgical procedures, or verify the results of surgical repairs. C-arm devices provide, for example, spot imaging and fluoroscopic imaging that allow for the generation of continuous, real-time video. Such images may be provided to a display of the C-arm device.

[0023] It is recognized herein that in some cases, the display of the C-arm system is not positioned in a manner to adequately assist the medical professional during surgery. In various embodiments described herein, images provided by the imaging device may be transmitted in real time to a display that may be mounted on a surgical instrument prior to the start of surgery, whereby fluoroscopic imaging provided by the imaging device may be processed in combination with such image sensor data such that alignment of the surgical instrument is based on the image sensor data, as described further below.

[0024] The display may receive the image in real-time such that the image is displayed by the display in real-time (e.g., at the same time as the image is generated by the imaging device). Displaying in real-time includes updating the displayed image at least every 0.2 seconds. For example, the display may be updated with the current image every 0.001 to 0.2 seconds. In some embodiments, displaying the orientation in real-time includes updating the displayed image at least every 1 second, 0.1 seconds, 0.01 seconds, 0.001 seconds, and / or 0.0001 seconds.

[0025] In one example, a display is attached to the surgical drill such that fluoroscopic images provided by the imaging device can be viewed before, during, and / or after an intramedullary (IM) nailing procedure. In some embodiments, when the display is displaying a representation of the image sensor data instead of the X-ray image data, a representation of the alignment of the surgical instrument can be displayed on the display attached to the surgical instrument to guide the medical professional during the IM nailing procedure.

[0026] The display may be interactive and may assist in various aspects of the IM nailing process. For example, the display may assist in determining and validating the proper entry point trajectory for a given IM nail, as well as determining and validating the proper placement and orientation of the distal locking screw for the IM nail.

[0027] First, because fluoroscopy is a type of medical imaging in which continuous x-ray images are shown on a monitor, the terms fluoroscopic data, fluoroscopic image, video data, and x-ray image may be used interchangeably herein, unless otherwise specified, but are not limited thereto. Thus, an x-ray image may refer to an image produced during a fluoroscopy procedure in which an x-ray beam passes through a patient's anatomy. Furthermore, it will be understood that fluoroscopic data may include x-ray images, video data, or computer-generated visual representations. Thus, fluoroscopic data may include still or moving images.

[0028] 1, the medical imaging system 102 may include a medical imaging device 104 and a surgical instrument assembly 202 in electrical communication with the imaging device 104. The medical imaging device 104, which may be a C-arm device, may include an X-ray generator or transmitter 106 configured to transmit X-rays through a body (e.g., bones) and an X-ray detector or receiver 108 configured to receive the X-rays from the X-ray transmitter 106. Thus, the medical imaging device 104 may define a direction of travel 128 of the X-rays from the X-ray transmitter 106 to the X-ray receiver 108. The X-ray transmitter 106 may define a flat surface 106a facing the X-ray receiver 108. The medical imaging device 104 may further include an arm 110 that physically connects the X-ray transmitter 106 to the X-ray receiver 108. The medical imaging device 104 may further communicate with a medical imaging device display 112 configured to display the x-ray image from the x-ray detector 108. In some cases, the medical imaging device display 112 may be physically integrated into the x-ray detector 108 such that the display 112 may be located in a fixed position relative to the arm 110.

[0029] The medical imaging device 104 is presented as a C-arm device to facilitate description of the disclosed subject matter and is not intended to limit the scope of the present disclosure. Additionally, the imaging system 102 and the imaging device 104 are presented as a medical imaging system and a medical imaging device, respectively, to facilitate description of the disclosed subject matter and are not intended to limit the scope of the present disclosure. Thus, it will be understood that other devices, systems, and configurations may be used in addition to or in place of a system such as the system 102 to implement the embodiments disclosed herein, and all such embodiments are contemplated within the scope of the present disclosure. It is recognized herein that the position of the display 112 may create problems for medical professionals. For example, in some cases, a medical professional may need to view images or data rendered by the display 112 while viewing a patient positioned between the x-ray generator 106 and the x-ray detector 108.

[0030] In one example, a medical professional may face challenges placing a distal locking screw during an IM nailing procedure due to insufficient auxiliary instruments or guidance systems, such as an aiming arm, used to place the proximal screw. The distal screw is typically inserted with a freehand technique under fluoroscopic guidance. The freehand technique is commonly referred to as the perfect circle technique. For example, once a perfect circle is established during an IM nailing procedure, it may be difficult to properly align the drill bit with the axis of the distal locking hole due to lack of visibility while using radiographic images. Improper alignment may cause the implant to break or crack during drilling of the pilot hole, which may result in implant breakage, poor reduction / fixation, delayed surgery, etc. It is further recognized herein that the orientation of the x-ray image rendered by the display 112 may not match the orientation of the patient's anatomy, thereby creating additional challenges for the medical professional.

[0031] As another example of a technical problem addressed by the embodiments described herein, a medical professional may face challenges placing an IM nail due to insufficient auxiliary instruments or guidance systems before the distal locking screw is placed. IM nails are typically inserted with a freehand technique under fluoroscopic guidance. However, improper placement may cause pain to the patient. For example, different bones and different IM nails require the IM nail to be inserted into the bone at different entry points and different trajectories to minimize pain. Furthermore, current approaches to determining the appropriate entry point and trajectory for a particular bone, for example by consulting a technical guide, may result in errors or delays. In various examples described herein, the surgical instrument assembly may be configured to guide and assist the medical professional during various operations, such as an IM nailing procedure.

[0032] The surgical instrument assembly 202 may include a surgical instrument 203 and a sensing unit 207 attached to a forward portion of the surgical instrument 203. The transmission unit 207 may include an image sensor 209 attached to the top of a measurement device 211, which may be attached to the surgical instrument. The image sensor 209 may be configured to face forward, away from the display 212. For example, the image sensor 209 may face forward at a downward angle toward the longitudinal axis A (e.g., about 30° relative to a plane defined by lateral and transverse axes L and T, each perpendicular to each other and to the longitudinal axis A).

[0033] The surgical tool assembly 202 may include a display 212 that faces away from the image sensor 209 and the measurement device 211. For example, the display 212 may face rearward.

[0034] In some embodiments, the sensing unit includes a second image sensor (e.g., a second camera) that may be configured to provide stereoscopic vision, redundant monocular vision, or a combination of the two to achieve greater accuracy or confidence in the image data generated by the image sensor.

[0035] In the illustrated embodiment, the image sensor is in electronic communication with the processor via a wired connection. In one embodiment, the image sensor is in electronic communication with the processor via a wireless connection.

[0036] The image sensor 209 may be small and lightweight enough to be added to a drill without impeding the mobility of the user. The image sensor 209 may be configured to generate image sensor data with sufficient image quality to be able to accurately detect a fiducial marker (e.g., the ArUco marker 213 shown in FIG. 4D). For example, the image sensor 209 may be configured to detect visible light, and examples of image sensor 209 characteristics are provided in the table below. The image sensor 209 may have any one or combination of the characteristics identified in the table below. The image sensor 209 characteristics may be balanced against size, cost, and other considerations when selecting a surgical instrument assembly 202.

[0037] [Table 1]

[0038] In one embodiment, the image sensor 209 may be combined with a lens to achieve one or more of the characteristics identified in the table above.

[0039] 3, in one embodiment, data (e.g., video or still images) provided by the medical imaging device 104 (an example of X-ray image data) may be received by an instrument application, e.g., an alignment application, which may be a program such as software or hardware or a combination of both that may be run on any suitable computing device. As described further below, a user may use the instrument application to view a representation of the alignment of the surgical instrument 203 and / or a representation of the images generated by the medical imaging device 104. The instrument application may receive, process, and / or display the representation of the alignment and the representation of the fluoroscopic images at various positions, e.g., positions aligned with a desired trajectory for drilling.

[0040] 2 and 3, any suitable computing device 204 may be configured to host the instrument application. It will be appreciated that the computing device 204 may include any suitable device, examples of which include a portable computing device such as a laptop, tablet, or smartphone. In another example, the computing device 204 may be internal to the surgical instrument 203.

[0041] In an exemplary configuration, the computing device 204 may include a processing unit or unit 206 (e.g., a processor), a power supply 208, an image sensor 209, an input unit 210, a display 212, a memory unit 214, a user interface unit 216, a wireless transceiver 217, and an accelerometer 215. It is emphasized that the block diagram depiction of the computing device 204 is an example and is not intended to suggest a particular implementation and / or configuration. The processing unit 206, the image sensor 209, the input unit 210, the display 212, the memory 214, the user interface 216, the wireless transceiver 217, and / or the accelerometer 215 may be coupled to each other to enable communication therebetween. The accelerometer 215 may be configured to generate accelerometer information corresponding to an orientation of the computing device 204. As will be appreciated, any of the above-mentioned components may be distributed across one or more separate devices and / or locations.

[0042] In various embodiments, the input 210 includes a receiver of the computing device 204, a transmitter of the computing device 204, or a combination thereof. The input 210 is capable of receiving information, such as fluoroscopic data, in real time from the medical imaging device 104. It should be understood that the transmitting and receiving functionality may also be provided by one or more devices external to the computing device 204, and thus to the surgical tool assembly 202. The input 210 may send and receive information to and from another component, such as the image sensor 209, via a wireless transceiver 217.

[0043] Depending on the exact configuration and type of processor, memory portion 214 may be volatile (such as some types of RAM), non-volatile (such as ROM, flash memory, etc.), or a combination thereof. Computing device 204 may include additional storage (e.g., removable and / or non-removable storage) including, but not limited to, tape, flash memory, smart cards, CD-ROMs, digital versatile disks (DVDs) or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, Universal Serial Bus (USB) compatible memory, or any other medium that can be used to store information and that can be accessed by computing device 204.

[0044] The computing device 204 may also include a user interface portion 216 that allows for user communication with the computing device 204. The user interface portion 216 may include inputs that provide the ability to control the computing device 204, for example, by buttons, soft keys, a mouse, voice-activated controls, a touch screen, movement of the computing device 204, visual cues (e.g., moving a hand in front of a camera on the computing device 204), and the like. The user interface portion 216 may provide outputs that include visual information (e.g., via a display), audible information (e.g., via a speaker), mechanical (e.g., via a vibration mechanism), or combinations thereof. In various configurations, the user interface portion 216 may include a display, a touch screen, a keyboard, a mouse, an accelerometer, a motion detector, a speaker, a microphone, a camera, a tilt sensor, or any combination thereof. The user interface portion 216 may further include any suitable device for inputting biometric information, such as, for example, fingerprint information, retinal information, voice information, and / or facial feature information. Thus, a computer system, such as computing device 204, may include a processor, a display coupled to the processor, and a memory in communication with the processor. The memory may have instructions stored therein that, when executed by the processor, cause the computer system to perform operations, such as those described herein. The display 212 may be configured to display visual information, such as those described with reference to Figures 4A-4O, 5A-5G, and 10A-20.

[0045] 1 and 3, the transmitter unit 114 may be electrically coupled to the medical imaging device 104 or may be part of the medical imaging device. The transmitter unit 114 may be any suitable computing device configured to receive and transmit video signals including images, e.g., fluoroscopic images. It will be appreciated that the transmitter unit 114 may include any suitable device, examples of which include portable computing devices such as laptops, tablets, or smartphones.

[0046] 3, in an exemplary configuration, the transmitter unit 114 may include a processing unit or unit 116, a power supply 118, an input 120, and an output 122. It is emphasized that the block diagram depiction of the transmitter unit 114 is an example and is not intended to suggest a particular implementation and / or configuration. The processing unit 116, the input 120, and the output 122 may be coupled together to enable communication therebetween. As will be appreciated, any of the above-mentioned components may be distributed across one or more separate devices and / or locations.

[0047] In various embodiments, the input 120 includes a receiver of the transmitter unit 114, and the output 122 includes a transmitter of the transmitter unit 114. The input 120 is capable of receiving information, such as fluoroscopic image or video data, from the medical imaging device 104, in particular from the output interface 105 of the medical imaging device 104. The output interface 105 may include a coaxial output, a usb output, a component output, a wireless output, and the like. It should be understood that the transmission and reception functions may also be provided by the medical imaging device 104. In one example, the transmitter unit 114 is electrically coupled to the output interface 105 of the medical imaging device 104 to establish a wired or wireless electrical connection between the transmitter unit 114 and the display 112. The output interface 105 may include more video output connectors using matching input modules. In one example, the processor 116, which may include more processors running on an embedded operating system, may detect the presence of a signal, for example a video signal including a fluoroscopic image, from the medical imaging device 104. The processor 116 may process the signal as needed for transmission to the surgical instrument assembly 202. For example, the processor 116 may compress the signal to reduce the bandwidth used to transmit the signal.

[0048] After the processing unit 116 performs processing on the video signal as required, the video signal, which may include the fluoroscopic image, may be transmitted by the output 122 of the transmitter unit 114 to the input 210 of the computing device 204. The output 122 of the transmitter unit 114 may be configured to transmit the fluoroscopic image according to any communication protocol as desired. For example, the output 122 may include a ZigBee module connected to the processing unit 206 via a Universal Serial Bus (USB) so that the output 122 can transmit data wirelessly (via a wireless communication channel) according to any ZigBee protocol. The output 122 may transmit the video signal, e.g., the fluoroscopic image, via Wi-Fi, Bluetooth, broadcast, or any other wireless communication channel as desired. The output 122 may transmit a single X-ray image (also called a "snapshot") at a time.

[0049] Thus, the input 210 of the device 204 can receive data or video signals, e.g., fluoroscopic images, in real time, transmitted from the medical imaging device 104 over a wireless communication channel. The input 210 can be configured to receive ZigBee messages, Wi-Fi messages, Bluetooth messages, broadcast messages, or messages formatted according to any wireless protocol, as desired. In one example, when the input 210 of the device 204 receives a fluoroscopic image from the medical imaging device 104, the image can be acquired and verified by the processing unit 206 of the computing device 204. For example, the processing unit 206 can verify that the received image is from the appropriate medical imaging device. The image can be transferred to the display 212, for example, when the image is verified. The processing unit 206 can also ensure that valid data is displayed. For example, if there is an interruption in the wireless communication channel or connection between the computing device 204 and the medical imaging device 104, the processing unit 206 can identify the interruption and send a message to the display 212 so that the interruption is communicated to a medical professional viewing the display 212. In some cases, the processor 206 may cause the surgical tool assembly 202 to display an indication of an error on the display 212 when the quality of the communication link between the imaging device 104 and the surgical tool assembly 202 falls below a predetermined threshold. Thus, a wireless point-to-point communication channel or connection between the transmitter unit 114 and the computing device 204 may be established, which may be managed by the input 210 and output 122 on the physical layer and the processing units 116 and 206 at the application layer.

[0050] 1 , the image sensor 209 can provide image sensor data to the processing unit 206. The image sensor 209 can be connected to the processing unit 206 via a cable. For example, the image sensor 209 can be directly connected to the processing unit 206. In one embodiment, the image sensor is housed with the processing unit 206.

[0051] In another embodiment, the image sensor is wirelessly connected (e.g., via wifi or a Bluetooth module) to the processing unit 206. For example, the image sensor data generated by the image sensor may be transmitted via wifi to a private URL that may be accessible to the processing unit 206 to obtain the image sensor data.

[0052] The processing unit 206 may receive the image sensor data from the image sensor and generate navigation graphics on the display 212 based on the image sensor data.

[0053] The display unit 212 may be reusable. For example, the display unit 212 may be sterilized prior to surgery. The sterilized display unit 212 may be sterilely removably attached to the surgical instrument 203 after such sterilization. For example, the display unit 212 may be removably attached via gravity, push locks, or magnets that secure the display unit 212. The display unit 212 and the surgical instrument 203 may include corresponding connectors that are aligned with one another to ensure that the display 212 is correctly attached to the surgical instrument. In one embodiment, the display and image sensor may be sterilized and packaged together. For example, the image sensor and display may be removed from their respective packages and each may be separately attached to a corresponding location on the surgical instrument.

[0054] In one embodiment, the display may be capable of being detached from the surgical instrument and placed elsewhere in the room for viewing. For example, the display may include a clamp, stand, or another component configured to hold or place the display on a table or other fixture. Removing the display may provide flexibility to the surgeon by reducing drill weight and / or reducing view obstruction.

[0055] 2A-2D and 13, the display 212 can be configured to be fixed relative to the surgical device 203 (eg, along the longitudinal axis A of the drill bit).

[0056] The surgical instrument 203 may define a proximal end 203b and a working end 203a opposite the proximal end 203b. The working end 203a may be configured to, for example, cut, pierce, or otherwise target, a structure, for example, an anatomical structure 124, of a patient receiving medical care. The display 212 may face the proximal end 203b. The display 212 may be positioned to provide a line of sight to both the working end 203a and the display 212 from a location proximal to the surgical instrument 203. Thus, in some cases, for example, a medical professional may be able to see both the display 212 and the working end 203a of the surgical instrument 203 while manipulating the surgical instrument 203.

[0057] In one example, the surgical instrument 203 includes a cutting instrument 226 including a proximal end 226b adjacent the body 205 of the surgical instrument 203 and a cutting tip 226a opposite the proximal end 226b of the cutting instrument 226. The cutting tip 226a can define an end of the cutting instrument opposite the proximal end 226b of the cutting instrument 226. The cutting instrument 226 can have the cutting tip 226a, which can be configured to remove anatomical material from an anatomical structure, such as the anatomical structure 124. In the illustrated example, the cutting instrument 226 is a drill bit and the cutting tip 226a is a tip of a drill bit, however, it should be understood that other instruments and configurations in addition to or in place of instruments such as the cutting instrument 226 may be used to achieve the embodiments disclosed herein, and all such embodiments are contemplated as falling within the scope of the present disclosure.

[0058] The image sensor 209 may be positioned in front of the display 212 so as to have an unobstructed view of the cutting tip 226a, the fiducial marker 213, and / or the fiducial markers described below with reference to Figures 4D-4Z. The image sensor 209 may be rotatable relative to the surgical instrument 203 and the display 212. The image sensor may be removable from the surgical instrument 203, which may allow the image sensor 209 to be sterilized and reused.

[0059] In one embodiment, the image sensor may be configured for disposable use. For example, the image sensor may be permanently attached to the surgical instrument.

[0060] The image sensor 209 may be calibrated. For example, the image sensor 209 may be calibrated by using a standard chessboard calibration method or any other widely accepted method of obtaining intrinsic parameters and distortion coefficients. In one embodiment, the image sensor calibration is performed using a calibration technique that uses a chessboard or other planar pattern. For example, in the technique proposed by Zhengyou Zhang, at least two orientations of a planar pattern are observed by the pattern to calibrate the camera in a closed-form manner. Zhang, Zhengyou. (2000). A Flexible New Technique for Camera Calibration. Pattern Analysis and Machine Intelligence, IEEE Transactions on. 22.1330-1334.10.1109 / 34.888718. In one embodiment, another suitable camera calibration method is used to calibrate the image sensor.

[0061] In one embodiment, the image sensor is calibrated to obtain the intrinsic parameters and distortion coefficients during device manufacturing, and the calibration is stored by the image sensor or another device in communication with processing unit 206 (FIG. 3).

[0062] 8, there is shown an exemplary flow chart of steps followed by an x-ray imaging application, e.g., an orientation application, which may be a program, such as software or hardware or a combination of both, that may be executed on the computing device 204 or another computing device. In step 302, the geometry of the implant may be input or detected. For example, a user may input a particular IM nail and select target holes in the IM nail via the user interface 216. The particular nail shape and target hole locations may be input or detected.

[0063] In one embodiment, a user inputs a model of a C-arm that generates an X-ray image of the IM nail and adjacent anatomical structures. The X-ray imaging application may be configured to access calibration data specific to the C-arm. In one embodiment, the C-arm includes an inertial motion unit (e.g., including an accelerometer) to determine the orientation of the C-arm. The orientation may be realized, for example, by the inertial motion unit in a computational device of the surgical instrument assembly.

[0064] C-arm calibration data that is model specific for the C-arm may be stored by the computing device 204 and / or medical imaging device 104. The calibration data may include the total volume of the C-arm imageable space. Calibration of the C-arm may be performed once per model and stored in the computing device 204 and / or medical imaging device 104 for later use.

[0065] Calibration of the C-arm may include using a volume fixture having identifiable radiopaque markers placed throughout the volume fixture at predetermined locations. Fluoroscopic images of the markers may be taken and the 2D image location of each marker may be mapped to a precise spatial location known from the fixture. This allows the entire volume of imageable space between the C-arm transmitter 106 and receiver 108 to be mapped and output as calibration data. X-ray images generated by the C-arm may be processed using the mapping to determine the 3D coordinates of features on the 2D image x-ray image.

[0066] Calibration can be done in one orientation and negligible changes in image geometry with c-arm angle can be ignored. In an embodiment, calibration can be done at different angles and an angle measurement device can be built into the c-arm for use in selecting or interpolating the calibration data that is most appropriate. Some modern c-arms may already have this functionality built-in. An example of volumetric calibration is described in Hosseinian S, Arefi H, Navab N. Toward an End-to-End Calibration for Mobile C-Arm in Combination with a Depth Sensor for Surgical Augmented Reality Applications. Sensors (Basel). 2019; 20(1): 36. Published 2019 Dec 19. doi: 10.3390 / s20010036.

[0067] A user may input the model of medical imaging device 102 (e.g., C-arm) into the user interface 216, for example when entering a particular IM nail. The display 212 may display a drop-down list of stored C-arm models to be selected. In one embodiment, the C-arm has an electronic or scannable identification label (such as a QR code) that can provide the system with any necessary model-specific information.

[0068] In one embodiment, a single fluoroscopic image can be used to approximate the C-arm imaging geometry as that of a pinhole camera. The intrinsic matrix and distortion coefficients can be estimated using a pattern on a rigid marker (e.g., the four outer corners of the ArUco marker), for example, using Zhang's method. In one embodiment, multiple fiducial markers can be detected in a single fluoroscopic image to estimate the matrix and distortion coefficients using Zhang's method. In some embodiments, in addition to the fiducial markers, the geometry of the IM nail, including the holes, can be used to provide more calibration points for using Zhang's method.

[0069] For ArUco markers, the markers can be detected using OpenCV's detectMarkers function, which can provide image points for each of the four corners of each ArUco marker. Object points for each ArUco marker can be predetermined as described below. Zhang's method can use ArUco markers using OpenCV's calibrateCamera function with the image points and object points. For example, ArUco markers can be detected using OpenCV's functions such as cv:aruco:detectMarkers, cv:aruco:estimatePoseBoard, and / or cv:aruco:estimatePoseSingleMarkers.

[0070] Aligning the fiducial marker to the desired trajectory (e.g., the central axis of the target hole in the IM nail) may include determining a world coordinate system in step 330. For example, a rotation or translation matrix is ​​calculated to map the C-arm coordinate system (e.g., if the C-arm is modeled as a pinhole camera) to a world coordinate system centered on the desired trajectory. In the world coordinate system, the Z-axis may be aligned with the desired trajectory (e.g., the central axis of the target through hole) and the X-axis may be aligned with the longitudinal axis of the implant (e.g., the IM nail).

[0071] The rotation and translation matrices may be determined by using the PnP method with the detected edge of the target through-hole (e.g., depicted as a perfect circle in the fluoroscopic image). Edge detection may be done by any suitable detection process. For example, OpenCV's Canny edge detector may perform the edge detection. In one embodiment, OpenCV's blob detector may be used to find circular shapes that meet certain shape prerequisites such as circularity and minimum / maximum area. The multiple points along the edge may be used as a 2D set of points to match with known 3D points around the through-hole (e.g., using a predefined radius of the through-hole) in optional step 324.

[0072] If the fluoroscopic image does not show a perfect circle, a transformation application can be used to determine the rotation angles required to align with a perfect circle, and those angles can be used to create a rotation matrix. The translation matrix can also be found by identifying the center point of the detected through-hole (e.g., a circle, or an arc in the case of a non-circular image), rather than using the edge points.

[0073] The 3D coordinates of each individual marker keypoint are stored in the computing device 204 based on the known geometry of the reference marker. For example, the four corners of an ArUco marker of side length L may be stored as the following array [[0,0,0],[L,0,0],[L,-L,0],[0,-L,0]] with the origin set as the top left corner. All the reference marker information may be combined into an array of arrays, which may be called an array of object points relative to the board of the reference marker.

[0074] The key points of each fiducial marker may be detected using PnP. The relative pose of the c-arm (e.g., in the form of rotation and translation vectors) may be stored in the computing device 204. For example, the ArUco markers may be detected, thereby providing four corner image points. The image point data may then be used with the known object points to determine the rotation and translation vectors. This determination may be made at the same time that the C-arm calibration is performed using a single C-arm image. The rotation and translation vectors may be stored in an array and correlated with the specific identity of the corresponding marker.

[0075] Object points at the corners of the markers can then be set by a change of basis from their local coordinate system (e.g., defined in the plane of the markers with origin in the upper left corner) to the C-arm coordinate system. The object points can then be set by another change of basis from the C-arm coordinate system to a world coordinate system centered on the desired trajectory (e.g., the central axis of the target through-hole).

[0076] The object points may then be used to create an ArUco board object (e.g., an array of arrays of world coordinates for each marker corresponding to another array of marker identifiers) in step 332. The ArUco board object may be used in subsequent marker board detection using image sensor 209.

[0077] The resulting pose of the image sensor 209 may be provided with respect to a world coordinate system centered on the image sensor 209. In one embodiment, the drill bit is targeted to the world origin via navigation graphics.

[0078] If the fiducial marker is observed in more than one fluoroscopic image (e.g., both the first x-ray with 30% of the two holes shown in step 304 and the second full circle x-ray shown in step 316), the fiducial marker from both images may be used to more accurately calibrate the C-arm with a calibration application in step 326. In one embodiment, the C-arm image is cropped to a central region (e.g., as illustrated in Figures 4G and 4L), which may typically have less distortion than the peripheral radial edges of the fluoroscopic image. The eigenmatrix and distortion coefficients may be determined based on the cropped image of the central region, rather than the entire image or any markers or data outside of the central region.

[0079] The reprojection error may be checked for the set of points and compared to a predefined threshold to determine if the quality of the calibration is sufficient to enable accurate navigation. The calibration may alternatively or additionally use the same methodology used in the MaxFrame project as described in U.S. Patent Application No. 2021 / 0077194, entitled "Othopedic Fixation Control and Manipulation."

[0080] At step 304, a fluoroscopic image may be taken. At step 306, it may be determined whether a sufficient portion of the target has been imaged. For example, a sufficient portion of the target area of ​​the IM nail may be at least 30% of the two distal holes of the IM nail shown in the fluoroscopic image. If at least 30%, an angular adjustment of the C-arm may be determined based on the orientation of the target (e.g., the target through holes) in the image. For example, an angular adjustment for the C-arm that results in a true circular image at step 312 may be determined at step 310. The adjustment information may be communicated to an appropriate application or user (e.g., a C-arm technician or surgeon) for adjustment at step 312.

[0081] In one embodiment, the orientation of the desired trajectory (e.g., the central axis of the target hole) is determined without achieving a full circle image. For example, the orientation of the central axis of the target hole may be determined based on predetermined characteristics of the IM nail and / or the target hole as described above.

[0082] If a sufficient portion of the target is not imaged, then in step 308 the C-arm may be adjusted to a different orientation and step 304 may be repeated with the C-arm at a different orientation relative to the implant.

[0083] In one embodiment, fiducial marker locations in the fluoroscopic image are detected and compared to the partial holes visible on the image (e.g., as shown in Figures 4E-4H). The spatial location of each fiducial marker may be calculated relative to the IM nail geometry provided in step 328. Determining the relationship between the fiducial markers and the IM nail may optionally accomplish the rendering steps 310-316. Thus, determining the relationship between the fiducial markers and the IM nail may provide complete navigation to any distal locking hole of the IM nail without additional full circle fluoroscopic images. In this case, the entirety of the distal locking may be performed based on only a single fluoroscopic image.

[0084] After adjusting the orientation of the C-arm to obtain a full circle fluoroscopic image in step 312 (eg, as shown in FIGS. 4J-4M), a further fluoroscopic image may be taken in step 314.

[0085] If one or more fiducial markers are not detected or a perfect circular fluoroscopic image is not achieved, then the orientation of the C-arm may be adjusted such that at least one fiducial marker is within the field of view of the C-arm image and perfect circularity is achieved, in step 318. In one embodiment, the orientation and / or position of the fiducial markers may be adjusted.

[0086] The orientation of the fiducial markers may be detected in the images in step 320 and the relative spatial position compared to the desired trajectory may be determined in step 322. Depth information (e.g., in a direction parallel to the central axis of the target through hole) may be determined by the processing unit based on the image sensor data. In one embodiment, depth information is not determined (e.g., if the user can drill along the axis of the target hole, the user feels like they are drilling bone and does not need Z-axis visual feedback). Fiducial marker detection in fluoroscopic images, image sensor images, or both may be less accurate with respect to depth (Z-axis) than X- and Y-coordinates.

[0087] In one embodiment, the C-arm intrinsic and extrinsic parameters may be determined using only a single fluoroscopic image. The C-arm calibration may be performed based on a single fluoroscopic image, as described above. The origin of the world coordinate system may be positioned such that the X-axis is parallel to the longitudinal axis of the IM nail, and the Z-axis may be parallel to the desired trajectory (e.g., the central axis of the target distal locking through hole).

[0088] If a fiducial marker falls off, is outside the field of view of the C-arm, or otherwise becomes obscured or ineffective to the view, there may still be a sufficient number of other fiducial markers in the array of fiducial markers to perform the steps described above. If insufficient markers remain, one or more additional fiducial markers may be placed in the field of view to supplement the existing fiducial markers in the field of view. After placing the additional fiducial markers in the field of view, an x-ray image may be taken to align the fiducial markers to the IM nail.

[0089] As shown in Figures 2A-2D, the measuring device 211 may be located in front of the display 212. The measuring device 211 may be releasably attached or fixed to the surgical instrument 203. For example, a mounting member 218 may removably attach the measuring device 211 to the surgical instrument 203. In one embodiment, the measuring device is constructed and used in the manner described in U.S. Patent No. 10,736,644, issued August 11, 2020 and entitled "Surgical power drill including a measuring unit suitable for bone screw length determination."

[0090] A sleeve 219 may be provided to interface with the measurement device 211. The sleeve may be configured to slide over the drill bit 226 such that the sleeve 219 is moveable along the longitudinal axis A. The sleeve 219 may include a fiducial marker 213, as discussed above. For example, a rearward facing surface of the sleeve 219 may include the fiducial marker 213.

[0091] The measuring device 211 may be configured to detect fiducial markers 213 on a rear-facing surface of the sleeve 219. The distance between the measuring device 211 and the sleeve may be determined based on the detected fiducial markers 213 of the sleeve 219. For example, the measuring device 211 may determine the distance between the sleeve 219 and the measuring device 211 based on the orientation and / or size of the detected fiducial markers 213.

[0092] The sleeve 219 may be positioned at a predetermined location on the drill bit (e.g., spaced a predetermined distance along the longitudinal axis A from the drill tip 226a or flush with the drill tip 226a). Thus, the orientation and length of the drill tip 226 may be determined based on the orientation of the fiducial marker 213 detected by the measurement device 211.

[0093] In one embodiment, the measurement device outputs data representative of the orientation and / or position of the sleeve and / or drill tip to the computing device 204. In an alternative embodiment, the sleeve may be displaceable at an angle relative to the longitudinal axis A, and the measurement device may be configured to detect such angular displacement.

[0094] In one embodiment, the sleeve includes a reflector for positioning the drill bit (e.g., the end of the drill bit tip). For example, the measurement device may include a laser device for detecting a linear displacement assessment between the measurement device and the reflector of the sleeve. The laser device may be configured to emit light and detect light reflected by the reflector.

[0095] The measurement device may be configured to determine a linear displacement between the sleeve and the measurement device based on the detected reflected light. For example, the measurement device may perform laser triangulation for the linear displacement assessment. In one embodiment, the linear displacement assessment may be performed by using an ultrasonic position sensor that detects ultrasonic waves emitted at the reflector.

[0096] The orientation, size, and / or position of the drill bit 226 relative to the image sensor 209 may be determined based on the orientation, size, and / or position of the drill bit in image data representing the drill bit 226 detected by the image sensor 209, for example based on the orientation, size, and / or position detected by the measurement device 211. The measurement device 211 may communicate the orientation, size, and / or position of the drill bit 226 to the computing device 204 (e.g., via a wired or wireless electronic connection).

[0097] The position / location of the drill bit relative to the image sensor 209 may be determined so that it can be referenced for navigation output to the display 212. Positioning the drill bit 226 may be based on the output of the measurement device 211. The positioning may be based on the orientation, size, and / or position detected by the measurement device 211, and may determine the orientation and position of the image sensor relative to the drill bit 226 based on a predetermined relationship between the image sensor 209 and the measurement device 211.

[0098] In one embodiment, the image sensor is configured to determine the distance between the sleeve and the image sensor based on the orientation and / or size of fiducial markers on the sleeve detected by the image sensor. For example, the image sensor may be configured to perform the functions of measurement device 211 discussed above. In some embodiments, the sleeve includes inherent geometry that is detected by the image sensor instead of or in addition to the fiducial markers.

[0099] In one embodiment, the image sensor is configured to determine an orientation, size, and / or position of the drill bit relative to the image sensor. The image sensor may be configured to detect the drill bit and automatically determine the orientation, size, and / or position of the drill bit relative to the image sensor using image recognition. For example, the image sensor and / or computing device may be trained (e.g., by using a neural network) to distinguish the drill bit from other objects in the environment.

[0100] The diameter of the drill bit may be determined based on the target IM nail and the target through hole input by the user (e.g., a 5.2 millimeter (mm) diameter distal locking hole may mean the user is using a 5.0 mm drill bit). The diameter may then be used to fit an expected point on the outer diameter of the drill bit to the drill bit edge detected by the image sensor. Thus, the image sensor and / or computing device may determine the 3D coordinates of the drill bit using, for example, a Points-N-Perspective (PnP) method described below.

[0101] In one embodiment, the spatial relationship between the image sensor, the drill bit, and the fiducial markers may be determined using any variation of the PnP method, which may use a given set of tightly aligned 3D points and their corresponding 2D image projections using a calibrated image sensor 209 to determine the image sensor pose. In some embodiments, one or more of the following functions or methods may be used: SOLVEPNP_ITERATIVE from OpenCV, Efficient PnP (EPnP) proposed by Vincent Lepetit, Perspective-Three-Point (P3P) proposed by Xiao-Shan Ga, Algebraic P3P (AP3P) proposed by Tong Ke, or Consistently Fast and Globally OptimalSolution PnP (SQPnP) proposed by George Terzakis.

[0102] In some embodiments, the image sensor provides sufficient information of the orientation of the drill relative to the image sensor and the reference markers such that neither the measurement device nor the accelerometer are utilized. For example, in one embodiment, the accelerometer and measurement device may be omitted from the surgical instrument assembly.

[0103] In some embodiments, the image sensor is the only component attached to the surgical instrument. For example, the image sensor may be configured to communicate with a computing device, which may be separate from the surgical instrument. Such communication may be wireless. In one embodiment, the image sensor and the display are the only components attached to the surgical instrument. The image sensor and the computing device may be configured to communicate with each other and / or with the display.

[0104] The surgical instrument assembly may be assembled by the user (e.g., a surgeon) or by another person (e.g., a scrub technician).

[0105] The fiducial marker 213a may be substantially the same as the fiducial markers discussed above and may be fixed relative to the target area for surgery. The fiducial marker 213a may be placed at any time prior to taking the x-ray image in step 304. The fiducial marker 213a may be placed after step 310 and before step 316 if steps 304-314 are being performed. The fiducial marker 213a may be fixed relative to the anatomy of the patient being operated on prior to the surgical procedure (e.g., while the patient is being prepared). The fiducial marker 213a may not interfere with other steps of the IM nailing procedure.

[0106] For example, an array of fiducial markers 213 may be affixed to the patient's skin (e.g., as shown in Figures 4U and 4W-4Z). In one embodiment, the fiducial markers are adhered to a layer of material that contacts the patient's skin; Adhered to the patient's skin or a layer adjacent to the patient's skin (e.g., drape, Ioban); May be incorporated into and glued onto the drape; May be taped to the patient using Ioban or other tape or strap-like device; May be latched onto the skin or layer using staples, tacks, or burr-like mechanisms; May include protrusions (i.e., bone tacks) that can be hammered or drilled into the patient's bone for fixation; May be attached to the proximal end of the nail or nail handle with an extended adjustable means to position the marker in the correct area near the distal locking portion of the nail.

[0107] In the embodiment illustrated in Figure 4T, the array of fiducial markers 213a is fixed relative to the anatomical structure 124 and implant 125 using stakes 340. In the embodiment illustrated in Figure 4V, the array fiducial markers 213a are fixed relative to the anatomical structure 124 and implant 125 via a jig 342 that is attached to the implant 125 and extends outside the anatomical structure 124.

[0108] In the embodiment illustrated in Figures 4D, 4P, and 4Q, for example, the fiducial marker is an ArUco marker. In some embodiments, other fiducial markers are provided instead of ArUco markers to determine pose and distance. For example, an array of small round markers 213b and large round markers 213c can be provided in a rectangular pattern, as illustrated in Figure 4R. In another embodiment, one or more spherical fiducials 213d can be fixed relative to the patient, as illustrated in Figure 4S. The spherical shape can provide improved detection of the fiducial markers, for example, when arranged on a curved surface and / or when the fiducial markers 213d are detected at different projection angles.

[0109] Regardless of the fiducial marker used, each individual fiducial marker and / or array of fiducial markers may be visually recognizable by the image sensor 209 to determine pose and distance relative to the image sensor 209. In one embodiment, the circular fiducial markers may be colored and / or have different diameters to distinguish features of the pattern to determine pose and distance. The different coloring of the markers may provide for rapid identification of areas of interest that may be noted to reduce the amount of image data that must be processed.

[0110] The fiducial markers 213 may be individually recognizable or may be recognizable as a recognizable array when placed on a curved surface, for example, when placed on the patient's skin, as illustrated in FIG. 4U and FIG. 4W-4Z. Distortion due to the shape and relative distance of the fiducial markers 213a may be detected by the image sensor 209. The curvature of the fiducial markers 213a may be determined based on the detected distortion. The pose of the fiducial markers may be determined based in part on the curvature. For example, the pose of the fiducial markers 213a may be determined based at least in part on the detected curvature from one or more viewpoints illustrated in each of FIG. 4W-4Z. In one embodiment, the x-ray image data may include the curvature and / or the relative distance of the fiducial markers from two or more viewpoints (e.g., the viewpoints illustrated in FIG. 4W-4Z).

[0111] 4W-4Z, the image sensor 209 may generate image data representing multiple viewpoints and including fiducial markers 213a that are not represented in the X-ray image data. For example, one or more of the fiducial markers 213a may be radiopaque and may be represented in a single X-ray image generated by the medical imaging device 104 (shown in FIG. 1). The orientation and / or position of each radiopaque marker 213a relative to a desired trajectory (e.g., a central axis of an implant bore and / or an anatomical structure bore) may be determined in the same manner as discussed above with reference to FIG. 8. The viewpoints may be perpendicular to each other such that navigational guidance may be provided in two perpendicular planes based on a single 2D X-ray image (i.e., snapshots).

[0112] Multiple images representing different viewpoints from the image sensor 209 may be daisy-chained to determine the orientation and / or position of fiducial markers 213a not represented in the X-ray image data relative to the desired trajectory. Fiducial markers 213a not detected by the medical imaging device 104 (e.g., fiducial markers 213a that are not radiopaque when a single X-ray image is generated or that cannot otherwise be detected by the medical imaging device 104 from that viewpoint) may be detected by the image sensor 209. For example, the lowest and highest fiducial markers 213a when viewed in FIG. 4W and the highest fiducial markers 213a when viewed in FIGS. 4X-4Z may each not be radiopaque. The image sensor 209 may detect each of the non-radiopaque fiducial markers 213a by generating image data from multiple different viewpoints relative to the fiducial markers 213a (e.g., from the viewpoints shown in FIGS. 4W-4Z).

[0113] With reference to FIG. 4W, the orientation of the non-radiopaque fiducial markers 213a depicted with the radiopaque fiducial markers 213a (e.g., fiducial markers 213a between the lowest and highest fiducial markers 213a in FIG. 4W) may be determined based on the image sensor data representing the fiducial markers 213a in FIG. 4W and a single X-ray image generated by the medical imaging device 104. For example, the orientation of the non-radiopaque fiducial markers 213a relative to the desired trajectory may be determined based on the detected orientation of the radiopaque fiducial markers 213a. With reference to FIG. 4X, the orientation of the top non-radiopaque fiducial marker 213a relative to the desired trajectory may be determined based on the determined orientation of the top non-radiopaque fiducial marker 213a in FIG. 4W. With reference to Figure 4Y, the orientation of the top non-radiopaque fiducial marker 213a relative to the desired trajectory may be determined based on the determined orientation of the top non-radiopaque fiducial marker 213a in Figure 4X. With reference to Figure 4Z, the orientation of the top non-radiopaque fiducial marker 213a relative to the desired trajectory may be determined based on the determined orientation of the top non-radiopaque fiducial marker 213a in Figure 4Y.

[0114] In some embodiments, the image sensor generates images from different viewpoints in different orders. For example, the image sensor may generate image data representing the viewpoint in Figure 4Z before generating image data representing the viewpoint in Figure 4X or the viewpoint in Figure 4Y. For example, the orientation of the non-radiopaque fiducial markers 213a shown in Figure 4Z relative to one another may be determined before determining the relative orientation of the non-radiopaque fiducial markers shown in Figure 4X or Figure 4Y.

[0115] The fiducial marker 213 may be made by cutting or machining metal, printing a pattern with radiopaque ink, and / or filling cavities with a curable radiopaque material. The radiopaque components of the fiducial marker 213 may be relatively thin, less than 0.3 mm.

[0116] Each of the fiducial markers 213 may have an area in the range of 25 mm to 2500 mm. For example, the fiducial markers 213 may have a square perimeter with sides of each fiducial marker 213 having lengths in the range of 5 mm to 50 mm. In some embodiments, each of the fiducial markers may have an area of ​​400 mm to 625 mm (e.g., having lengths in the range of 20 mm to 25 mm). In one embodiment, each of the fiducial markers has a square perimeter with sides having lengths of about 23 mm (i.e., an area of ​​about 529 mm).

[0117] The relatively large size of the fiducial markers may impact the ability of the C-arm and / or image sensor 209 to accurately detect the fiducial marker pattern. For example, the size may be limited by the anatomical space surrounding the surgical site (e.g., the locking site for an IM nail). A larger size may limit the amount of marker that can be placed within the field of view of the C-arm and / or image sensor 209, may obstruct the surgeon's view or range of movement, and / or may be more difficult to adhere securely to the patient's skin.

[0118] In one embodiment, a single fiducial marker is provided for detection by the C-arm and image sensor 209. That fiducial marker may be sufficient to determine the orientation of one fiducial marker in the X-ray image generated by the C-arm and in the image sensor image generated by the image sensor 209. In some embodiments, one or more fiducial markers are provided. For example, at least three fiducial markers may be provided for detection by the C-arm and image sensor 209. Detecting three fiducial markers may provide greater accuracy than when fewer fiducial markers are detected. In some embodiments, at least four fiducial marks are provided. Detecting four fiducial markers may provide greater accuracy than when fewer fiducial markers are detected.

[0119] 7, a user may select one or more operations by entering an option on an exemplary user interface 2100 that may be displayed by the display 212. For example, a user may select an IM trajectory option 2104 to perform an IM drilling operation. A user may select a plating option 2103 to perform an operation associated with securing a plate to a bone. A user may select a nailing option 2102 to perform an operation associated with securing a nail with a distal locking screw. It will be appreciated that alternative or additional options may be rendered by the user interface 2100 as desired. It will further be appreciated that upon entry of an option, further displays may be rendered to guide a user through a particular operation.

[0120] 2A-2D and 13, a surgical instrument assembly 202 can include a computing device 204 mounted to a surgical instrument 203. The surgical instrument 203 can be configured to act on an anatomical structure 124 (shown in FIG. 4A).

[0121] The surgical instrument 203 can define a body 205, and the computing device 204 can be attached anywhere on the body 205, as desired. In one example, with reference to FIGS. 2A-2D , the computing device 204 and display 212 can be supported by a mount 228. The mount 228 can be attachable to the mounting member 218. For example, the mounting member can include a rotatable base 230 to which the mount 228 can be attached (e.g., via a bolted connection). In some embodiments, the computing device 204 is permanently attached to the surgical instrument.

[0122] The rotatable base 230 may be configured to remain in a fixed position unless rotated by the user, regardless of the orientation of the surgical instrument 203. In one embodiment, the rotatable base may be configured to be locked in a desired position.

[0123] While the surgical instrument 203 is depicted as a surgical drill for illustrative purposes, it will be understood that the computing device 204 may be removably attached or permanently attached to other suitable instruments or tools. For example, the surgical instrument assembly 202 may include instruments or instruments configured to target bone regions or other portions of the anatomical structure, remove medical implants, perform osteotomies, or any other procedures, such as, for example, using a combination of fluoroscopy and image sensor images, as desired. Thus, while the anatomical structure 124 is presented as a bone, it will be understood that the structures that the surgical instrument assembly may be configured to act on are not limited to bones.

[0124] The computing device 204 can include a display 212, which can be attached to the surgical instrument 203. The display 212 can be configured to display a representation of the anatomical structure 124 (e.g., based on fluoroscopic data generated by the imaging device 104 and / or image sensor data generated by the image sensor 209). In an exemplary configuration, the display 212 can display the representation of the anatomical structure 124 in real-time, such that the representation of the anatomical structure 124 is displayed by the display 212 at the same time that corresponding image sensor data is generated by the image sensor 209 and / or images are generated by the imaging device 104. In some embodiments, the display can include multiple displays, e.g., a first display and a second display having a different orientation compared to the orientation of the first display.

[0125] 4A-4D, the representation of the anatomical structure 124 may include one or more target sites 126. The target sites 126 may represent sites on the anatomical structure 124 where the surgical instrument 203 may drill, cut, or otherwise target. According to the illustrated example, the target sites 126 may be defined by an implant 125, such as an IM nail or rod, within the bone. While the exemplary operation performed by the surgical instrument assembly is presented as an IM nailing operation to facilitate explanation of the disclosed subject matter, it will be understood that this exemplary IM operation is not intended to limit the scope of the present disclosure. Thus, it will be understood that the surgical instrument assembly 202 may be used to perform other operations in addition to or in place of operations such as the exemplary IM nailing operation.

[0126] With reference to Figure 4D, the array of fiducial markers 213a may be arranged in a rectangular pattern around one or more of the target sites 126. As illustrated in Figure 4D, the target sites 126 may be depicted as extending directly into the page when viewing Figure 4D. The view that Figure 4D represents may be said to be aligned with a perfect circle. For example, if the target site 126 is a cylindrical through-hole, an x-ray image aligned with a perfect circle will represent the target site 126 as a perfect circle.

[0127] The computing device 104 may receive such a circular X-ray image and determine a world coordinate system 220. For example, the computing device 104 may determine an orientation of each fiducial marker 213a in the array of fiducial markers 213a and, from the orientation, determine the world coordinate system 220 based on the X-ray image. The world coordinate system 220 may include an X1 axis, a Y1 axis, and a Z1 axis, each of which is perpendicular to one another and fixed relative to the array of fiducial markers 213a.

[0128] The world coordinate system 220 may be defined by the corners of the array of fiducial markers 213a. The X1 axis may extend along one side (e.g., the bottom side) of the rectangular array of fiducial markers 213a, the anatomical structure 124, and the implant 125. The Y1 axis may extend along another side (e.g., the left side) of the rectangular array of fiducial markers 213a. The Z1 axis may extend away from the corner of the rectangular array of fiducial markers 213a.

[0129] The computing device 104 may determine an image sensor coordinate system 222 based on the image sensor data. For example, the image sensor coordinate system 222 may include an X2 axis, a Y2 axis, and a Z2 axis, each of which is perpendicular to one another and determined based on the image sensor data. The image sensor coordinate system 222 may be fixed relative to the array of fiducial markers 213a, the anatomical structure 124, and the implant 125. For example, the Z2 axis may be centered on one of the target sites 126, and the X2 axis may extend along the length of the implant 125. The Z2 axis may extend radially away from the center of the target site 126. The desired trajectory may extend along the Z2 axis to the center of the opening of the target site 126.

[0130] In real time, the computing device 104 may determine a transformation matrix to determine an orientation and / or position of an object detected in the x-ray image relative to an object detected in the image sensor data. For example, the computing device 104 may determine a real time pose of the surgical instrument 203 (e.g., a drill bit 226 of the surgical instrument 204) in the camera coordinate system 222 based on the real time image sensor data. The computing device 104 may use the transformation matrix in real time to determine a real time pose of the surgical instrument 203, including the drill bit 226, in the world coordinate system 220. The computing system 104 may determine a pose of the surgical instrument 203 relative to the target site 126 based on the determined world coordinates of the surgical instrument 203, for example.

[0131] The orientation of the drill bit 226 and drill tip 226a relative to the target site 126 may be determined by the computing system 104 in substantially the same manner as described above.

[0132] In one embodiment, the image sensor coordinate system may be fixed relative to the image sensor.

[0133] 4E-4I, non-circular x-ray images of the implant 125 are illustrated and shown at various steps of a process for determining the coordinates of the target site 126. FIG 4E illustrates an initial x-ray image of the implant 125 along with the array of fiducial markers 213a. The x-ray image may be adjusted, for example, to more clearly identify the implant 125 and / or the fiducial markers 213a, e.g., the brightness, contrast, and / or noise may be adjusted.

[0134] The adjustment may result in an adjusted x-ray image as illustrated in FIG. 4F. A portion of the x-ray image illustrated in FIG. 4F may be cropped so that the computing device 204 may focus on an area near the target site 126, as illustrated in FIG. 4G, for example. As shown in FIG. 4H, the computing device 204 may determine a world coordinate system 220 based on the cropped x-ray image of FIG. 4G. As discussed further below, the computing device 104 may determine the orientation of the target site 126 based on the shape of the target site 126 depicted in FIG. 4G or 4H relative to the world coordinate system 220. The X1, Y1, and Z1 axis coordinates (e.g., true 3D coordinates) of the target site 126 (e.g., the center of the opening of the target site 126) relative to the world coordinate system 220 may be determined by the computing system 104.

[0135] 4J-4O, a circular x-ray image of the implant 125 is illustrated and shown at various steps of a process for determining the coordinates of the target site 126. FIG. 4J illustrates an initial x-ray image of the implant 125 along with the array of fiducial markers 213a. The x-ray image may be adjusted, for example, to more clearly identify the implant 125 and / or the fiducial markers 213a, e.g., the brightness, contrast, and / or noise may be adjusted.

[0136] The adjustment may result in an adjusted x-ray image illustrated in Figure 4K. A portion of the x-ray image illustrated in Figure 4K may be cropped out such that the computing device 204 may focus on an area near the target site 126, for example, as illustrated in Figure 4L. As shown in Figure 4M, the computing device 204 may determine a world coordinate system 220 based on the cropped x-ray image of Figure 4L. As discussed further below, the computing device 104 may determine the location of the target site 126 relative to the fiducial marker 213a depicted in Figure 4M, for example.

[0137] 4N, for example, the cylinder method may be used to determine the location of the center of the opening in the target site 126 and determine the X1 and Y1 axis coordinates of the target site relative to the world coordinate system. The desired trajectory for drilling may extend along the central axis of the cylinder.

[0138] 4I and illustrated in FIG. 0, the computing system 204 may determine the depth of the target site 126 based on the detected contour of the target site 126. Thus, the X1, Y1, and Z1 axis coordinates (e.g., true 3D coordinates) of the target site 126 relative to the world coordinate system 220 may be determined by the computing system 104.

[0139] The display 212 may, among other things, display a representation of a fluoroscopic image associated with the IM nailing operation. Additionally, the display 212 may display images or data associated with the depth of the drill bit 226. Additionally, the display 212 may display images or data associated with the depth at the same time that the display 212 is rendering the fluoroscopic image of the anatomy 124.

[0140] The display 212 may be configured to display, for example, representations 400a-400c of the anatomical structure 124 generated by the computing device 104 based on X-ray image data received from the medical imaging device 104 and image sensor data received from the image sensor 209. With particular reference to FIG. 4A, the display 212 may display a representation 400a of an implant 125 within the anatomical structure 124. The implant 125 may define one or more target sites 126 from which material may be removed from the anatomical structure 124. In an exemplary IM nailing operation, by viewing the display 212 displaying the representations based on the X-ray image data and the image sensor data, a medical professional may manipulate the patient or the surgical instrument 203 while simultaneously viewing the patient and the display 212 until the drill bit tip 226a is located at a desired entry point in the paint (e.g., as shown in FIG. 4C). In the example of IM nailing, when the drill bit tip 226a is centered over the respective target site 126, the drill bit tip 226a can be at the proper entry point for the locking screw.

[0141] 4B, the display 212 can display a representation 400b of the implant 125 and the anatomical structure 124. Accordingly, the display 212 can be configured to display the position of the cutting tip 226a of the cutting instrument 226 relative to the target site 126. The representation 400b can depict, for example, the position of the cutting tip 226a shown in FIG.

[0142] The cutting tip 226a can be configured to remove anatomical material from one or more target sites 126 of the anatomical structure 124. Additionally, as shown in FIG. 4C , the tip 226a of the cutting instrument 226 (e.g., a drill bit) can be positioned on the anatomical structure 124, such as centered at the target site 126. The display 212 can be positioned to provide a line of sight to both the tip 226a and the display 212 from a location proximal to the surgical instrument 203 such that a medical professional can view both the representations 400b and 400c, the tip 226a, and the anatomical structure 124 to center the tip 226a at the target site 126.

[0143] In some embodiments, for example, based on a user selection via the user interface 216, the surgical instrument assembly 202 can rotate the representational images 400a-400c displayed on the display 212 to a rotational orientation such that a vertical or horizontal direction of the display 212 corresponds to a vertical or horizontal direction, respectively, of the movement of the surgical instrument 203 relative to the anatomical structure 124. Thus, such representational images can be displayed rotated relative to the actual positions of the drill bit, implant, and / or anatomical structure.

[0144] 5A-5C, the display 212 may also be configured to provide a visual indication, e.g., an orientation image 129 of the alignment of the cutting tip 226a relative to a desired trajectory (e.g., a central axis of the target hole 126) based on the x-ray image data and the image sensor data. In one example, the display 212 is configured to display the representational images 400a-400c in real time based on the image sensor data generated in real time by the image sensor 209, and is configured to simultaneously display an orientation screen (e.g., orientation screens 500a-500c) that includes a visual indication of the orientation of the cutting instrument 226. In one embodiment, the display includes two or more displays, each of which displays a different one of the representational images or orientation screens.

[0145] In one embodiment, a user can select options via user interface 216 to select whether a representational image, an orientation screen, or depth gauge data is displayed by display 212. In some embodiments, display 212 can be divided, for example, into two or three, such that any combination of representational image, orientation screen, and depth information can be displayed simultaneously by display 212. It will be understood that the images that can be displayed by display 212 described herein (e.g., FIGS. 4A-4C, 5A-5C, and 10A-20) are not exhaustive. Display 212 can provide various information to a user through various configurations or alternative visual depictions.

[0146] The visual indication of the alignment, e.g., the orientation image 129, can be based on real-time image sensor data of the orientation of the drill bit 226 relative to the fiducial marker 213a (e.g., as shown in FIG. 4D). The visual indication of the alignment can be further based on the desired trajectory relative to the fiducial marker 213a (e.g., the central axis of the target 126).

[0147] For example, with reference to Figures 5A-5C, the orientation screens 500a-500c can include an orientation image 129, which can include a stationary region 130 and a movable indicator 132. The movable indicator 132 can represent an orientation of the cutting instrument 226. In one example, the cutting instrument 226 is oriented with a desired trajectory when the movable indicator 132 has a predetermined spatial relationship to the stationary region 130. In one example, the tip 226a (e.g., a drill bit) of the cutting instrument 226 is aligned with the target site 126 and a hole is drilled in the anatomical structure 124 while the movable indicator 132 has a predetermined spatial relationship to the stationary region 130. It will be appreciated that the predetermined spatial relationship can vary as desired.

[0148] In some cases, the cutting instrument 226 is oriented with a desired trajectory when the movable indicator 132 overlaps with the stationary region 130. As shown in FIG 5C, for example, the cutting instrument 226 is oriented with a desired trajectory when the movable indicator 132 is within the boundary defined by the stationary region 130.

[0149] 5D-5G, alternative alignment graphics are illustrated. For example, FIG. 5D illustrates a visual representation of a circle 240 and an arrow 242 of the alignment of the cutting instrument relative to the desired trajectory (and target site 126). The bullseye 244 may represent the aligned orientation, and the arrow 242 may guide the user to the aligned orientation. FIG. 5E illustrates a visual representation of a bubble level 246 with a stationary center 248 representing the aligned orientation. FIG. 5F illustrates a pair of circles 250, 252, each of which may represent a different portion of the drill bit 226 relative to the target site 126. The drill bit 226 may be aligned when each circle 250, 252 is centered on the target site 126. FIG. 5G illustrates a movable dot 254 that moves relative to a stationary dot 256 with the orientation of the drill bit 226. The movable dot 254 being centered on the stationary dot 256 may indicate that the drill bit 226 is aligned with the target site.

[0150] In one embodiment, the center of the stationary region target may change color when the drill bit tip is located within a predetermined distance from the center of the target 126. For example, the center of the stationary region may change to green. When both the drill bit tip and the drill bit axis alignment are within a predetermined range, the display outline may change color. For example, the display outline may change to green.

[0151] As described above with reference to Figures 4A-4C, the display 212 can display the representational images 400a-400c and a user interface associated with placing a locking screw to secure the IM nail. Referring now to Figures 13-20, the display 212 can additionally or alternatively display the representational images as well as a user interface associated with placing the implant 125, e.g., the IM nail. The display 212 can be configured to display the representational images based on the image sensor data combined with the orientation of the desired trajectory relative to the fiducial marker 213a fixed relative to the anatomical structure 124 as discussed above. For example, representational image 602 (Figures 13, 14A, 14B), representational image 604 (Figure 15), representational image 606 (Figure 16), representational image 608 (Figure 17), representational image 610 (Figure 18), and representational images 630a and 630b (Figure 20).

[0152] As used herein, unless otherwise specified, X-ray image data and X-ray image may be used interchangeably, without limitation. With particular reference to FIG. 14A, the display 212 may display a representation 602 of the anatomical structure 124 (e.g., a representation of X-ray image data of the anatomical structure). According to the illustrated example, the representation 602 may include a representation of the cutting instrument 226 in real time relative to the representation of the anatomical structure 124 (e.g., based on real-time image sensor data based on the orientation of the fiducial marker 213a relative to the image sensor 209). The cutting instrument 226 shown in FIG. 14A may be positioned to open a hole in the anatomical structure 224 for the implant 125.

[0153] In one example, a hole may be drilled to intersect the anatomical structure or IM canal of the bone 124. The hole may thus define an entry point into the bone and a trajectory between the entry point and the IM canal, and the implant 125 (e.g., an IM nail or rod) may be inserted into the hole sized to receive the implant 125. It is recognized herein that the desired trajectory (referred to herein as a "proper trajectory") of the drilling operation and the entry point (e.g., to minimize pain) may vary depending on the type of bone and / or implant to be inserted. It is further recognized herein that the proper trajectory and entry point may not be readily accessible in a given operating room, and thus a given medical professional may rely on personal knowledge to estimate the proper trajectory and entry point. Furthermore, even if the proper trajectory and entry point are known, the drilling operation is typically performed freehand, and thus the actual trajectory and entry point may differ from the proper trajectory and entry point.

[0154] 14B and 17, the processor of the surgical instrument assembly 202 may identify or determine a boundary 614 of the anatomical structure 124, such as a first or anterior-posterior (AP) boundary 615 (FIGS. 14B and 15) or a second or lateral boundary 617 (FIGS. 16 and 17). The boundary 614 may define a first outermost edge 614a of the anatomical structure 124 and a second outermost edge 614b of the anatomical structure 124 opposite the first outermost edge 614a. In some examples, the processor may determine boundary 614 by implementing an edge detection process as described in U.S. Patent Application Publication No. 2007 / 0274584, issued November 29, 2007, and entitled "Method and System for Detection of Bone Fractures," the disclosure of which is incorporated by reference as if set forth in its entirety herein.

[0155] It will be understood that other edge detection algorithms may be implemented as desired, and the above edge detection process is presented for illustrative purposes. In some cases, the processor may identify the boundary 614 based on a user selection via the user interface 216. For example, the display 212 may display an option such as a manual alignment option 646. A user, e.g., a medical professional, may activate the manual alignment option 646, e.g., by touch. When the manual alignment option 646 is activated, the user may manually overlay one or more images on the representational image 602 such that the display 212 displays one or more images (e.g., the longitudinal axis 618 of the anatomical structure 124) on the representational image.

[0156] An example of an image that the user may manually overlay is the boundary 614. By way of example, the user may manually overlay an image on the x-ray data using a stylus, finger, or the like. In one example, the user may activate a manual alignment option 646 to adjust the boundary 614 determined by the processing unit 206 of the surgical instrument assembly 202. For example, the processor unit 206 may perform an edge detection process to determine the boundary 614, but in some cases, the edge detection process may cause a portion of the boundary 614 to deviate from the actual outermost edge of the anatomical structure 124. For example, the edge detection process may erroneously identify a fracture in the anatomical structure 124 as part of the boundary 614. In this example, the user may adjust, via the user interface 216, the portion of the boundary 614 that is erroneously identified as representing the outermost edge of the anatomical structure 124. Thus, the surgical instrument assembly 202 may adjust at least a portion, e.g., all, of the boundary 614 in response to the user activating at least one of the options of the user interface 216.

[0157] As shown in Figures 14B, 15, 16, and 17, the display 212 can overlay a boundary 614 on a representation of the anatomical structure 124 to display the boundary 614 of the anatomical structure 124. With reference to Figures 18 and 20, the processing unit 206 of the surgical instrument assembly 202 can determine an axis 616 of the anatomical structure 124. The processing unit 206 of the surgical instrument assembly 202 can determine a representation of a trajectory 618 that defines an entry point 620 into the anatomical structure. With reference to Figures 14B-18 and 20, the display 212 can overlay a representation of the trajectory 618 on a representation of the anatomical structure 124 to display the representation of the trajectory 618 relative to the anatomical structure 124. The representation of the trajectory 618 can define a line along which a hole can be drilled to intersect with the IM canal of the anatomical structure 124.

[0158] The representation of the trajectory 618 may be determined based on X-ray image data of the anatomical structure 124. For example, the processing unit 206 may determine the trajectory 618 relative to the anatomical structure and the fiducial markers 213a.

[0159] 18 and 20, the display 212 may overlay the axis 616 onto the representations 610, 630a, 630b of the anatomical structure 124 to display the axis 616 of the anatomical structure 124.

[0160] In some embodiments, the axis 616 may define a centerline along the length of the anatomical structure. With reference to FIGS. 14B-17, the trajectory may coincide with the axis 616 such that the representation of the trajectory 618 and the axis 616 may overlap one another. For example, the first outermost edge 614a may be spaced from the second outermost edge 614b to define a width of the anatomical structure substantially perpendicular to the length of the anatomical structure. Thus, the axis 616 may be equidistant from the first outermost edge 614a and the second outermost edge 614b along the length of the anatomical structure 124. In some cases, the processing unit 206 may identify the axis 616 based on a user selection via the user interface 216. As an example, a user, e.g., a medical professional, may activate a manual alignment option 646, e.g., by touch. When manual alignment option 646 is activated, a user may manually overlay one or more images onto the x-ray data, causing display 212 to display one or more images onto the x-ray data. An example of an image that a user may manually overlay is axis line 616. As shown, axis line 616 is represented as a dashed line, although it will be understood that axis line 616 may alternatively be represented as desired, for example, by a solid line. By way of example, a user may manually overlay an image onto the x-ray data using a stylus, finger, or the like.

[0161] In one embodiment, a user can activate a manual alignment option 646 to adjust an axis 616 determined by the processing unit 206 of the surgical tool assembly 202 based on the boundary 614, specifically the first outermost edge 614a and the second outermost edge 614b. Thus, the surgical tool assembly 202 can adjust or determine at least a portion, such as all, of the axis 616 in response to a user activating at least one of the options of the user interface 216. Furthermore, the surgical tool assembly 202 can determine the axis 616 of the anatomical structure 124 based on the boundary 614 of the anatomical structure 124, such that if the boundary 614 of the anatomical structure changes, the axis 616 of the anatomical structure 124 changes according to the change in the boundary 614. For example, the second outermost edge 614b may be adjusted away from the first outermost edge 614a and the surgical instrument assembly 202 may move the axis line 616 toward the second outermost edge 614b such that the axis line 616 may be displayed further away from the first outermost edge 614a compared to when the axis line 616 was displayed before the boundary 614 was adjusted.

[0162] The present disclosure provides embodiments that can reduce the number of x-ray images taken in an operating room, thereby reducing the time it takes to perform a given procedure. In some embodiments described above, as few as a single x-ray image may be taken before the surgeon begins the procedure and is guided based on real-time image sensor data. In other embodiments, as few as two x-ray images may be taken before such a procedure begins.

[0163] 14A-15 and 20, the display 212 may display an x-ray image of the anatomical structure 124 from a first view or an anterior-posterior (AP) view. The processing unit 206 may determine a representation of a trajectory 618 that defines an entry point 620 into the anatomical structure 124. The display 212 may overlay the representation of the trajectory 618 on the representative image of the anatomical structure 124 to display the representation of the trajectory 618.

[0164] In use, the surgeon may use navigation based on real-time image sensor data to indent or otherwise mark the patient's skin with the cutting instrument 225 at an entry point (620). The surgeon may use another instrument to make an incision in the patient's skin to reach the bone.

[0165] In one embodiment, the surgical instrument assembly may include a blade attached to the surgical instrument for making an incision to reach an anatomical structure while continually using navigation displayed based on real-time image sensor data. In some embodiments, the cutting tool includes a separate computer vision module including a cutting image sensor configured to detect fiducial markers and a blade of the cutting tool substantially similar to the image sensor described above with reference to FIGS. 2A-2D. In one embodiment, the image sensor of FIGS. 2A-2D may be configured to be attached to the cutting tool to detect the blade substantially similar to how the image sensor detects the cutting instrument. In some embodiments, the user may use the fiducial markers as landmarks to help visually inform a decision regarding where the incision should be made (e.g., in an x-ray image where the target hole is halfway between marker A and marker B, the user may make the incision halfway between marker A and marker B).

[0166] When navigation is based on image sensor data, the image sensor 209 may detect at least one fiducial marker 213a. The processing unit 206 may, for example, determine the pose of each fiducial marker 213a in a world coordinate system.

[0167] For example, the processor unit 206 may compare the detected pose of each fiducial marker 213a and compare it to the pose determined from the X-ray image data. If the location of any marker(s) 213a is outside of the acceptable deviation (e.g., + / - 0.5mm), such marker(s) may not be used for further processing or navigation steps. If enough markers are within the acceptable deviation, the navigation process may proceed or continue as normal, but without input based on the markers that are outside the acceptable deviation. If the remaining markers are insufficient, the processing unit 206 may generate an alert that is provided to the user and may indicate mitigation options (e.g., by taking a new X-ray image, adjusting the orientation of the image sensor 209, and / or terminating the procedure freehand).

[0168] In one embodiment, the processing unit generates a visual alert, an audible alert, or some other feedback method to indicate to the user that the fiducial markers detected by the image sensor are no longer sufficient to provide accurate navigation. As an example, movement of any one of the fiducial markers detected by the image sensor relative to another fiducial marker may cause the processing unit to generate an alert.

[0169] The processing unit 206 may implement a best-fit algorithm to determine how the fiducial markers 213a detected by the image sensor 209 match a given object board. The processing unit 206 may execute a least-squares algorithm to minimize the reprojection error. If one or more fiducial markers 213a increase the error beyond a threshold amount, such fiducial markers 213a may be determined to have moved and, as a result, not be included in further calculations based on the fiducial markers 213a. If the amount of remaining (i.e., usable) fiducial markers is too small to provide accurate navigation, the processing unit 206 may generate an alert indicating to the user to take mitigating measures. For example, the processing unit 206 may generate a visual or audio alert indicating that the user should replace the fiducial markers 213a and take a new x-ray image to generate new x-ray image data based on the newly placed fiducial markers. In some embodiments, one or more markers may be placed on a static surface, such as an OR table, to provide a fixed reference point for the markers attached to the patient.

[0170] The pose of the cutting instrument 226 may be determined in a world coordinate system based on the pose of the fiducial marker 213a detected by and relative to the image sensor 209. The processing unit 206 may, for example, determine the pose of the cutting instrument 226 in the world coordinate system based on the determined pose of the fiducial marker 213a in the world coordinate system and based on the pose of the cutting instrument 226 relative to the fiducial marker 213a detected by the image sensor 209. The processing unit 206 may determine in real time the distance from the cutting instrument 226 to the desired trajectory (e.g., the axis of the anatomical structure 124 or implant 125) based on the real-time image sensor data.

[0171] The navigation graphics described above may guide the user in real time to shorten the distance from the cutting instrument 226 and the desired trajectory. In one embodiment, the processing unit determines the position of the drill tip and the axis of the cutting instrument in the world coordinate system in real time based on the image sensor data. Thus, the processing unit may determine an orientation or position change for the cutting instrument that will align the cutting instrument with the desired trajectory. The processing unit may generate a display that shows the user how to orient and / or position the cutting instrument to achieve alignment, for example, as described above.

[0172] In one embodiment, only the drill tip of the cutting instrument is determined by the processing unit. The axis of the cutting instrument relative to the image sensor may be predetermined and / or the axis may be determined based on feedback data from an accelerometer of the surgical instrument assembly. In some embodiments, the orientation of the cutting instrument may be determined based on the image sensor data, or vice versa, as a redundancy to double check the feedback data from the accelerometer.

[0173] During a drilling operation, the user may place the drill tip 226a on the bone and, based on the navigation graphics, verify that the cutting instrument 226 is aligned with the desired trajectory (e.g., depicted over the center of the target hole 126 and at a desired angle relative to the target hole 126, as shown in FIG. 18). If the cutting instrument 226 is oriented outside of a predetermined tolerance and / or positioned out of a predetermined position (e.g., as illustrated in FIGS. 15 and 16), the navigation graphics may indicate to the user how to adjust the cutting instrument 226 to achieve alignment.

[0174] In some embodiments, the processor unit 206 may determine a representation of the trajectory 618 in response to a user selection via the user interface 216. For example, the display 212 may display an option such as an auto-align option 622. A user, e.g., a medical professional, may activate the auto-align option 622, e.g., by touch. When the auto-align option 622 is entered, the processing unit 206 of the surgical instrument assembly 202 may determine a representation of the trajectory 618 that defines an entry point 620 into the anatomical structure 124. The surgical instrument assembly may also determine an axis 616 or a boundary 614, or both the axis 616 and the boundary 614, in response to the auto-align option 622 being entered. Further, in response to the automatic alignment option 622 being activated, the display 212 may overlay at least one of the representation of the trajectory 618, the axis line 616, and the boundary 614, e.g., only one of them, e.g., any combination thereof, onto the representation of the anatomical structure 124 to display the representation of the trajectory 618, the axis line 616, and / or the boundary 614.

[0175] In some examples, the surgical instrument assembly 202 can determine a representation of the trajectory 618 based on technical information, for example, technical information stored in the memory 214. Such technical information can include suitable trajectories for drilling holes in various bones for placement of the IM nail. Based on this technical information, the surgical instrument assembly 202 can determine a representation of the trajectory. As an example, the technical information can specify that the trajectory for a given bone from an AP view is 5 degrees lateral from an axis measured from a point just below the lesser trochanter. Continuing with this example, the technical information can specify that the trajectory for a given bone from a lateral view is centered on the greater trochanter and aligned with the medullary canal. In one example, the bone and nail type can be input to the processor via the user interface 216, and a view corresponding to the x-ray image (e.g., left / right view or AP view) can be input to the processor via the user interface 216. In response, the processor can obtain technical information corresponding to the view of the anatomical structure, the bone type, and the nail. Based on the obtained technical information, a trajectory may be determined. In some cases, the processor first determines a boundary 614 and then determines an axis 616 based on the boundary. A representation of a trajectory 618 may be determined based on the axis 616 and the technical information. For example, the technical information may indicate that the trajectory is coincident with the axis 616 in a first view and is angularly offset from the axis by a particular angle in a second view that is substantially perpendicular to the first view (see FIG. 19 ).

[0176] 19, a given user may obtain technical information from the surgical instrument assembly actuating a user selection via the user interface 216. For example, a user selection may cause the display 212 to display technical information 650a and 650b. The technical information 650a may include a graphical depiction of a proper trajectory 652a from an AP view. The technical information 650b may include a graphical depiction of a proper trajectory 652b from a lateral view. The technical information that may be displayed may include textual instructions 654 for placing an IM nail, among other operations. In one example, in response to a user selection, the user interface 216 may render audible instructions associated with, among other operations, an IM nailing operation.

[0177] In some cases, a given user, e.g., a medical professional, may utilize technical information rendered by the surgical instrument assembly 202 to manually overlay a representation of the trajectory 618 on the representational image. For example, the user may activate the manual alignment option 646, e.g., by touch. When the manual alignment option 646 is activated, the user may manually overlay a representation of the trajectory 618 such that the display 212 displays the trajectory 618 on the representational image. The representation of the trajectory 618 may define a solid line, a dashed line, or the like. In one example, the user may activate the manual alignment option 646 to adjust the axis line 616 determined by the processor of the surgical instrument assembly 202 after the automatic alignment option 622 is selected. The surgical instrument assembly 202 may adjust or determine at least a portion, e.g., all, of the representation of the trajectory in response to the user activating at least one of the options of the user interface 216. Thus, the processor of the surgical instrument assembly 202 can adjust the representation of the trajectory to define a new representation of the trajectory, and the display 212 can overlay the new representation of the trajectory on the representation of the anatomical structure to display the new representation of the trajectory. In one example, the processor can adjust the representation of the trajectory in response to a user activating at least one of the options of the user interface 216.

[0178] In some embodiments, the processing unit determines world coordinates of the representation of the trajectory, as discussed above with respect to the desired trajectory. The processing unit may also determine a pose of the cutting instrument relative to the representation of the trajectory based on real-time image sensor data, as discussed above with respect to the desired trajectory.

[0179] 14B, by viewing the representation of the trajectory 618 and cutting instrument 226 visible on the x-ray image 602, the user can move the cutting instrument 226 to align with the representation of the trajectory, as shown in the representation image 604 of FIG 15. In one embodiment, once the cutting instrument 226 is aligned with the representation of the trajectory 618, the processing unit 206 can instruct the display 212 to display a different representation image 606 or 608, which may represent a perspective of the anatomy 124 that is different from the perspective of the representation images 602 and 604. For example, the representation image 606 or 608 may represent a perspective that is perpendicular to the first view or AP view shown in FIG 14B and FIG 15.

[0180] With reference to Figures 14B, 15, and 20, the representation of the trajectory 618 may be referred to as a first representation 618a of the trajectory from a first perspective, e.g., from an AP perspective. In one example, with reference to Figures 16, 17, and 20, the surgical instrument assembly 202 may determine a second representation 618b of the trajectory that defines an entry point 620 into the anatomical structure 124. The second representation 618b may be from a second perspective. By way of example, the second perspective may be approximately periorbital relative to the first perspective, such that the first perspective may define an AP view and the second perspective may define a lateral view. The second representation 618b of the trajectory may be determined and displayed according to any of the embodiments described herein for determining and displaying a representation of the trajectory 618.

[0181] 14B-18, the display 212 can display the position of the cutting tip 226a relative to an entry point 620 of the anatomical structure. By viewing the second representation 618b of the trajectory and the cutting instrument 226 visible on the representational images 606 and 608, the user can move the cutting instrument 226, and thus the cutting tip 226a, to align with the second representation 618b of the trajectory. Alternatively, in an automated scenario, the cutting instrument 226 can be automatically moved to align with the second representation 618b of the trajectory.

[0182] In some cases, once the cutting instrument 226, and thus the cutting tip 226a, is aligned with the first representation of the trajectory 618a and the second representation of the trajectory 618b, the drilling operation may begin when the cutting instrument 226 is aligned with the appropriate entry point and trajectory, which may be determined from the technical information described herein. The display 212 may be positioned to provide a line of sight from a location proximal to the surgical instrument 203 to both the tip 226a and the display 212, such that the medical professional can view both representations, and thus the tip 226a and the anatomical structure 124, in order to center the tip 226a at the entry point 620.

[0183] 18, the display 212 may also be configured to provide a visual indication of the alignment of the cutting instrument 226 with respect to the first representation of the trajectory 618a and the second representation of the trajectory 618b, e.g., the orientation image 129. The visual indication of the alignment, e.g., the orientation image 129, may be based on image sensor data representing the relative orientation of the cutting instrument 226 with respect to the trajectory.

[0184] For example, referring to FIG. 18 , the orientation image 129 can include a stationary region 130 and a movable indicator 132. The movable indicator 132 can represent an orientation of the cutting instrument 226. In one example, the cutting instrument 226 is oriented using the first and second representations 618a and 618b of the trajectory when the movable indicator 132 has a predetermined spatial relationship to the stationary region 130. In one example, the cutting instrument 226 (e.g., a drill bit) is aligned with the first and second representations of the trajectory and a hole is drilled into the anatomical structure 124 while the movable indicator 132 has a predetermined spatial relationship to the stationary region 130. It will be appreciated that the predetermined spatial relationship can vary as desired. In some cases, the cutting instrument 226 is oriented using the first and second representations of the trajectory when the movable indicator 132 is overlaid on the stationary region 130. In some cases, when the movable indicator 132 is within the boundary defined by the stationary region 130, the cutting instrument 226 is oriented using the first and second representations of the trajectory.

[0185] 10A-12, the display 212 may also be configured to provide a visual representation, such as a depth gauge image 262, of the depth of the cutting tip 226a relative to one or more portions of the anatomical structure 124. The processing unit 206 may determine the depth of the cutting tip 226a in real time based on the real time image sensor data. For example, the processor may determine the orientation of the fiducial marker 213a relative to one or more portions of the anatomical structure 124, similar to that discussed above, based on the x-ray image data. The processor may determine the real time position of the cutting tip 226a relative to the fiducial marker 213a, and thereby the position of the cutting tip 226a relative to one or more portions of the anatomical structure 124, based on the real time image sensor data.

[0186] In one example, referring to FIG. 9 , the anatomical structure 124 defines a first cortex, i.e., proximal cortex 123, and a second cortex, i.e., distal cortex 127, opposite the first cortex 123 along a first direction D1, which may be the direction of drilling. The first cortex 123 may define a first surface, i.e., proximal surface 123a, and a second surface, i.e., distal surface 123b, opposite the first surface 123a along the first direction D1. Similarly, the second cortex 127 may define a first surface, i.e., proximal surface 127a, and a second surface, i.e., distal surface 127b, opposite the first surface 127a along the first direction D1, which may also be the direction of drilling and along the desired trajectory. The anatomical structure 124 may define a hollow portion 131. For example, the hollow portion 131 may be defined between the second surface 123b of the first cortex 123 and the first surface 127b of the second cortex 127.

[0187] An implant 125 may be disposed within hollow portion 131. A target hole 126 of implant 125 may be aligned with a resection instrument 226 and a desired trajectory. In one embodiment, the processor determines and directs the display to display a representation of the depth of the cutting tip for one or more portions of the implant based on the real-time image sensor data.

[0188] The visual indication of depth, such as the depth gauge image 262, can change as the cutting instrument 226, and in particular the cutting tip 226a, enters the anatomical structure 124 and / or the implant 125. In particular, the depth gauge image 262 can include data that can change as the cutting instrument tip 226a contacts the first and second surfaces of the first and second cortices 123 and 127, respectively.

[0189] In one embodiment, the depth measurement may be performed using a reflector system with a distance sensor, as discussed above. In some embodiments, the processing unit may track a fiducial marker on the reflector to determine the acceleration that occurs as the cutting instrument drills through the two cortices. The processing unit may determine the length of the drilled bone based on the acceleration. In some embodiments, the distance sensor may be used in conjunction with a processor that determines the acceleration to provide a redundant measurement. In one embodiment, a distance sensor may not be provided, and instead, a processor that determines the acceleration may be used for the depth measurement.

[0190] In an exemplary operation, referring first to FIGS. 10A and 12, which show an exemplary depth gauge screen 1000a and an exemplary split screen 1200, respectively, the depth gauge image 262 may be configured to measure a first distance of a reference site relative to a portion of the anatomical structure 124, and the display 212 may be configured to indicate a second distance of the cutting tip 226a relative to the portion of the anatomical structure 124. The processor may determine the depth based on the first distance as the surgical instrument 203 drills the hole. The display 212 may be configured to indicate the second distance as the surgical instrument drills to indicate the second distance in real time. The first cortex 123 may define a portion of the anatomical structure 124. In one example, the first cortex 123, and in particular the first surface 123a of the first cortex 123, may define a reference site, and the distance from the reference site is determined (e.g., using a distance sensor and / or based on x-ray image data and / or image sensor data) from the reference site. In one example, the cutting tip defines a reference site such that a first distance is equal to a second distance.

[0191] The display 212 can display the depth gauge screen 1000a and the exemplary split screen 1000. In the illustrated example, the total drill depth indication 264 indicates zero (0) when the cutting instrument tip 226a abuts the first surface 123a of the first cortex 123. In one embodiment, the processing unit can be configured such that the total drill depth indication 264 can indicate zero (0) when the drill sleeve abuts the first surface 123a of the first cortex 123.

[0192] The surgical instrument 203 may be configured to drill in a first direction D1 from the first cortex 123 toward the second cortex 127. Thus, the total drill depth display 264 may indicate zero (0) prior to the drilling operation whereby the cutting instrument tip 226a enters the anatomical structure 124 during the drilling operation. With further reference to FIGS. 10B and 11 , which respectively show an exemplary depth gauge screen 1000b and an exemplary split screen 1100 as the drilling operation progresses and the cutting instrument tip 226a moves through the first cortex 123, the total drill depth display 264 may increase to indicate a real-time distance the cutting instrument tip 226a has advanced relative to the first surface 123a of the first cortex 123. As shown, the display of the depth gauge image 262 is rendered in millimeters, although it will be understood that the display may be rendered in any alternative units.

[0193] The depth gauge image 262 may further include a nearest cortical exit point indication 266 indicating a distance from the cutting instrument tip 226a to the distal surface of the most recently pierced cortex. Thus, the display 212 may be configured to indicate a third distance as the cutting tip 226a exits the first cortex 123, which may represent a width of the first cortex 123 along the first direction D1. As an example, the nearest cortical exit point indication 266 indicates a distance from the first surface 123a of the first cortex 123 to the second surface 123b of the first cortex 123 as the cutting instrument tip 226a travels along the first direction D1, which may be a desired trajectory, to exit the second surface 123b of the first cortex 123. Thus, in one example, at the moment the cutting instrument tip 226a advances through the second surface 123b of the first cortex 123, the most recent cortical exit point indication 266 may show the same value as the total drill depth indication 264.

[0194] As drilling is performed, the processor may smooth the image sensor data to minimize errors in navigation guidance caused by vibration, noise, or drill bit deflection. For example, the processor may determine the pose of the cutting instrument 226a as a moving average. In one embodiment, the display may minimize alignment information shown during drilling. The processor may alert the user if the cutting instrument is out of tolerance, even when alignment information is minimized. In some embodiments, the tolerance is determined based on typical vibration, noise, and / or drill bit deflection seen during drilling.

[0195] In some embodiments, the display only displays alignment guidance before drilling, for example, the display may not display alignment or other navigation guidance feedback while drilling is occurring.

[0196] When the cutting instrument tip 226a advances along the first direction D1 to exit the second surface 127b of the second cortex 127, the nearest cortical exit point indication 266 may indicate a distance from the first surface 123a of the first cortex 123 to the second surface 127b of the second cortex 127. Thus, the display 212 may be configured to indicate a fourth distance when the cutting tip 226a exits the second cortex 127, the fourth distance representing the bone width of the bone along the first direction D1. The display 212 may be configured to simultaneously indicate the second distance, the third distance, and the fourth distance. Furthermore, at the moment when the cutting instrument tip 226a advances through the second surface 127b of the second cortex 127, the nearest cortical exit point indication 266 may indicate the same value as the total drill depth indication 264. The depth gauge image 262 may further include a previous cortical exit point indication 268 that displays an indication or value associated with a previous, but not immediately adjacent, cortical exit point.

[0197] Thus, continuing with the example, as the cutting instrument tip 226a exits the second surface 127b of the second cortex 127, the previous cortical exit point 268 may indicate the distance from the first surface 123a of the first cortex 123 to the second surface 123b of the first cortex 123. Thus, the value displayed in the recent cortical exit point indication 266 may transfer to the previous cortical exit point indication 268. As the cutting instrument tip 226a advances away from the second surface 127b of the second cortex 127, the total drill depth indication 264 may increase to indicate the real time distance that the cutting instrument tip 226a has advanced relative to the first surface 123a of the first cortex 123, as illustrated in Figures 10B and 11.

[0198] The user can view the depth gauge image 262 while the surgical instrument 203 is operating, either under user control or autonomously, to better perform the drilling operation. For example, the user can view the total drill depth indication 264 while the drilling operation is being performed to control the surgical instrument based on the total drill depth indication 264. The surgical instrument 203 can be controlled based on the information in the depth gauge image 262 to prevent the cutting instrument 203 from entering undesired portions of the anatomy, such as soft tissue or the far cortex where full or partial drilling is not intended. In some cases, the user can view the depth gauge image 262, particularly the total drill depth indication 264 or the most recent cortical exit point indication 266, to match the screw length with each drilled hole, rather than having to measure the hole after the drilling operation is performed. In one example, the computing device 204 stores an inventory of available screws so that the screws are automatically matched to the holes to be drilled based on the depth of the hole in the anatomy 124. In one example, the user can activate a screw selection option in the user interface 216 such that a screw corresponding to one of the indications on the depth gauge image 262, such as the nearest cortical exit point indication 266 or the total drill depth indication 262, is selected.

[0199] Thus, during operation, the display 212 can receive and display multiple representational images of the anatomical structure in real time based on the real time image sensor data. The display 212 can display the orientation image 129 and the depth gauge image 262, particularly the total drill depth indication 262, as the surgical instrument 203 is manipulated. For example, the depth gauge image 262 can represent the distance the cutting instrument 203 travels. In one embodiment, the representational image, the orientation image, and the depth gauge image can be displayed simultaneously by the display. The distance displayed by the display as the cutting instrument 203 travels along the drilling direction can change (e.g., based on the real time image sensor data) to update the distance in real time.

[0200] In one example, referring to FIG. 6A, the surgical instrument 203 may be manipulated to drill along a first direction D1 that is parallel to a desired trajectory. During drilling, for example, when the orientation of the cutting instrument 226 moves away from a zero value, the movable indicator 132 may move away from the stationary region 130. The movable indicator 132 may move relative to the stationary region 130 at the same time that the orientation of the cutting instrument 226 moves relative to the zero value, thereby providing a real-time representation of the orientation of the cutting instrument 226. For example, when the proximal end 226b of the cutting instrument 226 moves along a second direction D2 relative to the cutting tip 226a of the cutting instrument 226, the movable indicator 132 may move along the second direction D2 (see, for example, FIG. 5A). The second direction D2 may be perpendicular to the first direction D1. Similarly, when the proximal end 226b of the cutting instrument 226 moves along the third direction D3 relative to the cutting tip 226a of the cutting instrument 226, the movable indicator 132 may move along the third direction D3 (see, for example, FIG. 5B). The third direction D3 may be perpendicular to both the first and second directions D1 and D2, respectively. It will be further understood that when the proximal end 226b of the cutting instrument 226 moves along both the second and third directions relative to the cutting tip 226a of the cutting instrument 226, the movable indicator 132 may move along both the second and third directions D3. Additionally, the orientation screens 500a-500c may include a numerical representation 136 of the orientation of the cutting instrument 226 along the second and third directions D2 and D3.

[0201] 5C , when the cutting instrument 226 is oriented according to a zero value, the movable indicator 132 may be positioned within a boundary defined by the stationary region 130. Further, in some cases, when the cutting instrument 226 is precisely aligned with the desired trajectory, the numerical representation 136 may indicate a zero value associated with both the second and third directions. As an IM nailing example, the medical professional may maintain the orientation image 129 shown in FIG. 5C during drilling in order to drill a hole with a proper orientation at the target site 126.

[0202] The above process may be repeated to select or determine a new desired trajectory for drilling a new hole. For example, the C-arm may generate a new X-ray image including a new target portion of the implant or anatomical structure and a fiducial 213a fixed relative to the new target portion. The new desired trajectory may be determined relative to the fiducial 213a and oriented in the world coordinate system based on new X-ray image data of the new X-ray image. The pose of the cutting instrument 226 relative to the new desired trajectory may be determined based on real-time image sensor data. In one embodiment, the original X-ray image data shows a sufficient amount of the new target portion, so the pose of the cutting instrument 226 relative to the new desired trajectory may be determined without a new X-ray image. For example, the user may select a new target portion on the display and set a new desired trajectory without new X-ray image data.

[0203] After drilling is complete, a final x-ray image of the target portion (or portions) may be taken, for example, to confirm that distal locking has been completed as desired. In one embodiment, a fluoroscopic video of the target portion is taken to confirm desired completion by the user.

[0204] 21A-21D, a second embodiment of a surgical instrument assembly is shown. It should be understood that the second embodiment may be similar to the first embodiment of the surgical instrument assembly shown in FIGS. 1-2D. Accordingly, the same reference numbers used above with reference to the first embodiment may also be used in the "primary" notation with reference to the second embodiment. It should also be understood that, unless otherwise noted below, the components (and their features) of the surgical instrument assembly 202' of the second embodiment may be similar to those of the first embodiment.

[0205] The surgical instrument assembly 202' may include a surgical instrument 203, a computing device 204', an image sensor 209', and a mounting member 218' configured to mount the image sensor 209' to the surgical instrument 203. The computing device 204' may not include a measurement device (e.g., the measurement device 211 described above).

[0206] The image sensor 209' may be mounted on top of the mounting member 218' and may be attached to a surgical instrument. The image sensor 209' may be configured to face forward, away from the display 212'. For example, the image sensor 209' may face forward along the longitudinal axis A. The image sensor 209' may be rotatably mounted to the mounting member 218' such that the image sensor 209' may face forward at a downward angle toward the longitudinal axis A or at an upward angle away from the longitudinal axis A. In one embodiment, the image sensor 209' is fixed relative to the mounting member 218' such that the image sensor 209' cannot rotate relative to the mounting member 218'.

[0207] Any of the above processing steps may be performed by the processing unit 206 and / or stored as instructions by the memory unit 214. For example, each detecting, determining, generating, and / or outputting step discussed above may be performed by the processing unit 206 and / or stored as instructions by the memory unit 214. The memory unit 214 may be a non-transitory memory unit.

[0208] Although example embodiments of apparatus for performing the disclosed techniques are described herein, the underlying concepts may be applied to any computing device, processor, or system capable of communicating and presenting information as described herein. The various techniques described herein may be implemented in connection with hardware or software, or, where appropriate, a combination of both. Thus, the methods and apparatus described herein may be implemented, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in a tangible, non-transitory storage medium, such as a floppy diskette, CD-ROM, hard drive, or any other machine-readable storage medium (computer-readable storage medium), which when loaded into and executed by a machine, such as a computer, makes the machine an apparatus for performing the techniques described herein. In the case of program code executed on a programmable computer, a computing device generally includes a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device, such as a display. The display may be configured to display visual information. For example, the displayed visual information may include fluoroscopic data, such as an x-ray image, a fluoroscopic image, an orientation screen, or a computer-generated visual representation.

[0209] The program(s) may be implemented in assembly or machine language, if desired, which may be a compiled or interpreted language, and combined with hardware implementations.

[0210] The techniques described herein may also be implemented via communication embodied in the form of program code transmitted over some transmission medium, such as over electrical or cabling wiring, through optical fiber, or over any other form of transmission. When implemented on a general-purpose processor, the program code combines with the processor to provide a unique apparatus that operates to provide the functionality described herein. In addition, any storage techniques used in connection with the techniques described herein may invariably be a combination of hardware and software.

[0211] In the following description, certain terminology is used merely for convenience and is not limiting. The terms "right", "left", "lower", and "upper" designate directions within the drawings to which reference is made. The terms "inner", "internal", and "interior" refer to directions toward the geometric center of the anatomical structure and / or implant, and the terms "outer", "external", and "exterior" refer to directions away from the geometric center of the implant and / or anatomical structure. The terms "anterior", "posterior", "superior", "lower", "inner", "outer" and related terms and / or expressions are used to designate various positions and orientations within the human body to which reference is made. When these words are used in connection with an implant or anatomical structure, they should be understood to refer to the relative position of the respective anatomical structure or implant implanted within the body as shown in FIG. 9. The terms include the words listed above, derivatives thereof, and words of similar meaning.

[0212] [Embodiment] (1) A surgical instrument assembly comprising: a surgical instrument configured to act on an anatomical structure; an image sensor attached to the surgical instrument, the image sensor configured to detect an orientation of at least one fiducial marker relative to the image sensor, the image sensor configured to generate image sensor data based on the orientation of the at least one fiducial marker detected by the image sensor; A processor; a memory in communication with the processor having stored therein instructions, which, when executed by the processor, cause the processor to: 1) receive x-ray image data representative of a representation of the at least one fiducial marker and an anatomical structure and / or implant; 2) determine a trajectory for the at least one fiducial marker and the anatomical structure and / or implant based on the x-ray image data, the trajectory extending toward the at least one fiducial marker and the anatomical structure and / or implant; and 3) determine an orientation of the surgical instrument relative to the trajectory based on the image sensor data; a display coupled to the processor, the display configured to display a representation of the orientation of the surgical instrument relative to the trajectory. (2) A surgical instrument assembly as described in embodiment 1, wherein the display is attached to the surgical instrument. (3) A surgical instrument assembly as described in embodiment 1 or 2, wherein the display is configured to display in real time the representation of the orientation of the surgical instrument relative to the trajectory based on the image sensor data generated in real time by the image sensor. (4) The surgical instrument assembly of embodiment 3, wherein the instructions, when executed by the processor, further cause the processor to: 1) determine a world coordinate system fixed relative to the at least one fiducial marker based on the X-ray image data; 2) determine an image sensor coordinate system fixed relative to the image sensor; and 3) determine a real-time transformation matrix between the image sensor coordinate system and the world coordinate system based on the image sensor data, whereby the orientation of the surgical instrument relative to the trajectory is determined in real time based on the image sensor data and the real-time transformation matrix. (5) A surgical instrument assembly as described in embodiment 1, wherein the instructions, when executed by the processor, further cause the processor to calculate depth information representative of a depth of the surgical instrument based on the image sensor data.

[0213] (6) A surgical instrument assembly as described in embodiment 1, wherein an orientation of the surgical instrument relative to the image sensor is predetermined and stored in the memory. (7) A surgical instrument assembly as described in embodiment 1, wherein the orientation of the surgical instrument relative to the image sensor is detected by the image sensor and / or another sensor. (8) The surgical instrument assembly of claim 1, wherein the surgical instrument further comprises a cutting instrument. (9) A surgical instrument assembly as described in embodiment 8, wherein the image sensor has a predetermined orientation relative to a cutting instrument of the surgical instrument, and the memory stores the predetermined orientation of the image sensor. (10) A surgical instrument assembly according to claim 8, wherein the image sensor is configured to detect an orientation and / or position of a plate carried by the cutting instrument.

[0214] (11) The surgical instrument assembly of claim 1, wherein the instructions, when executed by the processor, further cause the processor to determine the orientation and position of the surgical instrument relative to the trajectory based on the image sensor data. (12) The surgical instrument assembly of embodiment 1, wherein the trajectory defines an entry point into the anatomical structure and into the opening of the implant. (13) The surgical instrument assembly of claim 12, wherein the implant is an intramedullary nail and the opening defines a distal locking hole of the intramedullary nail. (14) A surgical instrument assembly as described in embodiment 1, wherein the X-ray image data represents at most no more than two X-ray images of the same target area. (15) The surgical instrument assembly of claim 14, wherein the X-ray image data represents no more than a single X-ray image of the same target site.

[0215] (16) The surgical instrument assembly of claim 1, further comprising an X-ray imaging device configured to detect an orientation of the at least one fiducial marker and the anatomical structure and / or the implant, and generate the X-ray image data based on the orientation of the at least one fiducial marker and the anatomical structure and / or the implant. (17) The surgical instrument assembly of embodiment 1, further comprising the at least one reference marker, the at least one reference marker being detectable by an X-ray imaging device and the image sensor. (18) A surgical instrument assembly as described in embodiment 17, wherein the at least one reference marker is an array of ArUco markers. (19) The surgical instrument assembly of embodiment 1, wherein the display is configured to display a representation of a position of a cutting instrument of the surgical instrument relative to a representation of a target opening in the implant based on the image sensor data. (20) A surgical instrument assembly as described in embodiment 1, wherein the image sensor data represents the at least one reference marker, the at least one reference marker being at least one first reference marker, the image sensor data represents one or more second reference markers not represented by the X-ray image data, and the instructions, when executed by the processor, further cause the processor to determine a posture of the one or more second reference markers.

[0216] (21) The surgical instrument assembly of embodiment 20, wherein the image sensor data represents images taken from a plurality of different viewpoints, and wherein the instructions, when executed by the processor, further cause the processor to daisy-chain the images to determine a posture of the one or more second reference markers, such that a posture of a first one of the one or more second reference markers is determined based on a posture of a second one of the one or more second reference markers. (22) The surgical instrument assembly of claim 1, wherein the instructions, when executed by the processor, further cause the processor to: 1) determine an orientation of a distal locking hole of the intramedullary nail relative to the at least one fiducial marker based on the X-ray image data; 3) determine an orientation of a drill bit of the surgical instrument relative to the at least one fiducial marker based on the image sensor data; and 4) display a representation of the orientation of the drill bit relative to the locking hole. (23) A surgical instrument assembly as described in embodiment 1, further comprising a second image sensor. (24) The surgical instrument assembly of embodiment 1, wherein the instructions, when executed by the processor, further cause the processor to determine an orientation of the surgical instrument relative to the trajectory based on the image sensor data, a predetermined orientation of the image sensor relative to the surgical instrument, and a predetermined size and shape of the surgical instrument. (25) The surgical instrument assembly of claim 1, wherein the trajectory is a predetermined trajectory specific to the anatomical structure and stored in the memory.

[0217] (26) The anatomical structure is a bone including an intramedullary canal; the surgical instrument assembly is configured to drill a hole in the bone; 2. The surgical instrument assembly of claim 1, wherein the representation of the trajectory further defines a line along which the hole can be drilled to intersect with the intramedullary canal. (27) A method for indicating an orientation of a surgical instrument, comprising: receiving x-ray image data representative of at least one fiducial marker and representative of an anatomical structure and / or an implant; determining a trajectory for the at least one fiducial marker extending towards the anatomical structure and / or the implant based on the X-ray image data; generating image sensor data based on an orientation of the at least one reference marker relative to the image sensor using an image sensor; determining an orientation of the surgical instrument relative to the trajectory based on the image sensor data; and displaying with a display a representation of the orientation of the surgical instrument relative to the trajectory. (28) The method of claim 27, wherein the display is attached to the surgical instrument. (29) The method of any one of claims 27 to 28, wherein the display displays the representation of the orientation of the surgical instrument relative to the trajectory in real time based on the image sensor data generated in real time by the image sensor. (30) determining a world coordinate system fixed relative to the at least one fiducial marker based on the X-ray image data; determining an image sensor coordinate system based on the image sensor data; 28. The method of claim 27, further comprising determining a real-time transformation matrix between the image sensor coordinate system and the world coordinate system, wherein the orientation of the surgical instrument relative to the trajectory is determined in real time based on the image sensor data and the real-time transformation matrix.

[0218] (31) The method of embodiment 27, further comprising calculating depth information representative of a depth of the surgical instrument based on the image sensor data. (32) The method of embodiment 27, further comprising detecting, using an image sensor, an orientation of the at least one reference marker relative to the image sensor. (33) The method of embodiment 27, wherein the image sensor data is generated in real time by the image sensor. (34) The method of embodiment 27, wherein the X-ray image data is based on a single X-ray image of the at least one fiducial marker and the anatomical structure and / or the target site of the implant. (35) The method of embodiment 27, wherein the image sensor data represents the at least one reference marker, the at least one reference marker being at least one first reference marker, and the image sensor data represents one or more second reference markers not represented by the X-ray image data, and the method further includes determining a posture of the one or more second reference markers.

[0219] (36) The method of embodiment 35, wherein the image sensor data represents images taken from a plurality of different viewpoints, and the method further includes daisy-chaining the images to determine the orientation of the one or more second reference markers, such that an orientation of a first one of the one or more second reference markers is determined based on an orientation of a second one of the one or more second reference markers. (37) determining an orientation of a distal locking hole of the intramedullary nail relative to the at least one fiducial marker based on the X-ray image data; determining an orientation of a drill bit of the surgical instrument relative to the at least one fiducial marker based on the image sensor data; 28. The method of claim 27, further comprising displaying a representation of the orientation of the drill bit relative to the locking hole.

Claims

1. 1. A surgical instrument assembly comprising: a surgical instrument configured to act on an anatomical structure; an image sensor attached to the surgical instrument, the image sensor configured to detect an orientation of at least one fiducial marker relative to the image sensor, the image sensor configured to generate image sensor data based on the orientation of the at least one fiducial marker detected by the image sensor; A processor; a memory in communication with the processor having stored therein instructions, which, when executed by the processor, cause the processor to: 1) receive x-ray image data representative of a representation of the at least one fiducial marker and an anatomical structure and / or implant; 2) determine a trajectory for the at least one fiducial marker and the anatomical structure and / or implant based on the x-ray image data, the trajectory extending toward the at least one fiducial marker and the anatomical structure and / or implant; and 3) determine an orientation of the surgical instrument relative to the trajectory based on the image sensor data; a display coupled to the processor, the display configured to display a representation of the orientation of the surgical instrument relative to the trajectory.

2. The surgical instrument assembly of claim 1 , wherein the display is attached to the surgical instrument.

3. 3. The surgical instrument assembly of claim 1 or 2, wherein the display is configured to display the representation of the orientation of the surgical instrument relative to the trajectory in real time based on the image sensor data generated in real time by the image sensor.

4. 4. The surgical instrument assembly of claim 3, wherein the instructions, when executed by the processor, further cause the processor to: 1) determine a world coordinate system fixed relative to the at least one fiducial marker based on the X-ray image data; 2) determine an image sensor coordinate system fixed relative to the image sensor; and 3) determine a real-time transformation matrix between the image sensor coordinate system and the world coordinate system based on the image sensor data, whereby the orientation of the surgical instrument with respect to the trajectory is determined in real-time based on the image sensor data and the real-time transformation matrix.

5. The surgical instrument assembly of claim 1 , wherein the instructions, when executed by the processor, further cause the processor to calculate depth information representative of a depth of the surgical instrument based on the image sensor data.

6. The surgical instrument assembly of claim 1 , wherein an orientation of the surgical instrument relative to the image sensor is predetermined and stored in the memory.

7. The surgical instrument assembly of claim 1 , wherein the orientation of the surgical instrument relative to the image sensor is detected by the image sensor and / or another sensor.

8. The surgical instrument assembly of claim 1 , wherein the surgical instrument further comprises a cutting instrument.

9. The surgical instrument assembly of claim 8, wherein the image sensor has a predetermined orientation relative to a cutting instrument of the surgical instrument, and the memory stores the predetermined orientation of the image sensor.

10. The surgical instrument assembly of claim 8, wherein the image sensor is configured to detect an orientation and / or position of a plate carried by the cutting instrument.

11. The surgical instrument assembly of claim 1 , wherein the instructions, when executed by the processor, further cause the processor to determine the orientation and position of the surgical instrument relative to the trajectory based on the image sensor data.

12. The surgical instrument assembly of claim 1 , wherein the trajectory defines an entry point into the anatomical structure and into an opening in the implant.

13. The surgical instrument assembly of claim 12, wherein the implant is an intramedullary nail and the opening defines a distal locking hole of the intramedullary nail.

14. The surgical instrument assembly of claim 1 , wherein the x-ray image data represents at most no more than two x-ray images of the same target site.

15. The surgical instrument assembly of claim 14, wherein the x-ray image data represents no more than a single x-ray image of the same target site.

16. 10. The surgical instrument assembly of claim 1, further comprising an x-ray imaging device configured to detect an orientation of the at least one fiducial marker and the anatomical structure and / or the implant and generate the x-ray image data based on the orientation of the at least one fiducial marker and the anatomical structure and / or the implant.

17. The surgical instrument assembly of claim 1 , further comprising the at least one fiducial marker, the at least one fiducial marker detectable by an x-ray imager and the image sensor.

18. The surgical instrument assembly of claim 17, wherein the at least one fiducial marker is an array of ArUco markers.

19. The surgical instrument assembly of claim 1 , wherein the display is configured to display a representation of a position of a cutting instrument of the surgical instrument relative to a representation of a target opening in the implant based on the image sensor data.

20. 2. The surgical instrument assembly of claim 1, wherein the image sensor data represents the at least one fiducial marker, the at least one fiducial marker being at least one first fiducial marker, the image sensor data represents one or more second fiducial markers not represented by the X-ray image data, and the instructions, when executed by the processor, further cause the processor to determine a pose of the one or more second fiducial markers.

21. 21. The surgical instrument assembly of claim 20, wherein the image sensor data represents images taken from a plurality of different viewpoints, and the instructions, when executed by the processor, further cause the processor to daisy chain the images to determine a pose of the one or more second fiducial markers, such that a pose of a first one of the one or more second fiducial markers is determined based on a pose of a second one of the one or more second fiducial markers.

22. 2. The surgical instrument assembly of claim 1, wherein the instructions, when executed by the processor, further cause the processor to: 1) determine an orientation of a distal locking hole of an intramedullary nail relative to the at least one fiducial marker based on the x-ray image data; 3) determine an orientation of a drill bit of the surgical instrument relative to the at least one fiducial marker based on the image sensor data; and 4) display a representation of the orientation of the drill bit relative to the locking hole.

23. The surgical instrument assembly of claim 1 , further comprising a second image sensor.

24. 2. The surgical instrument assembly of claim 1, wherein the instructions, when executed by the processor, further cause the processor to determine an orientation of the surgical instrument relative to the trajectory based on the image sensor data, a predetermined orientation of the image sensor relative to the surgical instrument, and a predetermined size and shape of the surgical instrument.

25. The surgical instrument assembly of claim 1 , wherein the trajectory is a predetermined trajectory that is specific to the anatomy and stored in the memory.

26. the anatomical structure is a bone that includes an intramedullary canal; the surgical instrument assembly is configured to drill a hole in the bone; The surgical instrument assembly of claim 1 , wherein the representation of the trajectory further defines a line along which the hole can be drilled to intersect the intramedullary canal.

27. 1. A method for indicating an orientation of a surgical instrument, comprising: receiving x-ray image data representative of at least one fiducial marker and representative of an anatomical structure and / or an implant; determining a trajectory for the at least one fiducial marker extending towards the anatomical structure and / or the implant based on the X-ray image data; generating image sensor data based on an orientation of the at least one reference marker relative to the image sensor using an image sensor; determining an orientation of the surgical instrument relative to the trajectory based on the image sensor data; and displaying with a display a representation of the orientation of the surgical instrument relative to the trajectory.

28. 28. The method of claim 27, wherein the display is attached to the surgical instrument.

29. 29. The method of claim 27 or 28, wherein the display displays the representation of the orientation of the surgical instrument relative to the trajectory in real time based on the image sensor data generated in real time by the image sensor.

30. determining a world coordinate system fixed relative to the at least one fiducial marker based on the X-ray image data; determining an image sensor coordinate system based on the image sensor data; 28. The method of claim 27, further comprising determining a real-time transformation matrix between the image sensor coordinate system and the world coordinate system, wherein the orientation of the surgical instrument relative to the trajectory is determined in real-time based on the image sensor data and the real-time transformation matrix.

31. 28. The method of claim 27, further comprising calculating depth information representative of a depth of the surgical instrument based on the image sensor data.

32. The method of claim 27 , further comprising detecting, with an image sensor, an orientation of the at least one fiducial marker relative to the image sensor.

33. The method of claim 27 , wherein the image sensor data is generated in real time by the image sensor.

34. 28. The method of claim 27, wherein the X-ray image data is based on a single X-ray image of the at least one fiducial marker and the anatomical structure and / or the target site of the implant.

35. 28. The method of claim 27, wherein the image sensor data represents the at least one fiducial marker, the at least one fiducial marker being at least one first fiducial marker, and the image sensor data represents one or more second fiducial markers not represented by the X-ray image data, the method further comprising determining a pose of the one or more second fiducial markers.

36. 36. The method of claim 35, wherein the image sensor data represents images taken from a plurality of different viewpoints, the method further comprising daisy-chaining the images to determine a pose of the one or more second reference markers, such that a pose of a first one of the one or more second reference markers is determined based on a pose of a second one of the one or more second reference markers.

37. determining an orientation of a distal locking hole of the intramedullary nail relative to the at least one fiducial marker based on the x-ray image data; determining an orientation of a drill bit of the surgical instrument relative to the at least one fiducial marker based on the image sensor data; 28. The method of claim 27, further comprising: displaying a representation of the orientation of the drill bit relative to the locking hole.

Images