Apparatus and method for tracking a handheld surgical tool - Patent Application 20070122997

Abstract

An apparatus and method for tracking a handheld surgical tool is provided. A method for determining the location and orientation of a handheld surgical tool is provided. One embodiment captures image data including an image of a first detectable target on the handheld surgical tool and a second detectable target on the patient. The location and orientation of the first detectable target in 3D space are determined based on the image data. The current location and orientation of the handheld surgical tool in 3D space are determined based on the determined location and orientation of the first detectable target and based on retrieved model data representing the handheld surgical tool. The location of the second detectable target in 3D space is determined based on the image data. The location of the patient's tissue in 3D space relative to the location and orientation of the handheld surgical tool is then determined based on the determined location of the second detectable target on the patient.

Description

[Background technology]

[0001] During a medical intervention, there is a great need for precision in positioning a handheld surgical tool relative to the tissue with which the practitioner will interact. Typically, the practitioner simultaneously views both the working end of the handheld surgical tool and the observable portion of the tissue with which the practitioner is interacting. However, the practitioner may not be able to visually see any portion of the tissue of interest and / or handheld surgical tool or its working end that is not exposed for viewing.

[0002] Various electronic devices and systems have been developed to provide visualization assistance to practitioners while they are using handheld surgical tools to interact with patient tissue. Some systems provide visualization of the position of the working end of a handheld surgical tool by presenting an image on a display viewable by the practitioner. Sensing the location and orientation of the working end of a handheld surgical tool is typically performed using ultrasound devices, computed tomography (CT) scanning devices, X-ray devices, fluoroscopic scanning devices, and / or magnetic resonance imaging (MRI) systems. Each modality comes with its own limitations. For example, X-ray machines, MRI machines, and CT scanning machines are expensive to acquire and use. X-ray images present graphical information on a limited two-dimensional plane. MRI is unsatisfactorily slow, provides low-resolution images of bony structures, and is not very practical during procedures in an operating room or outpatient environment.

[0003] Additionally, X-ray imaging, fluoroscopy, and CT scans use ionizing radiation (X-rays), which can be harmful to the patient and the practitioner performing the surgical procedure in the operating room. The longer the surgical procedure, the more potentially harmful radiation both the patient and the practitioner are exposed to. For example, a patient undergoing a spinal epidural injection will have to undergo prolonged videofluoroscopy. The practitioner may have to wear cumbersome lead shielding suits. The patient will also have to use appropriate lead shielding. Furthermore, fluoroscopes are usually exceptionally large and may leave only a small amount of space available for the practitioner to work in. Intraoperative CT, such as cone beam, has similar downsides.

[0004] Other potentially less harmful electronic scanning systems and devices are available for obtaining patient information. For example, ultrasound devices project sound waves into a patient and detect reflected sound echoes, which are used to generate images called ultrasound images. Ultrasound devices used in ultrasound examination systems generate sound waves at frequencies above the human hearing range, approximately 20 kHz. Sound waves between 2 and 18 MHz are frequently used for ultrasound medical diagnostic applications. Currently, there are no known long-term side effects from examining the human body with ultrasound.

[0005] However, ultrasound scans can only cover a relatively small portion of a patient's body with each scan. Furthermore, ultrasound images are relatively narrow, covering a relatively small cross-section of only a few inches. Furthermore, objects found in ultrasound images can often be blurry. For example, to obtain sufficient image data for analysis of a complete human spine, 500 to 1000 ultrasound images must be captured. Therefore, when a large area of ​​a human subject, such as a patient's spine, must be examined, conventional ultrasound scanners are inadequate for obtaining image information about the patient's body because the ultrasound images are too small and it is not possible to easily analyze many ultrasound images to arrive at any meaningful information about the patient's condition.

[0006] Differentiation of tissues observed in ultrasound images is also problematic. That is, it is very difficult for the practitioner to distinguish which tissues are being observed in the ultrasound image. Typically, for example, when a surgical procedure is being performed on an area near the spine, ultrasound systems will only provide two-dimensional (2D) ultrasound images. Even when real-time acquired ultrasound images provide visual information indicating the current location and orientation of a handheld surgical tool during a surgical procedure, the practitioner will still struggle to identify some types of tissue in the presented ultrasound image. Furthermore, the practitioner will struggle to identify the location and / or orientation of the working end of the handheld surgical tool relative to the tissue (because the tissues are difficult to identify in a 2D ultrasound image).

[0007] The present inventor has created an ultrasound examination system operable to acquire ultrasound image information from a series of ultrasound scans of a patient, such as those disclosed in U.S. Patent Nos. 9,675,321 and 9,713,508, both of which are incorporated herein by reference in their entireties. In practice, a series of time-indexed ultrasound scans are taken over a portion of interest on a patient with underlying bone structures or other ultrasound-discernible organs under examination. The location and orientation of the ultrasound scanner and the location of the scanned portion of the patient are precisely identifiable in the time-indexed ultrasound images acquired from the patient's ultrasound image scanner during the scanning process. Data from the series of acquired time-indexed ultrasound image scans are combined into a single data file that is used to generate a three-dimensional (3D) image and / or a 3D model of the patient's underlying bone structures or organs being examined, referred to herein as a tissue model. However, this system is unsuitable for actual surgical procedures because, for a 3D model of the tissue of interest (a patient's bones and / or organs), many individual ultrasound image scans used to generate the 3D tissue model must be acquired before the surgical procedure begins. The ultrasound inspection systems disclosed in US Pat. Nos. 9,675,321 and 9,713,508 were not designed to sense the location and orientation of handheld surgical tools during a surgical procedure. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] U.S. Patent No. 9,675,321 [Patent Document 2] U.S. Patent No. 9,713,508 Summary of the Invention [Problem to be solved by the invention]

[0009] Therefore, there exists a need in the art for more efficiently acquiring image data that can be presented to a practitioner to indicate in real time the current location and orientation of a handheld surgical tool relative to the tissue of a human subject during a surgical procedure. [Means for solving the problem]

[0010]

[0003] Embodiments of a handheld surgical tool tracking system provide systems and methods for tracking the location and orientation of a handheld surgical tool being manipulated by a practitioner to interact with patient tissue. Exemplary embodiments determine the location and orientation of the handheld surgical tool by capturing image data including images of at least a first detectable target on the handheld surgical tool and a second detectable target on the patient. The location and orientation of the first detectable target in 3D space are determined based on the image data. The current location and orientation of the handheld surgical tool in 3D space are determined based on the determined location and orientation of the first detectable target and based on retrieved model data representing the handheld surgical tool. The location of the second detectable target in 3D space is determined based on the image data. The location of the patient's tissue in 3D space relative to the location and orientation of the handheld surgical tool is then determined based on the determined location of the second detectable target on the patient. In some embodiments, the relative location of the handheld surgical tool relative to the patient's tissue is determined. A composite image (based on a predetermined model of the tissue) showing the handheld surgical tool and an image of the tissue is then presented on a 2D and / or 3D display.

[0011] The elements in the drawings are not necessarily to scale relative to each other and like reference numerals indicate corresponding parts throughout the several views. [Brief explanation of the drawings]

[0012] [Figure 1A] 1A-1C are diagrams of various types of handheld surgical tools having one or more detectable targets at known locations on the face of the handheld surgical tool. [Figure 1B] 1A-1C are diagrams of various types of handheld surgical tools having one or more detectable targets at known locations on the face of the handheld surgical tool. [Figure 1C] 1A-1C are diagrams of various types of handheld surgical tools having one or more detectable targets at known locations on the face of the handheld surgical tool. [Figure 1D]1A-1C are diagrams of various types of handheld surgical tools having one or more detectable targets at known locations on the face of the handheld surgical tool. [Figure 1E] 1A-1C are diagrams of various types of handheld surgical tools having one or more detectable targets at known locations on the face of the handheld surgical tool. [Figure 2] FIG. 1 is a schematic diagram of a handheld surgical tool tracking system for assisting a practitioner performing a procedure on a human subject. [Figure 3] FIG. 1 is a conceptual diagram of a presented composite image showing the relative locations of a patient's spine and a handheld surgical tool. [Figure 4] FIG. 1 is a conceptual diagram of an embodiment of a handheld surgical tool tracking system and a robotic surgery system that work cooperatively together to generate a composite image including the robotic tool and the handheld surgical tool. DETAILED DESCRIPTION OF THE INVENTION

[0013] 1A-1E are illustrations of various non-limiting exemplary types of handheld surgical tools having one or more detectable targets 102a, 102b at known locations on the surface of the handheld surgical tools 104a-104e. In a preferred embodiment, the detectable targets 102a, 102b are optically detectable, although the detectable targets 102a, 102b may also be detectable using non-optical systems. FIGURE 2 is a schematic diagram of a handheld surgical tool tracking system 100 for assisting a practitioner performing a surgical procedure on a human subject 200 (interchangeably referred to herein as a patient 200).

[0014] During a surgical procedure, an embodiment of the handheld surgical tool tracking system 100 determines the location and orientation in three-dimensional (3D) space of the handheld surgical tool 104, particularly its working end (tool tip). Thus, the location and orientation in 3D space of the handheld surgical tool 104 being manipulated by the surgeon during the surgical procedure can be determined in real time or near real time. Based on a predetermined model of the tissue of interest on which the surgeon is operating, real-time or near-real-time renderings and presentations of images depicting the interaction of the working end 106 of the handheld surgical tool 104 with a two-dimensional (2D) or three-dimensional (3D) model of the tissue of the patient 200 are presented on the display. Thus, the surgeon can intuitively understand the actual location of the working end 106 of the handheld surgical tool 104 relative to the tissue being operated on.

[0015] The presented images of the tissue and handheld surgical tool 104 can be provided as 2D and / or 3D images. In some embodiments, the practitioner (or an assistant) wears virtual reality glasses to view a three-dimensional image of the working end 106 of the handheld surgical tool 104 interacting with the tissue of interest. Multiple displays can simultaneously present the 2D and / or 3D images so that other parties can observe the surgical procedure in progress. Furthermore, these time-series 2D and / or 3D images can be saved for later review.

[0016] The presently disclosed systems and methods for or assisting a practitioner in performing a procedure on a human subject 200 ( FIG. 2 ) using a handheld surgical tool tracking system 100 will be better understood by examining the following detailed description in conjunction with the figures. These detailed descriptions and figures provide examples of the various inventions described herein. Those skilled in the art will understand that changes, modifications, and adaptations can be made to the presently disclosed examples without departing from the scope of the inventions described herein. Many variations are anticipated to meet various application and design requirements, and the following detailed description will not, for the sake of brevity, describe each and every possible variation individually.

[0017] Various exemplary handheld surgical tool tracking systems 100 are provided throughout the following detailed description. Related features in these examples may be identical, similar, or dissimilar in the various examples. For purposes of brevity, related features will not be described redundantly in each example. Instead, related feature names are used to inform the reader that a feature having a related feature name may be similar to a feature in a related, previously described example. Features unique to a given example will be described below in that particular example. The reader should understand that a given feature need not match or resemble the specific description of the feature in any given figure or example.

[0018] As used herein, the following definitions apply unless otherwise indicated.

[0019] "Substantially" means roughly corresponding to a particular size, area, shape, concept, or other aspect modified by the term, and thus does not require that the features or components match exactly. For example, "substantially cylindrical" means that an object resembles a cylinder, but may have one or more deviations from a true cylinder.

[0020] The terms "comprising," "including," and "having" (and their conjugations) are used interchangeably to mean "including, but not necessarily limited to," and are open-ended terms that are not intended to exclude additional elements or method steps not expressly recited.

[0021] Terms such as "first," "second," and "third" are used to distinguish or identify various members of a group, etc., and are not intended to imply any ordering, chronological order, or numerical limitation.

[0022] "Coupled" means connected, either permanently or removably, whether directly or indirectly through intervening components. "Fixed to" means connected directly, without using intervening components.

[0023] "Communicatively coupled" means that an electronic device exchanges information with another electronic device, whether directly or indirectly through a communications network, either wirelessly or by a wire-based connector. "Controllably coupled" means that an electronic device controls the operation of another electronic device.

[0024] "Real time" means that it is immediate, whereas "near real time" means that there is a delay of some amount of time (whether that be by a few milliseconds or even less).

[0025] In various embodiments, like reference numbers represent like components. For example, reference number 104 may be used to generally represent any handheld surgical tool, and reference number 104a may be used to identify a particular handheld surgical tool, such as exemplary scalpel 104a (FIG. 1A).

[0026] The optical target 102 (interchangeably referred to herein as the detectable target 102) can be any conventional, custom-developed, or later-developed optical target that is distinguishable by the optical tracking system of the handheld surgical tool tracking system 100. In some examples, the optical target 102 can extend in three dimensions along three coordinate axes and include a separate optical target portion representing each axis. In other examples, the optical target 102 can extend in three dimensions along six axes and include a separate optical target portion representing each of the six axes. The optical target 102 is distinguishable to the human eye and distinguishable in a captured image (photographic or video image).

[0027] The detectable target 102 may be active, such as by emitting (or reflecting) an electromagnetic or other energy signal from itself. Alternatively, the detectable target 102 may be passive, such as a retroreflective marker that reflects radiation energy emitted by any interaction device. Although such active or passive targets 102 are generally referred to herein as detectable targets 102 for purposes of brevity, such detectable targets 102 may not be optically detectable by an image capture device. Any suitable detectable target 102, now known or later developed, is intended to be within the scope of this disclosure and protected by the accompanying claims.

[0028] In various embodiments, each of the detectable targets 102 is uniquely identifiable. In some embodiments, the visual characteristics of the optical target (or at least one optical target) can be compared to known identifying characteristics of that optical target, and therefore, a particular optical target 102 can be identified. For example, the optical targets 102 can be different in shape. If the optical targets 102 are small marks, such as dots, the number of marks on each optical target 102 can identify such a particular optical target 102. A unique alphanumeric identifier can be positioned around or on the optical target 102. An active detectable target 102 can emit various signals bearing identifier information. In embodiments using only a single optical target 102, such a single optical target 102 is inherently identifiable.

[0029] As conceptually illustrated in FIG. 2 , a non-limiting exemplary embodiment of the handheld surgical tool tracking system 100 includes a processor system 202, at least one handheld surgical tool 104, and an optional remote rendering and display system 204. The processor system 202 includes at least one image capture device 206, a target tracking unit 208, an image registration module 210, an image processing algorithm module 212, a 3D / 2D visualization module 214, a surgical tool database 216, a tissue model database 218, and a user input device 220. Some embodiments include an optional clock 222. Some embodiments may include an optional display 224. Any suitable processor system, now known or later developed, is intended to be within the scope of this disclosure and protected by the accompanying claims. In some embodiments, the processor system 202 can be communicatively coupled to the remote rendering and display system 204 via a wire-based or wireless connection 226. Other embodiments of the handheld surgical tool tracking system 100 may include some or exclude some of the components described above. Additionally, alternative embodiments may include additional components not described herein.

[0030] In an exemplary embodiment, the distinguishable target 102 is an optical target 102, where the optical target is distinguishable to a human observer and detectable in a photographic image, the image capture device 206 is a still camera device or a video camera device, and the acquired image data is still image photographic data or photographic video data.

[0031] 1A-1E depict a non-limiting selection of exemplary handheld surgical tools 104a-104e. Any particular handheld surgical tool may include one or more detectable targets 102. Each type of handheld surgical tool 104 is uniquely identifiable. An identifier for the handheld surgical tool 104 may be located on each handheld surgical tool 104. The identifier may be an alphanumeric identifier or bar or QR code, etc., that can be identified to the handheld surgical tool tracking system 100.

[0032] For example, a captured image or scan of the handheld surgical tool 104 can be analyzed by the handheld surgical tool tracking system 100 using any suitable alphanumeric text character recognition algorithm to determine the identifier of the handheld surgical tool 104. Alternatively or additionally, a radio frequency identification tag (RFID) can be placed on the handheld surgical tool 104 and / or its packaging that can be read by an RFID reader included in or communicatively coupled to the handheld surgical tool tracking system 100. In some embodiments, an object recognition algorithm can be used to identify the surgical tool 104 based on the captured image data. Alternatively or additionally, a practitioner or another party can manually input the identification of the handheld surgical tool 104's identifier into the handheld surgical tool tracking system 100 (e.g., if the identification is entered using a keyboard, a touch-sensitive screen, or spoken into an audible sound detection recognition system). Once the identification result of the practitioner handheld surgical tool 104 is input into the handheld surgical tool tracking system 100, handheld surgical tool model data corresponding to the identified surgical tool 104 (handheld surgical tool model) can be retrieved so that the processor system 202 can determine the precise location and orientation of the identified handheld surgical tool 104, particularly the working end 106 positioned at its distal end, during the surgical procedure.

[0033] Returning to FIG. 1A , a particular handheld surgical tool 104 can be identified as a scalpel 104a. The scalpel 104a has a working end 106a (interchangeably referred to herein as tool 106a or blade 102a), which will be understood by those skilled in the art to be a cutting edge or blade 1061, located at its distal end. In various preferred embodiments, prior to using the scalpel 104a during a surgical procedure, one or more identifiable optical targets 102 are placed (or fabricated in known locations) on the face of the scalpel 104a. Preferably, each of the optical targets 102 will be uniquely identifiable.

[0034] In various embodiments, a variety of different types of detectable targets 102 can be used. A first, non-limiting type is an exemplary optical target 102a having multiple distinct optical target portions. When the optical target 102a is observed in a captured image (or, more specifically, when image data corresponding to the optical target 102a is analyzed by the processor system 202), the location and orientation (i.e., relative locations to one another) of the optical target 102a can be precisely determined based on image characteristics of the target portions of the image. Because the orientation and location of the identified optical targets 102a relative to all portions of the scalpel 104a, particularly the location of the blade 106a, are known, a stored model corresponding to the scalpel 104a can be used to determine the precise location and orientation of the blade 106a in 3D space.

[0035] 1A, a model of the scalpel 104a with data corresponding to the exemplary optical target 102 is pre-determined and stored. The location of each part of the scalpel 104a relative to the optical target 102a, particularly the precise location and orientation of the blade 106a, is represented in the model data of the scalpel 104a. When acquired image data of the scalpel 104a is correlated with the scalpel model data, an embodiment of the handheld surgical tool tracking system 100 determines the precise location and orientation of the scalpel 104a in 3D space, particularly the precise location and orientation of the blade 106a.

[0036] Alternatively or additionally, multiple optical targets 102b can be positioned on the face of the handheld surgical tool 104. Such optical targets 102b can be simple marks, such as one or more small colored dots, squares, triangles, or stars, that can be identified in image data acquired by the image capture device 206. Various colors and / or patterns can be used on the marks. In various embodiments, any suitable shape, color, and / or pattern can be used, particularly for mark identification purposes. The precise location and identity of each identifiable optical target 102b on the face of the handheld surgical tool 104 is known using a stored model corresponding to the scalpel 104a. Two optical targets 102b and 102b' are illustrated for the scalpel 104a (where the characteristics of optical target 102b are different from those of optical target 102' for identification purposes). With at least two optical targets 102b identified in the captured image data, the precise location and orientation of the scalpel 104a in 3D space can be calculated (determined) by the processor system 202 based on the relative locations of the identified at least two optical targets 102b with respect to one another. Additional optical markers 102b can be used to improve the accuracy of the determination of the location and orientation of the scalpel 104a.

[0037] In this non-limiting example, conceptually shown in FIG. 1A , a model of the scalpel 104a having exemplary optical targets 102b and 102b′ is pre-determined and stored. The location of each part of the scalpel 104a relative to the optical targets 102b and 102b′, particularly the precise location and orientation of the blade 106a relative to the optical targets 102b and / or 102b′, is represented in model data of the scalpel 104a. When acquired data of the scalpel 104a is correlated with the scalpel model data, an embodiment of the handheld surgical tool tracking system 100 determines the precise location and orientation of the scalpel 104a in 3D space, particularly the precise location and orientation of the blade 106a. Those skilled in the art will recognize that the optical targets 102b and 102b′ are preferably uniquely identifiable so that the location and orientation of the working end 106 of the handheld surgical tool 104 can be determined. If the optical targets 102b and 102b′ are indistinguishable from one another, it can be assumed that an inaccurate orientation and location of the surgical tool 104 will be determined. The use of uniquely identifiable optical targets 102b and 102b' solves this potential problem.

[0038] In FIG. 1B , the particular handheld surgical tool 104 can be seen to be a general-purpose syringe 104b. The syringe 104b has a working end 106b, which those skilled in the art will understand to be the tip of a needle located at its distal end and used to puncture a patient to inject a medication or withdraw a tissue sample. In various preferred embodiments, prior to using the syringe 104b during a surgical procedure, one or more identifiable optical targets 102a and / or 102b are placed (or created) in known locations on the face of the syringe 104b. Preferably, each of the optical targets 102a and / or 102b will be uniquely identifiable. (It is appreciated that in other embodiments, one or more optical targets 102a and / or three or four or more optical targets 102b could be used.)

[0039] Additionally, some embodiments can be configured to determine the relative distance between optical targets 102b and 102b' during a surgical procedure. For example, optical target 102b can be positioned on plunger 110 of syringe 104b. A second optical component target 102b' can be positioned on barrel 112 of syringe 104b. In this case, changes in the distance between optical targets 102b and 102b' during the surgical procedure can be used to accurately calculate the distance traveled by barrel seal 114 at the distal end of plunger 110. Because the volume of barrel 112 is known and this volume information is stored as part of the syringe model data, the volume of medication injected from syringe 104b can be accurately determined based on the determined distance traveled by barrel seal 114. This information can be provided to the surgeon in real time during the surgical procedure.

[0040] 1C, the particular handheld surgical tool 104 can be seen to be a general-purpose surgical scissors 104c. The surgical scissors 104c have a first working end 106c and a second working end 106c' (located at the distal end of the surgical scissors 104c), which will be understood by those skilled in the art to be cutting ends used to cut tissue during a surgical procedure. In various preferred embodiments, prior to using the surgical scissors 104c during a surgical procedure, a plurality of identifiable optical targets 102 are placed (or fabricated in known locations) on the face of the surgical scissors 104c. Preferably, each of the optical targets 102 will be uniquely identifiable.

[0041] Those skilled in the art will recognize that precise determination of the location of the cutting edges of the surgical scissors 104c requires individually determining the precise location of each working end 106c, 106c'. To enable this determination of the precise location and orientation of the surgical scissors 104c and working ends 106c, 106c' during a surgical procedure, each of the two first surgical tool members 108c, 108c' of the surgical scissors 104c should have its own optical target 102. In the non-limiting exemplary embodiment shown in FIG. 1C, the first surgical tool member 108c includes a first optical target 102b and a second optical target 102b'. The second surgical tool member 108c' includes a third optical target 102b'' and a fourth optical target 102b'''. The determined locations of optical targets 102b", 102b'" can be used to determine the precise location and orientation of working end 106c. Similarly, the determined locations of optical targets 102b, 102b' can be used to determine the precise location and orientation of working end 106c'. Alternatively or additionally, the relative separation between optical targets 102b and 102b" and / or between optical targets 102b' and 102b'" can be used in this calculation.

[0042] Alternatively or additionally, multiple optical targets 102a (not shown) can be used to track members 108c, 108c'. Three optical targets 102b can be used (see, for example, FIG. 1D).

[0043] Indeed, at some point during a surgical procedure, the surgical scissors 104c may be closed, with the working ends 106c and 106c' adjacent and / or touching one another. At another point during a surgical procedure, the surgical scissors 104c may be opened such that the working ends 106c and 106c' are separated from one another by a determinable distance so that the practitioner can insert the working ends 106c, 106c' around the tissue to be cut. Furthermore, as the practitioner is cutting the tissue of interest, the working ends 106c and 106c' move toward one another in a cutting motion. Thus, the precise location and orientation of the surgical scissors 104c, and particularly the working ends 106c, 106c', can be determined in real time or near real time.

[0044] 1D , the particular handheld surgical tool 104 can be seen to be a general-purpose surgical clamp 104d. The surgical clamp 104d has a first working end 106d and a second working end 106d′ (located at the distal end of the surgical clamp 104d), which will be understood by those skilled in the art to be clamps used to fasten tissue during a surgical procedure. In various preferred embodiments, prior to using the surgical clamp 104d during a surgical procedure, multiple identifiable optical targets 102 are placed (or fabricated) in known locations on the surface of the surgical clamp 104d. Preferably, each of the optical targets 102 will be uniquely identifiable.

[0045] Those skilled in the art will recognize that, similar to the surgical scissors 104c described above, precise determination of the location of the fastening ends of the surgical clamp 104d requires individually determining the precise location of each working end 106d, 106d'. For example, the surgical clamp 104d may be opened so that the working ends 106d, 106d' are separated from one another by a determinable distance so that the practitioner can insert the working ends 106d, 106d' around the tissue to be fastened. At another point during the surgical procedure, the surgical clamp 104d may be closed, in which case the working ends 106d, 106d' are adjacent and / or in contact with one another to fasten the tissue of interest. Furthermore, as the practitioner fastens the tissue of interest, the working ends 106c, 106c' move toward one another in a fastening motion. Thus, the precise location and orientation of the surgical clamp 104d, and particularly the working ends 106d, 106d', may be determined in real time or near real time.

[0046] In this non-limiting example of surgical clamp 104d, each of the two surgical tool members 108d and 108d' of surgical clamp 104d is conceptually illustrated as having its own optical target 102a to enable determination of the precise location and orientation of surgical clamp 104d and working end 106d, 106d' during a surgical procedure. In the non-limiting exemplary embodiment shown in FIG. 1D , the first surgical tool member 108d includes a first optical target 102a. The second surgical tool member 108d' includes a second optical target 102b'. Once the precise location and orientation of each arm 108d, 108d' is determined, the precise location and orientation of surgical clamp 104d and associated working end 106d, 106d' can be determined by processor system 202.

[0047] Alternatively or additionally, to enable precise location and orientation determination of surgical clamp 104d and working end 106d, 106d' during a surgical procedure, each of two surgical tool members 108d and 108d' of surgical clamp 104d is conceptually illustrated as having its own optical target 102b and 102b', respectively. A third optical target 102b'' is positioned at the hinge location of surgical clamp 104d (where first surgical tool member 108d is hingedly coupled to second surgical tool member 108d''). In this case, the precise location and orientation of first surgical tool member 108d (first arm 108d) can be determined based on the determined locations of first optical target 102b and hinge optical target 102b''. Similarly, the precise location and orientation of the second surgical tool member 108d' can be determined based on the determined locations of the first optical target 102b' and the hinge optical target 102b". Once the precise location and orientation of each surgical tool member 108d, 108d' is determined, the precise location and orientation of the surgical clamp 104d and associated working end 106d, 106d' can be determined in real time or near real time by the processor system 202.

[0048] As noted herein, a handheld surgical tool 104 can have only one type of detectable target 102 or multiple different types of detectable targets 102. For example, non-limiting surgical clamp 104d uses two different types of optical targets 102a and 102b. Other embodiments of surgical clamp 104d may use only one type of optical target 102 (and / or may use other types of optical targets 102). The use of different types of optical targets 102 allows the same handheld surgical tool 104 to be used with different embodiments of handheld surgical tool tracking system 100 that use different image processing algorithm models 212 configured to identify specific types of optical targets 102. This feature of the detectable target 102 can be used on any handheld surgical tool 104.

[0049] 1E, the particular handheld surgical tool 104 can be seen to be a general-purpose surgical forceps 104e. The surgical forceps 104e have a first working end 106e and a second working end 106e' (located at the distal end of the surgical forceps 104e), which will be understood by those skilled in the art to be grasping tool ends used to grasp tissue during a surgical procedure. In various preferred embodiments, prior to using the surgical forceps 104e during a surgical procedure, multiple identifiable optical targets 102 are placed (or fabricated in known locations) on the surface of a surgical clamp 104d. Preferably, each of the optical targets 102 will be uniquely identifiable.

[0050] In this non-limiting example of surgical forceps 104e, each of the two surgical tool members 108e and 108e' of surgical forceps 104e is conceptually illustrated as having its own optical target 102a, 102a', respectively, to enable determination of the precise location and orientation of surgical forceps 104e and working ends 106e, 106e' during a surgical procedure. Once the precise location and orientation of each surgical tool member 108e, 108e' is determined, the precise location and orientation of surgical forceps 104e and associated working ends 106e, 106e' can be determined by processor system 202.

[0051] Those skilled in the art will recognize that there are a variety of different types of handheld surgical tools that may be used by a practitioner during a surgical procedure. Accordingly, the exemplary handheld surgical tool 104 of Figures 1A-1E is intended to represent only a selection of handheld surgical tools that may be tracked using the detectable target 102. Additionally, some handheld surgical tools may have working ends located at both their proximal and distal ends (such as those found in dental technology). All such manually operated handheld surgical tools 104, now known or later developed, are intended to be within the scope of this disclosure and protected by the accompanying claims.

[0052] Some embodiments may use multiple different detectable targets 102a, 102b positioned on other surfaces of the handheld surgical tool 104. For example, the detectable targets 102a, 102b may be positioned on opposite surfaces of the scalpel 104a and / or on the sides of the scalpel 104a. For example, if the practitioner's hand or another object obstructs the image capture device 206 from capturing an image of the handheld surgical tool 104 during a surgical procedure, multiple detectable optical targets 102a, 102b may be present on the same surface of the handheld surgical tool 104. This ensures that a sufficient number of detectable optical targets 102a, 102b are always distinguishable in the image captured by the image capture device 206 ( FIG. 2 ). Thus, the practitioner does not need to worry about how to grip the handheld surgical tool 104 during a surgical procedure.

[0053] As noted herein, the precise location and orientation of the working end 106 of any handheld surgical tool 104 is precisely determined based on the determined locations of one or more detectable targets 102. Those skilled in the art will understand how such location and orientation calculations are performed to identify the precise location and orientation of the handheld surgical tool 104 within a known 3D space based on the determined locations of detectable targets 102, such as exemplary optical targets 102 a, 102 b. Accordingly, such shape calculations will not be described herein for the sake of brevity.

[0054] Prior to performing a surgical procedure using any one of the available handheld surgical tools 104, the precise location of each detectable target 102 on the surface of the handheld surgical tool 104 is known. The known location of each detectable target 102 may be based on design specifications. During manufacturing, each of the detectable targets 102 is fabricated in a precisely known location and / or orientation. An unexpected advantage of incorporating detectable targets 102 as part of the handheld surgical tool 104 during manufacturing is that hundreds, and sometimes thousands, of similar handheld surgical tools 104 can be manufactured and distributed to different sites. If the surgical procedure site has an embodiment of the handheld surgical tool tracking system 100, the precise location and orientation of the handheld surgical tool 104 can be determined during the surgical procedure. However, even if the site does not have an embodiment of the handheld surgical tool tracking system 100, the practitioner can still perform the surgical procedure using the handheld surgical tool 104 in a conventional manner.

[0055] Alternatively or additionally, the detectable targets 102 can be placed on the surface of the handheld surgical tool 104 after its manufacture, where each detectable target 102 is affixed (such as with an adhesive or by painting) to the surface of the handheld surgical tool 104 in a precisely known location and / or orientation. Preferably, the detectable targets 102 are affixed in the same location to many similar handheld surgical tools 104 before distribution to different sites where surgical procedures will be performed.

[0056] An unexpected advantage of distributing many similar handheld surgical tools 104, each having a detectable target 102 identically and in the same location and / or orientation on the handheld surgical tool 104, is the generation and storage of a single model for a particular handheld surgical tool 104. The handheld surgical tool model data can be stored locally in the surgical instrument model database 216 (FIG. 2) and / or remotely in a remote database.

[0057] For example, thousands of scalpels 104a can be manufactured and distributed to many different surgical sites (e.g., hospitals, clinics, etc.). If the surgical site where the surgical procedure is being performed is equipped with an embodiment of the handheld surgical tool tracking system 100, the practitioner (or his or her assistant) can use any of the available scalpels 104a during the surgical procedure. Furthermore, different types of handheld surgical tools 104 can be used by the practitioner during the surgical procedure, as each different type of handheld surgical tool 104 is readily identifiable by an embodiment of the handheld surgical tool tracking system 100.

[0058] Yet another unexpected advantage is that for any particular type of handheld surgical tool 104 having a precisely positioned detectable target 102, different manufacturers can provide their own unique tools 104 to practitioners for different surgical procedures. If the manufacturers produce identical handheld surgical tools 104, the handheld surgical tools 104 distributed by different manufacturers can be represented by a single model. If there are differences between handheld surgical tools 104 of a particular type produced by different manufacturers, a handheld surgical tool model for each particular manufacturer can be created and saved.

[0059] Alternatively or additionally, the detectable target 102 can be affixed (using a suitable adhesive or paint) to the handheld surgical tool 104 of interest by the practitioner or another party prior to use. For example, after manual placement of the optical target 102, one or more images of the handheld surgical tool 104 are captured during a handheld surgical tool calibration process. An embodiment of the handheld surgical tool tracking system 100 then analyzes the captured image data to identify the precise location and / or orientation of the optical target 102 and the location of the working end 106 of the handheld surgical tool 104. An embodiment of the handheld surgical tool tracking system 100 can then generate a calibration model of the photographed handheld surgical tool 104. An unexpected advantage of this embodiment is that one or more optical targets 102 can be attached to any handheld surgical tool 104 of interest and then used during a surgical procedure.

[0060] 2, a patient 200 is lying prone on an operating table or the like. Assume that a practitioner (not shown) wishes to insert needle 106b (working end) of syringe 104b into an intended location within the patient's spinal column (not shown). While the precise location and orientation of syringe 104b in 3D space can be determined, the precise location of the patient's tissue of interest (in this case, the patient's spinal column) is obscured by the patient's covering skin.

[0061] Therefore, calibration of the patient 200 is necessary before the start of a surgical procedure. In the conceptual diagram shown in FIG. 2 , at least one detectable target 228 is placed on the patient 200 or at another suitable location around the patient 208. If an optical target 228 is used, the image capture device 206 acquires an image of the optical target 228 and then determines the precise location and orientation of the optical target 228. Because the optical target 228 is positioned at a known location relative to (and around) the patient, the location of the tissue of interest can be calculated based on information known about the patient 200. That is, the location relationship between the tissue of interest on the patient 200 and the fixed position of the detectable target 228 is known. Once the location of the tissue of interest is known, the relative location relationship between the tissue and the handheld surgical tool 104 can be determined during the surgical procedure. This patient tissue calibration scheme is believed to be sufficient when the tissue of interest is visible on the surface of the patient 200 and / or when high precision is not required. For example, this patient tissue calibration technique may be sufficient when the tissue of interest is a birthmark or the like on the skin of the patient 200. The localization of the tissue of interest may be improved by embodiments of the handheld surgical tool tracking system 100 that use object recognition techniques. For example, an image of the patient's skin may be analyzed to determine the location of the birthmark on the patient's skin.

[0062] However, those skilled in the art will recognize that the above-described patient tissue calibration may not be sufficient to identify the precise location and orientation of other types of tissue of interest, particularly those that are internal and / or particularly sensitive. For example, if needle 106b is to be used to puncture the spinal column of patient 200, a very precise patient calibration is required.

[0063] In some embodiments, patient calibration is performed using a separate device or an embodiment of the handheld surgical tool tracking system 100. By way of non-limiting example, an optical target 228 can be placed on the patient 200, preferably around the location of the tissue of interest. An ultrasound inspection system according to U.S. Pat. Nos. 9,675,321 and 9,713,508, or an embodiment of the handheld surgical tool tracking system 100 modified to incorporate features thereof, can be used to acquire one or more ultrasound scans of the tissue of interest. Because the precise location and orientation of the ultrasound scanner is known (due to the scanner also having one or more distinguishable optical targets 102 on its face), the precise location and orientation of the patient's tissue of interest relative to the patient's detectable target 228 can be determined. Once patient tissue calibration is complete, the surgical procedure can begin.

[0064] In yet another example of patient tissue calibration, the detectable target 228 can have a portion that is optically detectable and a portion that is detectable in an X-ray image, a CT image, a fluoroscopic image, or the like. For example, one or more metal beads or the like can be present on the detectable target 102 at known locations. Prior to the start of a surgical procedure, the detectable target 228 can be affixed to the patient 200. One or more images of the patient 200 can then be acquired. Because the tissue of interest and the detectable target 228 are distinguishable in the acquired images, the precise location and orientation of the optical target 228 relative to the tissue of interest can be determined by the processor system 202.

[0065] As yet another example of patient tissue calibration, a patient may be scanned using an MRI and / or CT system, and because the tissue of interest and the optical target 228 are distinguishable in the acquired MRI and / or CT data, the precise location and orientation of the detectable target 228 relative to the tissue of interest may be determined by the processor system 202.

[0066] Alternatively or additionally, the patient tissue calibration process can be performed in real time during the surgical procedure. For example, ultrasound or x-ray images, etc., of the patient 200 (and detectable target 228) can be acquired during the surgical procedure. Embodiments of the handheld surgical tool tracking system 100 can then perform the calibration in real time. This calibration scheme may be particularly desirable when the tissue of interest moves during the surgical procedure. For example, the tissue of interest may be the beating heart of the patient 200 or the lungs of the breathing patient 200 during the surgical procedure.

[0067] Any suitable system for obtaining patient tissue calibration information during a surgical procedure is intended to be within the scope of this disclosure and protected by the accompanying claims. Such patient tissue calibration may be obtained before and / or during the surgical procedure.

[0068] In practice, during a surgical procedure, a 2D or 3D model of a tissue of interest (organ, bone, etc.) is retrieved from the tissue model database 218. The retrieved target tissue model of the patient 200 currently undergoing the surgical procedure was previously generated from an examination of the patient 200 prior to the current surgical procedure and then stored in the tissue model database 218. As the practitioner begins the surgical procedure, the handheld surgical tool tracking system 100 determines the precise location and orientation of the handheld surgical tool 104 in 3D space.

[0069] In the exemplary embodiment, the image capture device 206 captures images in real time including the optical targets 102 a, 102 b on the handheld surgical tool 104 and the optical target 228 on the patient 200. Optionally, the clock 222 can time stamp the acquired images. The captured image data is then communicated from the image capture device 206 to the target tracking unit 208.

[0070] In some embodiments, multiple image capture devices 206 may be used to capture camera images from different viewpoints in a synchronized manner, where multiple image capture devices 206 provide the same time stamp to camera images captured simultaneously.

[0071] The target tracking unit 208 identifies, for each acquired image, one or more optical targets 102 on the handheld surgical tool 104 and an optical target 228 located on the surface of the body of the patient 200. The target tracking unit 208 then calculates or determines the precise location and orientation of the optical target 202 relative to the optical target 228 in 3D space with respect to the time index. In some embodiments, the image data is analyzed to identify the particular handheld surgical tool 104 currently being used by the practitioner.

[0072] In some embodiments, one or more of the detectable targets 102, 228 may not be optically detectable (because they may not reflect light in the visible spectrum). To detect the precise location and orientation of the detectable target 102, another detection device 206 is used to acquire image data from the detectable target 102, 228 (which emits energy from other, non-visible spectrums). The target tracking unit 208 can then determine the precise location and orientation of the detectable target 202 relative to the detectable target 228 in 3D space.

[0073] The image registration module 210 then accepts location and orientation information for the detectable targets 202 and 228. Additionally, the image registration module 210 retrieves model data for the particular handheld surgical tool 104 being used by the practitioner (and optionally identified and / or verified within the captured image) from the surgical tool model database 216. The position of the handheld surgical tool 104 and the position of the working end 106 of the handheld surgical tool 104 are determined based on the identified relative locations of the detectable targets 102 and 228 in the acquired camera image and based on a correlation with the retrieved model data of the corresponding handheld surgical tool 104. Information corresponding to the precise location and orientation of the handheld surgical tool 104 and its working end 106 in 3D space is then communicated to the image processing algorithm module 212.

[0074] The image processing algorithm module 212 retrieves previously generated and stored tissue model data for the tissue of interest of the patient 200 from the tissue model database 218. Preferably, the tissue model is based on the patient's tissue of interest during the most recent tissue model generation process. When the tissue of interest is stationary during the surgical procedure (such as a bone that does not move), the tissue model data may be retrieved when the surgical procedure begins. Now that the relative location and orientation of the handheld surgical tool 104 (and its working end 106) with respect to the detectable target 228 in 3D space has been determined, and the tissue of interest has been positioned with respect to the detectable target 228 (during patient calibration), the image processing algorithm module 212 can generate a real-time composite image that includes both an image of the handheld surgical tool 104 and an image of the tissue of interest (as represented by the tissue model).

[0075] Data corresponding to the generated real-time composite image is communicated to the 3D / 2D visualization module 214. The 3D / 2D visualization module 214 then generates image data for presentation on a 3D and / or 2D display. In embodiments with a 3D / 2D display 224, a composite image showing the precise location and orientation of the handheld surgical tool 104, and particularly its working end 106, relative to the tissue of interest (represented by a predetermined tissue model of the tissue of interest) is then rendered and presented on the display 224. The illustrated composite image can be either a 3D or 2D image, depending on the characteristics of the particular display 224.

[0076] In some embodiments of the handheld surgical tool tracking system 100, to adjust the presentation of the composite image, the practitioner or another user can provide input to the 3D / 2D visualization module 214 using a user input device 220. In various embodiments, any suitable type of user input device 220 can be used. In response to receiving the user input, the 3D / 2D visualization module 214 can modify the presentation of the composite image according to the user input.

[0077] In some situations, the user input may be a request to zoom in on a particular location or area within the composite image. For example, the practitioner may wish to view a close-up image that presents an enlarged view of the working end 106 of the handheld surgical tool 104 and the tissue adjacent to it. Thus, the practitioner may be able to precisely determine with a high degree of detail and precision where the working end 106 of the handheld surgical tool 104 is currently located relative to the tissue of interest.

[0078] In another exemplary situation, because the model of the tissue of interest is in 3D space and the precise location and orientation of the working end 106 of the handheld surgical tool 104 is known in 3D space, the practitioner or another user may wish to rotate the illustrated composite image display. For example, the practitioner may wish to examine a side view or a bottom view of the composite image. The 3D / 2D visualization module 214 may rotate the composite image so that a side view or a bottom view is presented to the practitioner or another user on the display 224.

[0079] Alternatively or additionally, the composite image information can be communicated to a remote rendering and display system 204. The composite information, rendered to present an image of the precise location and orientation of the handheld surgical tool 104 and the tissue of interest, can then be presented on a display of the remote rendering and display system 204. For example, the remote rendering and display system 204 can be a virtual reality system that uses a headset display device.

[0080] A practitioner or other user viewing the composite image on the display of the remote rendering and display system 204 may desire to adjust the display of the composite image. In some embodiments, information corresponding to the user's viewing request is communicated to the remote rendering and display system 204 and / or the 3D / 2D visualization module 214. The 3D / 2D visualization module 214 adjusts the composite display and then communicates the adjusted composite display to the remote rendering and display system 204 for presentation on the display. Alternatively or additionally, the composite image data communicated to the remote rendering and display system 204 may include 3D model data of both the handheld surgical tool 104 and the tissue of interest. The image presented on the display of the remote rendering and display system 204 may then be adjusted and presented by the remote rendering and display system 204.

[0081] Those skilled in the art will recognize that processes for rendering, manipulating, and presenting images based on 2D and 3D modeling techniques are known in the art. For purposes of brevity, such techniques will not be described herein. All such techniques, now known or later developed, are believed to be within the scope of this disclosure and protected by the accompanying claims.

[0082] 3 is a conceptual diagram of a presented composite image 302 showing the relative locations of a spinal column 304 of a patient 200 and a handheld surgical tool 104. Here, the spinal column 304 is an example of a stationary tissue of interest that does not move during a surgical procedure. It is contemplated that the stationary tissue of interest could be other types of organs, such as the liver, brain, muscle, or another bone. During some surgical procedures, a portion of the patient's body may be immobilized using a suitable immobilization device, such as, but not limited to, a clamp or strap, that immobilizes the tissue of interest during the surgical procedure.

[0083] In this simplified conceptual illustration, a practitioner inserts the needle 106b of a syringe 104b between two vertebrae of a patient's spine, represented by a rendered spine model 304. Those skilled in the art will appreciate the delicate nature of this surgical procedure and the importance of precisely positioning and orienting the tip 106b of the syringe 104b for injection or tissue sampling. Typically, as shown in FIG. 2 , the practitioner views only the skin surface of the patient 200. An embodiment of the handheld surgical tool tracking system 100 determines the precise location and orientation of the needle tip 106b relative to the patient's spine (based on a pre-generated tissue model 304 of the patient's spine) and presents a composite image 302 on a display viewable by the practitioner or another party. Furthermore, movement of the needle tip 106b can be presented in real time or near real time within the presented composite image 302. The practitioner can now observe the needle tip 106b and these vertebrae of the spinal column as the needle tip 106b punctures the patient 200. Once the practitioner is satisfied with the precise location and orientation of the needle tip 106b, they can manipulate the syringe 104b to inject a drug into the spinal column or to remove a tissue sample from the spinal column.

[0084] Non-stationary tissue (moving tissue) presents an even more complex problem of presenting a composite image of the tissue of interest and the handheld surgical tool 104 in real time. For example, the tissue of interest may be a patient's beating heart. Here, the pre-generated tissue model of the tissue of interest may further include tissue movement characteristics. For example, if the heart is the tissue of interest, the generated 3D model should have sufficient data to represent the beating of the heart. That is, the moving portions of the patient's heart must be visible in the presented composite image that represents the patient's beating heart and the precise location and orientation of the handheld surgical tool 104.

[0085] The generation of 2D and 3D dynamic models representing moving objects is well understood in the art. For purposes of brevity, the generation of such dynamic models will not be described in detail herein. All such systems and processes for generating dynamic models, now known or later developed, are intended to be within the scope of this disclosure and protected by the accompanying claims.

[0086] Embodiments of the handheld surgical tool tracking system 100 are configured to synchronize the presentation of the dynamic tissue model's movement with the actual movement of the patient's tissue in real time. For example, the presented composite image would show a beating model of the patient's heart that corresponds to the actual heartbeat. Similarly, other moving tissues would be synchronized with their dynamic tissue models.

[0087] In various embodiments, a detector 232 configured to detect tissue movement can be used to monitor movement of the tissue of interest during a surgical procedure. The detector 232 is communicatively coupled to the processor system 202 through a suitable wireless or wire-based connector. The tissue movement information from the detector 232 is communicated to the image processing algorithm module 212. The image processing algorithm module 212 synchronizes the movement of the dynamic tissue model with the movement of the patient's tissue in real time or near real time. Thus, the presented composite image accurately represents the movement of the tissue of interest.

[0088] In situations where the tissue of interest is a beating heart, an audio detector, such as a stethoscope, can be used to detect the beat of the patient's heart. Once a heart beat is detected, information corresponding to the detected heart beat is communicated to the processor system 202. Movement of the dynamic model corresponding to the patient's heart can then be synchronized with the received heart beat information.

[0089] Detection of moving tissue is known in the art and, for the sake of brevity, will not be described herein. Furthermore, synchronization of dynamic model movement based on actual movement of the modeled object (in this case, the patient's tissue) is known in the art and, for the sake of brevity, will not be described herein. All such tissue movement detection and / or dynamic model generation systems and processes, now known or later developed, are intended to be within the scope of this disclosure and protected by the accompanying claims.

[0090] 4 is a conceptual diagram of an embodiment in which a handheld surgical tool tracking system 100 and a robotic surgical system 402 cooperate to generate a composite image including a robotic tool 404 and a handheld surgical tool 104. In this simplified hypothetical example, a patient 200 lies on an operating table 406 during a surgical procedure. A conventional robotic surgical system 402 manipulates one or more robotic tools 404 according to commands specified by a practitioner 410 and received from a robotic controller 408.

[0091] During a surgical procedure, the robotic surgical system 402 determines the precise location and orientation of each of the robotic tools 404, as is known in the art of robotic tools. Graphical images of the tissue operative area 412 and the robotic tools 404 can be presented on a display 414 for viewing by the surgeon 410. Depending on the robotic system, 2D images of the operative area 412 and the robotic tools 404 can be presented on the display 414. Alternatively or additionally, 2D or 3D models of the tissue and robotic tools 404 can be presented on the display 414.

[0092] Occasionally, an assistant 416 may be required to assist or participate in a surgical procedure by using one or more handheld surgical tools 104. At its best, the robotic surgical system 402 is only able to acquire 3D image information that shows the involved handheld surgical tool 104 as it is being used. It is not possible for the robotic surgical system 402 to determine the exact location and orientation of the involved handheld surgical tool 104 being used by the assistant.

[0093] However, the precise location and orientation of the involved handheld surgical tool 104 can be determined by an embodiment of the handheld surgical tool tracking system 100, where the image capture device 206 is positioned to capture an image of the optically detectable target 102 on the face of the handheld surgical tool 104. The 3D space known by the handheld surgical tool tracking system 100 is the same as the 3D space known by the robotic surgical system 402. Thus, image information providing the precise location and orientation of the involved handheld surgical tool 104 can be generated by the handheld surgical tool tracking system 100. This information can be communicated to the remote rendering and display system 204 for presentation by the display 214 along with the simultaneous presentation of graphical information generated by the robotic surgical system 402. Thus, the practitioner 410 can simultaneously observe the robotic tool 404 being controlled by the robotic surgical system 402 and the involved handheld surgical tool 104 being used by the assistant 416. When a 2D or 3D model of the tissue of interest within the surgical area 412 is presented on the display 414 (either by the handheld surgical tool tracking system 100 or the robotic surgical system 402), the practitioner 410 can simultaneously observe the involved handheld surgical tool 104 and robotic tool 404 in relation to the presented tissue model.

[0094] Those skilled in the art will recognize that many robotic surgical systems 402 are currently known or will be developed in the future. Such robotic surgical systems 402 are capable of graphically presenting various information on a display 414 to a practitioner 410 operating the system. For the sake of brevity, many of these robotic surgical systems 402 will not be described herein. Furthermore, the integration of image information from multiple image sources into a single image is known in the art and will not be described herein for the sake of brevity. Here, image information generated by an embodiment of the handheld surgical tool tracking system 100 is integrated with image information generated by the robotic surgical system 402 so that the practitioner 410 can ascertain the precise location and orientation of any participating handheld surgical tools 104 being used during a surgical procedure. All such forms of robotic surgical systems 402, now known or later developed, and all such techniques, now known or later developed, are believed to be within the scope of this disclosure and protected by the accompanying claims.

[0095] In some embodiments of the handheld surgical tool tracking system 100, information corresponding to the generated and presented composite image (showing the tissue model and handheld surgical tool 104) can be stored in a local and / or remote memory medium (not shown), such as the exemplary surgical procedure history 230 ( FIG. 2 ). Preferably, the composite image information is time-stamped, so that participants can later review and analyze the surgical procedure by viewing the stored composite image. The stored composite image can be viewed individually or as a video.

[0096] Determining the precise location and orientation of handheld surgical tools and other objects takes a significant amount of time and is very computationally intensive in conventional object recognition systems. Such conventional object recognition systems may not have sufficient computing power (processor system speed and / or bandwidth) to determine the precise location and orientation of objects in real time or near real time based solely on object recognition techniques. Embodiments of the handheld surgical tool tracking system 100 address the above-mentioned problems by determining the precise location and orientation of one or more detectable optical targets 102 and then correlating the detected optical targets 102 with known locations of optical targets of a known model of the handheld surgical tool 104.

[0097] As noted herein, the detectable target 102 can be active, such as by emitting an infrared signal to an optical target, or passive, such as a retroreflective marker affixed to any interaction device. Although such active or passive detectable targets 102 are generally referred to herein as detectable targets for purposes of brevity, such detectable targets 102 may not be optically detectable by an image capture device. In other words, the active or passive detectable targets 102 are detectable using another detection device 206 or detection system 206.

[0098] It should be emphasized that the above-described embodiments of the handheld surgical tool tracking system 100 are merely possible examples of the present invention. Many variations and modifications can be made to the above-described embodiments. All such modifications and variations are intended to be included within the scope of this disclosure and protected by the following appended claims.

[0099] Moreover, the foregoing disclosure encompasses multiple separate inventions with independent utility. While each of these inventions has been disclosed in a specific form, many variations are possible, and the specific embodiments disclosed and exemplified above should not be construed in a limiting sense. The subject matter of the invention includes all novel and non-obvious combinations and subcombinations of the various elements, features, functions, and / or properties disclosed above and inherently understood by those skilled in the art to which such invention pertains. Where in the present disclosure or any claims to be filed thereafter a reference is made to "a" element, "a first" element, or any such equivalent term, the present disclosure or claims should be understood to incorporate one or more such elements, without requiring or excluding two or more such elements.

[0100] Applicant reserves the right to submit claims directed to inventive combinations and subcombinations of the present disclosure that are believed to be novel and unobvious. Inventions embodied in other combinations and subcombinations of features, functions, elements, and / or properties may be claimed by amending the claims above or by presenting new claims in this or a related application. Such amended or new claims, whether directed to the same or different inventions and whether different from, broader than, narrower than, or equal to the original claims, shall be considered to be within the inventive subject matter described herein.

[0101] The inventions described in this application can be made by industrial processes involving a variety of mechanical, electrical, and pneumatic assembly techniques. Additionally, the inventions described herein can be used in industrial settings, including surgical procedure development. [Explanation of symbols]

[0102] 100 Handheld Surgical Tool Tracking System 102a Detectable Target 104b Syringe 200 patients 228 Optical Target 232 detector

Claims

1. capturing image data including both a first detectable target on the handheld surgical tool and a second detectable target proximate to the patient using an image capture device; the first detectable target is positioned on the handheld surgical tool at a first predetermined target location; the second detectable target is positioned on the patient at a second predetermined target location; and a location relationship between the patient's tissue of interest and the second predetermined location of the second detectable target is known; said incorporating; determining a location of the first detectable target in three-dimensional (3D) space and an orientation of the first detectable target based on the image data; retrieving hand-held surgical tool model data representative of the hand-held surgical tool, the hand-held surgical tool model data including first detectable target location model data corresponding to the predetermined location of the first detectable target; determining a current location of the handheld surgical tool in the 3D space and a current orientation of the handheld surgical tool based on the determined location of the first detectable target and the determined orientation of the first detectable target; determining a location of the second detectable target in the 3D space based on the image data; and determining a location of the tissue of interest in the 3D space based on the determined location of the second detectable target on the patient; A surgical method including:

2. accessing tissue model data defining a tissue model corresponding to the tissue of interest of the patient; Further comprising: the tissue model was previously generated from an examination of the patient prior to the current surgical procedure; The surgical method according to claim 1 .

3. the tissue model data having been previously generated based on a series of acquired time-indexed ultrasound images; The surgical method according to claim 2.

4. generating a composite image, the generated composite image simultaneously presenting a first graphical representation of the tissue of interest of the patient based on the tissue model and a second graphical representation of the handheld surgical tool. said occurring; and presenting the generated composite image on a display; Further comprising: a surgical team member viewing the display intuitively understands the current location and orientation of the handheld surgical tool relative to the tissue of interest of the patient; The surgical method according to claim 2.

5. the handheld surgical tool model data includes working end location model data corresponding to a location of a working end of the handheld surgical tool; the working end of the handheld surgical tool is disposed on a distal end of the handheld surgical tool; The method is determining a location of the working end of the handheld surgical tool and an orientation of the working end of the handheld surgical tool based on the handheld surgical tool model data; Further comprising: the surgical team member viewing the display intuitively understands the current location of the working end of the handheld surgical tool and the current orientation of the working end of the handheld surgical tool relative to the tissue of interest of the patient; The surgical method according to claim 4.

6. the captured image data is one of a plurality of time-sequentially captured images of a video stream captured by the image capture device; The method is continuously updating the generated image presented on the display using currently acquired image data received in the video stream; the surgical team member viewing the display intuitively understands in real time the current location of the handheld surgical tool and the current orientation of the handheld surgical tool relative to the tissue of interest of the patient; said updating; Further comprising: The surgical method according to claim 4.

7. the first detectable target is a first optical target; the second detectable target is a second optical target; the image capture device is a camera; The surgical method according to claim 2.

8. a third detectable target is positioned on the handheld surgical tool at a second predetermined target location different from the first predetermined target location; the handheld surgical tool model data includes third detectable target location model data corresponding to the predetermined location of the third detectable target; The method is determining a location of the third detectable target in the 3D space based on the image data and based on the handheld surgical tool model data; Further comprising: determining the current location of the handheld surgical tool in the 3D space and the current orientation of the handheld surgical tool based on the determined location of the first detectable target and the determined location of the third detectable target. The surgical method according to claim 1 .

9. a first detectable target on the first surgical tool member of the handheld surgical tool at the first predetermined target location on the first surgical tool member of the handheld surgical tool; the handheld surgical tool includes a second surgical tool member coupled to the first surgical tool member; a third detectable target is positioned on the second member of the handheld surgical tool at a third predetermined target location; the handheld surgical tool model data further includes third detectable target location model data corresponding to the predetermined location of the third detectable target; The method is determining a current location of the first member of the handheld surgical tool and a current orientation of the first member of the handheld surgical tool in the 3D space based on the determined location of the first detectable target and the determined orientation of the first detectable target; determining a location of the third detectable target and an orientation of the third detectable target in the 3D space based on the image data and based on the handheld surgical tool model data; and determining a current location of the second member of the handheld surgical tool and a current orientation of the second member of the handheld surgical tool in the 3D space based on the determined location of the third detectable target and the determined orientation of the third detectable target; Further comprising: The surgical method according to claim 1 .

10. a first surgical tool member disposed at a distal end of the first surgical tool member; a second surgical tool member disposed at a distal end of said second surgical tool member; the first surgical tool member and the second surgical tool member cooperatively operate in a coordinated manner to perform a surgical operation on the tissue of the patient; The method is determining a current location of the first surgical tool member in the 3D space based on the determined location and orientation of the first surgical tool member and based on the handheld surgical tool model data; and determining a current location of the second surgical tool member in the 3D space based on the determined location and orientation of the second surgical tool member and based on the handheld surgical tool model data; Further comprising: The surgical method according to claim 9.

11. the first detectable target is on the first surgical tool member of the handheld surgical tool at the first predetermined target location on the first surgical tool member of the handheld surgical tool; a third detectable target is on the first surgical tool member of the handheld surgical tool at a third predetermined target location on the first surgical tool member that is different from the first predetermined target location; the handheld surgical tool includes a second surgical tool member coupled to the first surgical tool member; a fourth detectable target positioned on the second member of the handheld surgical tool at a fourth predetermined target location; a fifth detectable target on a second surgical tool member of the hand-held surgical tool at a fifth predetermined target location on the first surgical tool member that is different from the fourth predetermined target location; The handheld surgical tool model data includes: third detectable target location model data corresponding to the predetermined location of the third detectable target; fourth detectable target location model data corresponding to the predetermined location of the fourth detectable target; and fifth detectable target location model data corresponding to the predetermined location of the fifth detectable target; Further comprising: The method is determining a location of the third detectable target in the 3D space based on the image data and based on the handheld surgical tool model data; determining a location of the fourth detectable target in the 3D space based on the image data and based on the handheld surgical tool model data; determining a location of the fifth detectable target in the 3D space based on the image data and based on the handheld surgical tool model data; determining a current location of the first member of the handheld surgical tool and a current orientation of the first member of the handheld surgical tool in the 3D space based on the determined location of the first detectable target and the determined location of the third detectable target and based on the handheld surgical tool model data; and determining a current location of the second member of the handheld surgical tool and a current orientation of the second member of the handheld surgical tool in the 3D space based on the determined location of the fourth detectable target and the determined location of the fifth detectable target and based on the handheld surgical tool model data; Further comprising: The surgical method according to claim 1 .

12. the first detectable target is on the first surgical tool member of the handheld surgical tool at the first predetermined target location on the first surgical tool member of the handheld surgical tool; the handheld surgical tool includes a second surgical tool member coupled to the first surgical tool member; a third detectable target is positioned on the second member of the handheld surgical tool at a third predetermined target location; a fourth detectable target positioned at a location where the first surgical tool member is coupled to the second surgical tool member; The handheld surgical tool model data includes: third detectable target location model data corresponding to the predetermined location of the third detectable target; and fourth detectable target location model data corresponding to the predetermined location of the fourth detectable target; Further comprising: The method is determining a location of the third detectable target in the 3D space based on the image data and based on the handheld surgical tool model data; determining a location of the fourth detectable target in the 3D space based on the image data and based on the handheld surgical tool model data; determining a current location of the first member of the handheld surgical tool and a current orientation of the first member of the handheld surgical tool in the 3D space based on the determined location of the first detectable target and the determined location of the fourth detectable target and based on the handheld surgical tool model data; and determining a current location of the second member of the handheld surgical tool and a current orientation of the second member of the handheld surgical tool in the 3D space based on the determined location of the third detectable target and the determined location of the fourth detectable target and based on the handheld surgical tool model data; Further comprising: The surgical method according to claim 1 .

13. a first surgical tool member disposed at a distal end of the first surgical tool member; a second surgical tool member disposed at a distal end of said second surgical tool member; the first surgical tool member and the second surgical tool member cooperatively operate in a coordinated manner to perform a surgical operation on the tissue of the patient; The method is determining a current location of the first surgical tool member and a current orientation of the first surgical tool member in the 3D space based on the determined location and orientation of the first surgical tool member and based on the handheld surgical tool model data; and determining a current location of the second surgical tool member and a current orientation of the first surgical tool member in the 3D space based on the determined location and orientation of the second surgical tool member and based on the hand-held surgical tool model data; Further comprising: The surgical method according to claim 12.

14. the handheld surgical tool is a surgical scissors; the first surgical tool member is a first cutting edge of the surgical scissors; the second surgical tool member being a second cutting edge of the surgical scissors; The surgical method according to claim 13.

15. the manually operated surgical instrument is a surgical clamp; the first surgical tool member is a first clamp of the surgical clamp; the second surgical tool member is a second clamp of the surgical clamp; The surgical method according to claim 13.

16. 2. The surgical method of claim 1, wherein the handheld surgical tool is a scalpel having a surgical tool member that is a scalpel blade positioned at a distal end of the scalpel.

17. 10. The surgical method of claim 1, wherein the handheld surgical tool is a syringe having a surgical tool member that is a needle positioned at the distal end of the syringe.

18. the handheld surgical tool is a syringe having a surgical tool member that is a needle positioned at a distal end of the syringe; the first detectable target is on a plunger of the syringe; a second detectable target on the barrel of the syringe; The handheld surgical tool model data includes: second detectable target location model data corresponding to the predetermined location of the second detectable target; and volumetric data corresponding to the volume of the barrel of the syringe; Further comprising: The method is determining a change in location of the first detectable target relative to the location of the first detectable target; determining a change in a distance traveled by a barrel seal positioned at a distal end of the plunger, the distance traveled being equal to a distance of the change in location of the first detectable target relative to the second detectable target; and determining a volume defined within the barrel based on the determined travel distance of the barrel seal; Further comprising: The surgical method according to claim 1 .

19. a first surgical team member manually operating the manually operated surgical instrument and viewing the rendered image on the first display; a second surgical team member controlling operation of a robotic surgical system that robotically manipulates the robotic surgical instrument; the rendered image includes a first graphical representation of the tissue of interest of the patient, a second graphical representation of the manually operated surgical instrument being operated by the first surgical team member, and a third graphical representation of the robotic surgical instrument; The method is presenting the rendered image on the first display for viewing by the first surgical team member; and presenting the rendered image on a second display viewed by the second surgical team member; Further comprising: the rendered image is simultaneously presented to the first surgical team member on the first display and to the second surgical team member on the second display. The surgical method according to claim 1 .

20. receiving robotic surgical instrument model information from the robotic surgery system; receiving robotic surgical instrument location information from the robotic surgery system based on a current location of the robotic surgical instrument; and receiving robotic surgical instrument orientation information from the robotic surgical system based on a current orientation of the robotic surgical instrument; Further comprising: the rendered image, when presented on the first display and the second display, indicates a location and orientation of the robotic surgical instrument relative to a location and orientation of the manually operated surgical instrument and indicates a location and orientation of the robotic surgical instrument relative to an organ of interest.

20. The surgical method of claim 19.

Images