Devices and methods for use with bone surgery

Abstract

Various devices and methods for use with skeletal surgery are described, including acquiring three-dimensional image data of a skeletal portion. A plurality of two-dimensional X-ray images are acquired from respective views when a portion of a tool (36) is disposed in a first position relative to the bone portion. A computer processor (22) confirms the first position relative to the three-dimensional image data based on identifying the first position within the first two-dimensional X-ray image and the second two-dimensional X-ray image. After moving the portion of the tool to a second position, an additional two-dimensional X-ray image is acquired from a single image perspective. Based on identifying the second position of the portion of the tool within the additional two-dimensional X-ray image and the confirmed first position of the portion of the tool relative to the three-dimensional image data, the computer processor derives the second position relative to the three-dimensional image data. Other applications are also described.

Description

[0001] This application is a divisional application of the patent application filed on March 13, 2017, with application number 202210057286.1 and invention name “Device and method for use with bone surgery”; this patent application (application number: 202210057286.1) is a divisional application of the international application number PCT / IL2017 / 050314 filed on March 13, 2017, with application number 201780029652.X entering the Chinese national phase, and invention name “Device and method for use with bone surgery”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to:

[0004] U.S. Provisional Patent Application No. 62 / 307,514, filed by Tolkowsky on March 13, 2016, entitled “Freehand Assistant for Spinal Surgery”;

[0005] U.S. Provisional Patent Application No. 62 / 334,463, filed by Tolkowsky on May 11, 2016, entitled “Freehand Assistant for Spinal Surgery”;

[0006] U.S. Provisional Patent Application No. 62 / 362,607, filed by Tolkowsky on July 15, 2016, entitled “Freehand Assistant for Spinal Surgery”;

[0007] U.S. Provisional Patent Application No. 62 / 398,085, filed by Tolkowsky on September 22, 2016, entitled “Freehand Assistant for Spinal Surgery”;

[0008] U.S. Provisional Patent Application No. 62 / 439,495, filed by Tolkowsky on December 28, 2016, entitled “Handed Assistant for Spinal Surgery”; and

[0009] Tolkowsky’s U.S. Provisional Patent Application No. 62 / 463,747, filed on February 27, 2017, is titled “Freehand Assistant for Spinal Surgery.”

[0010] The above applications are incorporated into this specification by reference. Technical Field

[0011] Some applications of the present invention generally relate to medical devices and methods, and more particularly, some applications of the present invention relate to devices and methods for performing procedures on bony anatomy. Background Art

[0012] Approximately 5 million spinal surgeries are performed worldwide each year, with traditional manual procedures known as freehand surgery. Typically, prior to such procedures, a three-dimensional scan (e.g., a computed tomography (CT) and / or magnetic resonance imaging (MRI)) is performed. A CT scan is typically performed on bony tissue (e.g., vertebrae), while an MRI scan is typically performed on soft tissue (e.g., intervertebral discs).

[0013] refer to Figure 1A , Figure 1A is a schematic diagram of a typical setup of an orthopedic operating room for procedures performed in a freehand manner. Typically, in freehand procedures, the CT and / or MRI images are not used during surgery, although the surgeon reviews the CT and / or MRI scans when preparing for surgery. Instead, the surgery is typically performed under the guidance of two-dimensional X-ray images (also called fluoroscopic guidance), which are typically acquired using an X-ray C-arm. Figure 1A A surgeon 10 is shown using multiple intraoperative X-ray images acquired by a C-arm 34 and displayed on a monitor 12. Freehand surgery that makes extensive use of X-rays is known as fluoroscopically guided surgery, and X-ray C-arms are ubiquitous, familiar to surgeons, can be used to acquire multiple real-time images, are tool-agnostic (i.e., do not require the use of multiple orthopedic tools specifically designed for imaging with an X-ray C-arm), and are relatively inexpensive. An increasing number of spinal surgeries are performed using a minimally invasive procedure (also known as "MIS," or in the case of spinal surgery, minimally invasive spinal surgery, also known as "MISS") or "mini-open surgery." In contrast to open surgery, where a single incision is made along the level of the spine where the procedure is to be performed, in minimally invasive surgery, multiple very small incisions are made at the insertion points for multiple tools. In "mini-open" surgery, the multiple incisions made are smaller than in open surgery and larger than in minimally invasive surgery. Generally, the less invasive the type of procedure being performed, the greater the use of X-ray imaging to assist in the procedure. There is evidence that minimally invasive procedures performed under fluoroscopic guidance are more accurate than open procedures. However, the use of real-time fluoroscopic guidance often exposes the patient, surgeon, and support personnel to relatively large amounts of harmful radiation.

[0014] A small number of surgeries are performed using computer-assisted surgery (CAS) systems that provide navigation and / or robotics. Such systems typically utilize computed tomography and / or magnetic resonance imaging (MRI) images generated before the patient is in the operating room or while the patient is in the operating room but typically before an intervention begins. The CT and / or MRI images are registered to the patient's body, and during the procedure, various tools are navigated onto the images, with the tools being moved manually, robotically, or both.

[0015] Typically, in computer-assisted surgery procedures, a uniquely identifiable position sensor is attached to each tool that needs to be tracked by the computer-assisted surgery system, and each tool is identified and calibrated at the beginning of the procedure. In addition, a uniquely identifiable reference sensor is securely attached to the organ. In the case of spinal surgery, the reference sensor is typically drilled into the sacrum or spine, and if the surgery is performed along multiple vertebrae, the reference sensor is occasionally moved mid-operatively and drilled into a different part of the spine in order to remain close to the surgical site. The multiple images to be navigated (e.g., computed tomography, magnetic resonance imaging) acquired before the patient is in the operating room or while the patient is in the operating room but before an intervention begins are registered to the patient's body or part of the patient's body. To register the plurality of images to the patient's body, the reference sensor is used to bring the current position of the patient's body into the same reference coordinate system as the plurality of images, and then the plurality of position sensors on the plurality of tools and the reference sensor on the patient's body are tracked to confirm the plurality of positions of the plurality of tools relative to the patient's body, and a symbolic representation of the tools is displayed on the plurality of images being navigated. Typically, the tools are tracked within 5 to 6 degrees of freedom.

[0016] Various techniques exist for tracking the tools, each with corresponding position sensors. One technique is infrared ("IR") tracking, whereby a set of cameras tracks active infrared lights on the tools and the patient's body, or a plurality of beams and cameras tracks passive infrared reflectors on the tools and the patient's body. In both types of IR tracking, line of sight must always be maintained between the tracker and the tools. For example, if the surgeon's hand blocks the line of sight, this may interfere with tracking. Another technique is electromagnetic or magnetic radiation tracking, whereby a field generator tracks multiple receivers, typically coils, on the tools and the patient's body. With these latter techniques, environmental interference from other equipment can be avoided. In each of these techniques, the position sensors of the navigation system are tracked using tracking components that are not present in the operating room in the absence of the navigation system (i.e., the position sensors do not simply rely on imaging from imaging devices typically used in an orthopedic operating room in the absence of the navigation system).

[0017] Another technique that can be used with a mechanically driven tool is to start with the tool at a known starting point relative to the patient's body and then record the tool's motion from that starting point. Alternatively, the tools can be tracked using the techniques described above.

[0018] Due to the nature of computer-assisted surgery procedures, the equipment required for such procedures is generally more expensive than for non-computer-assisted procedures (including open procedures, mini-open procedures, or non-computer-assisted minimally invasive procedures with tool guidance). Typically, such procedures limit the selection of tools to those equipped with position sensors, as described above, and typically require individual identification and calibration of these tools at the beginning of each procedure. Summary of the Invention

[0019] According to some applications of the present invention, the following steps are typically performed during procedures performed on skeletal anatomy using a system including a computer processor. Such procedures may include joint (e.g., shoulder, knee, hip, and / or ankle) replacement, joint repair, fracture repair (e.g., femur, tibia, and / or fibula), a procedure performed on a rib (e.g., rib removal or rib resection), and / or other interventions in which three-dimensional image data is acquired prior to the intervention and two-dimensional images are acquired during the intervention. For some applications, the steps are performed during a procedure performed on one or more vertebrae of a subject's spine.

[0020] Typically, in a first step, target vertebra(s) are marked by an operator relative to three-dimensional image data of the subject's spine (e.g., a three-dimensional image, a two-dimensional cross-section derived from the three-dimensional image data, and / or a two-dimensional projection image derived from the three-dimensional image data). For some applications, in a second step, multiple combinations of multiple markers are placed on the subject, below the subject, on the operating table, or above the operating table in a vicinity of the subject. Typically, in a third step, multiple vertebrae of the spine are identified to verify that the procedure was performed on the correct vertebrae (a step referred to as "segment verification"), and multiple radiographic images of the spine and the multiple markers are used to facilitate the identification. For some applications, in a fourth step, an incision site (in the case of minimally invasive surgery) or a tool entry point into a vertebra (in the case of open surgery) is confirmed. In a fifth step, the first tool in the sequence of the multiple tools (the first tool is typically a needle, such as a Jamshidi TM A needle) is typically inserted into the subject (e.g., at the subject's back) through the incision site or the tool entry point and is lightly secured in the vertebra. In a sixth step, two or more two-dimensional radiographic images are typically acquired from a plurality of corresponding views, the plurality of corresponding views typically differing by at least 10 degrees (and typically more by 30 degrees or more), and one of the plurality of corresponding views is typically from the insertion direction of the tool. Typically, a generally anterior-posterior image and a generally lateral image are acquired. Alternatively or additionally, a plurality of images are acquired from a plurality of different viewing angles. In a seventh step, typically, the computer processor registers the three-dimensional image data to the plurality of two-dimensional images.

[0021] Typically, a three-dimensional image data of an individual vertebra is registered with a two-dimensional image. Further, typically, the three-dimensional image data is registered with the plurality of two-dimensional images by generating a plurality of two-dimensional projections from the three-dimensional image data and identifying a corresponding first two-dimensional projection and a second two-dimensional projection that match the plurality of two-dimensional x-ray images of the vertebra, as described in further detail below. Typically, a first two-dimensional x-ray image and a second two-dimensional x-ray image of the vertebra are acquired using an x-ray imaging device that is not registered to the subject's body, and by (a) acquiring a first two-dimensional x-ray image of the vertebra (and at least a portion of the tool) from a first perspective, with the x-ray imaging device positioned in a first posture relative to the subject's body, (b) moving the x-ray imaging device to a second posture relative to the subject's body, and (c) acquiring a second two-dimensional x-ray image of at least the portion of the tool and the vertebra from a second perspective while the two-dimensional x-ray imaging device is in the second posture. For some applications, more than two 2D x-rays are acquired from a plurality of corresponding x-ray image views, and the 3D image data are typically registered with the 2D x-ray images by identifying a corresponding number of 2D projections of the 3D image data that match the plurality of corresponding 2D x-ray images.

[0022] For some applications, the computer processor acquires a two-dimensional x-ray image of a tool within the vertebra from only a single x-ray image viewpoint and registers the two-dimensional x-ray image to the three-dimensional image data by generating a plurality of two-dimensional projections from the three-dimensional image data and identifying a two-dimensional projection that matches the two-dimensional x-ray image of the vertebra. In response to registering the two-dimensional x-ray image to the three-dimensional image data, the computer processor drives a display to display a cross-section derived from the three-dimensional image data at a current position of a tip of the tool, as identified from the two-dimensional x-ray image, and optionally, to show a vertical line on the cross-sectional image, the vertical line indicating a line within the cross-sectional image at a location where the tip of the tool is currently positioned.

[0023] As described above, generally, two or more 2D X-ray images are acquired from a plurality of corresponding X-ray views, and the 3D image data is registered with the plurality of 2D images by identifying a corresponding number of 2D projections of the 3D image data that match the plurality of corresponding 2D X-ray images. After registering the 3D image data to the plurality of 2D X-ray images, the computer processor applies a number of additional features of the system. For example, the computer processor can drive the display to display the expected (i.e., inferred) path of the tool with reference to a target position and / or with reference to a desired insertion vector. For some applications, the computer processor simulates the progression of the tool within a secondary 2D imaging view based on the progression of the tool observed in a primary 2D imaging view. Alternatively or additionally, the computer processor overlays an image of the tool, a representation of an image of the tool, and / or a representation of the path of the tool on the 3D image data (e.g., a 3D image, a 2D cross-section derived from the 3D image data, and / or a 2D projection image derived from the 3D image data), the position of the tool or the path of the tool having been derived from the plurality of current 2D images.

[0024] As described above, for some applications, multiple combinations of multiple markers are placed on, beneath, or above the subject. Typically, the markers, placed at corresponding locations relative to the subject, are identifiable in multiple X-ray images, multiple optical images, and are visible to the human eye. For example, multiple related radiopaque alphanumeric characters can be placed at the corresponding locations. For some applications, the markers placed at the corresponding locations can be identified based on other characteristics, such as the placement of the markers relative to other markers. Using a radiographic imaging device, multiple radiographic images of the combination of the multiple radiopaque markers are acquired, the multiple corresponding images being at corresponding locations along at least a portion of the subject's spine, and each of the multiple images including at least some of the multiple radiopaque markers. Using a computer processor, multiple locations of the multiple radiopaque markers within the multiple radiographic images are identified through image processing. Based on the identified locations of the multiple radiopaque markers within the multiple radiographic images, at least some of the multiple radiographic images are combined relative to one another. Typically, such a combination of multiple images is similar to stitching together multiple images. However, the multiple images are typically stitched together imprecisely so that multiple portions of the object's anatomical structure in multiple adjacent images are stitched together. Instead, the multiple images are combined with sufficient accuracy to be able to confirm a position of the given vertebra within the multiple combined radiographic images. Therefore, the computer processor automatically confirms (or facilitates manual confirmation) a position of the given vertebra within the multiple combined radiographic images. Based on the position of the given vertebra within the multiple combined radiographic images, as further described in detail below, a position of the given vertebra is confirmed in relation to a combination of the multiple radiopaque markers placed on the object. As further described in detail below, the multiple markers are typically used to provide multiple additional functions or to facilitate multiple functions in some cases.

[0025] Thus, according to some applications of the present invention, there is provided an apparatus and method for use with bone surgery for performing a procedure using a tool configured to be advanced along a longitudinal insertion path into a bone portion within a body of a subject, the method comprising:

[0026] Acquiring three-dimensional image data of the bone part;

[0027] When a portion of the tool is disposed in a first position relative to the bone portion and along the longitudinal insertion path, sequentially:

[0028] acquiring a first two-dimensional X-ray image of at least the portion of the tool and the bone portion from a first viewing angle using a two-dimensional X-ray imaging device, the two-dimensional X-ray imaging device not being registered to the subject's body and the two-dimensional X-ray imaging device being positioned in a first posture relative to the subject's body;

[0029] moving the two-dimensional X-ray imaging device to a second position relative to the subject's body; and

[0030] When the two-dimensional X-ray imaging device is in the second posture, acquiring a second two-dimensional X-ray image of at least the portion of the tool and the bone portion from a second viewing angle;

[0031] Using at least one computer processor to:

[0032] Registering the first X-ray image and the second X-ray image to the three-dimensional image data, the registering comprising:

[0033] generating a plurality of two-dimensional projections from the three-dimensional image data; and

[0034] identifying a first two-dimensional projection and a second two-dimensional projection corresponding to the first two-dimensional X-ray image and the second two-dimensional X-ray image data of the skeletal portion;

[0035] identifying, by image processing, a position of the portion of the tool relative to the bone portion in the first two-dimensional X-ray image and the second two-dimensional X-ray image;

[0036] confirming the first position of the portion of the tool relative to the three-dimensional image data based on the identified position of the portion of the tool within the first and second two-dimensional x-ray images and registering the first and second two-dimensional x-ray images to the three-dimensional image data;

[0037] Next, moving the portion of the tool relative to the bone portion and along the longitudinal insertion path to a second position;

[0038] after moving the portion of the tool to the second position, acquiring one or more additional two-dimensional X-ray images of at least the portion of the tool and the bone portion from a single image view;

[0039] Using the computer processor to:

[0040] identifying, by image processing, the second position of the portion of the tool within the one or more additional two-dimensional x-ray images;

[0041] deriving the second position of the portion of the tool relative to the three-dimensional image data based on the second position of the portion of the tool within the one or more additional two-dimensional x-ray images and the identified first position of the portion of the tool relative to the three-dimensional image data; and

[0042] An output is generated at least in part responsive thereto.

[0043] In some applications, obtaining the one or more additional two-dimensional X-ray images of at least the portion of the tool and the bone portion from the single image perspective includes: obtaining the one or more additional two-dimensional X-ray images of at least the portion of the tool and the bone portion from one of the first image perspective and the second image perspective.

[0044] In some applications, obtaining the one or more additional two-dimensional X-ray images of at least the portion of the tool and the bone portion includes obtaining the one or more additional two-dimensional X-ray images of at least the portion of the tool and the bone portion from a third image perspective that is different from the first image perspective and the second image perspective.

[0045] According to some applications of the present invention, there is also provided an apparatus and method for use with bone surgery for performing a procedure using a tool configured to be advanced along a longitudinal insertion path into a bone portion within a body of a subject, the method comprising:

[0046] When a portion of the tool is disposed at a first position along the longitudinal insertion path, sequentially:

[0047] acquiring a first two-dimensional X-ray image of at least the portion of the tool and the bone portion from a first viewing angle, the two-dimensional X-ray imaging device not being registered to the subject's body, and the two-dimensional X-ray imaging device being positioned in a first posture relative to the subject's body;

[0048] moving to a second posture relative to the subject's body; and

[0049] When the two-dimensional X-ray imaging device is in the second posture, acquiring a second two-dimensional X-ray image of at least the portion of the tool and the bone portion from a second viewing angle;

[0050] identifying, using at least one computer processor, the portion of the tool within the first two-dimensional X-ray image and the second two-dimensional X-ray image through image processing;

[0051] using the computer processor, determining a relationship between the first two-dimensional X-ray image and the second two-dimensional X-ray image;

[0052] advancing the tool relative to the bone portion along the longitudinal insertion path such that the portion of the tool is disposed at a second position along the longitudinal insertion path;

[0053] Next, acquiring one or more additional two-dimensional X-ray images of at least the portion of the tool and the bone portion from a single image view; and

[0054] Using the computer processor to:

[0055] identifying, by image processing, the second position of the portion of the tool within the one or more additional two-dimensional x-ray images;

[0056] deriving the second position of the portion of the tool relative to at least one of the first two-dimensional x-ray image and the second two-dimensional x-ray image based on the second position of the portion of the tool identified in the one or more additional two-dimensional x-ray images and the determined relationship between the first two-dimensional x-ray image and the second two-dimensional x-ray image; and

[0057] An output is generated at least in part responsive thereto.

[0058] In some applications, confirming the relationship between the first two-dimensional X-ray image and the second two-dimensional X-ray image includes: aligning the first two-dimensional X-ray image and the second two-dimensional X-ray image with each other by aligning the first two-dimensional X-ray image and the second two-dimensional X-ray image to a three-dimensional image data of the bone part.

[0059] In some applications, confirming the relationship between the first two-dimensional X-ray image and the second two-dimensional X-ray image includes: using a three-dimensional non-transmissive rigid fixture visible in the first two-dimensional X-ray image and the second two-dimensional X-ray image to align the first two-dimensional X-ray image and the second two-dimensional X-ray image.

[0060] In some applications, confirming the relationship between the first two-dimensional X-ray image and the second two-dimensional X-ray image includes: using two or more non-radiopaque features of the tool that are visible in the first two-dimensional X-ray image and the second two-dimensional X-ray image to confirm the relationship between the first two-dimensional X-ray image and the second two-dimensional X-ray image.

[0061] In some applications, obtaining the one or more additional two-dimensional X-ray images of at least the portion of the tool and the bone portion from the single image perspective includes obtaining the one or more additional two-dimensional X-ray images of at least the portion of the tool and the bone portion from one of the first image perspective and the second image perspective.

[0062] In some applications, obtaining the one or more additional two-dimensional X-ray images of at least the portion of the tool and the bone portion from the single image perspective includes: obtaining the one or more additional two-dimensional X-ray images of at least the portion of the tool and the bone portion from a third image perspective different from the first image perspective and the second image perspective.

[0063] According to some applications of the present invention, there is also provided a method for performing a procedure on a given vertebra of a spine of a subject, the method comprising:

[0064] placing a combination of a plurality of radiopaque markers in a vicinity of the object such that the plurality of markers placed at respective locations relative to the object are identifiable;

[0065] acquiring, using a radiocontrast imaging device, a plurality of radiocontrast images of the combination of the plurality of radiopaque markers, the plurality of corresponding images being at a plurality of corresponding locations along at least a portion of the subject's spine, and each of the plurality of images including at least some of the plurality of radiopaque markers;

[0066] Using at least one computer processor to:

[0067] identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0068] combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0069] automatically confirming a position of the given vertebra within the combined plurality of radiographic images through image processing; and

[0070] An output is generated at least in part responsive thereto.

[0071] In some applications, generating the output includes driving a display to display the combined plurality of radiographic images and an indication of the given vertebra displayed together with the combined plurality of radiographic images, the method further comprising manually confirming a physical location of the given vertebra relative to the combination of radiopaque markers placed in the vicinity of the subject based on the location within the given vertebra in the combined plurality of radiographic images.

[0072] In some applications, generating the output includes driving a display to display the combined plurality of radiographic images and an indication of the given vertebra displayed relative to the combined plurality of radiographic images, the method further comprising driving the display to display an indication of the given vertebra relative to three-dimensional image data of at least a portion of the subject's spinal column.

[0073] In some applications, automatically confirming, by image processing, the position of the given vertebra within the combined plurality of radiographic images comprises:

[0074] identifying an identifiable feature within the combined plurality of radiographic images;

[0075] identifying individual vertebrae within the combined plurality of radiographic images; and

[0076] Vertebrae are counted from the identifiable features.

[0077] In some applications, identifying an identifiable feature within the combined radiographic images includes identifying a sacrum of the subject within the combined radiographic images.

[0078] According to some applications of the present invention, there is also provided an apparatus for performing a procedure on a given vertebra of a spine of a subject and for use with a radiographic imaging device, the apparatus comprising:

[0079] a combination of a plurality of radiopaque markers configured to be placed in a vicinity of the object such that the plurality of markers placed at respective locations relative to the object are identifiable; and

[0080] At least one computer processor configured to:

[0081] receiving a plurality of radiocontrast images of the plurality of radiopaque markers from the radiocontrast imaging device, the plurality of corresponding images being a plurality of corresponding locations along at least a portion of the subject's spine, and each of the plurality of images including at least some of the plurality of radiopaque markers;

[0082] identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0083] combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0084] automatically confirming a position of the given vertebra within the combined plurality of radiographic images through image processing; and

[0085] An output is generated at least in part responsive thereto.

[0086] In some applications, the apparatus is for use with a display, and the at least one computer processor is configured to generate the output by driving the display to display the combined plurality of radiographic images and an indication of the given vertebra displayed relative to the combined plurality of radiographic images.

[0087] In some applications, the device is for use with a display, and the at least one computer processor is configured to drive the display to display the combined plurality of radiographic images and an indication of the given vertebra displayed relative to the combined plurality of radiographic images, and to drive the display to display an indication of the given vertebra relative to three-dimensional image data of at least a portion of the subject's spine.

[0088] In some applications, the at least one computer processor is configured to automatically confirm the position of the given vertebra within the combined plurality of radiographic images by:

[0089] identifying a recognizable feature within the combined radiographic images;

[0090] identifying individual vertebrae within the combined radiographic images; and

[0091] Vertebrae are counted from the identifiable features.

[0092] In some applications, the at least one computer processor is configured to identify the identifiable feature within the combined plurality of radiographic images by identifying a sacrum of the subject within the combined plurality of radiographic images.

[0093] According to some applications of the present invention, there is also provided a computer software product for performing a procedure on a given vertebra of a spine of a subject, the computer software product being used in conjunction with a radiographic imaging device and a combination of a plurality of radiopaque markers, the combination of the plurality of radiopaque markers being configured to be placed in a vicinity of the subject such that the plurality of markers placed at respective positions relative to the subject are identifiable, the computer software product comprising a non-transitory computer-readable medium storing a plurality of program instructions that, when read by a computer, cause the computer to perform the following steps:

[0094] receiving a plurality of radiocontrast images of the combination of the plurality of radiopaque markers from the radiocontrast imaging device, the plurality of corresponding images being a plurality of corresponding locations along at least a portion of the subject's spine, and each of the plurality of images including at least some of the plurality of radiopaque markers;

[0095] identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; and combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0096] automatically confirming a position of the given vertebra within the combined plurality of radiographic images through image processing; and

[0097] An output is generated at least in part responsive thereto.

[0098] According to some applications of the present invention, there is also provided a method for performing a procedure on a given rib of a thorax of a subject, the method comprising:

[0099] placing a combination of a plurality of radiopaque markers in a vicinity of the object such that the plurality of markers placed at respective locations relative to the object are identifiable;

[0100] acquiring, using a radiocontrast imaging device, a plurality of radiocontrast images of a combination of the plurality of radiopaque markers, the plurality of corresponding images being a plurality of corresponding locations along at least a portion of the subject's thorax, and each of the plurality of images including at least some of the plurality of radiopaque markers; and

[0101] Using at least one computer processor to:

[0102] identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0103] combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0104] automatically confirming a position of the given rib within the combined plurality of radiographic images through image processing; and

[0105] An output is generated at least in part responsive thereto.

[0106] According to some applications of the present invention, there is also provided an apparatus for performing a procedure on a given rib of a thorax of a subject and for use with a radiographic imaging device, the apparatus comprising:

[0107] a combination of a plurality of radiopaque markers configured to be placed in a vicinity of the object such that the plurality of markers placed at respective locations relative to the object are identifiable; and

[0108] At least one computer processor configured to:

[0109] receiving a plurality of radiocontrast images of the plurality of radiopaque markers from the radiocontrast imaging device, the plurality of corresponding images being a plurality of corresponding locations along at least a portion of the subject's thorax, and each of the plurality of images including at least some of the plurality of radiopaque markers;

[0110] identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0111] combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0112] automatically confirming a position of the given rib within the combined plurality of radiographic images through image processing; and

[0113] An output is generated at least in part responsive thereto.

[0114] According to some applications of the present invention, there is also provided a computer software product for performing a procedure on a given rib of a thorax of a subject, the computer software product being used in conjunction with a radiographic imaging device and a combination of a plurality of radiopaque markers, the combination of the plurality of radiopaque markers being configured to be placed in a vicinity of the subject such that the plurality of markers placed at a plurality of corresponding positions relative to the subject are identifiable, the computer software product comprising a non-transitory computer-readable medium storing a plurality of program instructions, which, when read by a computer, cause the computer to perform the following steps:

[0115] receiving a plurality of radiocontrast images of the combination of the plurality of radiopaque markers from the radiocontrast imaging device, the plurality of corresponding images being a plurality of corresponding locations along at least a portion of the subject's thorax, and each of the plurality of images including at least some of the plurality of radiopaque markers;

[0116] identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; and combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0117] automatically confirming a position of the given rib within the combined plurality of radiographic images through image processing; and

[0118] An output is generated at least in part responsive thereto.

[0119] According to some applications of the present invention, there is also provided an apparatus for performing a procedure on a spine of a body of a subject, the apparatus comprising:

[0120] a first set of radiopaque markers;

[0121] a second set of radiopaque markers; and

[0122] one or more surfaces through which the first set of radiopaque markers and the second set of radiopaque markers are coupled to each other, the one or more surfaces being configured to:

[0123] positioning the first set of markers and the second set of markers on respective sides of the subject's spine in a plurality of predefined orientations relative to one another by placing the one or more surfaces on a portion of the subject's spine on which the procedure is to be performed; and

[0124] After positioning the first set of markers and the second set of markers on the multiple corresponding sides of the subject's spine, the one or more surfaces can be removed from the subject's body to facilitate performing the procedure on the subject's spine on the one or more surfaces placed on the subject's spine while maintaining the first set of markers and the second set of markers on the multiple corresponding sides of the subject's spine in the predefined orientation relative to each other.

[0125] In some applications, at least some of the plurality of markers in each of the first set of markers and the second set of markers are rigid and have known dimensions.

[0126] In some applications, each of the first set of markers and the second set of markers includes a combination of multiple radiopaque characters, and the multiple combinations are different from each other.

[0127] In some applications, the device also includes a third set of radiopaque markers, which are coupled to the first set of markers and the second set of markers through the one or more surfaces, such that the third set of markers is configured to be positioned along a center of the subject's spine by the one or more surfaces placed over the portion of the subject's spine on which the procedure is to be performed when the first set of markers and the second set of markers are positioned relative to each other in the multiple predefined positions on multiple corresponding sides of the subject's spine.

[0128] According to some applications of the present invention, there is also provided a method for performing a procedure on a spine of a body of a subject, the method comprising:

[0129] supply:

[0130] a first set of radiopaque markers;

[0131] a second set of radiopaque markers; and

[0132] one or more surfaces by which the first set of radiopaque markers and the second set of radiopaque markers are coupled to each other;

[0133] positioning the first set of markers and the second set of markers on respective sides of the subject's spine in a plurality of predefined orientations relative to one another by placing the one or more surfaces on a portion of the subject's spine on which the procedure is to be performed; and

[0134] After positioning the first set of markers and the second set of markers on the multiple corresponding sides of the subject's spine, the one or more surfaces are removed from the subject's body to facilitate performing the procedure on the subject's spine on the one or more surfaces placed on the subject's spine, while maintaining the first set of markers and the second set of markers on the multiple corresponding sides of the subject's spine in the predefined orientation relative to each other.

[0135] In some applications, providing the first set of markers and the second set of markers includes providing the first set of markers and the second set of markers, at least some markers of each of the first set of markers and the second set of markers being rigid and having a plurality of known dimensions.

[0136] In some applications, providing the first set of markers and the second set of markers includes providing the first set of markers and the second set of markers, each of the first set of markers and the second set of markers comprising a combination of multiple radiopaque characters, the multiple combinations being different from each other.

[0137] In some applications, the method also includes providing a third set of radiopaque markers, which are coupled to the first set of markers and the second set of markers via the one or more surfaces, wherein the one or more surfaces placed over the portion of the spine of the subject on which the procedure is to be performed include: when the first set of markers and the second set of markers are positioned relative to each other at the multiple predefined positions on multiple corresponding sides of the spine of the subject, the third set of radiopaque markers are positioned along a center of the spine of the subject.

[0138] According to some applications of the present invention, there is also provided an apparatus for use with a radiographic imaging device configured to acquire a plurality of radiographic images of a spine of a body of a subject, the apparatus comprising:

[0139] a flexible material configured to be placed over the subject's body along at least a portion of the subject's spine and to generally conform to the contours of the subject's body; and

[0140] a combination of a plurality of radiopaque markers disposed on the flexible material, at least some of the radiopaque markers being rigid and having a plurality of corresponding known dimensions, the plurality of rigid radiopaque markers being disposed on the flexible material such that the rigidity of the plurality of rigid markers does not prevent the flexible material from generally conforming to the contours of the subject's body;

[0141] At least one computer processor configured to:

[0142] identifying, by image processing, at least one of the rigid radiopaque markers within a radiographic image of the spine acquired by the radiographic imaging device; and

[0143] Based on the identified plurality of rigid radiopaque markers, sizes of features within the radiocontrast image are determined through image processing.

[0144] According to some applications of the present invention, there is also provided a method for use with a radiographic imaging device configured to acquire a plurality of radiographic images of a spine of a subject's body, the method comprising:

[0145] placing a flexible material on the subject's body along at least a portion of the subject's spine such that the flexible material substantially conforms to the contours of the subject's body, a combination of a plurality of radiopaque markers disposed on the flexible material, at least some of the radiopaque markers being rigid and having a plurality of corresponding known dimensions, the plurality of rigid radiopaque markers disposed on the flexible material such that the rigidity of the plurality of rigid markers does not prevent the flexible material from substantially conforming to the contours of the subject's body; and

[0146] Using at least one computer processor to:

[0147] identifying, by image processing, at least one of the rigid radiopaque markers within a radiographic image of the spine acquired by the radiographic imaging device; and

[0148] Based on the identified plurality of rigid radiopaque markers, sizes of features within the radiocontrast image are determined through image processing.

[0149] According to some applications of the present invention, there is also provided a method for performing surgery on a spine of a subject's body, the method comprising:

[0150] supply:

[0151] a first set of radiopaque markers; and

[0152] a second set of radiopaque markers;

[0153] Positioning the first set of markers and the second set of markers relative to the subject's spine such that the first set of markers appears in a plurality of radiographic images of the subject's spine acquired from a first imaging perspective, and such that the second set of markers appears in a plurality of radiographic images of the subject's spine acquired from a second imaging perspective;

[0154] acquiring a plurality of radiographic images of the subject's spine from the first image view and the second image view; and

[0155] A given vertebra appearing in the multiple radiographic images of the spine from the first image view is associated with a given vertebra of the spine in the multiple radiographic images of the spine from the second image view by identifying multiple markers that have a known correlation with other markers in the multiple radiographic images acquired from the first image view and the second image view.

[0156] In some applications, associating the given vertebra appearing in the multiple radiographic images of the spine from the first image view with a given vertebra of the spine from the multiple radiographic images of the spine from the second image view includes: using at least one computer processor to associate the given vertebra appearing in the multiple radiographic images of the spine from the first image view with the given vertebra of the spine from the multiple radiographic images of the spine from the second image view by identifying multiple markers that have a known correlation with other markers in the multiple radiographic images acquired from the first image view and the second image view.

[0157] In some applications, associating the given vertebra appearing in the multiple radiographic images of the spine from the first image view with a given vertebra of the spine from the multiple radiographic images of the spine from the second image view includes manually associating the given vertebra appearing in the multiple radiographic images of the spine from the first image view with the given vertebra of the spine from the multiple radiographic images of the spine from the second image view by identifying multiple markers that have a known correlation with other markers in the multiple radiographic images acquired from the first image view and the second image view.

[0158] According to some applications of the present invention, there is also provided an apparatus for performing a procedure on a spine of a body of a subject, the apparatus comprising:

[0159] a first set of radiopaque markers configured for placement relative to the subject's spine such that the first set of radiopaque markers appear in a plurality of radiographic images of the subject's spine acquired from a first imaging view; and

[0160] a second set of radiopaque markers,

[0161] The first and second sets of markers are coupled to one another such that when the first set of radiopaque markers are positioned relative to the subject's spine such that the first set of radiopaque markers appear in a radiographic image of the subject's spine acquired from a first imaging perspective, the second set of markers appear in a radiographic image of the subject's spine acquired from a second imaging perspective;

[0162] The first set of markers and the second set of markers thereby facilitate associating a given vertebra appearing in the multiple radiographic images of the spine from the first image view with a given vertebra of the spine in the multiple radiographic images of the spine from the second image view by facilitating identification of multiple markers having a known association with other markers in the multiple radiographic images acquired from the first image view and the second image view.

[0163] In some applications, the device also includes at least one computer processor configured to associate a given vertebra appearing in the multiple radiographic images of the spine from the first image view with a given vertebra of the spine in the multiple radiographic images of the spine from the second image view by identifying multiple markers that have a known correlation with other markers in the multiple radiographic images acquired from the first image view and the second image view.

[0164] According to some applications of the present invention, there is also provided a method for performing a procedure on a skeletal portion in a body of a subject, the method comprising:

[0165] placing a combination of a plurality of radiopaque markers in a vicinity of the subject;

[0166] obtaining a radiographic image of the bone portion;

[0167] acquiring an optical image of the subject's body;

[0168] Using at least one computer processor to:

[0169] identifying the plurality of radiopaque markers in the radiographic image and the optical image through image processing;

[0170] bidirectionally mapping the plurality of radiocontrast images and the optical image relative to one another based on the identification of the plurality of radiopaque markers within the plurality of radiocontrast images and the optical image;

[0171] displaying the radiographic image and the optical image separately from each other on one or more displays;

[0172] receiving an input indicating a location in the radiographic image and a first one of the plurality of optical images; and

[0173] An output is generated in response thereto, the output indicating the location of the other of the radiographic and optical images.

[0174] In some applications, receiving the input, the input indicating the location in the radiographic image and the first of the multiple optical images, includes: identifying an object placed at a suggested entry point into the skeletal portion within the optical image of the subject's body, and generating an output in response thereto, the output indicating the suggested entry point into the skeletal portion relative to the radiographic image.

[0175] In some applications, receiving the input, the input indicating the location in the radiographic image and the first of the multiple optical images, includes: identifying an object in the optical image of the subject's body that is placed at a suggested incision site into the bone portion, and generating an output in response thereto, the output indicating the suggested incision site into the bone portion in the radiographic image.

[0176] In some applications, receiving the input, the input indicating the location in the radiographic image and the first of the multiple optical images includes: receiving an input indicating a location in the optical image, the method further comprising, in response to receiving the input, using at least one computer processor, driving a display to display a cross-section of the bone portion corresponding to the indicated location.

[0177] In some applications, the method further includes, in response to receiving the input, driving, using at least one computer processor, a display to display a line on the cross-section of the bone portion indicating that the indicated location is somewhere on the line.

[0178] According to some applications of the present invention, there is also provided an apparatus for performing a procedure on a skeletal portion of a body of a subject, using a two-dimensional radiographic imaging device, an optical camera, and one or more displays, the apparatus comprising:

[0179] a set of radiopaque markers configured to be placed in a vicinity of the subject; and

[0180] At least one computer processor configured to:

[0181] receiving a radiographic image of the bone portion from the two-dimensional radiographic imaging device;

[0182] receiving an optical image of the subject's body from the optical camera;

[0183] identifying the plurality of radiopaque markers in the plurality of radiocontrast images and the optical image through image processing;

[0184] bidirectionally mapping the radiocontrast images and the optical image relative to one another based on the identification of the plurality of radiopaque markers within the plurality of radiocontrast images and the optical image;

[0185] displaying the radiographic image and the optical image separately from each other on one or more displays;

[0186] receiving an input indicating a location in the radiographic image and a first one of the plurality of optical images; and

[0187] An output is generated in response thereto, the output indicating the location of the other of the radiographic and optical images.

[0188] In some applications, the at least one computer processor is configured to receive the input by identifying an object placed at a suggested entry point into the bony portion within the optical image of the subject's body, the input indicating the location in the radiographic image and the first of the plurality of optical images, and the at least one computer processor is configured to generate the output by generating an output indicating the suggested entry point into the bony portion relative to the radiographic image.

[0189] In some applications, the at least one computer processor is configured to receive the input by identifying an object within the optical image of the subject's body to be placed at a suggested incision site into the bony portion, the input indicating the location in the radiographic image and the first of the plurality of optical images, and the at least one computer processor is configured to generate the output by generating an output indicating the incision site into the bony portion relative to the radiographic image.

[0190] In some applications, the at least one computer processor is configured to receive the input by receiving an input indicating a location in the optical image, the input indicating the location in the first of the radiographic image and the optical image, and the at least one computer processor is further configured to drive the one or more displays to display a cross-section of the bone portion corresponding to the indicated location in response to receiving the input.

[0191] In some applications, the at least one computer processor is further configured to, in response to receiving the input, drive the one or more displays to display a line on the cross-section of the skeletal portion indicating that the indicated location is somewhere on the line.

[0192] According to some applications of the present invention, there is also provided a computer software product for executing a program on a skeletal portion of a body of a subject using a two-dimensional radiographic imaging device, an optical camera, one or more displays, and a combination of a plurality of radiopaque markers, wherein the combination of the plurality of radiopaque markers is configured to be placed in a vicinity of the subject, the computer software product comprising a non-transitory computer-readable medium storing a plurality of program instructions, which, when read by a computer, cause the computer to perform the following steps:

[0193] receiving a radiographic image of the bone portion from the two-dimensional imaging device; receiving an optical image of the subject's body from the optical camera;

[0194] identifying, through image processing, the plurality of radiopaque markers in the radiocontrast image and the optical image; and bidirectionally mapping the radiocontrast image and the optical image relative to each other based on the identification of the plurality of radiopaque markers within the radiocontrast image and the optical image;

[0195] displaying the radiographic image and the optical image separately from each other on one or more displays; receiving an input indicating a location in the radiographic image and a first one of the plurality of optical images; and

[0196] An output is generated in response thereto, the output indicating the location of the other of the radiographic and optical images.

[0197] According to some applications of the present invention, there is also provided a method for performing a procedure using a tool configured to be advanced along a longitudinal insertion path into a bone portion within a body of a subject, the method comprising:

[0198] Acquiring three-dimensional image data of the bone part;

[0199] When a portion of the tool is disposed at a position along the longitudinal insertion path, sequentially:

[0200] acquiring a first two-dimensional X-ray image of at least the portion of the tool and the bone portion from a first viewing angle using a two-dimensional X-ray imaging device, the two-dimensional X-ray imaging device not being registered to the subject's body and the two-dimensional X-ray imaging device being positioned in a first posture relative to the subject's body;

[0201] moving the two-dimensional X-ray imaging device to a second position relative to the subject's body, and

[0202] When the two-dimensional X-ray imaging device is in the second posture, acquiring a second two-dimensional X-ray image of at least the portion of the tool and the bone portion from a second viewing angle;

[0203] Using at least one computer processor to:

[0204] Registering the first two-dimensional X-ray image and the second two-dimensional X-ray image to the three-dimensional image data, the registering comprising:

[0205] generating a plurality of two-dimensional projections from the three-dimensional imaging device; and

[0206] identifying a first two-dimensional projection and a second two-dimensional projection that match the first two-dimensional X-ray image and the second two-dimensional X-ray image;

[0207] identifying, by image processing, the position of the portion of the tool relative to the bone portion in the first X-ray image and the second X-ray image;

[0208] deriving a relationship between the position of the portion of the tool relative to the three-dimensional image data and a given position within the three-dimensional image data based on the identified position of the portion of the tool within the first and second x-ray images and registering the first and second two-dimensional x-ray images to the three-dimensional image data; and

[0209] An output is generated that indicates the relationship between the position of the portion of the tool relative to the three-dimensional image data and a given position within the three-dimensional image data.

[0210] In some applications, generating the output to derive the relationship between the position of the portion of the tool relative to the three-dimensional image data and the given position within the three-dimensional image data includes generating the output on a two-dimensional cross-section of the bone portion derived from the three-dimensional image data.

[0211] In some applications, generating the output to derive the relationship between the position of the portion of the tool relative to the three-dimensional image data and the given position within the three-dimensional image data includes generating the output on a two-dimensional projection of the bone portion derived from the three-dimensional image data.

[0212] In some applications, deriving the relationship between the position of the portion of the tool relative to the three-dimensional image data and the given position within the three-dimensional image data includes generating the output on a three-dimensional image of the skeletal portion derived from the three-dimensional image data.

[0213] In some applications, deriving the relationship between the position of the portion of the tool relative to the three-dimensional image data and the given position within the three-dimensional image data includes deriving an expected longitudinal insertion path of the tool relative to the given position within the three-dimensional image data.

[0214] In some applications, deriving the relationship between the position of the portion of the tool relative to the three-dimensional image data and the given position within the three-dimensional image data includes deriving a relationship between the first position of the portion of the tool relative to the three-dimensional image data and a pre-specified target position within the three-dimensional image data.

[0215] In some applications, deriving the relationship between the position of the portion of the tool relative to the three-dimensional image data and the given position within the three-dimensional image data includes deriving a relationship between the first position of the portion of the tool relative to the three-dimensional image data and a plurality of corresponding volumes within the three-dimensional image data, the plurality of corresponding volumes specifying a plurality of corresponding acceptable degrees of protrusion of the tool relative to the plurality of corresponding volumes.

[0216] According to some applications of the present invention, there is also provided a method for performing a procedure using a tool configured to be advanced along a longitudinal insertion path into a bone portion within a body of a subject, the method comprising:

[0217] Acquiring three-dimensional image data of the bone part;

[0218] specifying a position within the skeletal portion relative to the three-dimensional image data;

[0219] acquiring a first two-dimensional X-ray image of at least the portion of the tool and the bone portion from a first viewing angle using a two-dimensional X-ray imaging device, the two-dimensional X-ray imaging device not being registered to the subject's body and the two-dimensional X-ray imaging device being positioned in a first posture relative to the subject's body;

[0220] then moving the two-dimensional X-ray imaging device to a second posture relative to the subject's body;

[0221] When the two-dimensional X-ray imaging device is in the second posture, acquiring a second two-dimensional X-ray image of at least the portion of the tool and the bone portion from a second viewing angle;

[0222] Using at least one computer processor to:

[0223] Registering the first two-dimensional X-ray image and the second two-dimensional X-ray image to the three-dimensional image data, the registering comprising:

[0224] generating a plurality of two-dimensional projections from the three-dimensional image data; and

[0225] identifying a first two-dimensional projection and a second two-dimensional projection corresponding to the first two-dimensional X-ray image and the second two-dimensional X-ray image data of the skeletal portion;

[0226] identifying, by image processing, the position of the portion of the tool relative to the bone portion in the first two-dimensional X-ray image and the second two-dimensional X-ray image;

[0227] deriving an orientation of the designated location within at least one of the plurality of two-dimensional x-ray images of the skeletal portion based on the identified location of the portion of the tool within the first and second two-dimensional x-ray images and registering the first and second two-dimensional x-ray images to the three-dimensional image data,

[0228] identifying, by image processing, a position of at least a portion of the tool relative to the at least one of the plurality of two-dimensional rays of the skeletal portion;

[0229] confirming a relationship between an intended longitudinal insertion path of the tool and the designated location within at least one of the plurality of two-dimensional x-rays of the bone portion based on the identified location of the tool portion; and

[0230] An output is generated at least in part responsive thereto.

[0231] In some applications, specifying the position within the skeletal component relative to the three-dimensional image data includes specifying a target position within the skeletal component relative to the three-dimensional image data.

[0232] In some applications, specifying the location within the bone portion relative to the three-dimensional image data includes specifying one or more locations within the bone portion that the tool should avoid relative to the three-dimensional image data.

[0233] In some applications, specifying the position within the skeletal component relative to the three-dimensional image data includes specifying the position relative to a cross-section of the skeletal component derived from the three-dimensional image data.

[0234] In some applications, specifying the position within the skeletal component relative to the three-dimensional image data includes specifying the position relative to a two-dimensional projection of the skeletal component derived from the three-dimensional image data.

[0235] In some applications, specifying the position within the skeletal component relative to the three-dimensional image data includes specifying the position relative to a three-dimensional image of the skeletal component derived from the three-dimensional image data.

[0236] According to some applications of the present invention, there is also provided a method for performing a procedure using a tool configured to be advanced along a longitudinal insertion path into a bone portion within a body of a subject, the method comprising:

[0237] Acquiring three-dimensional image data of the bone part;

[0238] acquiring a first two-dimensional X-ray image of at least the portion of the tool and the bone portion from a first viewing angle using a two-dimensional X-ray imaging device, the two-dimensional X-ray imaging device not being registered to the subject's body and the two-dimensional X-ray imaging device being positioned in a first posture relative to the subject's body;

[0239] then moving the two-dimensional X-ray imaging device to a second posture relative to the subject's body;

[0240] When the two-dimensional X-ray imaging device is in the second posture, acquiring a second two-dimensional X-ray image of at least the portion of the tool and the bone portion from a second viewing angle;

[0241] Using at least one computer processor to:

[0242] Registering the first two-dimensional X-ray image and the second two-dimensional X-ray image to the three-dimensional image data, the registering comprising:

[0243] generating a plurality of two-dimensional projections from the three-dimensional image data; and

[0244] identifying a first two-dimensional projection and a second two-dimensional projection corresponding to the first two-dimensional X-ray image and the second two-dimensional X-ray image data of the skeletal portion;

[0245] identifying, by image processing, a position of a tip of the tool relative to the bone portion in the first two-dimensional X-ray image and the second two-dimensional X-ray image;

[0246] confirming a position of the tip of the tool relative to the tool in the three-dimensional image data based on the identified position of the portion of the tool in the first and second two-dimensional x-ray images and registering the first and second two-dimensional x-ray images to the three-dimensional image data; and

[0247] In response thereto, a display is driven to display a cross-section of the bone portion, the cross-section being derived from the three-dimensional image data and corresponding to the position of the tip of the tool.

[0248] In some applications, the method further includes driving the display to display an indication of a position of the tool on the cross-section of the bone portion.

[0249] In some applications, driving the display to display the cross-section of the bone portion includes: driving the display to display a cross-sectional view based on a direction of insertion of the tool, the cross-sectional view being selected from a group consisting of an axial cross-section, a coronal cross-section, a sagittal cross-section, and a cross-sectional view.

[0250] According to some applications of the present invention, there is also provided a method for performing a procedure using a tool configured to be advanced along a longitudinal insertion path into a given vertebra within a body of a subject, the method comprising:

[0251] acquiring three-dimensional image data of at least a portion of a spinal column of the subject, the spinal column including the given vertebra;

[0252] acquiring a first two-dimensional x-ray image of at least the portion of the tool and a spine of the subject, the spine including the given vertebra, from a first viewing angle using a two-dimensional x-ray imaging device, the two-dimensional x-ray imaging device not being registered to the subject's body and the two-dimensional x-ray imaging device being positioned in a first posture relative to the subject's body;

[0253] then moving the two-dimensional X-ray imaging device to a second posture relative to the subject's body;

[0254] acquiring a second two-dimensional X-ray image of at least the portion of the tool and the subject's spine from a second viewing angle when the two-dimensional X-ray imaging device is in the second posture, the spine including the given vertebra;

[0255] Using at least one computer processor to:

[0256] receiving an input indicating a location of the given vertebra within the three-dimensional image data;

[0257] automatically identifying a position of the given vertebra in the first two-dimensional X-ray image and the second two-dimensional X-ray image through image processing;

[0258] Registering the given vertebra in the first two-dimensional X-ray image and the second two-dimensional X-ray image to the given vertebra in the three-dimensional image data, the registering comprising:

[0259] generating a plurality of two-dimensional projections of the given vertebra from the three-dimensional image data; and

[0260] identifying a first two-dimensional projection and a second two-dimensional projection that match the given vertebra within the first two-dimensional x-ray image and the second two-dimensional x-ray image of the skeletal portion, respectively;

[0261] identifying, by image processing, a position of the portion of the tool relative to the given vertebra within the first two-dimensional X-ray image and the second two-dimensional X-ray image of the portion of the spine;

[0262] confirming the position of the portion of the tool relative to the three-dimensional image data based on the identified position of the portion of the tool within the first and second x-ray images and registering the first and second two-dimensional x-ray images to the three-dimensional image data; and

[0263] In response thereto, an output is generated.

[0264] According to some applications of the present invention, there is also provided a method for use with a radiographic imaging device and an instrument having a straight radiopaque component, the method comprising:

[0265] using the radiocontrast imaging device, acquiring a radiocontrast image in which at least a portion of the straight radiopaque component of the device is visible;

[0266] Using at least one computer processor to:

[0267] identifying, by image processing, the straight radiopaque component of the device within the radiocontrast image;

[0268] at least partially correcting distortion in at least a portion of the radiographic image by deforming the portion of the radiographic image so that the straight radiopaque portion of the device within the radiographic image appears straight; and

[0269] An output is generated in response thereto.

[0270] In some applications, the method further includes correcting an additional portion of the image based on the correction applied to a portion of the image that is present within the straight radiopaque component of the device.

[0271] In some applications, the method further comprises registering the corrected radiographic image to three-dimensional image data using the at least one computer processor.

[0272] In some applications, acquiring the radiocontrast image includes acquiring a radiocontrast image of the straight radiopaque component of the instrument and a body portion of a subject, and registering the corrected radiocontrast image to the three-dimensional image data includes:

[0273] generating a plurality of two-dimensional projections of the portion of the subject's body from the three-dimensional image data; and

[0274] A two-dimensional projection is identified that matches the portion of the subject's body within the radiographic image.

[0275] According to some applications of the present invention, there is also provided an apparatus for use with a radiographic imaging device, an output device, and an instrument having a straight radiopaque component, the apparatus comprising:

[0276] At least one computer processor configured to:

[0277] receiving a radiocontrast image from the radiocontrast imaging device in which at least a portion of the straight radiopaque member of the device is visible;

[0278] identifying, by image processing, the straight radiopaque component of the device within the radiocontrast image;

[0279] at least partially correcting distortion in at least a portion of the radiographic image by deforming the portion of the radiographic image such that the straight radiopaque portion of the device within the radiographic image appears straight; and

[0280] In response thereto, an output is generated on the output device.

[0281] In some applications, the at least one computer processor is further configured to correct an additional portion of the image based on the correction applied to a portion of the image present within the straight radiopaque component of the device.

[0282] In some applications, the at least one computer processor is further configured to register the corrected radiographic image to three-dimensional image data.

[0283] In some applications, the radiographic image includes the straight radiopaque component of the device and a body portion of a subject, and the at least one computer processor is configured to register the corrected radiographic image to the three-dimensional image data by:

[0284] generating a plurality of two-dimensional projections of the portion of the subject's body from the three-dimensional image data, and

[0285] A two-dimensional projection is identified that matches the portion of the subject's body within the radiographic image.

[0286] According to some applications of the present invention, there is also provided a computer software product for use with a radiographic imaging device, an output device, and an instrument having a straight radiopaque component.

[0287] The computer software product includes a non-transitory computer-readable medium storing a plurality of program instructions that, when read by a computer, cause the computer to perform the following steps: receiving a radiocontrast image from the radiocontrast imaging device in which at least a portion of the straight radiopaque component of the device is visible;

[0288] identifying, by image processing, the straight radiopaque portion of the device within the radiocontrast image; at least partially correcting distortion in at least a portion of the radiocontrast image by deforming the portion of the radiocontrast image so that the straight radiopaque portion of the device within the radiocontrast image appears straight, and

[0289] In response thereto, an output is generated on the output device.

[0290] According to some applications of the present invention, there is also provided a method for performing a plurality of interventional procedures on at least a first vertebra and a second vertebra of a spine of a subject, the method comprising:

[0291] acquiring imaging data of the subject's spine using an imaging device;

[0292] Using at least one computer processor to:

[0293] generating on a display a spinal roadmap image of at least a portion of the spine, the spine including the first vertebra and the second vertebra;

[0294] automatically labeling vertebrae within the spinal roadmap image;

[0295] confirming that an intervention has been performed on the first vertebra such that an appearance of the first vertebra has been altered; and

[0296] The spinal roadmap is automatically updated to reflect the change in the appearance of the first vertebra such that when the intervention is performed on the second vertebra, the updated spinal roadmap is displayed.

[0297] In some applications, acquiring the imaging data includes acquiring three-dimensional imaging data, and generating the spinal pathmap image includes generating a three-dimensional spinal pathmap image.

[0298] In some applications, acquiring the imaging data includes acquiring 2D imaging data, and generating the spinal roadmap image includes generating a 2D spinal roadmap image.

[0299] In some applications, confirming that an intervention has been performed on the first vertebra such that an appearance of the first vertebra has been changed includes confirming that a tool has been inserted into the first vertebra, and updating the spinal roadmap includes updating the spinal roadmap to display the tool within the vertebra.

[0300] According to some applications of the present invention, there is also provided an apparatus for use in performing a procedure for performing multiple interventions on at least a first vertebra and a second vertebra of a spine of a subject, a display, and an imaging device, the imaging device being configured to acquire imaging data of the spine of the subject, the apparatus comprising:

[0301] At least one computer processor configured to:

[0302] receiving the imaging data from the imaging device;

[0303] generating on the display a spinal roadmap image of at least a portion of the spinal column, the spinal column including the first vertebra and the second vertebra;

[0304] automatically labeling vertebrae within the spinal roadmap image;

[0305] confirming that an intervention has been performed on the first vertebra such that an appearance of the first vertebra has been altered; and

[0306] The spinal roadmap is automatically updated to reflect the change in the appearance of the first vertebra such that when the intervention is performed on the second vertebra, the updated spinal roadmap is displayed.

[0307] In some applications, the imaging device includes an imaging device configured to acquire three-dimensional imaging data, and the at least one computer processor is configured to generate the spinal roadmap image by generating a three-dimensional spinal roadmap image.

[0308] In some applications, the imaging device includes an imaging device configured to acquire two-dimensional imaging data, and the at least one computer processor is configured to generate the spinal roadmap image by generating a two-dimensional spinal roadmap image.

[0309] In some applications, the at least one computer processor is configured to confirm that an intervention has been performed on the first vertebra by confirming that a tool has been inserted into the first vertebra such that an appearance of the first vertebra has been changed, and the at least one computer processor is configured to update the spinal roadmap by updating the spinal roadmap to display the tool within the vertebra.

[0310] According to some applications of the present invention, there is also provided a computer software product having a program for performing a plurality of interventions on at least a first vertebra and a second vertebra of a subject's spine, a display, and an imaging device configured to acquire imaging data of the subject's spine, the computer software product comprising a non-transitory computer-readable medium storing a plurality of program instructions, which, when read by a computer, cause the computer to perform the following steps:

[0311] receiving the image data from the imaging device;

[0312] generating a spinal roadmap image of at least a portion of the spine on the display;

[0313] automatically labeling vertebrae within the spinal roadmap image;

[0314] confirming that an intervention has been performed on the first vertebra such that an appearance of the first vertebra has been altered; and

[0315] The spinal roadmap is automatically updated to reflect the change in the appearance of the first vertebra such that when the intervention is performed on the second vertebra, the updated spinal roadmap is displayed.

[0316] According to some applications of the present invention, there is also provided a method of applying a tool to a body part of a subject during a medical intervention, the method comprising:

[0317] acquiring a plurality of images of the tool and the subject's body during the medical intervention;

[0318] determining, using at least one computer processor, a position of the tool relative to the subject's body by analyzing a plurality of images;

[0319] detecting, using a motion detection sensor, that movement of at least a portion of the subject's body has occurred exceeding a threshold amount between acquisition of two or more images, the motion detection sensor being disposed on the subject's body and configured to detect movement of the motion detection sensor itself compared to a previous orientation of the motion detection sensor; and

[0320] An alarm is generated in response thereto, the alarm indicating that the motion has occurred.

[0321] In some applications, generating the alarm includes generating the alarm by driving an output device of the motion detection sensor to generate an alarm.

[0322] In some applications, generating the alert includes: receiving a signal from the motion detection sensor at the computer processor indicating that movement of at least a portion of the subject's body has occurred exceeding the threshold amount, and

[0323] Using the computer processor, an output is generated in response thereto.

[0324] In some applications, generating the alert includes generating an output indicating that one or more images should be reacquired.

[0325] According to some applications of the present invention, there is also provided an apparatus for applying a tool to a part of a body of a subject during a medical intervention, and an imaging device configured to acquire a plurality of images of the subject's body during the intervention, the apparatus comprising:

[0326] at least one computer processor configured to determine a position of the tool relative to the subject's body by analyzing a plurality of images;

[0327] a motion detection sensor configured to be disposed on the subject's body and to detect movement of at least the portion of the subject's body by detecting movement of the motion detection sensor itself compared to a previous position of the motion detection sensor, the motion detection sensor being configured to detect movement of at least the portion of the subject's body that exceeds a threshold amount between acquisition of two or more images; and

[0328] An output device is configured to generate an alarm in response to the motion detection sensor detecting that movement of at least the portion of the subject's body has occurred exceeding the threshold amount, the alarm indicating that movement has occurred.

[0329] In some applications, the motion detection sensor is configured to drive the output device to generate the alarm.

[0330] In some applications, the at least one computer processor is configured to receive a signal from the sensor indicating that movement of at least a portion of the subject's body has occurred exceeding the threshold amount, and to drive the output device to generate the alarm in response thereto.

[0331] In some applications, the output device is configured to generate an output indicating that one or more images should be reacquired when the motion detection sensor detects that movement of at least the portion of the subject's body has occurred exceeding the threshold amount.

[0332] According to some applications of the present invention, there is also provided a method for performing a procedure using a tool configured to be advanced along a longitudinal insertion path into a bone portion within a body of a subject, the method comprising:

[0333] Acquiring three-dimensional image data of the bone part;

[0334] acquiring one or more two-dimensional x-ray images of at least the portion of the tool from a single viewing angle using a two-dimensional x-ray imaging device when a tip of the tool is positioned at a given position along the longitudinal insertion path, the two-dimensional x-ray imaging device not being registered to the subject's body and the two-dimensional x-ray imaging device being positioned in a first posture relative to the subject's body;

[0335] Using a computer processor to:

[0336] registering one of the one or more two-dimensional X-ray images acquired from the single view to the three-dimensional image data, the registering comprising:

[0337] generating a plurality of two-dimensional projections from the three-dimensional image data; and

[0338] identifying a two-dimensional projection that matches the one of the one or more two-dimensional X-ray images of the skeletal portion acquired from the single view;

[0339] identifying, by image processing, a position of the tip of the tool relative to the bone portion within the one of the one or more two-dimensional x-ray images;

[0340] confirming a position of the tip of the tool relative to the two-dimensional projection that matches the one of the one or more two-dimensional x-ray images of the bone portion based on the identified position of the tip of the tool relative to the bone portion within the one of the one or more two-dimensional x-ray images and registering the one of the one or more two-dimensional x-ray images to the three-dimensional image data; and

[0341] In response thereto, a display is driven to display a cross-section of the bone portion, the cross-section being derived from the three-dimensional image data and corresponding to the position of the tip of the tool of the two-dimensional projection matched to the one or more two-dimensional X-ray images of the bone portion.

[0342] In some applications, the method further includes driving the display to display a line on the cross section, the line indicating where the position of the tip of the tool within the cross section is on the line.

[0343] According to some applications of the present invention, there is also provided a method of applying a tool to a body part of a subject during a medical intervention, the method comprising:

[0344] Acquiring three-dimensional image data of the bone part;

[0345] Using at least one computer processor to:

[0346] calculating a position of the proximal portion of the tool disposed outside the subject's body;

[0347] deriving a position of the distal portion of the tool relative to the portion of the subject's body based on the calculated position of the proximal portion of the tool relative to the three-dimensional image data; and

[0348] driving a display to display the indication of the distal portion of the tool relative to the portion of the subject's body relative to the three-dimensional image data based on the derived position;

[0349] In order:

[0350] acquiring a first two-dimensional X-ray image of the distal portion of the tool and the subject's body from a first viewing angle using a two-dimensional X-ray imaging device, the two-dimensional X-ray imaging device not being registered to the subject's body and the two-dimensional X-ray imaging device being disposed in a first posture relative to the subject's body;

[0351] Next, moving the two-dimensional X-ray imaging device to a second posture relative to the subject's body; and

[0352] acquiring a second two-dimensional X-ray image of the distal portion of the tool and the subject's body from a second viewing angle when the two-dimensional X-ray imaging device is in the second posture;

[0353] Using at least one computer processor to:

[0354] registering the portion of the subject's body within the first and second two-dimensional x-ray images to the portion of the subject's body within the three-dimensional image data;

[0355] identifying, by image processing, a position of the distal end portion of the tool relative to the bone portion in the first two-dimensional X-ray image and the second two-dimensional X-ray image;

[0356] confirming the position of the distal portion of the tool relative to the tool in the three-dimensional image data based on registering the portion of the subject's body within the first and second two-dimensional x-ray images to the portion of the subject's body within the three-dimensional image data and the identified position of the distal portion of the tool within the first and second two-dimensional x-ray images; and

[0357] Based on the confirmed position of the distal portion of the tool, the display is driven to update the indication of the position of the distal portion of the tool relative to the portion of the subject's body relative to the three-dimensional image data.

[0358] In some applications, registering the portion of the subject's body within the first and second two-dimensional x-ray images to the portion of the subject's body within the three-dimensional image data includes:

[0359] generating a plurality of two-dimensional projections of the portion of the subject's body from the three-dimensional image data; and

[0360] A first two-dimensional projection and a second two-dimensional projection are identified that match the portion of the subject's body within the first two-dimensional X-ray image and the second two-dimensional X-ray image of the portion of the subject's body, respectively.

[0361] In some applications, calculating the position of the proximal portion of the tool disposed outside the body of the subject includes calculating the position of the proximal portion of the tool disposed outside the body of the subject via one or more position sensors coupled to the proximal portion of the tool.

[0362] In some applications, calculating the position of the proximal portion of the tool disposed outside the body of the subject includes calculating the position of the proximal portion of the tool disposed outside the body of the subject by tracking the proximal portion of the tool through image tracking.

[0363] In some applications, calculating the position of the proximal portion of the tool disposed outside the body of the subject includes calculating the position of the proximal portion of the tool disposed outside the body of the subject by tracking a portion of a robot, the portion of the robot being at a known previous position relative to the portion of the robot to which the proximal portion of the tool is coupled.

[0364] The present invention will be more fully understood from the following detailed description of various embodiments of the present invention taken in conjunction with the accompanying drawings, in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0365] Figure 1A is a schematic diagram of an orthopedic operating room used in the prior art;

[0366] Figure 1B is a schematic diagram of a system for performing multiple procedures on skeletal anatomy according to some applications of the present invention;

[0367] Figure 2 Two tools are inserted into one vertebra as used in the prior art (e.g. Jamshidi TM Needle) and a schematic diagram of multiple insertion windows required for inserting such multiple tools;

[0368] Figure 3A and 3B is a three-dimensional computed tomography image of a vertebra as used in the prior art ( Figure 3A ) and a two-dimensional axial slice derived from the three-dimensional computed tomography image ( Figure 3B ) multiple schematic diagrams;

[0369] Figure 4A and 4B A method for obtaining an anterior-posterior ("AP") two-dimensional radiographic image and a combined anterior-posterior image ( Figure 4A ), and the C-arm ( Figure 4B );

[0370] Figure 5A 、 5B 5C, 5D, and 5E are schematic diagrams of combinations of radiopaque markers placed on a subject according to some applications of the present invention;

[0371] Figure 6A and 6B are multiple schematic diagrams of a first set of radiopaque and a second set of radiopaque markers configured for placement on a subject according to some applications of the present invention;

[0372] Figure 7 is a diagram showing the general use of some applications according to the present invention Figure 1B a flow chart of a plurality of steps performed by the system;

[0373] Figure 8A showing a vertebra designated on a plurality of cross-sectional images of a spinal column of a subject derived from three-dimensional image data according to some applications of the present invention;

[0374] Figure 8B shows an example of a three-dimensional computed tomography image of a subject's spine displayed together with a combined radiographic image of the subject's spine in accordance with some applications of the present invention;

[0375] Figure 8C FIG. 1 shows indicating the designated vertebra on a three-dimensional computed tomography image and a two-dimensional X-ray image according to some applications of the present invention, wherein the computed tomography image and the X-ray image are displayed side by side with each other;

[0376] Figure 9 An example of an optical image displayed together with a two-dimensional radiographic image according to some applications of the present invention is shown;

[0377] Figure 10 An example of a two-dimensional radiographic (e.g., X-ray) image displayed together with a cross-sectional image of a subject's vertebra derived from three-dimensional image data of the vertebra according to some applications of the present invention is shown;

[0378] Figure 11A and 11B The Jamshidi TM multiple examples of anterior-posterior and lateral x-ray images of a needle inserted into a subject's spine;

[0379] Figure 12A and 12B shows examples of correspondence between a plurality of respective views of a three-dimensional image of a vertebra and corresponding first and second x-ray images of the vertebra according to some applications of the present invention;

[0380] Figure 13A 、 13B and 13C show a three-dimensional image of an object used in some applications according to the present invention (where Figure 13A The example is a cone) and the side to side of the object ( Figure 13B ) and from bottom to top ( Figure 13C )'s two-dimensional projections;

[0381] Figure 14A and 14B Shows some applications according to the present invention that are typically used Figure 1B a plurality of flow charts of the plurality of steps performed by the system;

[0382] Figure 15A An example of multiple axial cross-sections of a vertebra is shown, as shown in multiple corresponding two-dimensional X-ray images obtained from a single X-ray image view, the multiple axial cross-sections corresponding to first and second positions, respectively, of a tip of a tool advanced into the vertebra along a longitudinal insertion path, according to some applications of the present invention;

[0383] Figure 15B An example of multiple axial cross-sections of a vertebra is shown, according to some applications of the present invention, at first and second positions, respectively, of a tip of a tool advanced into the vertebra along a shown longitudinal insertion path, the multiple positions being derived using multiple X-ray images acquired from two or more X-ray image views;

[0384] Figure 16A and 16B Several examples of a display according to some applications of the present invention are shown, the display showing a relationship between a given location specified on three-dimensional (e.g., computed tomography or magnetic resonance imaging) image data and an intended longitudinal insertion path of a tool and the given location on two-dimensional anterior-posterior and lateral X-ray images, respectively;

[0385] Figure 17A An anterior-posterior x-ray of two tools inserted into a vertebra through the 10-11 o'clock and 1-2 o'clock insertion windows, respectively, is shown, the anterior-posterior x-ray being generated using various prior art techniques;

[0386] Figure 17B Shown Figure 17A a corresponding lateral X-ray image of the anterior-posterior X-ray image, the anterior-posterior X-ray image being generated using a plurality of prior art techniques;

[0387] Figure 18 A Jamshidi having a radiopaque clip attached thereto according to some applications of the present invention. TM A schematic diagram of the needle;

[0388] Figure 19A An anterior-posterior X-ray image and a corresponding lateral X-ray image of a vertebra are shown during a first stage of inserting a tool into the vertebra according to some applications of the present invention;

[0389] Figure 19Bshowing an indication of the derived current position of the tip of the tool displayed on an anterior-posterior X-ray image of a vertebra and on a lateral X-ray image of the vertebra during a second stage of inserting the tool into the vertebra according to some applications of the present invention;

[0390] Figure 20 is a schematic diagram of a three-dimensional rigid fixture according to some applications of the present invention, the three-dimensional rigid fixture including a plurality of portions that are at least radiopaque and function as a plurality of radiopaque markers disposed in a predefined three-dimensional arrangement;

[0391] Figure 21A Examples of X-ray images of a calibration jig produced by a C-arm using an image intensifier and by a C-arm using a flat panel detector are shown, such images reflecting prior art techniques; and

[0392] Figure 21B An example of an X-ray image acquired by a C-arm using an image intensifier according to some applications of the present invention is shown, the image including a radiopaque component corresponding to a portion of a tool known to be straight, and a dashed line overlaid on the image and indicating how a line defined by the straight portion would appear if multiple distortions in the image were corrected. DETAILED DESCRIPTION

[0393] Now refer to Figure 1B , Figure 1B is a schematic diagram of a system 20 for performing multiple procedures on skeletal anatomy according to some applications of the present invention. For some applications, the system is used for a procedure performed on one or more vertebrae of a subject's spine. However, the scope of the present invention includes applying any of the multiple devices and multiple methods described herein to multiple procedures performed on other portions of a subject's skeletal anatomy, mutatis mutandis. Such multiple procedures can include joint (e.g., shoulder, knee, hip, and / or ankle) replacement, joint repair, fracture repair (e.g., femur, tibia, and / or fibula), a procedure performed on a rib (e.g., rib removal or rib resection), and / or other interventions in which three-dimensional image data is acquired prior to the intervention and multiple two-dimensional images are acquired during the intervention.

[0394] The system 20 typically includes a computer processor 22 interacting with a memory 24 and one or more user interface devices 26. Typically, the user interface devices include one or more input devices, such as a keyboard 28 (shown), and one or more output devices, such as a display 30 (shown). The inputs and outputs of the computer processor described herein are typically performed through the user interface devices. For some applications, the computer processor, the memory, and the user interface devices are combined into a single unit, such as a tablet device and / or a laptop computer.

[0395] For some applications, the user interface device includes a mouse, a joystick, a touch screen device (e.g., a smartphone or a tablet), a touchpad, a trackball, a voice command interface, and / or other types of user interfaces known in the art. For some applications, the output device includes a heads-up display and / or a head-mounted display, such as Google Glass. ). For some applications, the computer processor generates an output on different types of visual, textual, graphical, tactile, audio and / or video output devices (e.g., speakers, headphones, smartphones, or tablet computers). For some applications, a user interface device serves as an input device and an output device. For some applications, the computer processor 22 generates an output on a computer-readable medium (e.g., a non-transitory computer-readable medium), such as a disk or a portable USB drive. For some applications, the computer processor comprises a portion of a picture archiving and communication system (PACS) and is configured to receive multiple inputs from other components of the system via, for example, memory 24. Alternatively or additionally, the computer processor is configured to receive an input on a computer-readable medium (e.g., a non-transitory computer-readable medium), such as a disk or a portable USB drive. It should be noted that for some applications, more than one computer processor is used to perform the multiple functions described herein as being performed by the computer processor 22.

[0396] Typically, three-dimensional image data is acquired before the subject enters the operating room, or while the subject is in the operating room, but before the intervention begins. For example, a computed tomography scanner 32 can be used to acquire three-dimensional computed tomography image data of the portion of the anatomical structure of the bone on which a procedure is to be performed (and / or multiple adjacent portions of the anatomical structure). Alternatively or additionally, a magnetic resonance imaging scanner can be used to acquire three-dimensional magnetic resonance imaging image data of the portion of the anatomical structure of the bone on which a procedure is to be performed (and / or multiple adjacent portions of the anatomical structure). For some applications, three-dimensional X-ray data is acquired. Typically, the three-dimensional image data is transferred to a memory 24 and accessed from the memory by the computer processor 22. It should be noted that for illustrative purposes, Figure 1B The CT scanner, the C-arm, and the system 20 are shown together. However, as described above, for some applications, the CT scanner is not located in the same room as the system 20 and / or the C-arm 34.

[0397] During the procedure, a plurality of real-time two-dimensional images are acquired by a radiographic imaging device (e.g., C-arm 34 (shown)) that acquires a plurality of two-dimensional x-ray images. For some applications, the plurality of two-dimensional images are captured in real time by a frame grabber of system 20, the frame grabber of system 20 being connected to an output port of the C-arm. Alternatively or additionally, system 20 and the C-arm are interconnected via a picture archiving and communication system (PACS) network, system 20 and C-arm 34 being connected to the PACS network, and once acquired, the two-dimensional images are transmitted to system 20 (e.g., via memory 24) via the PACS network. Alternatively or additionally, the C-arm sends the plurality of image files directly to system 20 (e.g., via memory 24), for example, in a Digital Imaging and Communications Technology (DICOM) format.

[0398] Typically, the interventional portion of a procedure performed on bony anatomy such as the spine begins with the insertion of a tool, such as a Jamshidi TM Needle 36. Jamshidi TM The needle usually consists of an inner tube and an outer tube. TM The needle is usually inserted into a target location, at which point the Jamshidi TM The needle is inserted into other tools and / or implants. Typically, in open surgery, for multiple smaller diameter tools and / or multiple implants, the Jamshidi TMThe inner tube of the needle is removed and passed through the Jamshidi TM The outer tube of the needle is inserted into the tool and / or implant, and for multiple tools and / or multiple implants of larger diameter, by removing the Jamshidi TM The inner tube of the needle is used to insert the tool and / or implant, a rigid guide wire is inserted through the outer tube, the outer tube is removed, and then the tool and / or implant is inserted along the rigid guide wire. For minimally invasive surgery, the multiple steps described above (or similar steps) are usually performed through multiple small incisions.

[0399] It should be noted that generally throughout the specification and claims of this application, the term "tool" should be interpreted to include any tool or implant inserted into any part of the bony anatomy during a procedure performed on the bony anatomy. Such multiple tools may include flexible, rigid and / or semi-rigid probes, and may include multiple diagnostic probes, therapeutic probes and / or imaging probes. For example: the multiple tools may include Jamshidi TM Needles, other needles, K-wires, pedicle markers, screws, nails, other implants, implant delivery probes, drills, endoscopes, probes inserted through an endoscope, tissue ablation probes, laser probes, balloon probes, injection needles, tissue removal probes, drug delivery probes, stimulation probes, dilators, etc. Typically, such procedures include various spinal stabilization procedures, such as: vertebroplasty (i.e., injection of synthetic or biologic bone cement to stabilize spinal fractures), kyphoplasty (i.e., injection of synthetic or biologic bone cement to stabilize spinal fractures with an additional step of inflating a balloon in the area of the fracture prior to injecting the bone cement), fixation (i.e., anchoring two or more vertebrae to each other by inserting a device such as a screw into each of the vertebrae and connecting the screws to a plurality of studs), fixation and fusion (i.e., fixation with an additional step of an implant, such as a cage placed between the vertebral bodies of the vertebrae), and / or endoscopy (i.e., inserting an endoscope into a vertebra and / or an intervertebral disc, for example, to remove tissue (e.g., disc tissue or vertebrae) that is compressing nerves).

[0400] Now refer to Figure 2 , Figure 2 are two Jamshidi as used in the prior art TMA schematic diagram of a needle 36 inserted into a vertebra 38. Typically, spinal intervention targeting a vertebral body is performed with multiple tools positioned with multiple insertion windows at 10-11 o'clock and 1-2 o'clock relative to the subject's spine. Insertion of tools into a vertebra must avoid the spinal cord 42 and, additionally, avoid exiting the vertebra from the multiple sides, leaving only two narrow insertion windows 44 on either side of the vertebra. As described below, and with reference to Figures 3A to 4B Typically, the most important images for confirming the positions of the insertion windows are the images derived from three-dimensional image data and cannot be obtained from the real-time two-dimensional images typically acquired during the intervention.

[0401] Now refer to Figure 3A and 3B , Figure 3A and 3B are multiple schematic diagrams of a three-dimensional computed tomography image of a vertebra, and a two-dimensional axial slice derived from the three-dimensional computed tomography image ( Figure 3B ), such multiple images are used in the prior art. Figure 4A and 4B , Figure 4A and 4B FIG. 3 shows a C-arm 34 used in the prior art to obtain an anteroposterior ("AP") two-dimensional radiographic image and the resulting anteroposterior image ( FIG. Figure 4A ), and a C-arm 34 for acquiring a two-dimensional radiographic image of one side and the resulting image of one side ( Figure 4B ).

[0402] As can be observed, Figure 3B The axial two-dimensional image slices show the views of the vertebrae that are important for confirming the entry point, insertion direction, and insertion depth of the tool. In contrast, the multiple two-dimensional radiographic images acquired by the C-arm are the sum of three dimensions and do not show multiple cross-sectional views of the vertebrae. As described above, multiple computer-assisted surgery (CAS) systems typically utilize computed tomography and / or magnetic resonance imaging images generated before the subject is placed in the operating room, or once the subject is placed in the operating room but typically before an intervention has begun. However, such multiple procedures are typically more expensive than non-computer-assisted surgical procedures (e.g., multiple non-computer-assisted surgical procedures, including open procedures, mini-open procedures, and minimally invasive procedures), limit the choice of tools to multiple tools equipped with multiple position sensors as described above, and require that the multiple tools be individually identified and calibrated at the beginning of each operation.

[0403] According to some applications of the present invention, the intraoperative position of a tool is determined relative to three-dimensional image data (e.g., a three-dimensional image, a two-dimensional cross-section derived from the three-dimensional image data, and / or a two-dimensional projection derived from the three-dimensional image data) during a non-computer-assisted surgical procedure (e.g., in open, mini-open, and / or minimally invasive surgery). The various techniques described herein are generally implemented without requiring position sensors (e.g., infrared emitters or infrared reflectors, or magnetic or electromagnetic sensors) to be mounted on the tool or the subject, and without requiring identification and / or calibration of the tools prior to the procedure. The various techniques described herein generally do not require tracking of the position of the subject's body or the applicable portion of the subject's body, and do not assume knowledge of the position coordinates of the subject's body in some reference frame. The techniques described herein generally do not require reliance on position sensors of tracking technologies (e.g., electromagnetic tracking technology or infrared tracking technology) that are not typically used in an orthopedic operating room. Furthermore, generally, practicing the techniques described herein does not require knowledge of the precise parameters of any individual pose of the two-dimensional radiographic imaging device (e.g., C-arm 34), nor does it require tracking the poses of the two-dimensional radiographic imaging device (e.g., C-arm 34) relative to each other and / or the position of the subject. For some applications, multiple two-dimensional radiographic images (e.g., multiple two-dimensional X-ray images) are acquired from two or more viewpoints by moving the radiographic imaging device to multiple poses between acquisitions of the images from multiple corresponding viewpoints. Typically, a single X-ray source is used to acquire the multiple two-dimensional X-ray images, although for some applications, multiple sources are used. Generally, descriptions of the multiple viewpoints of the two-dimensional radiographic imaging device herein as anterior-posterior, lateral, oblique, etc. should not be interpreted as meaning that multiple images must be acquired from the precise multiple viewpoints, but rather that acquiring multiple images from approximately such viewpoints is generally sufficient. Generally, the techniques described herein are tool-independent, meaning that the techniques can be implemented using any applicable tool and generally do not require any modifications and / or additions to the tool.

[0404] It should be noted that while some applications of the present invention have been described with reference to three-dimensional computed tomography, the scope of the present invention includes the use of any three-dimensional imaging, such as magnetic resonance imaging, three-dimensional x-ray imaging, and / or other modalities of three-dimensional imaging, mutatis mutandis. Such imaging can be performed prior to an intervention, at the outset of the intervention, and / or at some point during the intervention. For example, the three-dimensional imaging can be performed before the subject is placed in the operating room, when the subject is first placed in the operating room, or at some point in time when the subject is in the operating room but before a given tool is inserted into the given target part. Similarly, while some applications of the present invention have been described with reference to two-dimensional radiographic or x-ray imaging, the scope of the present invention includes the use of any two-dimensional imaging, such as ultrasound and / or other forms of two-dimensional imaging, mutatis mutandis. Although some applications of the present invention are described with reference to procedures performed on bony anatomy and / or vertebrae of the spine, the scope of the invention includes applying the apparatus and methods described herein to other orthopedic procedures (e.g., a joint (e.g., shoulder, knee, hip and / or ankle) replacement, joint repair, fracture repair (e.g., femur, tibia and / or fibula), a procedure performed on a rib (e.g., rib removal or rib resection), vascular interventions, cardiovascular interventions, neurovascular interventions, abdominal interventions, therapeutic irradiation and / or interventions performed on other portions of a subject, including interventions in which three-dimensional image data is acquired prior to the interventions and two-dimensional images are acquired during the interventions, mutatis mutandis.

[0405] Now refer to Figure 5A 、 5B 5C, 5D, and 5E are schematic diagrams of multiple combinations 50 of multiple radiopaque markers 52 typically placed on an object according to some applications of the present invention. For some applications, the multiple combinations of multiple markers are disposed on a pendant 53, as shown. Pendant 53 is typically sterile and disposable. For some applications, the multiple marker combinations include an authentication and / or anti-copying component, such as radio frequency identification (RFID), barcode(s), etc.

[0406] Typically, multiple combinations 50 of multiple markers 52 are placed on or near a region of the subject where an intervention is to be performed, such that at least some of the multiple markers appear in multiple two-dimensional radiographic images of the intervention region taken from multiple typical imaging views for such an intervention. For example, for a procedure performed on the subject's spine, the multiple markers are typically placed on the subject's back near the region of the spinal intervention, such that at least some of the multiple markers appear in multiple two-dimensional radiographic images of the intervention region taken from multiple typical imaging views for such an intervention. For example, for a procedure performed on the subject's spine, the multiple markers are typically placed on the subject's back near the region of the spinal intervention, such that at least some of the multiple markers appear in multiple two-dimensional radiographic images of the intervention region taken from multiple anterior-posterior imaging views (e.g., Figure 5B 、 5C , 5D, and 5E). For some applications, the plurality of markers are placed on the side of the subject in a vicinity of the site of intervention of the spine such that at least some of the markers appear in a two-dimensional radiographic image of the site of intervention acquired from a lateral imaging view ( Figure 5A For some applications, the plurality of markers are positioned on the subject's back such that at least some of the plurality of markers are level with the subject's sacrum.

[0407] As shown in the figure, the multiple markers can be arranged in a line (for example, Figure 5A A ruler in the middle), a grid ( Figure 5B ), a pair of parallel lines ( Figure 5C ), a frame around the surgical site (e.g., a circular ( Figure 5D ), or a rectangle ( Figure 5E )) and / or any other shape. For some applications, the plurality of markers comprises one or more grids and / or a combination of a plurality of discrete elements, wherein the plurality of discrete elements are connected (virtually or physically) into one or more grids. For some applications, the combination of the plurality of markers comprises an arrangement, wherein the arrangement has multiple parts visible in multiple different image perspectives, for example, the arrangement may include two rulers, one of the two rulers is positioned on the back of the subject (for example, as shown in FIG. Figure 5C ), so that it is visible in an image taken from a front-to-back perspective, and another one located to the side of the object (e.g., as shown in FIG. Figure 5A For some applications (not shown), the plurality of markers is a rigid arrangement of identifiable radiopaque features, such as a notched radiopaque ruler placed on or attached to the operating table.

[0408] Typically, surgery performed on skeletal anatomy begins with attaching a sterile surgical drape around the site of the surgery. In the case of spinal surgery, the approach to the surgery can be front, back, side, oblique, etc., and the surgical drape is placed accordingly. For such applications, multiple groups 50 of multiple markers 52 are typically placed above the surgical drape. Alternatively, multiple combinations of multiple markers are placed on the skin of the object (e.g., if no surgical drape is used). For some applications, multiple combinations of multiple markers are placed below the body of the object, on the operating table (e.g., attached to the operating table) and / or so that some of the multiple markers are above the operating table in the vicinity of the body of the object. For some applications, multiple combinations of multiple markers are used. For example, multiple combinations of multiple markers can be placed adjacent to each other. Alternatively or additionally, one or more groups of multiple markers can be placed on the subject's body such that at least some of the markers are visible in each of a plurality of X-ray image views, e.g., on the back or stomach and / or chest for an anteroposterior or posteroanterior (PA) view, and on the side of the body for a lateral view. For some applications, a single overhang 53 with a marker positioned thereon extends from, e.g., the back to the side such that multiple markers are visible in both the anteroposterior and lateral X-ray image views.

[0409] Also refer to Figure 6A and 6B , Figure 6A and 6BSchematic diagrams of first and second groups 50A and 50B of a plurality of radiopaque markers 52 configured for placement on a subject according to some applications of the present invention. For some applications, the first group of radiopaque markers and the second group of radiopaque markers are coupled to each other via one or more surfaces 54. For example, the one or more surfaces may comprise a portion of an overhang 53. The one or more surfaces are configured to position the first group of markers and the second group of markers on respective sides of the subject's spine in a plurality of predefined orientations relative to each other (e.g., parallel and / or aligned with one or both ends of the groups) by placing the one or more surfaces on a portion of the subject's spine where the procedure is to be performed. After positioning the first set of markers and the second set of markers on the multiple corresponding sides of the subject's spine, the one or more surfaces are configured to be removable from the subject's body (e.g., the multiple surfaces can be peeled off) to facilitate performing the procedure on the portion of the subject's spine where the one or more surfaces are placed, while leaving the first set of markers and the second set of markers on the multiple corresponding sides of the subject's spine in multiple predefined orientations relative to each other.

[0410] Typically, the plurality of combinations of markers are thereby positioned relative to each other in a desired relationship (e.g., parallel and / or with one or both ends of each of the plurality of combinations aligned with each other). However, by removing the surface(s) 54, fragments of the surface(s) are prevented from entering the body and / or interfering with the intervention. Typically, the plurality of combinations of markers are positioned on either side of the subject's spine such that at least the markers belonging to the plurality of combinations are visible even in multiple oblique X-ray image perspectives of the intervention site (and adjacent portions of the spine). Further typically, the plurality of combinations of markers are positioned on either side of the subject's spine such that at least the markers belonging to the plurality of combinations are visible even in multiple magnified views taken from the direction of insertion of the tool or from multiple perspectives oblique (i.e., diagonal) relative to the direction of insertion of the tool. Typically, the combination of the multiple radiopaque markers is placed on the object so that the multiple radiopaque markers do not obstruct the anterior-posterior or lateral X-ray images of the vertebrae, so that the multiple radiopaque markers do not interfere with the surgeon's viewing angle during the procedure, and do not interfere with the registration of two-dimensional and three-dimensional image data relative to each other (typically based on the geometry of the vertebrae, as described below).

[0411] Typically, multiple combinations of the multiple markers used for multiple open surgical procedures are described. Figure 6A As shown, for such multiple programs, a relatively large center window is required to perform the program between two combinations of the multiple markers.

[0412] Typically, minimally invasive spinal interventions are performed through multiple small incisions with multiple insertion windows at the 10-11 o'clock and 1-2 o'clock directions. For some such applications, an additional marker assembly 50C is placed along a portion between the 11 o'clock and 1 o'clock positions, such as Figure 6B As shown, the plurality of additional marker combinations are positioned relative to the first and second marker groups such that, while the first and second marker groups are positioned along respective sides of the spine, the plurality of additional marker combinations are positioned along a center of the subject's spine. For such applications, an intervention is typically performed through multiple incisions with multiple insertion windows in the 10-11 o'clock and 1-2 o'clock directions, while removing surfaces 54 disposed between the plurality of marker combinations 50A and 50C, respectively.

[0413] The plurality of radiopaque markers 52 typically include a plurality of marks (e.g., lines, notches, numbers, characters, shapes) visible to the naked eye and the imaging applied. Typically, the plurality of markers are radiopaque such that the plurality of markers are visible in a plurality of radiographic images. Further typically, the plurality of markers are identifiable at a plurality of corresponding positions relative to the object. For example, Figure 6A and Figure 6B A plurality of corresponding radiopaque alphanumeric characters are shown disposed at a plurality of corresponding locations. For some applications, the plurality of markers disposed at the plurality of corresponding locations can be identified based on other characteristics, such as, for example, the plurality of arrangements of the plurality of markers relative to other markers. Using a radiographic imaging device (e.g., C-arm 34), a plurality of radiographic images of the combination of the plurality of radiopaque markers are acquired, the plurality of corresponding images having a plurality of corresponding locations along at least a portion of the subject's spine, and each of the plurality of images including at least some of the plurality of radiopaque markers.

[0414] like Figures 6A to 6BAs shown, typically, each set of the plurality of markers includes a plurality of radiopaque notches. For some applications, to facilitate interpretation of the plurality of images and, for example, to avoid confusion when only one scale is visible, each of the plurality of sets of markers is additionally labeled with a corresponding combination of radiopaque characters, the plurality of sets being distinct from one another. For example, as shown, set 50A is labeled with a plurality of numbers, while set 50B is labeled with a plurality of letters.

[0415] Typically, the overhang 53 is made of a flexible material that is configured to generally conform to the contours of the subject's body. For some applications, at least some of the plurality of radiopaque markers 52 are rigid and have a plurality of corresponding known dimensions. For example: Figure 6A and 6B The plurality of markers 58 shown in the figure can be rigid. The plurality of rigid radiopaque markers are disposed on the flexible material such that generally the rigidity of the plurality of rigid markers does not prevent the flexible material from substantially conforming to the contours of the subject's body. Typically, the computer processor 22 captures at least one of the plurality of rigid radiopaque markers within a radiographic image of the spine via the radiographic imaging device (e.g., C-arm 34) through image processing recognition, and based on the identified plurality of rigid radiopaque markers, determines, through image processing, a plurality of dimensions of a plurality of features within the radiographic image (e.g., a plurality of automatically recognized features and / or a plurality of features specified by an operator of the system 20). Depending on a plurality of corresponding applications, the plurality of rigid markers are two-dimensional or three-dimensional, and all or a portion of the plurality of rigid markers are radiopaque.

[0416] For some applications, generally according to the various techniques described herein, a plurality of rigid radiopaque markers (and / or a rigid radiopaque fixture) that appear in a plurality of different X-ray image views of the subject are used to facilitate registration of the plurality of X-ray images with respect to one another. For some applications, generally according to the various techniques described herein, a plurality of rigid radiopaque markers (and / or a rigid radiopaque fixture) that appear in a plurality of different X-ray image views of the subject are used to facilitate registration of the plurality of X-ray images to three-dimensional image data (e.g., to computed tomography image data) and / or to one another.

[0417] For some applications, similar to reference Figure 6AThe arrangement shown is applied to a first set of radiopaque markers and a second set of radiopaque markers, with the first set 50A positioned such that the first set 50A is visible from an intended interventional direction (e.g., dorsal) corresponding to a first intended viewing angle (e.g., anterior-posterior) of the x-ray imaging, and the second set 50B positioned such that the second set 50B is visible from a secondary direction (e.g., lateral) corresponding to a second intended viewing angle (e.g., left or right side view) of the x-ray imaging. The spatial relationship of the plurality of markers is known, for example, as defined by one or more surfaces 54 and by the plurality of markers coupled to the surfaces 54 (for some applications, as described above, the plurality of surfaces are detachable). Once applied to the subject, the two marker sets 50A and 50B generally facilitate the association of specific vertebrae, as seen in one x-ray view (e.g., anterior-posterior view), with the same vertebrae as seen in a second x-ray view (e.g., a left or right side view). For some applications, such association is performed manually by the surgeon with reference to the plurality of radiopaque markers, e.g., the surgeon identifying markers that have a known association with one another in the plurality of X-ray images, e.g., by matching alphanumeric characters. Alternatively or additionally, the association is performed automatically by the computer processor 22 of the system 20 through image processing, e.g., by identifying markers that have a known association with one another in the plurality of X-ray images, e.g., by pattern matching or by optical character recognition.

[0418] For some applications, combinations 50 of markers 52 and / or a rigid radiopaque fixture as described above are used to facilitate any of the following functions:

[0419] Vertebral segment verification, as described below.

[0420] • Reaching a desired vertebra intraoperatively without inserting multiple needles into the patient and / or counting along a series of non-combined X-rays.

[0421] • Using vertebral segment verification, a three-dimensional image of the spine is displayed, the three-dimensional image including a plurality of indications of vertebrae on the three-dimensional image.

[0422] • Confirmation of the correct incision site(s) prior to the actual incision(s).

[0423] Providing a reference scale with some known dimensions / intervals, for example by using a plurality of rigid markers 58 to provide reference to multiple dimensions and / or by comparing multiple known shapes and / or dimensions, for example a plurality of rigid markers 58 seen in an image (for example to confirm the degree of foreshortening in the image).

[0424] · Performing multiple measurements on multiple images of the object, for example, by using the multiple rigid markers to provide multiple reference sizes and / or by comparing multiple known shapes and / or sizes (for example, the shapes and / or sizes of the multiple rigid markers 58) with what is seen in an image (for example, to confirm the degree of foreshortening in the image).

[0425] Registering a plurality of two-dimensional radiographic images to the three-dimensional image data. In this regard, it should be noted that if the three-dimensional image data is acquired in the operating room, a plurality of markers 52 and a plurality of rigid markers 58 may be observed in the three-dimensional image data and in at least some of the plurality of two-dimensional images, typically further facilitating registration. If the three-dimensional image data has been previously acquired, the three-dimensional imaging data typically includes some built-in dimensions that the two-dimensional image data typically does not. The plurality of rigid markers 58 provide a plurality of reference dimensions in the plurality of two-dimensional images that, for some applications, can be used as an additional input for registering the plurality of two-dimensional images to the three-dimensional image data.

[0426] Identifying changes in a posture of the two-dimensional imaging device (e.g., an X-ray C-arm) and / or a positioning of the patient. Typically, if the positioning of the two-dimensional imaging device relative to the object or the positioning of the object relative to the two-dimensional imaging device has changed, there will be a noticeable change in the appearance of the markers 52 relative to the anatomical structures within the images in the two-dimensional images. For some applications, in response to detecting such a change, the computer processor generates an alarm. Alternatively or additionally, the computer processor can calculate the change in position and take the change in position into account, for example, in the application of the algorithms described herein. Further alternatively or additionally, the computer processor assists the surgeon in returning the two-dimensional imaging device to a previous orientation relative to the object. For example, the computer processor can generate instructions as to where to move an X-ray C-arm to replicate a previous imaging position, or the computer processor can facilitate a visual comparison by an operator.

[0427] • Provide the surgeon with a reference throughout a procedure that serves to provide general orientation.

[0428] Provide information to the computer processor regarding the orientation of image acquisition and / or tool insertion, e.g., anterior to posterior ("AP") or posterior to anterior ("PA"), left or right, etc.

[0429] • Generate and update a visual roadmap of the subject's spine, as described in further detail below.

[0430] For some applications, at least some of the functions listed above as facilitated by the use of the plurality of sets 50 of markers 52 and / or a rigid fixture can be performed by the computer processor 22 even in the absence of the plurality of sets 50 of markers 52 and / or a rigid fixture, for example using the techniques described herein. Typically, the plurality of sets 50 of markers 52 and / or a rigid fixture are used for segment verification, confirmation of a tool entry point or incision site, performing measurements using the rigid markers as a reference, identifying changes in the relative posture of the two-dimensional imaging device (e.g., an X-ray C-arm) and the subject, and providing general orientation. All other functions of the system 20 (e.g., registering two-dimensional images to three-dimensional image data and other functions derived from the registration) generally do not require the use of the plurality of sets 50 of markers 52 and / or a rigid fixture. The functions described above can be performed automatically and / or manually by the computer processor 22.

[0431] The present invention has numerous applications generally applicable to one or more procedural tasks in non-computer-assisted spinal surgery, including but not limited to:

[0432] During the intervention, using preoperative 3D visibility (e.g., from computed tomography and / or magnetic resonance imaging). It should be noted that 3D visibility provides a plurality of desired cross-sectional images (described in further detail below), and that these cross-sectional images are generally more informative and / or of better quality than a plurality of intraoperative 2D images (it should be noted that for some applications, intraoperative 3D imaging is performed).

[0433] • Identify the vertebra(s) to be operated on.

[0434] • Confirming the point(s) for inserting one or more tools.

[0435] • Confirming the insertion direction of one or more tools.

[0436] • Monitor the progress of the tool during insertion, typically relative to the patient's anatomy.

[0437] • Reaching the target(s) or target area(s).

[0438] Repeat any of the above steps to replace multiple tools.

[0439] • Confirmation of tool / implant position within the anatomy, including in three dimensions.

[0440] • Generate and update a visual roadmap of the subject's spine, as described in further detail below.

[0441] Now refer to Figure 7 , Figure 7 is a flow chart showing a number of steps typically performed by the use system 20 according to some applications of the present invention. It should be noted that Figure 7 Some of the steps shown in are optional, and some of the steps may be performed in the same manner as described above. Figure 7 . In a first step 70, target vertebra(s) are marked by an operator relative to three-dimensional image data of the subject's spine (e.g., a three-dimensional image, a two-dimensional cross-section derived from the three-dimensional image data, and / or a two-dimensional projection image derived from the three-dimensional image data). For some applications, in a second step 72, multiple combinations 50 of multiple markers are placed on the subject, below the subject, on the operating table, or above the operating table in a vicinity of the subject. For some applications, step 72 is performed before step 70. Typically, in a third step 74, the vertebrae of the spine are identified to verify that the procedure is being performed on the correct vertebra (a step known as "level verification"), and multiple radiographic images of the spine and the multiple markers are used to facilitate the identification. In a fourth step 76, an incision site (in the case of minimally invasive surgery) or a tool entry point (in the case of open surgery) is confirmed. In a fifth step 78, the first tool in the sequence of multiple tools (typically a needle, such as a Jamshidi needle) is placed. TM A needle) is typically inserted into the subject (e.g., in the subject's back) and slightly fixed in the vertebra. In a sixth step 80, two or more two-dimensional radiographic images are acquired from a plurality of corresponding viewing angles, the plurality of corresponding viewing angles typically differing by at least 10 degrees (and more typically differing by 30 degrees or more), and one of the plurality of corresponding viewing angles typically being from a direction of insertion of the tool. Typically, a generally anterior-posterior image and a generally lateral image are acquired. Alternatively or additionally, a plurality of images are acquired from a plurality of different viewing angles. In a seventh step, typically, the computer processor registers the three-dimensional image data to the plurality of two-dimensional images.

[0442] After registering the three-dimensional image data to the plurality of two-dimensional images, a number of additional features of the system 20, as described in detail below, may be applied by the computer processor 22. For example, in step 84, the computer processor drives the display 30 to display a cross-section derived from the three-dimensional image data at a current position of a tip of a tool identified in a two-dimensional image, and optionally, to show a vertical line on the cross-sectional image indicating a line within the cross-sectional image at a location where the tip of the tool is currently located.

[0443] It should be noted that, as described in further detail below, for some applications, to perform step 84, the computer processor acquires only one or more two-dimensional x-ray images of a tool at a first position within the vertebra acquired from a single x-ray image viewpoint, registers the one or more two-dimensional x-ray images to the three-dimensional image data by generating a plurality of two-dimensional projections from the three-dimensional image data, and identifies a two-dimensional projection that matches the plurality of two-dimensional x-ray images of the vertebra. In response to registering the one or more two-dimensional x-ray images acquired from the single x-ray image viewpoint to the three-dimensional image data, the computer processor drives a display to display a cross-section derived from the three-dimensional image data at a current position of a tip of the tool, as identified from the one or more two-dimensional x-ray images, and optionally, to show a vertical line on the cross-sectional image, the vertical line indicating a line within the cross-sectional image at a location where the tip of the tool is located at the first position. Typically, when the tip of the tool is positioned at an additional position relative to the vertebra, multiple two-dimensional x-ray images of the tool at the additional position are further acquired from the same single x-ray image viewpoint or a different single x-ray image viewpoint, and the above steps are repeated. Typically, for each position of the tip of the tool to which the above techniques are applied, only multiple two-dimensional x-ray images acquired from a single x-ray image viewpoint are required, which may remain the same as or may be different from the multiple corresponding positions of the tip of the tool. Typically, two or more two-dimensional x-rays are acquired from multiple corresponding views, and the three-dimensional image data and the two-dimensional x-ray images are registered with each other (and with each other) by typically identifying a corresponding number of two-dimensional projections of the three-dimensional image data that match the multiple corresponding two-dimensional x-ray images. In step 86, the computer processor drives the display 30 to display the expected (i.e., inferred) path of the tool with reference to a target position and / or with reference to a desired insertion vector.

[0444] In step 88, the computer processor simulates the progression of the tool within a secondary 2D imaging view based on the progression of the tool observed in a primary 2D imaging view. In step 90, the computer processor overlays an image of the tool, a representation of an image of the tool, and / or a representation of the tool's path onto the 3D image data (e.g., a 3D image derived from the 3D image data, a 2D cross-section, and / or a 2D projection image derived from the 3D image data). The position of the tool or the tool's path has been derived from the current plurality of 2D images.

[0445] Now refer to Figure 8A , Figure 8A A vertebra 91 is shown designated on a coronal cross-sectional image 92 and a sagittal cross-sectional image 94 of a subject's spine according to some applications of the present invention, the cross-sectional images being derived from three-dimensional image data. Figure 7 As described in step 70, typically, before the subject is placed in the operating room, or while the subject is in the operating room but before an intervention begins, an operator marks the target vertebra(s) relative to three-dimensional image data (e.g., a three-dimensional image, a two-dimensional cross-section derived from the three-dimensional image data, and / or a two-dimensional projection image derived from the three-dimensional image data). For some applications, in response to the operator marking a vertebra, the computer processor designates additional vertebra(s). For some applications, the operator marks any one or any combination of the following relative to the three-dimensional image data: a specific target within the vertebra (e.g., a fracture, a tumor, etc.), directions / vectors for desired paths for inserting tools, and / or desired placement locations for implants (e.g., pedicle screws). For some applications, the operator marks the target vertebra(s) relative to a two-dimensional X-ray image having a sufficiently large view to include an identifiable portion of the anatomy (e.g., the sacrum) and the target vertebra(s). For some applications, more than one target vertebra is marked, and for some applications, two or more vertebrae that are not adjacent to each other are marked.

[0446] For some applications, the computer processor automatically counts the number of vertebrae in the image from an identifiable anatomical reference (e.g., the sacrum) to the labeled target vertebra(s). The target vertebra(s) are then known to be the Nth vertebra from the identifiable anatomical reference (even if the anatomical labels of the vertebra(s) are unknown). For some applications, the vertebra(s) are automatically counted using a plurality of image processing techniques. For example, the plurality of image processing techniques may include shape recognition of a plurality of anatomical features (e.g., considering the vertebrae as a whole, transverse processes, and / or spinous processes). Alternatively, the plurality of image processing techniques may include detection of the outer edge of the spine (in a two-dimensional image of the spine) followed by counting the number of bulges along the spine (each corresponding to a vertebra). For some applications, the plurality of image processing techniques include the techniques described in US 2010-0161022 to Tolkowsky, which is incorporated herein by reference.

[0447] For more details, refer to Figure 7 At step 72, for some applications, wherein a procedure is performed on a given vertebra of a spine in the subject, one or more sets of radiopaque markers 52 are placed on or near the subject such that multiple markers placed at multiple corresponding positions relative to the subject are identifiable, e.g., as Figures 5A to 5E and Figures 6A to 6B For example: Figure 6A and 6BAs shown, for some applications, multiple markers placed at multiple corresponding locations can be identified based on other characteristics, such as the placement of the markers relative to other markers. Using a radiographic imaging device (e.g., a C-arm 34), multiple radiographic images of the combination of the multiple radiopaque markers are acquired, the multiple corresponding images having multiple corresponding locations along at least a portion of the subject's spine, and each of the multiple images including at least some of the multiple radiopaque markers. Using a computer processor 22, multiple locations of the multiple radiopaque markers within the multiple radiographic images are identified through image processing. Based on the identified multiple locations of the multiple radiopaque markers within the multiple radiographic images, at least some of the multiple radiographic images are combined relative to each other. Generally, such combining of multiple images is similar to stitching together multiple images. However, the multiple images are typically not precisely stitched together to stitch together multiple portions of the subject's anatomy in multiple adjacent images. Instead, the multiple images are combined with sufficient accuracy to confirm a location of a given vertebra within the combined multiple radiographic images.

[0448] For some applications, based on the combined radiographic images, the computer processor automatically determines a location of the given vertebra (e.g., the previously marked target vertebra) within the combined radiographic images. For some applications, the computer processor automatically determines the location of the given vertebra within the combined radiographic images by counting the number of vertebrae in the images relative to an identifiable anatomical reference (e.g., the sacrum). For some applications, the counting is performed up to the Nth vertebra. For some applications, the counting is performed up to a value defined relative to the Nth vertebra. For some applications, the vertebra(s) are automatically counted using various image processing techniques. For example, the various image processing techniques may include shape recognition of various anatomical features (e.g., considering the vertebrae as a whole, transverse processes, and / or spinous processes). Alternatively, the various image processing techniques may include detecting the outer edge of the spine (in a two-dimensional image of the spine) and then counting the number of bulges along the spine (each corresponding to a vertebra). For some applications, the plurality of image processing techniques include the plurality of techniques described in US 2010-0161022 to Tolkowsky, which is incorporated herein by reference. For some applications, the computer processor facilitates manual confirmation of the position of the given vertebra within the combined plurality of radiographic images by displaying the combined plurality of radiographic images.

[0449] Based on the position of the given vertebra within the combined plurality of radiographic images, a position of the given vertebra relative to the combination of a plurality of radiopaque markers placed on or near the subject is determined, as described in further detail below.

[0450] For some applications, a technique similar to the techniques described above with respect to multiple vertebrae is performed on the ribs of a subject. For example, a combination of multiple radiopaque markers is placed on or near the subject such that the multiple markers placed at multiple corresponding positions relative to the subject are identifiable. Using a radiographic imaging device (e.g., C-arm 34), multiple radiographic images of the combination of the multiple radiopaque markers are acquired, the multiple corresponding images having multiple corresponding positions along at least a portion of the subject's thorax, and each of the multiple images including at least some of the multiple radiopaque markers. Using the computer processor, multiple positions of the multiple radiopaque markers within the multiple radiographic images are identified through image processing. Based on the identified multiple positions of the multiple radiopaque markers within the multiple radiographic images, at least some of the multiple radiographic images are combined relative to each other.

[0451] For some applications, based on the combined radiographic images, the computer processor automatically identifies a location of the given rib within the combined radiographic images (e.g., a target rib previously marked and counted in the computed tomography image data, magnetic resonance imaging image data, and / or an x-ray image with a sufficiently large field of view). For some applications, the computer processor automatically identifies the location of the given rib within the combined radiographic images by counting the number of ribs on the images based on an identifiable anatomical reference. For some applications, the ribs are automatically counted using image processing techniques. For example, the image processing techniques may include shape recognition of ribs. For some applications, the image processing techniques include techniques described in US2010-0161022 to Tolkowsky, which is incorporated herein by reference. For some applications, the computer processor facilitates manual identification of the location of the given rib within the combined radiographic images by displaying the combined radiographic images. Based on the position of the given rib within the combined plurality of radiographic images, a position of the given rib relative to the combination of the plurality of radiopaque markers placed on or near the subject is determined.

[0452] It should be noted that, without the multiple combinations 50 of multiple markers 52, the typical method for confirming the position of a given vertebra involves obtaining a series of x-rays along the patient's spine from the sacrum and inserting multiple radiopaque needles into the subject to match the x-rays to each other. Typically, in each x-ray spine image, there are only 3 to 4 vertebrae within the field of view, and multiple overlapping images must be obtained so that the vertebrae can be manually counted using the multiple overlapping images. This technique may involve switching back and forth between anterior-posterior x-ray images and lateral x-ray images, which sometimes results in surgery at the wrong level, as described in "Spine Surgery at the Wrong Site: An Underreported Problem? AAOS Current," American Academy of Orthopaedic Surgeons, March 2010.

[0453] Now refer to Figure 8B , Figure 8B An example of a three-dimensional computed tomography image 95 of a subject's spine is shown along with a combined radiographic image 96 of the subject's spine, according to some applications of the present invention. According to some applications of the present invention, a plurality of markers are positioned on or near the subject such that a base of the plurality of markers is positioned above or near the sacrum. A series of x-ray images are acquired along the spine from substantially the same viewing angle, typically, but not necessarily, with some overlap between adjacent images. Typically, the series of x-ray images is acquired from a generally anterior-posterior viewing angle, but a series of x-ray images may also be acquired from a different viewing angle, such as a generally lateral view. Using a computer processor 22, a plurality of locations of the plurality of radiopaque markers within the plurality of radiopaque images are identified through image processing. Based on the plurality of identified locations of the plurality of radiopaque markers within the plurality of radiopaque images, at least some of the plurality of radiopaque images are combined relative to each other. For example: a combined radiographic image 96 is produced by combining (a) and (b), wherein (a) is a first X-ray image 97 obtained from a generally anterior-posterior perspective, and the first X-ray image 97 starts from the sacrum of the subject and includes markers H to E in the right marker combination and markers 8-5 in the left marker combination, and (b) is a second X-ray image 98 obtained from a perspective generally similar to the first perspective (but not exactly the same perspective), and includes markers E to B in the right marker combination and markers 5 to 2 in the left marker combination.

[0454] (It should be noted that in Figure 8B In some embodiments, the alphanumeric markers appear white in the image. Generally, the markers may appear generally white or generally black, depending on (a) contrast settings of the image (e.g., radiopaque portions appear white on a black background, or vice versa), and (b) according to some applications of the present invention, whether the markers themselves are radiopaque or whether the markers are comprised of cutouts in a radiopaque backing material.

[0455] Generally, the combination of the multiple images is similar to stitching together multiple images. However, the multiple images are typically not stitched precisely to stitch together multiple portions of the subject's anatomy in multiple adjacent images. Instead, the multiple images are combined with sufficient accuracy to facilitate counting multiple vertebrae along the spine within the combined images. The physical position of a given vertebra is then known by being adjacent to, or in the vicinity of, or relative to a given one of the multiple identifiable markers. It should be noted that in order to combine the multiple radiographic images with each other, it is generally not necessary for each image to be acquired from a precise perspective (e.g., a precise anterior-posterior perspective or a precise lateral perspective) or to be an exact replica of a given reference point in multiple consecutive images. Instead, it is generally sufficient to maintain a given imaging orientation and to have at least some markers that are generally visible in the multiple images.

[0456] As described above, for some applications, to confirm the location of the previously marked target vertebra(s) or other vertebra(s), the computer processor automatically counts (and, for some applications, labels, e.g., automatically labels and / or digitally labels) the vertebra(s) within the combined radiographic images. Alternatively, the computer processor drives the display to display the combined radiographic images to facilitate confirmation of the location of the previously marked target vertebra(s) by an operator. The operator can count the vertebrae within the combined radiographic images to confirm which of the radiopaque markers within the combined images is adjacent to or in the vicinity of the vertebrae, and then physically locate the vertebrae within the subject by locating the corresponding physical markers.

[0457] Now refer to Figure 8C , Figure 8CAn example of a three-dimensional computed tomography image 100 of a subject's spine is shown along with a two-dimensional radiographic image 102 of the subject's spine, according to some applications of the present invention. As shown, a plurality of markers 52 appear on the combined radiographic image. As shown, vertebra 91 is identified in the two-dimensional radiographic image using the various techniques described above, and the vertebra 91 is identified by an operator relative to the three-dimensional image data (as described above with reference to FIG. 1 ). Figure 8A ), and is indicated by a cursor 104 .

[0458] For some applications, the computer processor 22 of the system 20 counts and / or marks the plurality of vertebrae within the three-dimensional image data (e.g., a three-dimensional image, a two-dimensional cross-section derived from the three-dimensional image data, and / or a two-dimensional projection image derived from the three-dimensional image data) based on counting and / or marking the plurality of vertebrae in the combined plurality of radiographic images. For some applications, the computer processor drives the display to display the plurality of marked vertebrae while simultaneously acquiring and displaying a plurality of relative, corresponding two-dimensional images. Alternatively or additionally, the computer processor drives the display to display the plurality of marked vertebrae after the combined radiographic image has been generated and / or displayed. It should be noted that, generally, the computer processor counts, marks, and / or identifies the plurality of vertebrae in the three-dimensional image data and the two-dimensional radiographic image without determining relative scales of the three-dimensional image data and the plurality of two-dimensional images. Instead, it is sufficient for the computer processor to identify individual vertebrae at a level sufficient to perform the counting, marking, and / or identification of the plurality of vertebrae.

[0459] It should be noted that the identification of the plurality of vertebrae described above, facilitated by the plurality of markers 52, is not limited to being performed by the computer processor at the beginning of an intervention. Rather, the computer processor may perform similar steps at subsequent stages of the procedure. Typically, because the computer processor utilizes knowledge of an already identified vertebra to identify additional vertebrae, the computer processor does not have to repeat the entire series of steps at the subsequent stages. For example, after identifying a first vertebra and subsequently performing a procedure, the computer processor may utilize the combined radiographic image to derive a position of another target vertebra (which may be separated from the first vertebra by a gap) based on the already identified first vertebra. For some applications, to derive the position of the additional target vertebra, the computer processor first expands the combined radiographic image (typically, using the plurality of markers according to the techniques described above).

[0460] For some applications, the operator marks a single vertebra (or in some applications, multiple vertebrae) within the three-dimensional image data of the spine. Based on marking the vertebra(s), the computer processor automatically marks other vertebrae within the three-dimensional image data based on the known anatomical arrangement of the multiple vertebrae along the spine. For some applications, based on the one or more vertebrae marked within the three-dimensional image data, the computer processor marks (e.g., anatomically and / or digitally) the multiple vertebrae within the combined radiographic image. In this manner, a spinal roadmap is created within the three-dimensional image data (e.g., a three-dimensional image, a two-dimensional cross-sectional image derived from the three-dimensional image data, and / or a two-dimensional projection image derived from the three-dimensional image data) and within the combined multiple radiographic images. For some applications, the spinal roadmap is generated on an image that is a fused combination of the three-dimensional image data and the multiple two-dimensional radiographic images. For some applications, the spinal roadmap is automatically updated during surgery. For example, in response to detecting a tool within a given vertebra in a two-dimensional x-ray image, the three-dimensional spinal roadmap can be updated to display the tool within the vertebra using a number of registration techniques described in detail below.

[0461] Generally, the scope of the present invention includes acquiring imaging data of the subject's spine using an imaging device (e.g., a two-dimensional imaging device and / or a three-dimensional imaging device). During a procedure of performing multiple interventions on at least a first vertebra and a second vertebra of the subject's spine, a spinal roadmap image of at least a portion of the spine including the first vertebra and the second vertebra is generated by a computer processor 22 and displayed on a display 30. Generally, the computer processor automatically marks the vertebrae within the spinal roadmap image. For some applications, the computer processor determines that an intervention has been performed on the first vertebra (e.g., a tool has been inserted and / or implanted into the first vertebra) such that an appearance of the first vertebra has changed. In response, the computer processor automatically updates the spinal roadmap to reflect the change in the appearance of the first vertebra, such that the updated spinal roadmap is displayed when the intervention is performed on the second vertebra.

[0462] Now refer to Figure 9 , Figure 9 An example of an optical image 110 displayed together with a two-dimensional radiographic (eg, X-ray) image 112 is shown in accordance with some applications of the present invention. Figure 7As described in step 76 of , after identifying a target vertebra along the subject's spine, typically, the operator confirms a desired site for an incision (in the case of minimally invasive surgery) or tool insertion (in the case of open or surgical surgery). For some applications, to facilitate confirmation of the incision site or tool insertion site, an optical camera 114 is positioned in the operating room so that the optical camera has a viewing angle that is substantially similar to a viewing angle of the two-dimensional radiographic imaging device. For example, as shown in FIG1 , the camera can be positioned on an X-ray C-arm 34. Alternatively or additionally, the camera can be positioned on a separate arm, can be handheld, can be a rear-mounted camera of a display such as a tablet or a mini-tablet device and / or can be handheld by another member of the operating room staff. For some applications, the camera is placed on the surgeon's head. Typically, for such applications, the surgeon uses a head-mounted display.

[0463] For some applications, a two-dimensional radiographic image 112 of a portion of the subject's body is acquired using the two-dimensional radiographic imaging device (e.g., C-arm 34) in a radiographic imaging mode, and an optical image 110 (as shown in FIG. 1 ) of the subject's body is acquired using an optical camera 114 in an optical imaging mode. Through image processing, the computer processor 22 of the system 20 identifies a plurality of radiopaque markers (e.g., markers 52) in the radiographic image and in the optical image. For example, in Figure 9 In the case of multiple radiopaque grid lines (e.g. Figure 5B ) and a plurality of alphanumeric radiopaque markers associated with the plurality of radiopaque grid lines (also as shown Figure 5B The computer processor maps the radiographic images and the optical images bidirectionally relative to each other based on identifying the plurality of radiopaque markers in the radiographic images and the optical images. It should be noted that the radiographic images and the optical images acquired from a plurality of substantially similar views (but not necessarily a plurality of identical views) are generally sufficient to facilitate bidirectional mapping of the plurality of images relative to each other, with the plurality of radiopaque markers visible in both the radiographic images and the optical images.

[0464] like Figure 9As shown, for some applications, the computer processor drives the display 30 to display the radiographic image and the optical image separately from each other on one or more displays. Then, in response to receiving an input indicating a location in a first one of the radiographic image and the optical image, the computer processor generates an output indicating the location in the other one of the radiographic image and the optical image. For example, in response to a line or point marked on the two-dimensional X-ray image 112, the computer processor indicates a plurality of corresponding lines or points overlaid on the optical image 110. Similarly, in response to a line or point marked on the optical image 110, the computer processor indicates a plurality of corresponding lines or points overlaid on the two-dimensional X-ray image 112. Further similarly, in response to a line or point being marked on the subject's body as seen in a contemporaneous optical image 110, or a K-wire or cutting blade being placed on the subject's body (e.g., in the case of a planned dorsal insertion tool), the computer processor identifies such line, point, or object (or applicable portion of the object) and indicates a corresponding line or point overlaid on the two-dimensional X-ray image 112. For some applications, a line or point is drawn on the subject's body (e.g., on the subject's back in the case of a planned dorsal insertion tool) using radiopaque ink.

[0465] Conventionally, to confirm the location of an incision site, a radiopaque guide wire is placed at a series of locations on the subject's back, and X-rays are taken on the guide wire at the multiple locations until the incision site is confirmed. A cutter is then placed at the confirmed incision site, and a final X-ray image is acquired for verification. In contrast, according to the various techniques described herein, a single X-ray image can initially be acquired and bidirectionally mapped to the optical image. The guide wire is then placed at a location, and the corresponding position of the guide wire relative to the X-ray image can be observed (using the bidirectional mapping) without the need to acquire a new X-ray image. Similarly, when a cutting knife is placed at a location, the corresponding position of an applicable portion of the knife (typically the distal tip of the knife) relative to the X-ray image can be observed (using the bidirectional mapping) without the need to acquire a new X-ray image. Alternatively or additionally, a line can be drawn on the X-ray image (e.g., a vertical line passing through the centers of the vertebrae, anatomically along the spinous processes of the vertebrae), and the corresponding line can be observed in the optical image overlaid on the patient's back.

[0466] For some applications, a surgeon places a radiopaque cutter 116 (or another radiopaque tool or object) at a desired incision site (and / or places a tool at a desired tool insertion location) using the optical image and the bidirectional mapping between the optical images, and verifies the location of the incision site (and / or tool insertion location) by observing the position of the tip of the cutter (or a portion of another tool) relative to the X-ray (e.g., via cursor 117). For some applications, the above and referenced methods are performed using multiple markers (typically sterile) in addition to marker 52. Figure 9 and / or functions described with reference to other figures. For example, a radiopaque shaft 118, a ruler, radiopaque notches, and / or radiopaque ink may be used.

[0467] Now refer to Figure 10 , Figure 10 An example of a two-dimensional radiographic (e.g., X-ray) image 120 displayed together with a cross-sectional image 122 of a subject's vertebra derived from three-dimensional image data of the vertebra according to some applications of the present invention is shown. For some applications, even before registering the two-dimensional image to the three-dimensional image data (as described below), the following steps are performed: Acquire an X-ray image 120 of the subject's spine at a given viewpoint (e.g., anterior-posterior, as shown). Indicate a point on the image, e.g., at Figure 10The point indicated by cursor 124 in the image is displayed. The computer processor 22 of system 20 automatically identifies the endplates of the vertebra and calculates relative distances between the indicated point and the endplates. (Note that the computer processor typically does not calculate absolute distances to perform this function.) From the three-dimensional (e.g., computed tomography) image of the same vertebra, the computer processor generates and displays a cross-section in a given plane (typically axial) at the indicated location (e.g., image 122). For some applications, the computer processor drives the display to display a line 126 (e.g., a vertical line) within the cross-section, indicating where the indicated location falls on the line. For some applications, the line is drawn vertically in an axial cross-section of the spine, as shown. The computer processor determines where to place the line based on the distance of the indicated point from the left and right edges of the vertebra and / or the position of the indicated point relative to features visible in the X-ray image (e.g., spinous process, transverse processes, pedicles). Typically, the cross-sectional image with the lines, combined with the surgeon's tactile sense of how far the skin is from the vertebrae (and / or deriving such information from three-dimensional images), assists the surgeon in calculating the desired angle of insertion for a tool.

[0468] Refer again Figure 7 In step 78, the first tool (usually a needle, such as Jamshidi) in the arrangement of multiple tools is placed. TM A needle) is typically inserted into the subject (e.g., in the subject's back) and slightly fixed in the vertebra. Next, in step 80, two or more two-dimensional radiographic images are acquired from a plurality of corresponding viewing angles, typically differing by at least 10 degrees (and more typically by 30 degrees or more), and one of the plurality of corresponding viewing angles typically being from a direction of insertion of the tool. A plurality of common combinations of these viewing angles include anterior-posterior and left or right lateral, anterior-posterior with a left tilt or right tilt, left tilt with a left side, and right tilt with a right side. It should be noted that, as described in further detail below, reference is made to Figure 15A For some applications, multiple two-dimensional radiographic images of the tool and the vertebra are acquired from only a single x-ray image perspective.

[0469] Now refer to Figure 11A and 11B , showing some applications of the present invention will be a Jamshidi TMMultiple examples of anterior-posterior and lateral X-ray images of a needle 36 inserted into a subject's spine. As shown, multiple sets 50 of markers 52 typically appear in at least the posterior-posterior image.

[0470] Now refer to Figure 12A and 12B , respectively, illustrate multiple examples of correspondences between multiple viewing angles of a three-dimensional image of a vertebra and a first two-dimensional X-ray image 132 and a second two-dimensional X-ray image 136 of the vertebra according to some applications of the present invention. FIG12 illustrates the correspondence between a first viewing angle 130 of a three-dimensional image of a vertebra and an X-ray image of an anterior and posterior portion of the vertebra. Figure 12B The correspondence between a second viewing angle 134 of a three-dimensional image of a vertebra and an X-ray image of a side of the vertebra is shown.

[0471] For some applications, according to Figure 7 In step 82, after securing the tool within the subject's vertebra, the three-dimensional image data is registered with the two-dimensional image. However, it should be noted that, according to various techniques described below, as further detailed below, registering the three-dimensional image data with the two-dimensional image can be performed even when a tool is not present within the plurality of images. Generally, the three-dimensional image data is registered with the two-dimensional image by generating a plurality of two-dimensional projections from the three-dimensional image data and identifying a first two-dimensional projection and a second two-dimensional projection that match the first two-dimensional X-ray image and the second two-dimensional X-ray image of the vertebra. (For some applications, two-dimensional x-ray images are acquired from more than two-dimensional x-ray image views, and the three-dimensional image data is typically registered with the plurality of two-dimensional x-ray images by identifying a corresponding number of two-dimensional projections of the three-dimensional image data that match the plurality of corresponding two-dimensional x-ray images.) Typically, the first two-dimensional x-ray image and the second two-dimensional x-ray image of the vertebra are acquired using an x-ray imaging device that is not registered to the subject's body by the following steps: (a) acquiring a first two-dimensional x-ray image of the vertebra (and at least a portion of the tool) from a first viewpoint, with the x-ray imaging device positioned in a first posture relative to the subject's body; (b) moving the x-ray imaging device to a second posture; and (c) acquiring a second two-dimensional x-ray image of at least the portion of the tool and the bone portion from a second viewpoint while the two-dimensional x-ray imaging device is in the second posture.

[0472] For some applications, the 3D imaging used is computed tomography (CT), and the following description of registering the 3D image data to the multiple 2D images will focus on CT images. However, the scope of the present invention includes applying the various techniques described herein to other 3D imaging modalities, such as magnetic resonance imaging (MRI) and 3D X-ray, with corresponding application.

[0473] Both X-ray imaging and computed tomography (CT) use ionizing radiation to image an object, such as a body part or an organ. Two-dimensional X-ray imaging produces a projection image of the object being imaged, while CT scans utilize computer processing to combine many X-ray images taken from different angles to generate multiple cross-sectional images (virtual "slices") of the scanned object, allowing the user to see the interior of the object without cutting. Digital geometry is used to generate a three-dimensional image of the object's interior from a large number of two-dimensional images.

[0474] Now refer to Figure 13A 、 13B and 13C, showing a three-dimensional image of an object utilized in accordance with some applications of the present invention (where Figure 13A The example is a cone) and the side to side of the object ( Figure 13B ) and from bottom to top ( Figure 13C ) is a schematic diagram of a relationship between the two-dimensional projection of the object. As shown in the figure, for the example of the cone, the two-dimensional image from the bottom to the top (such as Figure 13C As schematically shown in FIG. 1 , the two-dimensional image acquired by the C-arm 34 is similar to an X-ray image of an anterior-posterior portion of an object ( FIG. 1 ), and the side-to-side image ( FIG. 1 ) is a circular image. Figure 13C As schematically shown in FIG, the image acquired by C-arm 34 (similar to an X-ray image of a side of an object) is a triangle. Thus, in the example shown, if the circle and the triangle can be registered to the cone in three-dimensional space, they are also aligned with each other in this three-dimensional space. Therefore, for some applications, multiple two-dimensional X-ray images of a vertebra from multiple corresponding viewpoints are registered with each other and with the three-dimensional image data of the vertebra by generating multiple two-dimensional projections from the three-dimensional image data and identifying corresponding first and second projections that match the two-dimensional X-ray image of the vertebra.

[0475] In the case of multiple 3D computed tomography (CT) images, the derived 2D projections are referred to as digitally reconstructed radiographs (DRRs). Considering 3D CT data and a 2D X-ray image of the same vertebra, a simulated X-ray camera position (i.e., viewing angle and viewing distance) can be virtually positioned at any location in space relative to the 3D image of the vertebra, and the corresponding DRR that the simulated camera view would produce can be determined. At a given simulated X-ray camera position relative to the 3D image of the vertebra, the simulated camera view would produce the same DRR as the 2D X-ray image. For purposes of the present application, such a DRR is referred to as being matched to an X-ray image of the vertebra. Typically, multiple 2D X-ray images of a vertebra from multiple corresponding viewing angles are aligned with each other and with the 3D image data of the vertebra by generating multiple DRRs from the 3D CT image data and identifying corresponding first and second DRRs (i.e., multiple 2D projections) that match the multiple 2D X-ray images of the vertebra. The multiple X-ray images are registered to the three-dimensional image data by identifying multiple corresponding digitally reconstructed radiographic images that match two or more X-ray images acquired from respective viewing angles, and the multiple X-ray images are registered to each other by, in turn, registering the multiple X-ray images to the three-dimensional image data.

[0476] For some applications, to register the multiple 2D images to the 3D image data, a number of additional registration techniques are used in combination with the various techniques described herein. For example, the registration can be performed using intensity-based methods, feature-based methods, similarity metrics, transformations, spatial domains, frequency domains, etc.

[0477] Typically, the plurality of X-ray images are registered to the 3D image data using the above-described techniques, bringing the 3D image data and the plurality of 2D X-ray images into a common reference coordinate system, with the 3D image data and the plurality of 2D X-ray images all aligned and scaled to the common reference coordinate system. It should be noted that the registration does not require tracking of the subject's body or portion of the subject's body (e.g., by affixing one or more position sensors, such as infrared light, infrared reflectors, optical sensors, or magnetic or electromagnetic sensors, to the body or portion of the body and tracking the plurality of position sensors).

[0478] Typically, the position and / or orientation of a vertebra relative to the subject's body and adjacent vertebrae may change between preoperative 3D imaging (e.g., 3D imaging performed before entering the operating room or before performing a given intervention) and intraoperative 2D imaging. For example, this may be due to the patient lying on their back during preoperative imaging but lying on their stomach or side during intraoperative imaging, or the patient's back being straight during preoperative imaging but folded (e.g., in a Wilson brace) during intraoperative imaging. Furthermore, in some cases, the position of the spine changes (e.g., sinking) due to anesthesia, and once tools are inserted into a vertebra, this may also change the position of the vertebra relative to adjacent vertebrae. However, because a vertebra is a single bone, the shape of the vertebra generally does not change between the preoperative 3D imaging and the intraoperative 2D imaging. Therefore, registration of the 3D image data to the 2D images is typically performed for each vertebra. For some applications, registering the 3D image data to the plurality of 2D images is performed on a per-vertebral basis within the segmented image of the vertebra, even when some components of a segment of a vertebra remain in the 3D image data (e.g., portions of the plurality of spinous processes of adjacent vertebrae). Furthermore, for some applications, registering the 3D image data to the plurality of 2D images is performed relative to a spinal segment comprising a plurality of vertebrae. For example, registering the 3D image data to the plurality of 2D images can be performed relative to a spinal segment while the 3D image data is being acquired while the subject is already in the operating room and positioned on the operating table for the intervention.

[0479] As described above, typically during a planning phase, an operator indicates a target vertebra or a portion of the target vertebra within the three-dimensional image data of the spine (e.g., as described above with reference to Figure 8A For some applications, the computer processor automatically identifies the target vertebrae in the plurality of X-ray images through image processing (e.g., using the techniques described above). For some applications, when a target vertebra is indicated by the operator relative to the three-dimensional image data of the spine or a portion of the spine (e.g., as described above with reference to FIG. 1 ), the computer processor automatically identifies the target vertebrae in the plurality of X-ray images through image processing (e.g., using the techniques described above). Figure 8A to C), automatically identifying, by the computer processor, the registration of the three-dimensional image data to the plurality of two-dimensional images performed for an individual vertebra.

[0480] Typically, since the registration is performed on a single vertebra, the registration is not affected by movements of the vertebra that occur between the acquisition of the two X-ray images (e.g. due to movement of the subject on the operating table, movement caused by breathing, etc.), since the movements of the C-arm and the vertebra can be assumed to be rigid deformations (and therefore a link between two rigid transformations can be assumed if both movements occur between the acquisition of the two X-ray images).

[0481] For some applications, the patient's motion is detected as an input for some functions of the computer processor 22. For example, a motion detection sensor 56 can be coupled to a combination 50 of multiple markers 52 (e.g., by coupling to a pendant 53, such as in Figure 5A ) and / or coupled to a portion of the subject's body. It should be noted that the motion detection sensor is typically not a position sensor, and the movement of the motion detection sensor is tracked by a tracker or a tracker configured to track the movement of a position sensor located elsewhere. Typically, the motion detection sensor detects in an independent manner that a portion of the subject's body has moved (i.e., the motion detection sensor detects that movement of the motion detection sensor has occurred relative to a previous position of the motion detection sensor itself (as opposed to detecting that movement of the motion detection sensor has occurred relative to an external element)). Furthermore, typically, the motion detection sensor is configured to detect that movement has occurred, but not necessarily to detect that a specific movement has occurred. For example, the motion detection sensor may be configured to detect movement greater than a threshold. For some applications, an accelerometer is used. For example, the accelerometer may be configured to detect sudden movement. In response to detecting that movement has occurred, the motion detection sensor may transmit a signal to the computer processor. For some applications, in response to receiving an input indicating that the subject has experienced movement, the computer processor generates an alarm and / or generates an output indicating that one or more images should be reacquired. Alternatively or additionally, the motion detection sensor is independently powered and configured to generate an alarm, such as a visual alarm or an audio alarm (e.g., by a light or a buzzer connected to the motion detection sensor) in response to detecting that motion has occurred.

[0482] Generally, the scope of the present invention includes acquiring a sequence of two or more images of a subject's body to confirm the position of a tool relative to the body during a medical intervention. For some applications, during such a procedure, a motion detection sensor is configured to detect that movement of the subject (or a portion of the subject) has occurred that is greater than a given threshold. For example, the motion detection sensor may be configured to detect that such movement has occurred between the acquisition of two or more images. Generally, the motion detection sensor detects that the portion of the subject's body has moved in an independent manner (i.e., the motion detection sensor detects that movement has occurred relative to a previous position of the motion detection sensor (as opposed to detecting that movement has occurred relative to an external element)). In response, the motion detection sensor generates an alert indicating to a user that such movement has occurred. For some applications, the motion detection sensor generates an output by actuating an output device itself. Alternatively or additionally, in response to an input received from a motion detection sensor indicating that such movement has occurred, the computer processor (e.g., computer processor 22) generates an alert indicating to a user that such movement has occurred. For some applications, the computer processor generates an output suggesting that the user acquire additional images (eg, reacquire an image from a given imaging perspective).

[0483] As described above, generally, a plurality of two-dimensional x-ray images of a vertebra from a plurality of corresponding views are registered with each other and with a three-dimensional image data of the vertebra by generating a plurality of digitally reconstructed radiographs from a three-dimensional computed tomography image and identifying a first digitally reconstructed radiograph and a second digitally reconstructed radiograph that match the plurality of two-dimensional images of the vertebra. The plurality of x-ray images are registered to the three-dimensional image data by identifying a plurality of corresponding digitally reconstructed radiographs that match two or more x-ray images acquired from the plurality of corresponding views, and the plurality of x-ray images are registered with each other by, in turn, registering the plurality of x-ray images to the three-dimensional image data.

[0484] For some applications, to avoid duplication of solutions when searching for a digitally reconstructed radiograph to match a given x-ray image, the computer processor 22 first determines whether the x-ray image is, for example, anterior-posterior, posterior-anterior, left lateral, right lateral, left oblique, or right oblique and / or which quadrant a tool was inserted from. The computer processor can determine this automatically using various techniques described herein, for example, by using multiple sets 50 of multiple markers 52. Alternatively, this information can be manually entered into the computer processor.

[0485] For some applications, to identify a digitally reconstructed radiographic image that matches a given x-ray image, the computer processor 22 first restricts the search space to a matching digitally reconstructed radiographic image, for example, by using a number of techniques such as those described in US Pat. No. 9,240,046 to Carrell, incorporated herein by reference.

[0486] For some applications, the steps of generating a plurality of digitally reconstructed radiographic images from a three-dimensional computed tomography image and identifying corresponding first and second digitally reconstructed radiographic images that match the two-dimensional x-ray images of the vertebrae are assisted by multiple deep learning algorithms. Typically, for such applications, during a learning phase, many combinations of three-dimensional computed tomography images of vertebrae and multiple x-ray images of those same vertebrae are input into a computer processor serving as a deep learning engine. The registration results for each set, i.e., the plurality of digitally reconstructed radiographic images that match the plurality of x-rays, are confirmed. The plurality of results of the deep learning are then input into a computer processor 22. Then, intraoperatively, the computer processor 22 uses the plurality of results of the deep learning phase to facilitate matching the plurality of digitally reconstructed radiographic images from the computed tomography images of the subject's vertebrae to the plurality of x-ray images.

[0487] For some applications, as described in the following paragraphs, multiple deep learning techniques are performed as part of image processing of a subject's vertebrae. By performing the multiple deep learning techniques, the search space of digitally reconstructed radiographs of the subject's vertebrae that are matched to the multiple X-ray images is limited, reducing the intraoperative processing requirements, reducing the time spent performing the matching, and / or reducing instances of double solutions for the matching.

[0488] For some applications, in a first deep learning phase, a moderate number (e.g., fewer than 10,000 or fewer than 1000, which is moderate relative to the larger data sets typically required for deep learning) of spinal computed tomography scans are processed, each of the plurality of computed tomography scans comprising a plurality of vertebrae, e.g., greater than 20. For each vertebra, a large number of pairs (or triplets or a larger multiple) of digitally reconstructed radiographs are generated, each pair being generated from a plurality of simulated viewing distances and a plurality of simulated camera angles, typically separated by at least 10 degrees, the plurality of simulated camera angles being those typically used for x-ray acquisition during spinal surgery, e.g., generally anterior-posterior, generally left oblique, generally right oblique, generally left lateral, and / or generally right lateral. Each of these combinations typically comprises a 3D computed tomography scan and a pair of digitally reconstructed radiographs, and the plurality of simulated camera viewing distances and plurality of simulated camera angles from which the digitally reconstructed radiographs were generated are fed to a deep learning analysis engine. Thus, given a 3D computed tomography of a vertebra and a pair of digitally reconstructed radiographs, the engine learns to suggest a plurality of simulated camera viewing distances and simulated camera angles corresponding to those digitally reconstructed radiographs. The deep learning data is then fed as an input to the computer processor 22 of the system 20. Intraoperatively, to register the plurality of 2D X-ray images to the 3D image data, the computer processor uses the deep learning data to limit the search space for digitally reconstructed radiographs of the 3D image data that match the plurality of X-ray images to be searched. The computer processor 22 then searches for the matching digitally reconstructed radiographs only within the search space specified by the deep learning data.

[0489] Alternatively or additionally, during the deep learning phase, a large database of two-dimensional X-ray images and their known parameters relative to the vertebrae (at least some of which) are input into a deep learning engine. These parameters typically include viewing angle, viewing distance, and a plurality of optional additional camera parameters. For some applications, the aforementioned parameters are exact. Alternatively, the parameters are approximate parameters. The parameters can be recorded when the images are initially generated or annotated by a radiologist. Thus, given a certain two-dimensional projection image, the engine learns to suggest a plurality of simulated camera viewing distances and a plurality of simulated camera angles corresponding to the projection image. The deep learning data is then fed as an input to the computer processor 22 of the system 20. Intraoperatively, in order to register the two-dimensional X-ray images to the three-dimensional image data, the computer processor uses the deep learning data to limit the search space for a plurality of digitally reconstructed radiographic angiography of the three-dimensional image data that matches the X-ray images to be searched. The computer processor 22 then searches for the matching digitally reconstructed radiographic angiography only within the search space specified by the deep learning data.

[0490] The above multiple registration steps are summarized in Figure 14A middle, Figure 14A A flow chart illustrating a plurality of steps performed by a computer processor to register three-dimensional image data of a vertebra to two or more two-dimensional x-ray images of the vertebra.

[0491] In a first step 140, the search space for matching digitally reconstructed radiographic images to corresponding x-ray images is restricted, for example, using deep learning data and / or using techniques such as those described in US Pat. No. 9,240,046 to Carrell, which is incorporated herein by reference. Alternatively or additionally, to avoid duplicate solutions when searching for a digitally reconstructed radiographic image that matches a given x-ray image, the computer processor determines whether the x-ray image is, for example, anterior-posterior, lateral, lateral, left oblique, or right oblique and / or from which quadrant a tool was inserted.

[0492] In step 141 , a plurality of digitally reconstructed radiographic images are generated within the search space.

[0493] In step 142, a plurality of digitally reconstructed radiographs are compared to the X-ray images from various views of the vertebrae.

[0494] Based on the comparison, the digitally reconstructed radiograph that best matches each X-ray image of the vertebra is selected in step 143. Typically, the computer processor identifies the viewing angle and the viewing distance of the camera from the three-dimensional image of the vertebra for the simulated camera position that will produce the best-matching digitally reconstructed radiograph.

[0495] It should be noted that the above steps are performed separately for each of the plurality of 2D X-ray images used for the registration. For some applications, each time one or more new 2D X-ray images are acquired, the image(s) are automatically registered to the 3D image data using the above techniques, thereby updating the 2D to 3D registration based on the new 2D X-ray acquisition(s).

[0496] Now refer to Figure 14B , Figure 14B is a flow chart illustrating the steps of an algorithm executed by the computer processor 22 of the system 20 according to some applications of the present invention. As described above, for each of the plurality of X-ray images (labeled XI and X2), the computer processor identifies a corresponding digitally reconstructed radiographic image from a simulated camera perspective (the simulated cameras for XI are labeled CI and X2 C2).

[0497] The 3D scan and the two 2D images are now co-registered, and the following 3D-2D bidirectional relationship generally exists:

[0498] Geometrically, a point P3D (in three coordinates) in the three-dimensional scan of the body part is located at the intersection in three-dimensional space of two straight lines.

[0499] i. A line is drawn from the simulated camera C1 and passes through the corresponding point PX1 (in both image coordinates) in the two-dimensional image X1.

[0500] ii. A line drawn from the simulated camera C2 passes through the corresponding point PX2 (in both image coordinates) in the two-dimensional image X2.

[0501] Therefore, reference Figure 14B For some applications, for a portion of a tool visible in the two-dimensional images, such as the tip of the tool or a distal portion of the tool, the computer processor determines the position of the computer processor within the three-dimensional image data (denoted as TP3D) using the following algorithmic steps:

[0502] Step 145: Identify the tip TPX1 of the tool in image XI through image processing (e.g., using the multiple image processing techniques described above). For some applications, to better define the tip point of the tool, the computer processor first generates a centerline for the tool, and then the distal tip TPX1 of the tool is located on the centerline.

[0503] Typically, the computer processor identifies the plurality of locations of a tool or portion of the tool in the two-dimensional X-ray image, typically using image processing alone. For example, the computer processor may identify the tool by using a filter that detects pixel darkness (the tool is typically dark), using a filter that detects a given shape (e.g., an elongated shape), and / or by using multiple masks. For some applications, the computer processor compares a given region within the image with the same region within a previous image. In response to detecting a change in some pixels within the region, the computer processor identifies the pixels as corresponding to a portion of the tool. For some applications, the comparison is performed for a desired region where the tool may be inserted, which may be based on a known approach direction for the tool. For some applications, the computer processor identifies the portion of the tool in the plurality of two-dimensional images using image processing alone, using the algorithmic steps described in US 2010-0161022 to Tolkowsky, which is incorporated herein by reference.

[0504] Step 146: Generate a typical straight line L1 from C1 to TPX1. (It should be noted that, as with the other steps described as being performed by the computer processor, the generation of a line refers to a processing step equivalent to drawing a line and should not be interpreted as implying that a physical line is drawn; rather, the line is generated as a processing step).

[0505] Step 147: Identify the tip TPX2 of the tool in image X2 by image processing (e.g., using the multiple image processing techniques described above). For some applications, to better define the tip point of the tool, the computer processor first generates a centerline for the tool, and then the distal tip TPX2 of the tool is located on the centerline. The multiple image processing techniques used to identify the tip TPX2 of the tool in image X2 are generally similar to the multiple image processing techniques described above with reference to step 145.

[0506] Step 148: Generate a typical straight line L2 from C2 to TPX2.

[0507] Step 149: Identify the intersection of L1 and L2 in three-dimensional space as the position of the tool tip relative to the three-dimensional scan data.

[0508] Step 150: Assuming the shape of the tool is known (e.g., if the tool is a rigid or at least partially rigid tool, or if the tool can be assumed to have a given shape due to being placed in tissue), the computer processor derives the positions of the plurality of additional portions of the tool in three-dimensional space. For example, in the case of a tool having a straight shaft, or a distal portion thereof, or a tool that can be assumed to be straight once inserted into bone, or at least the distal portion thereof, once inserted into bone, then the shaft, or at least the distal portion thereof, is located at the intersection of two planes, each of which extends from the simulated camera to the shaft (or portion thereof) in the corresponding two-dimensional image. For some applications, the direction of the shaft from the tip of the shaft to the proximal side of the shaft and along the intersection of the two planes is determined by selecting a point on any of the plurality of X-ray images near the tip of the tool and observing the position at which a line generated between this point and the corresponding simulated camera intersects the intersection of the two planes.

[0509] It should be noted that because the co-registration of the three-dimensional image data to the plurality of two-dimensional images is bidirectional, for some applications, as described in further detail below, the computer processor identifies a plurality of features identifiable within the three-dimensional image data and determines the plurality of positions of the plurality of features relative to the two-dimensional x-rays. The position of each such feature relative to the plurality of two-dimensional x-rays is typically determined by: (a) generating a representative straight line from the analog camera used to generate the digitally reconstructed radiographic image corresponding to the x-ray image through the feature within the three-dimensional image data, and (b) thereby determining the plurality of positions of the feature relative to the x-ray image itself. For some applications, the positions of the plurality of features relative to the plurality of two-dimensional x-ray images are determined by determining the positions of the plurality of features within the plurality of digitally reconstructed radiographic images that are matched to a plurality of corresponding x-ray images, and assuming a plurality of corresponding positions of the plurality of features within the plurality of matched x-ray images.

[0510] For some applications, as described in further detail below, the 3D image data is overlaid on a 12D image based on the registration. However, typically, the 3D image data (e.g., a 3D image, a 12D cross-section derived from the 3D image data, and / or a 12D projection image derived from the 3D image data) is displayed together with multiple 2D images.

[0511] Now refer to Figure 15A , shows an example of cross-sections 160 and 162 of a vertebra as shown on a plurality of corresponding two-dimensional X-ray images, the cross-sections corresponding to first and second positions, respectively, of a tip 164 of a tool advanced into the vertebra along a longitudinal insertion path, according to some applications of the present invention. Typically, the entirety of the tool or the distal portion of the tool has a straight shaft and / or can be assumed to be straight once inserted into the bone, or at least straight at the distal portion of the tool once inserted into the bone. Also refer to Figure 7 At step 84, for some applications, based on the identified position of the tool tip relative to one or more two-dimensional x-ray images of the vertebra acquired from a single image viewpoint, and an x-ray registered from the single two-dimensional x-ray image viewpoint to the three-dimensional image data (e.g., by matching a digitally reconstructed radiographic image from the three-dimensional image data to the two-dimensional x-ray image), the computer processor 22 determines a position of the tool tip relative to a digitally reconstructed radiographic image derived from the three-dimensional image data (e.g., the digitally reconstructed radiographic image matched to the two-dimensional x-ray image), and in response thereto, drives the display to display a cross-section of the vertebra derived from the three-dimensional image data and corresponding to the position of the tool tip. The cross-section is typically a given plane at the identified position. Typically, the cross-section is an axial cross-section, but for some applications, the cross-section is a sagittal cross-section, a coronal cross-section, and / or a cross-section perpendicular to or parallel to the direction of insertion of the tool.

[0512] For some applications, in the cross-section, the computer processor drives the display to display a line 166 (e.g., a vertical line) indicating where the tip of the tool is located on the line segment. For some applications, a line is drawn vertically on an axial cross-section of the spine as shown. For some applications, the surgeon can confirm the tool is at the likely location on the line based on their sense of touch. Alternatively or additionally, based on the three-dimensional image data, the computer processor drives the display to show the depth at which the vertebra is positioned beneath the skin, which further assists the surgeon in confirming the tool's location along the line.

[0513] As described above, generally, a 3D image data may be generated by acquiring one or more 2D X-ray images from only a single X-ray image view of the tool and the vertebra and registering one of the plurality of 2D X-ray images to the 3D image data using the various registration techniques described herein. Figure 15A Typically, by registering the two-dimensional x-ray image acquired from the single image view to the three-dimensional image data, the computer processor 22 identifies with respect to the three-dimensional image data (e.g., with respect to the digitally reconstructed radiography identified as matching the two-dimensional x-ray image): (a) the plane in which the tip of the tool is located, and (b) a line within the plane at which the tip of the tool is located, such as Figure 15A As described above, generally, when the tip of the tool is positioned at an additional position relative to the vertebra, multiple additional two-dimensional X-ray images of the tool at the additional position are acquired from the same single X-ray image viewpoint or a different single X-ray image viewpoint, and the above steps are repeated. Generally, for each position of the tip of the tool to which the above techniques are applied, it is only necessary to acquire multiple two-dimensional X-ray images from a single X-ray image viewpoint, which can remain the same for the multiple corresponding positions of the tip of the tool. Alternatively, it can be different for the multiple corresponding positions of the tip of the tool.

[0514] Now refer to Figure 15B , is a schematic diagram of the position of the tip 168 of the tool according to some applications of the present invention, wherein the position of the tip 168 of the tool corresponds to the position of the tool along a longitudinal insertion path (such as Figure 15AA plurality of cross-hairs are indicated on the cross-sections 160 and 162 of a first position and a second position of a tip 164 of a tool (as shown) advanced into the vertebra. For some applications, the precise position of the tip of the tool within the 3D image data is confirmed by initially registering the tool with two or more 2D X-ray images of the vertebra acquired from a plurality of corresponding 2D X-ray image views to the 3D image data and indicating the position on the cross-sections, as shown. Figure 15B As shown below, refer to Figures 19A to 19B For some applications, after initially confirming the position of the tip of the tool relative to the three-dimensional image data using two or more two-dimensional x-ray images of the tool and the vertebra acquired from a plurality of corresponding two-dimensional x-rays, multiple subsequent positions of the tip of the tool relative to the three-dimensional image data are confirmed using a plurality of additional x-ray images acquired from only a single x-ray image perspective.

[0515] Now refer to Figure 16A and 16B , shows several examples of a display showing the relationship between an intended longitudinal insertion path 170 and a designated position 174 of a tool 172 on an anterior-posterior 2D X-ray image and a lateral 2D X-ray image, respectively, according to some applications of the present invention. Also refer to Figure 7 Step 86.

[0516] For some applications, a location of a vertebra is specified within the three-dimensional image data. For example, an operator can specify a target portion (e.g., a fracture, a tumor, a virtual pedicle screw, etc.) and / or an area that the tool should avoid (e.g., the spinal cord) on the three-dimensional image data (e.g., a three-dimensional image, a two-dimensional cross-section derived from the three-dimensional image data, and / or a two-dimensional projection image derived from the three-dimensional image data). Alternatively or additionally, the computer processor can automatically identify such a location, for example, by identifying the portion through image processing. Based on registering the first two-dimensional X-ray image to the second two-dimensional X-ray image to the three-dimensional image data, the computer processor derives a location of the specified location within at least one of the multiple X-ray images using the techniques described above. Furthermore, the computer processor identifies an intended path for the tool within the X-ray image. Typically, the computer processor identifies an intended path by identifying an orientation of an elongated portion of the tool (and / or a centerline of the elongated portion) within the X-ray image. Since the tool is typically advanced along a longitudinal insertion path, the computer processor infers the intended path by inferring a line along the identified direction.

[0517] For some applications, the computer processor performs a substantially similar process, but with respect to a desired approach vector (e.g., for inserting and implanting a screw), such as manually input into the computer processor and / or automatically derived from the processor. For example, such an approach vector may have been generated during a planning phase, typically based on the three-dimensional image data, and based on a simulation of tool insertion into the vertebra. Typically, such an approach vector is one that reaches a desired target while avoiding the spinal cord or exiting the sides of the vertebrae.

[0518] For some applications, in response to the above steps, the computer processor generates an output indicating a relationship between the expected longitudinal insertion path of the tool and the specified position. For some applications, the computer processor generates an output on the display, such as: Figure 16A and 16B Alternatively or additionally, the computer processor can generate instructions to the operator to redirect the tool. Further alternatively or additionally, the computer processor can generate an alarm (e.g., an audio or visual alarm) in response to detecting that the tool is expected to enter an area to be avoided (e.g., the spinal cord) or move laterally away from the vertebrae in another direction.

[0519] Refer again Figure 7 In step 90, for some applications, as described above, computer processor 22 identifies a position of a portion of the tool relative to the vertebra within the plurality of X-ray images through image processing. Based on the identified position of the portion of the tool within the plurality of X-ray images and registering the first and second two-dimensional X-ray images to the three-dimensional image data, the computer processor identifies the position of the portion of the tool relative to the three-dimensional image data. For some applications, in response, the computer processor displays an image of the tool itself or a symbolic representation of the tool overlaid on the three-dimensional image data. Alternatively or additionally, the computer processor derives a relationship between the position of the portion of the tool relative to the three-dimensional image data and a given position within the three-dimensional image data, and generates an output indicating the relationship. As described above, the registration of the plurality of two-dimensional images to the three-dimensional image data is typically performed for a plurality of individual vertebrae. Thus, the techniques described herein are generally effective even if the subject has moved between the acquisition of the three-dimensional image data and the acquisition of the plurality of two-dimensional images.

[0520] For some applications, the computer processor generates an output indicating the distance of the tip of the tool from the spinal cord and / or outer vertebral border, e.g., using a plurality of numbers or colors displayed relative to the three-dimensional image data. For some applications, the computer processor outputs a plurality of instructions (e.g., textual, graphical, or audio instructions) indicating that the tool should be reoriented. For some applications, as an input to the process, the computer processor confirms or receives a manual input indicating a direction or orientation of the tool to be inserted (e.g., from the top or bottom, or from the left or right).

[0521] Now refer to Figure 17A An anterior-posterior x-ray of two tools 176L and 176R inserted into a vertebra through insertion windows at the 10-11 o'clock and 1-2 o'clock directions, respectively, is shown. Figure 17B Shown Figure 17A As described above, in many cases, during spinal surgery, two or more tools are inserted into a vertebra, for example, an insertion window from 10 o'clock to 11 o'clock and an insertion window from 1 o'clock to 2 o'clock, and the process is repeated for one or more additional vertebrae if applicable. Figure 17A As shown, in the anterior-posterior X-ray view, the tools inserted into the respective windows are generally distinguishable from one another. Figures 6A to 6B For some applications, within the anterior-posterior viewing angle, the computer processor identifies the tools inserted through the corresponding insertion windows based on the multiple arrangements of the multiple marker combinations 50A and 50B (and in some cases 50C). However, if the multiple tools have the same or similar appearance, it may be challenging to identify which tool is which from some imaging directions. In particular, if Figure 17B It can be observed that identifying which tool is which in the lateral X-ray view is challenging. Generally, tools can be distinguished in multiple images taken along the direction of insertion, and it is more difficult to distinguish multiple tools in multiple images taken along other directions.

[0522] For some applications, as a solution to the aforementioned challenges, the computer processor provides the operator with an interface for identifying one or more tools within an X-ray image, for example, by matching the tools within a first image acquired from a perspective in which the tools are distinguishable from one another (e.g., the anterior-posterior perspective) with the tools within a second image acquired from a perspective in which the tools are not distinguishable from one another (e.g., the lateral perspective). Alternatively or additionally, because the tools are typically inserted sequentially and all tool insertions are performed under X-ray imaging, when the first tool is inserted, multiple images are acquired from a first image perspective in which the tools are distinguishable from one another (e.g., the anterior-posterior perspective) and a second image perspective in which the tools are not distinguishable from one another (e.g., the lateral perspective). Thus, the computer processor identifies the tool as the first tool even in the image acquired from the second image perspective in which the tools are not distinguishable from one another. Then, when the second tool is inserted and the plurality of images (from the same or similar two views) are reacquired, the computer processor is able to identify the second tool even in the plurality of images acquired from the second image viewpoint where the plurality of tools are not distinguishable from each other, because the identification of the first tool has already been performed. The computer processor then tracks which tool is which along the sequence of the plurality of X-ray images. For some applications, once the computer processor has identified which tool is which, the computer processor indicates which tool is which (e.g., by color coding or labeling the plurality of tools) with respect to all X-ray viewpoints and / or with respect to the three-dimensional image data at all relevant stages throughout the procedure.

[0523] Refer again Figure 7In step 90, for some applications, the computer processor drives the display to display the tool, a representation of the tool, and / or a path on a two-dimensional cross-section of the vertebra derived from the three-dimensional image, rather than displaying the tool, a representation of the tool, and / or a path on a three-dimensional image on a three-dimensional image. For some applications, the computer processor determines the position of the centerline of the tool shaft through image processing. For example, the computer processor may use a variety of techniques for automatically identifying a centerline of an object, such as those described in US 2010-0161022 to Tolkowsky, which is incorporated herein by reference. For some applications, the computer processor drives the display to display the centerline of the tool on the three-dimensional image data, the end of the centerline indicating the position of the tool tip within the three-dimensional image data. Alternatively or additionally, the computer processor drives the display to display an extrapolation of the centerline of the tool on the three-dimensional image data, the extrapolation of the centerline indicating an expected path of the tool relative to the three-dimensional image data. For some applications, the computer processor drives the display to display a dot at the end of the extrapolated centerline on the three-dimensional image data, the dot representing the expected position of the tip of the tool.

[0524] For some applications, the computer processor drives the display to display a three-dimensional image of the vertebra in a semi-transparent format, with the tool, a representation of the tool, and / or a path of the tool positioned within the three-dimensional image. Alternatively or additionally, the computer processor drives the display to automatically rotate the three-dimensional image of the vertebra (e.g., rotating the three-dimensional image approximately 30 degrees front to back). For some applications, the computer processor retrieves an image of a tool of the type being inserted from a library and overlays the image on the derived centerline of the three-dimensional image data. Typically, the tool is positioned along the centerline at an appropriate scale, with dimensions derived from the three-dimensional image data. For some applications, a cylindrical representation of the tool is overlaid on the derived centerline of the three-dimensional image data. For some applications, any of the plurality of representations described above is displayed relative to a pre-specified tool path, such as automatically derived by processor 22 or manually entered by the surgeon during a planning phase.

[0525] Refer again Figure 2A tool inserted into a vertebra must avoid the spinal cord 42 while avoiding exiting the vertebra from the side, leaving only two narrow tool insertion windows 44 on either side of the vertebra. Generally, the greater the degree of protrusion of a tool or implant into the spinal cord, the more severe the clinical significance. For some applications, multiple volumes within the three-dimensional image of the vertebra (and / or a cross-sectional image derived from the three-dimensional image) are color-coded (e.g., highlighted) to indicate acceptable (or unacceptable) degrees of protrusion within the multiple volumes. For some applications, during the procedure, the computer processor determines the position of the tool relative to the three-dimensional image data and, in response, drives the display to highlight a vertebral volume where an unacceptable protrusion exists. For some applications, the computer processor drives the display to display multiple (e.g., 2-6) generally concentric cylinders within the three-dimensional image of the vertebra, the cylinders indicating corresponding acceptable degrees of protrusion of a tool into the multiple volumes defined by the multiple cylinders. During the procedure, the computer processor determines the position of the tool relative to the three-dimensional image data, and in response thereto, the computer processor drives the display to highlight the cylinder in which the tool or a portion of the tool is located and / or is expected to enter. For some applications, the computer processor performs the aforementioned functions, but not with respect to the tool currently being inserted (which may be a narrow tool, such as a needle), but rather with respect to positioning the final implant (e.g., a pedicle screw, typically having a larger diameter) later using the current tool. For some applications, the computer processor performs the aforementioned steps with respect to a two-dimensional cross-sectional image derived from the three-dimensional image data. For such cases, rectangles are typically used rather than cylinders to represent the respective acceptable degrees of protrusion of a tool into the regions defined by the rectangles.

[0526] For some applications, the processor allows a three-dimensional image of the vertebra with the tools, a representation of the tools, and / or a path of the tools indicated within the image to be rotated, or the processor automatically rotates the image, to allow the user to better understand the three-dimensional placement of the tools. It should be noted that because the multiple images of the vertebrae and the tools are input from multiple different imaging sources, the segmented data indicating the vertebrae and the tools (or representations of the tools) is built-in (i.e., already known to the computer processor). For some applications, the computer processor utilizes this built-in segmentation to allow the operator to virtually manipulate the multiple tools relative to the vertebrae. For example, the operator can virtually advance or retract the tools along the tool insertion path and observe the tool's movement relative to the vertebrae. For some applications, the computer processor can automatically virtually advance or retract the tools along the tool insertion path relative to the vertebrae in the three-dimensional image data.

[0527] For some applications, the accuracy of confirming the position of the portion of the tool within the three-dimensional image data is enhanced by registering three two-dimensional x-ray images to the three-dimensional image data, the multiple images being acquired from corresponding mutually different perspectives. Typically, for such applications, an oblique x-ray image perspective is used in addition to the anterior-posterior and lateral views. For some applications, the accuracy of confirming the positioning of the portion of the tool within the three-dimensional image data is enhanced by using multiple x-ray images, wherein the multiple portions of the tool or multiple portions of multiple tools are visible and distinguishable from each other in the multiple x-ray images. For some applications, the multiple tools are distinguished from each other based on a manual input by the operator or automatically input by the computer processor. For some applications, the accuracy of confirming the positioning of the portion of the tool within the three-dimensional image data is enhanced by reference to multiple known shapes and / or multiple sizes of multiple radiopaque markers 52 as described above.

[0528] Now refer to Figure 18 , Figure 18 A Jamshidi having a radiopaque clip 180 attached thereto according to some applications of the present invention TMA schematic diagram of needle 36. For some applications, the accuracy of confirming the position of a portion of the tool within the three-dimensional image data is enhanced by adding an additional radiopaque component to the tool (e.g., clip 180) such that the tool has at least two identifiable features in each two-dimensional image, namely, the distal tip of the tool and the additional radiopaque component. For some applications, the additional radiopaque component is configured to have a defined three-dimensional arrangement such that the additional radiopaque component provides an understanding of the orientation of the tool. For example, the additional radiopaque component may include an array of multiple radiopaque spheres. For some applications, the additional radiopaque component facilitates multiple additional functions, such as those described below. For some applications, the tool itself includes one or more radiopaque features that are identifiable in each two-dimensional X-ray image. For such applications, an additional radiopaque component (e.g., clip 180) is typically not attached to the tool.

[0529] For some applications, the plurality of imaging functions described above with reference to the three-dimensional image data are performed on the plurality of two-dimensional x-ray images based on the co-registration of the plurality of two-dimensional images to the three-dimensional image data. For example, the tool may be color-coded in the plurality of x-ray images based on the degree of placement of the tool. For some applications, if the tool is incorrectly placed, the computer processor drives the display to display in the plurality of two-dimensional x-ray images what the tool would appear to be when correctly placed.

[0530] Now refer to Figure 19A and 19B , Figure 19A and 19B 1 shows several examples of anterior-posterior X-ray images of a vertebra and corresponding lateral X-ray images at respective stages of insertion of a tool into the vertebra according to some applications of the present invention. Figure 7 Step 88. A common practice in spinal surgery performed under X-ray is to use two separate C-arm positions (usually any two of the anterior-posterior, lateral, and oblique) to obtain a three-dimensional understanding of the part during tool insertion and / or manipulation. This typically requires moving the C-arm back and forth and exposes the patient to a high radiation dose.

[0531] For some applications of the present invention, multiple images are initially acquired from two poses corresponding to respective image views. For example, FIG. 19A shows an example of AP and lateral X-ray images of a tool inserted dorsally into a vertebra. The C-arm is then maintained in a single pose to repeat multiple acquisitions during tool insertion and / or manipulation, but the computer processor derives the position of the tool relative to the vertebra in multiple additional X-ray imaging views and drives the display to display the derived position of the tool relative to the vertebra in the additional X-ray image views. For example: Figure 19B Shown Figure 19A An example of the tool and the anterior-posterior image of the vertebra, but the tool is relative to Figure 19A Based on the anterior-posterior image in which the tool has been advanced, the computer processor derives the new calculated position of the tool relative to the lateral X-ray imaging view and drives the display to display a representation of the new tool position on the lateral image 190. Typically, as Figure 19B In one embodiment, the computer processor displays the new calculated tool position on the lateral image in addition to the previously imaged position of the tool tip within the lateral image. Generally, the computer processor derives the position of the tool portion relative to one of the two original two-dimensional X-ray image views based on the current position of the tool portion identified within a current two-dimensional X-ray image and a relationship determined between a plurality of images acquired from the two original two-dimensional X-ray image views, as described in further detail below.

[0532] For some applications, the plurality of repeated acquisitions are performed from the same 2D X-ray image viewpoint as one of the plurality of original 2D X-ray image views, while for some applications, the plurality of repeated acquisitions are performed from a 2D X-ray image viewpoint different from the plurality of original 2D X-ray image views. For some applications, in a subsequent step, the tool within the vertebra is still periodically imaged from one or more additional 2D X-ray image views to verify the accuracy of the position of the tool within the plurality of additional views derived by the computer processor and, if necessary, to correct the positioning of the tool within the plurality of additional 2D X-ray image views. For some applications, the C-arm is maintained in a single pose (e.g., anterior-posterior) for repeated acquisitions during insertion and / or manipulation of the tool, and the computer processor automatically derives the position of a portion of the tool relative to the 3D image data of the vertebra and updates the image of the tool (or a representation of the tool) within the 3D image data.

[0533] Usually, reference Figures 19A to 19B Applications described herein are for use with a tool that is inserted into the bony anatomy along a longitudinal (i.e., a straight or substantially straight) insertion path. For some applications, the techniques are used with a tool that is inserted into the bony anatomy along a non-linear insertion path. For such applications, the computer processor typically identifies the non-linear expected path of advancement of the tool by analyzing previous advancements of the tool and / or by observing anatomical constraints along the tool insertion path and predicting the effects of the anatomical constraints. For such applications, the algorithms described below are modified accordingly.

[0534] For some applications, refer to Figures 19A to 19B The techniques described are performed with a primary X-ray imaging view, typically from the direction in which the intervention is to be performed (and typically with multiple combinations 50 of markers 52 placed on or near the subject so that the markers appear in the imaging view), and a secondary direction from which images are acquired to provide additional three-dimensional understanding. In the case of interventions performed dorsally, the primary X-ray imaging view is typically anterior-posterior, while the secondary view is typically generally lateral.

[0535] For some applications, the computer processor 22 performs the reference calculation using one of the following algorithms: Figures 19A to 19B The multiple techniques described.

[0536] Algorithm 1:

[0537] 1. Register the original two two-dimensional X-ray images XI and X2 to three-dimensional image data using the various techniques described above.

[0538] 2. Based on the registration, a substantially straight line of the tool TL (eg, the centerline or the tool's axis) derived from the plurality of 2D X-ray images is positioned relative to the 3D image data as TL-3D.

[0539] 3. Extrapolating the approximate straight line of the tool relative to the 3D image data to generate a forward line F-TL3D relative to the 3D image data.

[0540] 4. As the tool advances, acquire a new 2D X-ray X from only one of the previous poses, e.g., the same pose from which the original X was acquired. (Typically, to avoid moving the C-arm, this is the pose from which the most recent of the two previous 2D X-rays was acquired.)

[0541] For some applications, the computer processor verifies that there has been no movement of the C-arm relative to the subject and / or vice versa between the acquisitions of X1 and X1^ by comparing the appearance of markers 52 in the two images. For some applications, if there has been movement, Algorithm 2 described below is used.

[0542] 5. Through image processing, the computer processor identifies the position of the distal tip of the tool in image X1^, which is marked as TPX1^.

[0543] 6. The computer processor registers the two-dimensional X-ray image X 1 ^ to the three-dimensional image data using the digitally reconstructed radiographic image that matches the first X-ray view angle. Note that because the pose does not change between the multiple acquisitions of X 1 and X 1 ^ , the digitally reconstructed radiographic image that matches the X-ray X 1 ^ is identical to the X-ray X 1 ^ . Therefore, there is no need to re-search for the digitally reconstructed radiographic image that best matches the X-ray X 1 ^ .

[0544] 7. The computer processor draws a line from C1 to TPX1^ relative to the three-dimensional image data.

[0545] 8. The computer processor identifies the intersection of the line segment and the F-TL3D line relative to the three-dimensional image data as the new position of the tip. It should be noted that if the tool has been retracted, the computer processor identifies the intersection of the line segment and the straight line of the tool relative to the three-dimensional image data TL-3D, rather than the forward line F-TL3D relative to the three-dimensional image data.

[0546] 9. The computer processor drives the display relative to the three-dimensional image data or relative to the X-ray image X2 to display the tip of the tool (or a representation of the tool) at the new position of the tool.

[0547] Algorithm 2:

[0548] 1. Register the original two two-dimensional X-ray images XI and X2 to three-dimensional image data using the various techniques described above.

[0549] 2. Based on the registration, a substantially straight line of the tool TL (eg, the centerline or the tool's axis) derived from the plurality of X-ray images is positioned relative to the 3D image data as TL-3D.

[0550] 3. Extrapolating the substantially straight line of the tool relative to the 3D image data to generate a forward line F-TL3D relative to the 3D image data.

[0551] 4. As the tool advances, typically, a new two-dimensional X-ray X1^ is acquired from any pose, and not necessarily from one of the two previous poses.

[0552] 5. Through image processing, the computer processor identifies the position of the distal tip of the tool in image X3, which is marked as TPX3.

[0553] 6. The computer processor registers the two-dimensional X-ray image X3 to the three-dimensional image data of the vertebra by finding a digitally reconstructed radiography that best matches the two-dimensional X-ray image X3 using the multiple techniques described above, the new digitally reconstructed radiography having a corresponding simulated camera position C3.

[0554] 7. The computer processor draws a line from C1 to TPX3 relative to the three-dimensional image data.

[0555] 8. The computer processor identifies the intersection of the line segment and the F-TL3D line relative to the three-dimensional image data as the new position of the tip. It should be noted that if the tool has been retracted, the computer processor identifies the intersection of the line segment and the straight line of the tool relative to the three-dimensional image data TL-3D, rather than the forward line F-TL3D relative to the three-dimensional image data.

[0556] 9. The computer processor drives the display relative to the three-dimensional image data or relative to the X-ray images X1 and / or X2 to display the tip of the tool (or a representation of the tool) at the new position of the tool.

[0557] Algorithm 3:

[0558] Now refer to Figure 20 , Figure 20is a schematic diagram of a three-dimensional rigid fixture 194 according to some applications of the present invention, the three-dimensional rigid fixture 194 including at least a plurality of portions 196 of the three-dimensional rigid fixture 194 that are radiopaque and function as a plurality of radiopaque markers arranged in a predetermined three-dimensional arrangement. For some applications, as shown, the plurality of radiopaque portions 196 are a plurality of radiopaque spheres (as shown, for some applications, the plurality of radiopaque spheres have different sizes from each other), and the plurality of spheres are coupled to each other by a plurality of arms 198 that are generally radiolucent. Typically, the plurality of spheres are coupled to each other by the plurality of arms such that the plurality of spatial relationships between the plurality of spheres are precisely known.

[0559] Even in the case where the multiple x-ray images are not registered to the three-dimensional image data of the vertebra, the following algorithm is typically implemented by the computer processor 22. Typically, the algorithm is used with a three-dimensional radiopaque fixture, such as fixture 194, sufficient portions of which are visible in all applicable x-ray images and can be used to correlate the multiple x-ray images with each other. For some applications, such as Figure 20 As shown, the clamp comprises a three-dimensional array of a plurality of radiopaque spheres. For example, the clamp can be attached to the operating table.

[0560] 1. Using the multiple markers of the fixture as an anchor to register the two original two-dimensional X-ray images X1 and X2 to each other to provide a three-dimensional reference system.

[0561] 2. As the tool advances, acquire a new X-ray X^ from one of the previous poses, e.g., the same pose from which the original X^ was acquired. (Typically, to avoid moving the C-arm, this is the pose from which the two previous X-rays were acquired most recently.)

[0562] For some applications, the computer processor verifies that there has been no movement of the C-arm relative to the subject and / or vice versa between the acquisitions of X1 and X2 by comparing the appearance of markers 52 (typically, relative to the visible bone portion of the subject) and / or portions 196 of the fixture 194 (typically, relative to the visible bone portion of the subject) in the two images. For some applications, if there has been movement, one of the other algorithms described herein is used.

[0563] 3. Through image processing, the computer processor identifies the position of the distal tip of the tool in image X1^, which is marked as TPX1^.

[0564] 4. The computer processor registers the two-dimensional X-ray image X1^ to X2 using the fixture.

[0565] 5. The computer processor calculates the new position of the tool on X2 based on the registration.

[0566] 6. The computer processor drives the display relative to the X-ray image X2 to show the tip of the tool (or a representation of the tool) in the new position of the tool.

[0567] Algorithm 4:

[0568] Even in the case where the plurality of X-ray images are not registered to the three-dimensional image data of the vertebra, the following algorithm is typically implemented by the computer processor 22. Typically, the algorithm is used with a tool having two or more identifiable points in each two-dimensional X-ray image. For example, the algorithm can be used with a tool to which a clip or another radiopaque feature is attached, such as Figure 18 shown.

[0569] 1. In the two original two-dimensional X-ray images X1 and X2, the computer processor identifies the two identifiable points of the tool, such as the distal tip and the clip, through image processing.

[0570] 2. The computer processor determines a relationship between XI and X2, for example:

[0571] a. In XI, the multiple two-dimensional distances between the tip of the tool and the clamp are dxl pixels horizontally and dyl pixels vertically.

[0572] b. In X2, the two-dimensional distances between the tip of the tool and the gripper are dx2 pixels horizontally and dy2 pixels vertically.

[0573] c. Therefore, the computer processor determines a two-dimensional relationship between the two images based on the multiple ratios dx2:dx1 and dy2:dyl.

[0574] 3. As the tool advances, acquire a new X-ray X1^ from one of the multiple previous poses, for example, from the same pose used to acquire the original X-ray X1. (Typically, to avoid moving the C-arm, this is the pose most recently acquired between the two previous X-rays.)

[0575] For some applications, the computer processor verifies that there has been no movement of the C-arm relative to the subject and / or vice versa between the acquisitions of X1 and X1^ by comparing the appearance of the markers 52 in the two images. For some applications, if there has been movement, one of the other algorithms described herein is used.

[0576] 4. The computer processor identifies the tip of the tool in the image X1^ through image processing.

[0577] 5. The computer processor determines how many pixels the tip has moved between the multiple acquisitions of images XI and X1^.

[0578] 6. Based on the two-dimensional relationship between multiple images XI and X2 and the number of multiple pixels that the tip moves between the multiple acquisitions of multiple images X1 and X1^, the computer processor confirms the new position of the tip of the tool in image X2.

[0579] 7. The computer processor drives the display relative to the X-ray image X2 to show the tip of the tool (or a representation of the tool) in its new position.

[0580] refer to Figure 19A and 19BIn general, the scope of the present invention includes acquiring three-dimensional image data of a bone portion, and acquiring a first two-dimensional X-ray image and a second two-dimensional X-ray image of the bone portion and a portion of a tool from a plurality of corresponding X-ray image views. The tool is configured to be advanced into the bone portion along a longitudinal insertion path, with the portion of the tool positioned at a first position relative to the insertion path. The position of the portion of the tool relative to the bone portion is identified within the first and second two-dimensional X-ray images by a computer processor 22 of a system 20 through image processing, and the computer processor registers the two-dimensional X-ray images to the three-dimensional image data, for example, using the techniques described herein. Thus, a first position of the portion of the tool relative to the three-dimensional image data is determined. The tool is then advanced relative to the bone portion along the longitudinal insertion path, such that the portion of the tool is positioned at a second position along the longitudinal insertion path. After moving the portion of the tool along the insertion path to the second position, one or more additional two-dimensional X-ray images of at least the portion of the tool and the bone portion are acquired from a single image view. Depending on various applications, the single image perspective is the same as one of the plurality of original two-dimensional X-ray image perspectives or a different third two-dimensional X-ray image perspective. The computer processor 22 of the system 20 identifies, through image processing, the second position of the portion of the tool within the one or more additional two-dimensional X-ray images, and derives the second position of the portion of the tool relative to the three-dimensional image data based on the second position of the portion of the tool within the one or more additional two-dimensional X-ray images and the determined first position of the portion of the tool relative to the three-dimensional image data. Typically, an output is generated at least in part responsive thereto (e.g., by displaying the derived position of the tool relative to the X-ray image perspective from which the position was derived).

[0581] According to some applications, a first two-dimensional X-ray image and a second two-dimensional X-ray image of the bone portion and a portion of the tool are acquired from a plurality of corresponding X-ray image views, the portion of the tool being configured to be advanced into the bone portion along a longitudinal insertion path, with the portion of the tool positioned at a first position relative to the insertion path. Through image processing, the computer processor 22 of the system 20 identifies the position of the portion of the tool relative to the bone portion within the first and second two-dimensional X-ray images, and the computer processor determines a relationship between the first and second two-dimensional X-ray images. For example, any of the algorithms 1 to 4 described above may be used. Next, the tool is advanced relative to the bone portion and along the longitudinal insertion path, such that the portion of the tool is positioned at a second position along the longitudinal insertion path. After moving the portion of the tool along the insertion path to the second position, one or more additional two-dimensional X-ray images of at least the portion of the tool and the bone portion are acquired from a single image view. According to various applications, the single image view is the same as one of the plurality of original two-dimensional X-ray image views or a different third two-dimensional X-ray image view. The computer processor 22 of the system 20 identifies, through image processing, the second position of the portion of the tool within the one or more additional two-dimensional x-ray images, and derives the second position of the portion of the tool relative to one of the two-dimensional x-ray images based on the second position of the portion of the tool identified within the additional two-dimensional x-ray images and the determined relationship between the first two-dimensional x-ray image and the second two-dimensional x-ray image, and generates an output at least partially responsive thereto (e.g., by displaying the derived position of the tool relative to the viewing angle of the x-ray image from which the position was derived).

[0582] refer to Figure 19A and 19BSome examples of the various applications of the various techniques described are as follows: For an intervention performed on the back, multiple X-rays may initially be acquired from a lateral view and an anterior-posterior view. Typically, subsequent X-rays may be acquired only from an anterior-posterior view (and multiple optional periodic checks from the lateral view, as described above), and the multiple updated positions of the tool relative to the lateral view are derived and displayed. It should be noted that although the C-arm may interfere with the intervention in this configuration, the anterior-posterior view provides the best indication of the position of the tool relative to the spinal cord. Alternatively, typically, subsequent X-rays may be acquired only from a lateral view (and multiple optional periodic checks from the anterior-posterior view), and the multiple updated positions of the tool relative to the anterior-posterior view are derived and displayed. Typically, for such applications, multiple combinations 50 of multiple markers 52 are placed on the patient so that at least one group of the multiple markers is visible from the lateral view. Further alternatively, subsequent X-rays may typically be acquired from only an oblique view (and optionally periodic inspections from the lateral and / or anterior-posterior views), and the updated positions of the tool relative to the anterior-posterior and / or lateral views derived and displayed. It should be noted that the above applications are shown as examples and that the scope of the present invention includes applications using reference Figure 19A and 19B The various techniques described and the various interventions performed on any portion of the bony anatomy from any approach direction and any type of X-ray image view apply mutatis mutandis.

[0583] For some applications, the image of the tool (a representation of the tool and / or a tool path) derived from the two-dimensional image is overlaid as a hologram on the three-dimensional image data of the vertebra. As described above, according to these applications, because the multiple images of the vertebra and the tool (or representations of the tool) are input from multiple different imaging sources, the segmented data of what the tool (or representation of the tool) is and what the vertebra is is built-in (i.e., already known to the computer processor). For some applications, the computer processor utilizes this built-in segmentation to allow the operator to virtually manipulate the multiple tools relative to the vertebra within the hologram. For example, the operator can virtually advance or retract the tool along the tool's insertion path and observe the tool's movement relative to the vertebra. Alternatively, the computer processor can automatically drive the hologram display to virtually advance or retract the tool along the tool's insertion path. For some applications, similar techniques are applied to other tools and various body organs, and similar techniques apply accordingly. For example, this technique can be applied to a computed tomography image of the heart and combined with multiple two-dimensional angiographic images of a catheter within the heart.

[0584] For some applications, an optical camera is used to acquire multiple optical images of a tool. For example, an optical camera 114 mounted on an X-ray C-arm 34 can be used, as shown in FIG1 . Alternatively or additionally, an optical camera can be mounted on a separate arm, can be handheld, can be a rear-mounted camera on a display such as a tablet or mini-tablet device, can be placed on the surgeon's head, can be placed on another part of the surgeon's arm, and / or can be held by another member of the operating room staff. Typically, based on observing multiple two-dimensional images of the tool and using the various registration techniques described above, the computer processor derives the position of the tool relative to the three-dimensional image data. Furthermore, for some applications, the computer processor identifies the tool within an optical image acquired by the optical camera. For some such applications, the computer processor then overlays the three-dimensional image data onto the optical image by aligning the position of the tool within the three-dimensional image data with the position of the tool within the optical image.

[0585] The computer processor then drives an augmented reality display to display the three-dimensional image data overlaid on the optical image. This technique can be performed using any viewing direction of the optical camera that can view the tool, and generally does not require tracking the position of the object relative to the optical camera.

[0586] For some applications, the position of the tool within the optical image space is confirmed by using two or more optical cameras and / or one or more three-dimensional optical cameras. For some applications, even if a two-dimensional optical camera is used, the three-dimensional image data is overlaid on the optical image by aligning two or more tools from each of the multiple imaging modes. For some applications, even if a two-dimensional optical camera and a single tool are used, the three-dimensional image data is overlaid on the optical image by obtaining additional information about the orientation (e.g., rotation) of the tool and / or the depth of the tool under the skin. For some applications, such information is derived from the three-dimensional image data, wherein the position of the skin surface relative to the vertebrae is derived from the three-dimensional image data. Alternatively or additionally, such information is derived from an X-ray image, wherein the tool and the anatomy of the object are visible. Alternatively or additionally, such information is derived from a combination of the markers seen in an X-ray image, wherein the tool and the anatomy of the object are visible.

[0587] As described above, because the multiple images of the vertebrae and the tools (or representations of the tools) are input from different imaging sources, the segmented data of what the tools (or representations of the tools) are and what the vertebrae are is built-in (i.e., already known to the computer processor). For some applications, the computer processor utilizes this built-in segmentation to allow the operator to virtually manipulate the multiple tools relative to the vertebrae within an augmented reality display. For example, the operator can virtually advance or retract the tools along the tool insertion path and observe the movement of the tools relative to the vertebrae. Alternatively, the computer processor can automatically drive the augmented reality display to virtually advance or retract the tools along the tool insertion path.

[0588] While some applications of the present invention have been described with reference to three-dimensional computed tomography (CT) image data, the scope of the present invention includes applying the described techniques to three-dimensional magnetic resonance imaging (MRI) image data. For such applications, multiple two-dimensional projection images (geometrically similar to multiple digitally reconstructed radiographic images generated from the CT images) are typically generated from the MRI image data using the techniques described above and matched to the multiple two-dimensional images. For some applications, other techniques are used to register the MRI image data to multiple two-dimensional X-ray images. For example, pseudo-CT image data can be generated from the MRI image data (e.g., using the techniques described in van der Bom et al., "Registration of Multiple Two-Dimensional X-Ray Images to Three-Dimensional MRIs by Generating Pseudo-CT Data," Physics in Medicine and Biology, Vol. 56, No. 4), and multiple digitally reconstructed radiographic images generated from the pseudo-CT data are matched to the multiple X-ray images using the techniques described above.

[0589] For some applications, magnetic resonance imaging is used during spinal endoscopy along with the various techniques described herein (including with respect to Figure 7 Any of the steps described) is used to facilitate the performance of the spinal endoscopy. Spinal endoscopy is an emerging procedure, used, for example, in spinal decompression surgery. Generally, by using an endoscope, multiple tools can be inserted and manipulated through a smaller incision than in current comparable procedures used for similar purposes, resulting in a smaller access space providing a larger treatment volume than in traditional surgeries. Typically, the procedures are used to perform interventions on soft tissues, such as intervertebral discs. Such tissues are typically visible in magnetic resonance imaging images but are less visible or not visible at all in computed tomography images or two-dimensional x-ray images. Traditionally, the procedures begin with the insertion of a needle under C-arm imaging, followed by the insertion of a series of dilators to gradually expand the access path. Finally, an outer tube having a diameter of approximately 1 centimeter (cm) is held in place and an endoscope is inserted through the outer tube. From this point on, the procedure is performed under the endoscopic field of view.

[0590] For some applications, segment verification as described above is applied in a spinal endoscopy procedure in order to confirm the position of the vertebra at which an endoscopic examination of the spine is to be performed. Alternatively or additionally, as described above, the entry point for the spinal endoscopy can be confirmed using a bidirectional mapping of multiple optical images and multiple X-ray images. For some applications, the MRI image data is registered to multiple intraoperative two-dimensional X-ray images. Based on the registration, multiple additional steps as described above are typically performed. For example, the needle, dilator and / or endoscope (and / or a representation of the endoscope and / or the path of the endoscope) can be displayed relative to a target within the MRI image data (e.g., a three-dimensional MRI image, a two-dimensional cross-section derived from the three-dimensional MRI image data and / or a two-dimensional projection image derived from the three-dimensional MRI image data). For some applications, the endoscopic image data is collectively registered to multiple corresponding positions within the multiple intraoperative images. For example, a plurality of corresponding endoscopic image data points may be collectively registered to a plurality of corresponding locations within the plurality of intraoperative images. For some applications, the collectively registered endoscopic image data is displayed together with the plurality of intraoperative images and an indication of the collective registration of the plurality of corresponding endoscopic image data points to the plurality of corresponding locations within the plurality of intraoperative images. Alternatively or additionally, the endoscopic image data is collectively registered to magnetic resonance imaging image data. For example, a plurality of corresponding endoscopic image data points may be collectively registered to a plurality of corresponding locations within the magnetic resonance imaging image data. For some applications, the collectively registered endoscopic image data is displayed together with the magnetic resonance imaging image data and an indication of the collective registration of the plurality of corresponding endoscopic image data points to the plurality of corresponding locations within the magnetic resonance imaging image data.

[0591] For some applications, the various techniques described herein are performed in conjunction with a robotic arm, such as a relatively low-cost robotic arm having five to six degrees of freedom. According to some applications, the robotic arm is used to hold, manipulate, and / or activate a tool and / or to operate the tool along a preprogrammed path. For some applications, as described above, the computer processor 22 drives the robotic arm in response to imaging data to perform any of the operations described above.

[0592] Now refer to Figure 21A , Figure 21A Examples of X-ray images of a calibration fixture produced by a C-arm using an image intensifier and by a C-arm using a flat panel detector (on the right) are shown, reflecting the prior art. Figure 21B , Figure 21BAn example of an X-ray image acquired by a C-arm using an image intensifier according to some applications of the present invention is shown, the image including a radiopaque component 200 corresponding to a portion of a tool known to be straight, and a dashed line overlaid on the image and indicating how a line (e.g., a centerline) defined by the straight portion would appear if multiple distortions in the image were corrected.

[0593] As in Figure 21A As can be observed in the example shown in FIG, X-ray images generated by a C-arm using an image intensifier generally exhibit image distortion that increases toward the periphery of the image. In contrast, images generated using a planar detector generally exhibit no distortion. For some applications of the present invention, distortions in X-ray images generated by a C-arm using an image intensifier are at least partially automatically corrected through image processing. For example, the techniques described above can be used to correct the distortions in the images in order to subsequently register the corrected images to three-dimensional image data.

[0594] Reference Figure 21B For some applications, such an X-ray image is at least partially corrected by a computer processor 22 that identifies, through image processing, a radiopaque component 200 of a device within a portion of the radiographic image. For some applications, the radiopaque component is a portion of the tool that is known to be straight, a component having a different known shape, and / or two or more features disposed in a known arrangement relative to one another. For example, a Jamshidi TM The straight shaft of the needle.

[0595] For some applications, to at least partially correct an X-ray image including a radiopaque component known to be straight, the computer processor uses a plurality of techniques for automatically identifying a centerline of an object, such as those described in US 2010-0161022 to Tolkowsky, which is incorporated herein by reference, to generate a centerline of the component. Typically, the computer processor then at least partially corrects the image distortion in at least a portion of the image in which the known straight component is located by deforming the portion of the radiographic image so that the centerline of the radiopaque component of the device known to be straight appears straight within the radiographic image.

[0596] Figure 19B Shows the correction Figure 19BFIG2 is an example of an X-ray image before distortion in FIG2 , wherein the X-ray image includes a component 200 that is known to be straight and does not appear directly within the image, as can be observed relative to line 210. For some applications, two or more of the components are identified in respective portions of the image, and the distortion in the portions of the image is corrected accordingly. For some applications, based on distortion correction parameters determined for the radiopaque components that are known to be straight or known to have a different known shape, the distortions in the portions of the image where the components are not located are corrected.

[0597] For some applications of the present invention, the techniques described above are combined with a system that determines the position of the tip of a tool relative to a portion of a subject's body by: (a) calculating a position of a proximal portion of the tool disposed outside the subject's body, and (b) deriving a position of the tip of the tool relative to the portion of the subject's body relative to the three-dimensional image data based on the calculated position of the proximal portion of the tool. For example, the techniques can be used with a navigation system, such as one that can include one or more position sensors attached to a portion of a tool, which is typically disposed outside the subject's body, even during the procedure. (It should be noted that the position sensors disposed on the tool can be sensors tracked by a tracker disposed elsewhere, or they can be a tracker that tracks sensors disposed elsewhere, thereby serving as a position sensor for the tool.) For example, a tool can be inserted into a vertebra of the subject such that the distal tip of the tool (or a distal portion of the tool) is disposed within the vertebra, and a position sensor can be disposed on a proximal portion of the tool, wherein the tool is disposed outside the subject's body. The navigation system typically derives the position of the tool tip (or a distal portion of the tool) by detecting the position(s) of the position sensor(s) disposed on the proximal portion of the tool. The position of the tool tip (or a distal portion of the tool) is then derived based on an assumed position of the distal tip of the tool (or a distal portion of the tool) relative to the position sensor(s). The navigation system then overlays the derived position of the tool tip (or a distal portion of the tool) relative to the vertebra on previously acquired three-dimensional image data (e.g., images acquired before or while the subject is positioned in the operating room, but typically before the intervention begins). Alternatively or additionally, the position of the proximal portion of the tool disposed outside the subject's body can be calculated relative to a previously known position by video tracking the proximal portion of the tool and / or by tracking the motion of a portion of a robot coupled to the proximal portion of the tool. For example: the multiple values of the multiple joints of the robot based on the multiple corresponding values of the multiple joints of the robot relative to the previously known positions.

[0598] In such cases, the assumed position of the tool's distal tip (or a distal portion of the tool) based on the position sensor(s) is erroneous, and errors associated with confirming the position of the tool's tip (or a distal portion of the tool) may occur. This may be due, for example, to slight bending of the tool during insertion into the vertebra. Therefore, for some applications, multiple two-dimensional X-ray images are acquired, typically periodically, during the procedure, within the vertebra where the actual tip of the tool (or the distal portion of the tool) is visible. The position of the tool's tip (or the distal portion of the tool) relative to the vertebra as observed in the two-dimensional X-ray images is confirmed relative to the three-dimensional image data by registering the multiple two-dimensional X-ray images to the multiple two-dimensional image data. For example, the multiple two-dimensional X-ray images can be registered to the three-dimensional image data using the techniques described above. In this way, the actual position of the tool's tip (or the distal portion of the tool) relative to the vertebra is confirmed relative to the three-dimensional image data. For some applications, in response thereto, errors in the confirmation of the position of the tool tip (or the tool distal portion) relative to the vertebra within the three-dimensional image space generated by the confirmation navigation system are periodically corrected by system 20. For example, based on the confirmed position of at least the tool tip (or the tool distal portion), the computer processor can drive the display to update the indication of the position of the tool tip (or the tool distal portion) relative to the vertebra relative to the three-dimensional image data. For some applications, the navigation system includes the use of augmented reality, virtual reality, or mechanical manipulation of multiple tools, or any combination thereof.

[0599] By way of illustration and not limitation, it should be noted that the scope of the present invention includes applying the various apparatuses and methods described herein to any of the following applications:

[0600] • Multiple tool insertions in the same vertebra (eg, into both pedicles).

[0601] Any type of medical device or implant including: Jamshidi TM Needles, K-wires, pedicle markers, screws, endoscopes, RF probes, laser probes, injection needles, etc.

[0602] • An intervention performed from a lateral approach, in which case the functional roles of the anterior-posterior and lateral X-ray views described above are usually switched.

[0603] • Various interventions using multiple X-ray views other than the lateral and anterior-posterior views as an alternative or additional view. For example, multiple oblique imaging views can be used.

[0604] • An intervention performed from an anterior, oblique and / or posterior interventional approach.

[0605] • Interventions performed on multiple vertebrae, even in which case the intraoperative X-ray images of the vertebrae are typically registered to the three-dimensional image data of the corresponding vertebrae on a volumetric basis.

[0606] • Various interventions performed on multiple discs between multiple vertebrae.

[0607] • Various interventions performed neurologically.

[0608] Tool insertion under X-ray in a video imaging mode.

[0609] • Use of certain features of system 20 that utilize intraoperative two-dimensional X-ray imaging, but not preoperative three-dimensional imaging.

[0610] • Certain features of the system 20 without the use of some or all of the disposable items described above, such as a pendant 53.

[0611] Various orthopedic surgeries, e.g., multiple surgeries performed on the limbs and / or multiple joints.

[0612] Involvement of various other body organs.

[0613] For some applications, system 20 includes additional functionality to that described above. For example, the computer processor can generate an output indicating a current accuracy (e.g., the current accuracy of the verification of the vertebral segment, the confirmation of the insertion site, and / or the registration of the three-dimensional image data to the two-dimensional image), for example, based on a calculation of a statistic of the possible errors. For some applications, the computer processor generates a prompt indicating a new x-ray from one or more of a plurality of views that should be acquired. For example, the computer processor can generate the prompt based on the time that has passed since the acquisition of a previous x-ray from a given view and / or based on the distance a tool has moved since the acquisition of a previous x-ray from a given view and / or based on the observed changes in the positioning of a plurality of markers 52 relative to the C-arm.

[0614] Many applications of the present invention described in this specification can take the form of a computer program product accessible from a computer-usable medium or a computer-readable medium (e.g., a non-transitory computer-readable medium) that provides program code for use by or in connection with a computer or any instruction execution system, such as computer processor 22. For the purposes of this specification, a computer-usable medium or a computer-readable medium can include any medium capable of storing, communicating, propagating, or transmitting the program for use by or in connection with a computer or any instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus or device) or a propagation medium. Typically, the computer-usable medium or the computer-readable medium is a non-transitory computer-usable medium or a non-transitory computer-readable medium.

[0615] Examples of computer-readable media include semiconductor or solid-state memory, magnetic tape, removable computer disks, random access memory (RAM), read-only memory (ROM), rigid disks, and optical disks. Current examples of optical disks include compact disk-read-only memory (CD-ROM), compact disk-read / write (CD-R / W), and digital versatile disks (DVD). For some applications, cloud storage and / or storage on a remote server is used.

[0616] A data processing system suitable for storing and / or executing program code includes at least one processor (e.g., computer processor 22) coupled directly or indirectly to a plurality of memory components (e.g., memory 24) via a system bus. To reduce the number of times code must be retrieved from bulk storage during execution, the plurality of memory components may include local memory employed during actual execution of the program code, bulk storage, and cache memories providing temporary storage of at least some program code. The system may read the inventive instructions on the program storage devices and follow the instructions to perform the methods of the various embodiments of the present invention.

[0617] Multiple network adapters can be coupled to the processor to enable the processor to couple to other processors or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the currently available types of network adapters.

[0618] Computer program code for performing operations of the present invention may be written in any combination of one or more programming languages, including object oriented languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as the C programming language or similar programming languages.

[0619] It should be understood that Figure 7 、 14A The blocks of the flowcharts shown in FIG14B , combinations of the blocks in the flowcharts, and any of the algorithms described in this specification may be implemented by computer program instructions. The computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions are executed by the processor of the computer (e.g., computer processor 22) or other programmable data processing device, or by multiple creation devices for implementing the functions / actions set forth in the flowcharts and / or algorithms described in this application. The computer program instructions may also be stored in a computer-readable medium (e.g., a non-transitory computer-readable medium), which may instruct a computer or other programmable data processing device to operate in a specific manner so that the instructions stored in the computer-readable medium produce an article of manufacture comprising instruction devices that implement the functions / actions set forth in the flowchart blocks and algorithms. The multiple computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the multiple instructions executed on the computer or other programmable device provide multiple processes for implementing the multiple functions / actions proposed in the multiple flowcharts and / or multiple algorithms described in this application.

[0620] The computer processor 22 and other computer processors described herein are typically hardware devices programmed with computer program instructions to create a special-purpose computer. For example, when programmed to execute the algorithms described herein, the computer processor typically functions as a special-purpose bone surgery assistance computer processor. Typically, the operations described herein performed by computer processors transform the physical state of a memory, where memory is a physical entity having different magnetic polarity, charge, etc. depending on the memory technology used.

[0621] The embodiments of the present invention disclose but are not limited to the following schemes:

[0622] Solution 1. A method for use in conjunction with bone surgery for performing a procedure using a tool configured to be advanced along a longitudinal insertion path into a bone portion within a body of a subject, characterized in that the method comprises:

[0623] Acquiring three-dimensional image data of the bone part;

[0624] When a portion of the tool is disposed in a first position relative to the bone portion and along the longitudinal insertion path, sequentially:

[0625] acquiring a first two-dimensional X-ray image of at least the portion of the tool and the bone portion from a first viewing angle using a two-dimensional X-ray imaging device, the two-dimensional X-ray imaging device not being registered to the subject's body and the two-dimensional X-ray imaging device being positioned in a first posture relative to the subject's body;

[0626] moving the two-dimensional X-ray imaging device to a second position relative to the subject's body; and

[0627] When the two-dimensional X-ray imaging device is in the second posture, acquiring a second two-dimensional X-ray image of at least the portion of the tool and the bone portion from a second viewing angle;

[0628] Using at least one computer processor to:

[0629] Registering the first X-ray image and the second X-ray image to the three-dimensional image data, the registering comprising:

[0630] generating a plurality of two-dimensional projections from the three-dimensional image data; and

[0631] identifying a first two-dimensional projection and a second two-dimensional projection corresponding to the first two-dimensional X-ray image and the second two-dimensional X-ray image data of the skeletal portion;

[0632] identifying, by image processing, a position of the portion of the tool relative to the bone portion in the first two-dimensional X-ray image and the second two-dimensional X-ray image;

[0633] confirming the first position of the portion of the tool relative to the three-dimensional image data based on the identified position of the portion of the tool within the first and second two-dimensional x-ray images and registering the first and second two-dimensional x-ray images to the three-dimensional image data;

[0634] Next, moving the portion of the tool relative to the bone portion and along the longitudinal insertion path to a second position;

[0635] after moving the portion of the tool to the second position, acquiring one or more additional two-dimensional X-ray images of at least the portion of the tool and the bone portion from a single image view;

[0636] Using the computer processor to:

[0637] identifying, by image processing, the second position of the portion of the tool within the one or more additional two-dimensional x-ray images;

[0638] deriving the second position of the portion of the tool relative to the three-dimensional image data based on the second position of the portion of the tool within the one or more additional two-dimensional x-ray images and the identified first position of the portion of the tool relative to the three-dimensional image data; and

[0639] An output is generated at least in part responsive thereto.

[0640] Option 2. The method according to Option 1 is characterized in that: obtaining the one or more additional two-dimensional X-ray images of at least the part of the tool and the bone part from the single image perspective includes: obtaining the one or more additional two-dimensional X-ray images of at least the part of the tool and the bone part from one of the first image perspective and the second image perspective.

[0641] Option 3. The method according to Option 1 is characterized in that: obtaining the one or more additional two-dimensional X-ray images of at least the part of the tool and the bone part includes: obtaining the one or more additional two-dimensional X-ray images of at least the part of the tool and the bone part from a third image perspective that is different from the first image perspective and the second image perspective.

[0642] 4. A method for use in conjunction with bone surgery for performing a procedure using a tool configured to be advanced along a longitudinal insertion path into a bone portion within a body of a subject, the method comprising:

[0643] When a portion of the tool is disposed at a first position along the longitudinal insertion path, sequentially:

[0644] acquiring a first two-dimensional X-ray image of at least the portion of the tool and the bone portion from a first viewing angle, the two-dimensional X-ray imaging device not being registered to the subject's body, and the two-dimensional X-ray imaging device being positioned in a first posture relative to the subject's body;

[0645] moving to a second posture relative to the subject's body; and

[0646] When the two-dimensional X-ray imaging device is in the second posture, acquiring a second two-dimensional X-ray image of at least the portion of the tool and the bone portion from a second viewing angle;

[0647] identifying, using at least one computer processor, the portion of the tool within the first two-dimensional X-ray image and the second two-dimensional X-ray image through image processing;

[0648] using the computer processor, determining a relationship between the first two-dimensional X-ray image and the second two-dimensional X-ray image;

[0649] advancing the tool relative to the bone portion along the longitudinal insertion path such that the portion of the tool is disposed at a second position along the longitudinal insertion path;

[0650] Next, acquiring one or more additional two-dimensional X-ray images of at least the portion of the tool and the bone portion from a single image view; and

[0651] Using the computer processor to:

[0652] identifying, by image processing, the second position of the portion of the tool within the one or more additional two-dimensional x-ray images;

[0653] deriving the second position of the portion of the tool relative to at least one of the first two-dimensional x-ray image and the second two-dimensional x-ray image based on the second position of the portion of the tool identified in the one or more additional two-dimensional x-ray images and the determined relationship between the first two-dimensional x-ray image and the second two-dimensional x-ray image; and

[0654] An output is generated at least in part responsive thereto.

[0655] Option 5. The method according to Option 4 is characterized in that: confirming the relationship between the first two-dimensional X-ray image and the second two-dimensional X-ray image includes: aligning the first two-dimensional X-ray image and the second two-dimensional X-ray image with each other by aligning the first two-dimensional X-ray image and the second two-dimensional X-ray image to a three-dimensional image data of the bone part.

[0656] Option 6. The method according to Option 4 is characterized in that: confirming the relationship between the first two-dimensional X-ray image and the second two-dimensional X-ray image includes: using a three-dimensional non-transmissive rigid clamp visible in the first two-dimensional X-ray image and the second two-dimensional X-ray image to align the first two-dimensional X-ray image and the second two-dimensional X-ray image with each other.

[0657] Option 7. The method according to Option 4 is characterized in that: confirming the relationship between the first two-dimensional X-ray image and the second two-dimensional X-ray image includes: using two or more non-transmissive features of the tool visible in the first two-dimensional X-ray image and the second two-dimensional X-ray image to confirm the relationship between the first two-dimensional X-ray image and the second two-dimensional X-ray image.

[0658] Option 8. The method according to Option 4 is characterized in that: obtaining the one or more additional two-dimensional X-ray images of at least the part of the tool and the bone part from the single image perspective includes: obtaining the one or more additional two-dimensional X-ray images of at least the part of the tool and the bone part from one of the first image perspective and the second image perspective.

[0659] Option 9. The method according to Option 4 is characterized in that: obtaining the one or more additional two-dimensional X-ray images of at least the part of the tool and the bone part from the single image perspective includes: obtaining the one or more additional two-dimensional X-ray images of at least the part of the tool and the bone part from a third image perspective different from the first image perspective and the second image perspective.

[0660] 10. A method for performing a procedure on a given vertebra of a spine of a subject, the method comprising:

[0661] placing a combination of a plurality of radiopaque markers in a vicinity of the object such that the plurality of markers placed at respective locations relative to the object are identifiable;

[0662] acquiring, using a radiocontrast imaging device, a plurality of radiocontrast images of the combination of the plurality of radiopaque markers, the plurality of corresponding images being at a plurality of corresponding locations along at least a portion of the subject's spine, and each of the plurality of images including at least some of the plurality of radiopaque markers;

[0663] Using at least one computer processor to:

[0664] identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0665] combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0666] automatically confirming a position of the given vertebra within the combined plurality of radiographic images through image processing; and

[0667] An output is generated at least in part responsive thereto.

[0668] Option 11. A method according to Option 10, characterized in that: generating the output includes: driving a display to display the combined multiple radiographic images and an indication of the given vertebra displayed together with the combined multiple radiographic images, and the method also includes manually confirming a physical position of the given vertebra relative to the combination of the non-transmissive markers placed in the vicinity of the object based on the position within the given vertebra in the combined multiple radiographic images.

[0669] Option 12. A method according to Option 10, characterized in that: generating the output includes: driving a display to display the combined multiple radiographic images and an indication of the given vertebra displayed together with the combined multiple radiographic images, and the method also includes driving the display to display an indication of the given vertebra relative to a three-dimensional image data of at least a portion of the spinal column of the subject.

[0670] Solution 13. The method according to any one of solutions 10 to 12, wherein automatically confirming the position of the given vertebra within the combined plurality of radiographic images by image processing comprises:

[0671] identifying an identifiable feature within the combined plurality of radiographic images;

[0672] identifying individual vertebrae within the combined plurality of radiographic images; and

[0673] Vertebrae are counted from the identifiable features.

[0674] Option 14. The method according to Option 13, wherein identifying an identifiable feature within the combined plurality of radiographic images comprises identifying a sacrum of the subject within the combined plurality of radiographic images.

[0675] 15. An apparatus for performing a procedure on a given vertebra of a spine of a subject and for use with a radiographic imaging device, comprising:

[0676] a combination of a plurality of radiopaque markers configured to be placed in a vicinity of the object such that the plurality of markers placed at respective locations relative to the object are identifiable; and

[0677] At least one computer processor configured to:

[0678] receiving a plurality of radiocontrast images of the plurality of radiopaque markers from the radiocontrast imaging device, the plurality of corresponding images being a plurality of corresponding locations along at least a portion of the subject's spine, and each of the plurality of images including at least some of the plurality of radiopaque markers;

[0679] identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0680] combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0681] automatically confirming a position of the given vertebra within the combined plurality of radiographic images through image processing; and

[0682] An output is generated at least in part responsive thereto.

[0683] Option 16. The device according to Option 15, characterized in that the device is used in conjunction with a display, and the at least one computer processor is configured to generate the output by driving the display to display the combined multiple radiographic images and an indication of the given vertebra displayed together with the combined multiple radiographic images.

[0684] Option 17. The device according to Option 15 is characterized in that: the device is used in conjunction with a display, and the at least one computer processor is configured to drive the display to display the combined multiple radiographic images and an indication of the given vertebra displayed together with the combined multiple radiographic images, and to drive the display to display an indication of the given vertebra relative to a three-dimensional image data of at least a portion of the spinal column of the subject.

[0685] Solution 18. The apparatus according to any one of solutions 15 to 17, wherein the at least one computer processor is configured to automatically confirm the position of the given vertebra within the combined plurality of radiographic images by:

[0686] identifying a recognizable feature within the combined radiographic images;

[0687] identifying individual vertebrae within the combined radiographic images; and

[0688] Vertebrae are counted from the identifiable features.

[0689] Option 19. The apparatus of Option 18, wherein the at least one computer processor is configured to identify the identifiable feature within the combined plurality of radiographic images by identifying a sacrum of the subject within the combined plurality of radiographic images.

[0690] 20. A computer software product for performing a procedure on a given vertebra of a spine of a subject, the computer software product being used in conjunction with a radiographic imaging device and a combination of a plurality of radiopaque markers, the combination of the plurality of radiopaque markers being configured to be placed in a vicinity of the subject such that the plurality of markers placed at a plurality of corresponding positions relative to the subject are identifiable, the computer software product comprising a non-transitory computer-readable medium storing a plurality of program instructions that, when read by a computer, cause the computer to perform the following steps:

[0691] receiving a plurality of radiocontrast images of the combination of the plurality of radiopaque markers from the radiocontrast imaging device, the plurality of corresponding images being a plurality of corresponding locations along at least a portion of the subject's spine, and each of the plurality of images including at least some of the plurality of radiopaque markers;

[0692] identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; and combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0693] automatically confirming a position of the given vertebra within the combined plurality of radiographic images through image processing; and

[0694] An output is generated at least in part responsive thereto.

[0695] 21. A method for performing a procedure on a given rib of a thorax of a subject, the method comprising:

[0696] placing a combination of a plurality of radiopaque markers in a vicinity of the object such that the plurality of markers placed at respective locations relative to the object are identifiable;

[0697] acquiring, using a radiocontrast imaging device, a plurality of radiocontrast images of a combination of the plurality of radiopaque markers, the plurality of corresponding images being a plurality of corresponding locations along at least a portion of the subject's thorax, and each of the plurality of images including at least some of the plurality of radiopaque markers; and

[0698] Using at least one computer processor to:

[0699] identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0700] combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0701] automatically confirming a position of the given rib within the combined plurality of radiographic images through image processing; and

[0702] An output is generated at least in part responsive thereto.

[0703] 22. An apparatus for performing a procedure on a given rib of a thorax of a subject, for use with a radiographic imaging device, comprising:

[0704] a combination of a plurality of radiopaque markers configured to be placed in a vicinity of the object such that the plurality of markers placed at respective locations relative to the object are identifiable; and

[0705] At least one computer processor configured to:

[0706] receiving a plurality of radiocontrast images of the plurality of radiopaque markers from the radiocontrast imaging device, the plurality of corresponding images being a plurality of corresponding locations along at least a portion of the subject's thorax, and each of the plurality of images including at least some of the plurality of radiopaque markers;

[0707] identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0708] combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0709] automatically confirming a position of the given rib within the combined plurality of radiographic images through image processing; and

[0710] An output is generated at least in part responsive thereto.

[0711] 23. A computer software product for performing a procedure on a given rib of a thorax of a subject, wherein the computer software product is used in conjunction with a radiographic imaging device and a combination of a plurality of radiopaque markers, the combination of the plurality of radiopaque markers being configured to be placed in a vicinity of the subject such that the plurality of markers placed at a plurality of corresponding positions relative to the subject are identifiable, the computer software product comprising a non-transitory computer-readable medium storing a plurality of program instructions, which, when read by a computer, cause the computer to perform the following steps:

[0712] receiving a plurality of radiocontrast images of the combination of the plurality of radiopaque markers from the radiocontrast imaging device, the plurality of corresponding images being a plurality of corresponding locations along at least a portion of the subject's thorax, and each of the plurality of images including at least some of the plurality of radiopaque markers;

[0713] identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; and combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images;

[0714] automatically confirming a position of the given rib within the combined plurality of radiographic images through image processing; and

[0715] An output is generated at least in part responsive thereto.

[0716] 24. An apparatus for performing a procedure on a spine of a body of a subject, comprising:

[0717] a first set of radiopaque markers;

[0718] a second set of radiopaque markers; and

[0719] one or more surfaces through which the first set of radiopaque markers and the second set of radiopaque markers are coupled to each other, the one or more surfaces being configured to:

[0720] positioning the first set of markers and the second set of markers on respective sides of the subject's spine in a plurality of predefined orientations relative to one another by placing the one or more surfaces on a portion of the subject's spine on which the procedure is to be performed; and

[0721] After positioning the first set of markers and the second set of markers on the multiple corresponding sides of the subject's spine, the one or more surfaces can be removed from the subject's body to facilitate performing the procedure on the subject's spine on the one or more surfaces placed on the subject's spine while maintaining the first set of markers and the second set of markers on the multiple corresponding sides of the subject's spine in the predefined orientation relative to each other.

[0722] Option 25. The device according to Option 24, characterized i...

Claims

1. An apparatus for performing a procedure in which a tool is advanced along an insertion path into a bony portion within a subject's body, a portion of the tool being coupled to a portion of a robot and used with: (i) a 3D imaging device configured to acquire 3D image data of the bone portion, and (ii) a 2D imaging device configured to, when a portion of the tool is disposed at a first position relative to the bone portion along the insertion path, sequentially: acquiring a first 2D x-ray image of at least a portion of the tool and the bone portion from a first perspective while the 2D imaging device is positioned in a first posture relative to the subject's body, moving to a second posture relative to the subject's body, and acquiring a second 2D x-ray image of the at least a portion of the tool and the bone portion from a second viewing angle when the 2D imaging device is in the second posture, and (iii) displays, The device comprises: at least one computer processor configured to: (a) receiving the 3D image data from the 3D imaging device; (b) receiving the first 2D x-ray image and the second 2D x-ray image of the at least a portion of the tool and the bone portion from the 2D imaging device; (c) registering the first 2D x-ray image and the second 2D x-ray image to the 3D image data; (d) identifying a position of the portion of the tool relative to the bone portion within the first 2D x-ray image and the second 2D x-ray image; (e) based on the identified positions of the portion of the tool within the first and second 2D x-ray images and the registration of the first and second 2D x-ray images with the 3D image data, (i) determining a first position of the portion of the tool relative to the 3D image data, and (ii) deriving a relationship between the first position of the portion of the tool relative to the 3D image data and a given position within the 3D image data; (f) tracking movement of the portion of the tool by tracking movement of a portion of the robot coupled to the portion of the tool relative to the first position when the robot is used to move the portion of the tool to a second position relative to the bone portion along the insertion path in response to the derived relationship between the first position of the portion of the tool and the given position within the 3D image data; (g) deriving the second position of the part of the tool relative to the 3D image data based on tracking the motion of the part of the robot coupled to the part of the tool; as well as (h) driving the display to display the second position of the part of the tool relative to the 3D image data, Where (f) is repeated iteratively.

2. The device according to claim 1, wherein The at least one computer processor is configured to derive a relationship between the first position of the part of the tool relative to the 3D image data and the given position within the 3D image data by deriving a relationship between the first position of the part of the tool relative to the 3D image data and a pre-specified target position within the 3D image data.

3. The device according to claim 1, wherein The 2D imaging device is not registered relative to the subject's body, and the computer processor is configured to receive the first and second 2D x-ray images of at least a portion of the tool and the bone portion from the 2D imaging device that is not registered relative to the subject's body.

4. A non-transitory computer software product for facilitating execution of a procedure in which a tool is advanced along an insertion path into a bone portion within a body of a subject, a portion of the tool being coupled to a portion of a robot, the computer software product for use with: (i) a 3D imaging device configured to acquire 3D image data of the bone portion, and (ii) a 2D imaging device configured to, when a portion of the tool is disposed at a first position relative to the bone portion along the insertion path, sequentially: acquiring a first 2D x-ray image of at least a portion of the tool and the bone portion from a first perspective while the 2D imaging device is positioned in a first posture relative to the subject's body, moving to a second posture relative to the subject's body, and acquiring a second 2D x-ray image of the at least a portion of the tool and the bone portion from a second viewing angle when the 2D imaging device is in the second posture, and (iii) displays, The computer software product includes a non-transitory computer-readable medium having program instructions stored therein, which, when read by a computer, causes the computer to perform the following steps: (a) receiving the 3D image data from the 3D imaging device; (b) receiving the first 2D x-ray image and the second 2D x-ray image of the at least a portion of the tool and the bone portion from the 2D imaging device; (c) registering the first 2D x-ray image and the second 2D x-ray image to the 3D image data; (d) identifying a position of the portion of the tool relative to the bone portion within the first 2D x-ray image and the second 2D x-ray image; (e) based on the identified positions of the portion of the tool within the first and second 2D x-ray images and the registration of the first and second 2D x-ray images with the 3D image data, (i) determining a first position of the portion of the tool relative to the 3D image data, and (ii) deriving a relationship between the first position of the portion of the tool relative to the 3D image data and a given position within the 3D image data; (f) tracking movement of the portion of the tool by tracking movement of a portion of the robot coupled to the portion of the tool relative to the first position when the robot is used to move the portion of the tool to a second position relative to the bone portion along the insertion path in response to the relationship derived between the first position of the portion of the tool and the given position within the 3D image; (g) deriving the second position of the part of the tool relative to the 3D image data based on tracking the motion of the part of the robot coupled to the part of the tool; as well as (h) driving the display to display the second position of the part of the tool relative to the 3D image data, Where (f) is repeated iteratively.

5. The non-transitory computer software product according to claim 4, wherein: Exporting the relationship between the first position of the part of the tool relative to the 3D image data and a given position within the 3D image data includes: exporting the relationship between the first position of the part of the tool relative to the 3D image data and a pre-specified target position within the 3D image data.

6. The non-transitory computer software product according to claim 4, wherein: The 2D imaging device is not aligned with respect to the body of the subject, and wherein receiving the first and second 2D x-ray images of at least a portion of the tool and the bone portion from the 2D imaging device comprises receiving the first and second 2D x-ray images of at least a portion of the tool and the bone portion from the 2D imaging device that is not aligned with respect to the body of the subject.

7. A method for performing a procedure on a given vertebra of a spine of a subject, characterized in that: The method comprises: placing a combination of a plurality of radiopaque markers in a vicinity of the subject such that the plurality of markers placed at a plurality of corresponding locations relative to the subject are identifiable; acquiring, using a radiocontrast imaging device, a plurality of radiocontrast images of the combination of the plurality of radiopaque markers, the plurality of corresponding images being at a plurality of corresponding locations along at least a portion of the subject's spine, and each of the plurality of images including at least some of the plurality of radiopaque markers; Using at least one computer processor to: identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; automatically confirming a position of the given vertebra within the combined plurality of radiographic images through image processing; and An output is generated at least in part responsive thereto.

8. The method according to claim 7, wherein: Generating the output includes driving a display to display the combined plurality of radiographic images and an indication of the given vertebra displayed together with the combined plurality of radiographic images, the method further comprising manually confirming a physical location of the given vertebra relative to the combination of radiopaque markers placed in the vicinity of the subject based on the location within the given vertebra in the combined plurality of radiographic images.

9. The method according to claim 7, wherein: Generating the output includes driving a display to display the combined plurality of radiographic images and an indication of the given vertebra displayed relative to the combined plurality of radiographic images. The method further includes driving the display to display an indication of the given vertebra relative to three-dimensional image data of at least a portion of the subject's spinal column.

10. The method according to any one of claims 7 to 9, characterized in that: Automatically confirming the position of the given vertebra within the combined plurality of radiographic images by image processing comprises: automatically confirming the position of the given vertebra within the combined plurality of radiographic images by image processing: identifying an identifiable feature within the combined plurality of radiographic images; identifying individual vertebrae within the combined plurality of radiographic images; and Vertebrae are counted from the identifiable features.

11. The method according to claim 10, characterized in that: Identifying an identifiable feature within the combined radiographic images includes identifying a sacrum of the subject within the combined radiographic images.

12. An apparatus for performing a procedure on a given vertebra of a spine of a subject and for use with a radiographic imaging device, characterized in that: The device comprises: a combination of a plurality of radiopaque markers configured to be placed in a vicinity of the object such that the plurality of markers placed at respective locations relative to the object are identifiable; and At least one computer processor configured to: receiving a plurality of radiocontrast images of the plurality of radiopaque markers from the radiocontrast imaging device, the plurality of corresponding images being a plurality of corresponding locations along at least a portion of the subject's spine, and each of the plurality of images including at least some of the plurality of radiopaque markers; identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; automatically confirming a position of the given vertebra within the combined plurality of radiographic images through image processing; and An output is generated at least in part responsive thereto.

13. The device according to claim 12, characterized in that: The apparatus is for use with a display, and the at least one computer processor is configured to generate the output by driving the display to display the combined plurality of radiographic images and an indication of the given vertebra displayed together with the combined plurality of radiographic images.

14. The device according to claim 12, characterized in that: The apparatus is for use with a display, and the at least one computer processor is configured to drive the display to display the combined plurality of radiographic images and an indication of the given vertebra displayed relative to the combined plurality of radiographic images, and to drive the display to display an indication of the given vertebra relative to three-dimensional image data of at least a portion of the subject's spine.

15. The device according to any one of claims 12 to 14, characterized in that: The at least one computer processor is configured to automatically confirm the position of the given vertebra within the combined plurality of radiographic images by: identifying a recognizable feature within the combined radiographic images; identifying individual vertebrae within the combined radiographic images; and Vertebrae are counted from the identifiable features.

16. The device according to claim 15, characterized in that: The at least one computer processor is configured to identify the identifiable feature within the combined plurality of radiographic images by identifying a sacrum of the subject within the combined plurality of radiographic images.

17. A computer software product for performing a procedure on a given vertebra of a spinal column of a subject, characterized in that: The computer software product is for use with a radiographic imaging device and a combination of radiopaque markers configured to be placed in a vicinity of the object such that the markers placed at corresponding positions relative to the object are identifiable, the computer software product comprising a non-transitory computer-readable medium storing program instructions that, when read by a computer, cause the computer to perform the following steps: receiving a plurality of radiocontrast images of the combination of the plurality of radiopaque markers from the radiocontrast imaging device, the plurality of corresponding images being a plurality of corresponding locations along at least a portion of the subject's spine, and each of the plurality of images including at least some of the plurality of radiopaque markers; identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; automatically confirming a position of the given vertebra within the combined plurality of radiographic images through image processing; as well as An output is generated at least in part responsive thereto.

18. A method for performing a procedure on a given rib of a thorax of a subject, characterized in that: The method comprises: placing a combination of a plurality of radiopaque markers in a vicinity of the object such that the plurality of markers placed at respective locations relative to the object are identifiable; acquiring, using a radiocontrast imaging device, a plurality of radiocontrast images of a combination of the plurality of radiopaque markers, the plurality of corresponding images being a plurality of corresponding locations along at least a portion of the subject's thorax, and each of the plurality of images including at least some of the plurality of radiopaque markers; and Using at least one computer processor to: identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; automatically confirming a position of the given rib within the combined plurality of radiographic images through image processing; and An output is generated at least in part responsive thereto.

19. An apparatus for performing a procedure on a given rib of a thorax of a subject and for use with a radiographic imaging device, characterized in that: The device comprises: a combination of a plurality of radiopaque markers configured to be placed in a vicinity of the object such that the plurality of markers placed at respective locations relative to the object are identifiable; and At least one computer processor configured to: receiving a plurality of radiocontrast images of the plurality of radiopaque markers from the radiocontrast imaging device, the plurality of corresponding images being a plurality of corresponding locations along at least a portion of the subject's thorax, and each of the plurality of images including at least some of the plurality of radiopaque markers; identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; automatically confirming a position of the given rib within the combined plurality of radiographic images through image processing; and An output is generated at least in part responsive thereto.

20. A computer software product for performing a procedure on a given rib of a thorax of a subject, characterized in that: The computer software product is for use with a radiographic imaging device and a combination of radiopaque markers configured to be placed in a vicinity of the object such that the markers placed at corresponding positions relative to the object are identifiable, the computer software product comprising a non-transitory computer-readable medium storing program instructions that, when read by a computer, cause the computer to perform the following steps: receiving a plurality of radiocontrast images of the combination of the plurality of radiopaque markers from the radiocontrast imaging device, the plurality of corresponding images being a plurality of corresponding locations along at least a portion of the subject's thorax, and each of the plurality of images including at least some of the plurality of radiopaque markers; identifying, through image processing, a plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; combining at least some of the plurality of radiocontrast images relative to one another based on the identified plurality of locations of the plurality of radiopaque markers within the plurality of radiocontrast images; automatically confirming a position of the given rib within the combined plurality of radiographic images through image processing; as well as An output is generated at least in part responsive thereto.

21. An apparatus for performing a procedure on a spine of a body of a subject, characterized in that: The device comprises: a first set of radiopaque markers; a second set of radiopaque markers; and one or more surfaces through which the first set of radiopaque markers and the second set of radiopaque markers are coupled to each other, the one or more surfaces being configured to: positioning the first set of markers and the second set of markers on respective sides of the subject's spine in a plurality of predefined orientations relative to one another by placing the one or more surfaces on a portion of the subject's spine on which the procedure is to be performed; and After positioning the first set of markers and the second set of markers on the multiple corresponding sides of the subject's spine, the one or more surfaces can be removed from the subject's body to facilitate performing the procedure on the subject's spine on the one or more surfaces placed on the subject's spine while maintaining the first set of markers and the second set of markers on the multiple corresponding sides of the subject's spine in the predefined orientation relative to each other.

22. The device according to claim 21, characterized in that: At least some of the plurality of markers in each of the first set of markers and the second set of markers are rigid and have a plurality of known dimensions.

23. The device according to claim 21, characterized in that: Each of the first set of markers and the second set of markers includes a combination of a plurality of radiopaque characters, and the plurality of combinations are different from each other.

24. The device according to any one of claims 21 to 23, characterized in that: The device also includes a third set of radiopaque markers, which are coupled to the first set of markers and the second set of markers through the one or more surfaces, so that the third set of markers is configured to be positioned along a center of the subject's spine through the one or more surfaces placed above the portion of the subject's spine on which the procedure is to be performed when the first set of markers and the second set of markers are positioned relative to each other in the multiple predefined positions on multiple corresponding sides of the subject's spine.

25. A method for performing a procedure on a spine of a body of a subject, characterized by: The method comprises: supply: a first set of radiopaque markers; a second set of radiopaque markers; and one or more surfaces by which the first set of radiopaque markers and the second set of radiopaque markers are coupled to each other; positioning the first set of markers and the second set of markers on respective sides of the subject's spine in a plurality of predefined orientations relative to one another by placing the one or more surfaces on a portion of the subject's spine on which the procedure is to be performed; and After positioning the first set of markers and the second set of markers on the multiple corresponding sides of the subject's spine, the one or more surfaces are removed from the subject's body to facilitate performing the procedure on the subject's spine on the one or more surfaces placed on the subject's spine, while maintaining the first set of markers and the second set of markers on the multiple corresponding sides of the subject's spine in the predefined orientation relative to each other.

26. The method according to claim 25, characterized in that: Providing the first and second sets of markers includes providing the first and second sets of markers, at least some markers of each of the first and second sets of markers being rigid and having a plurality of known dimensions.

27. The method according to claim 25, characterized in that: Providing the first set of markers and the second set of markers includes providing the first set of markers and the second set of markers, each of the first set of markers and the second set of markers including a combination of a plurality of radiopaque characters, the plurality of combinations being different from each other.

28. The method according to any one of claims 25 to 27, characterized in that: The method also includes providing a third set of radiopaque markers, which are coupled to the first set of markers and the second set of markers through the one or more surfaces, wherein the one or more surfaces placed over the portion of the spine of the subject on which the procedure is to be performed include: when the first set of markers and the second set of markers are positioned relative to each other at the multiple predefined positions on multiple corresponding sides of the spine of the subject, the third set of radiopaque markers are positioned along a center of the spine of the subject.

29. An apparatus for use with a radiographic imaging device configured to acquire a plurality of radiographic images of a spine of a body of a subject, characterized in that: The device comprises: a flexible material configured to be placed over the subject's body along at least a portion of the subject's spine and to generally conform to the contours of the subject's body; and a combination of a plurality of radiopaque markers disposed on the flexible material, at least some of the radiopaque markers being rigid and having a plurality of corresponding known dimensions, the plurality of rigid radiopaque markers being disposed on the flexible material such that the rigidity of the plurality of rigid markers does not prevent the flexible material from generally conforming to the contours of the subject's body; At least one computer processor configured to: identifying, by image processing, at least one of the rigid radiopaque markers within a radiographic image of the spine acquired by the radiographic imaging device; and Based on the identified plurality of rigid radiopaque markers, sizes of features within the radiocontrast image are determined through image processing.

30. A method for use with a radiocontrast imaging device configured to acquire a plurality of radiocontrast images of a spine of a subject's body, characterized in that: The method comprises: placing a flexible material on the subject's body along at least a portion of the subject's spine such that the flexible material substantially conforms to the contours of the subject's body, a combination of a plurality of radiopaque markers disposed on the flexible material, at least some of the radiopaque markers being rigid and having a plurality of corresponding known dimensions, the plurality of rigid radiopaque markers disposed on the flexible material such that the rigidity of the plurality of rigid markers does not prevent the flexible material from substantially conforming to the contours of the subject's body; and Using at least one computer processor to: identifying, by image processing, at least one of the rigid radiopaque markers within a radiographic image of the spine acquired by the radiographic imaging device; and Based on the identified plurality of rigid radiopaque markers, sizes of features within the radiocontrast image are determined through image processing.

31. A method for performing a procedure on the spine of a subject's body, characterized by: The method comprises: supply: a first set of radiopaque markers; and a second set of radiopaque markers; Positioning the first set of markers and the second set of markers relative to the subject's spine such that the first set of markers appears in a plurality of radiographic images of the subject's spine acquired from a first imaging perspective, and such that the second set of markers appears in a plurality of radiographic images of the subject's spine acquired from a second imaging perspective; acquiring a plurality of radiographic images of the subject's spine from the first image view and the second image view; and A given vertebra appearing in the multiple radiographic images of the spine from the first image view is associated with a given vertebra of the spine in the multiple radiographic images of the spine from the second image view by identifying multiple markers that have a known correlation with other markers in the multiple radiographic images acquired from the first image view and the second image view.

32. The method according to claim 31, characterized in that: Associating the given vertebra appearing in the multiple radiographic images of the spine from the first image view with a given vertebra of the spine from the multiple radiographic images of the spine from the second image view includes: using at least one computer processor to associate the given vertebra appearing in the multiple radiographic images of the spine from the first image view with the given vertebra of the spine from the multiple radiographic images of the spine from the second image view by identifying multiple markers that have a known correlation with other markers in the multiple radiographic images obtained from the first image view and the second image view.

33. The method according to claim 31, wherein: Associating the given vertebra appearing in the multiple radiographic images of the spine from the first image view with a given vertebra of the spine from the multiple radiographic images of the spine from the second image view includes manually associating the given vertebra appearing in the multiple radiographic images of the spine from the first image view with the given vertebra of the spine from the multiple radiographic images of the spine from the second image view by identifying multiple markers that have a known correlation with other markers in the multiple radiographic images acquired from the first image view and the second image view.

34. An apparatus for performing a procedure on a spine of a body of a subject, characterized in that: The device comprises: a first set of radiopaque markers configured for placement relative to the subject's spine such that the first set of radiopaque markers appear in a plurality of radiographic images of the subject's spine acquired from a first imaging view; and a second set of radiopaque markers, The first and second sets of markers are coupled to one another such that when the first set of radiopaque markers are positioned relative to the subject's spine such that the first set of radiopaque markers appear in a radiographic image of the subject's spine acquired from a first imaging perspective, the second set of markers appear in a radiographic image of the subject's spine acquired from a second imaging perspective; The first set of markers and the second set of markers thereby facilitate associating a given vertebra appearing in the multiple radiographic images of the spine from the first image view with a given vertebra of the spine in the multiple radiographic images of the spine from the second image view by facilitating identification of multiple markers having a known association with other markers in the multiple radiographic images acquired from the first image view and the second image view.

35. The device according to claim 34, characterized in that: The device also includes at least one computer processor configured to associate a given vertebra appearing in the multiple radiographic images of the spine from the first image view with a given vertebra of the spine in the multiple radiographic images of the spine from the second image view by identifying multiple markers that have a known correlation with other markers in the multiple radiographic images acquired from the first image view and the second image view.

36. A method for performing a procedure using a tool configured to be advanced along a longitudinal insertion path into a bony portion within a body of a subject, characterized in that: The method comprises: Acquiring three-dimensional image data of the bone part; When a portion of the tool is disposed at a position along the longitudinal insertion path, sequentially: acquiring a first two-dimensional X-ray image of at least the portion of the tool and the bone portion from a first viewing angle using a two-dimensional X-ray imaging device, the two-dimensional X-ray imaging device not being registered to the subject's body and the two-dimensional X-ray imaging device being positioned in a first posture relative to the subject's body; moving the two-dimensional X-ray imaging device to a second position relative to the subject's body, and When the two-dimensional X-ray imaging device is in the second posture, acquiring a second two-dimensional X-ray image of at least the portion of the tool and the bone portion from a second viewing angle; Using at least one computer processor to: Registering the first two-dimensional X-ray image and the second two-dimensional X-ray image to the three-dimensional image data, the registering comprising: generating a plurality of two-dimensional projections from the three-dimensional imaging device; and identifying a first two-dimensional projection and a second two-dimensional projection that match the first two-dimensional X-ray image and the second two-dimensional X-ray image; identifying, by image processing, the position of the portion of the tool relative to the bone portion in the first X-ray image and the second X-ray image; deriving a relationship between the position of the portion of the tool relative to the three-dimensional image data and a given position within the three-dimensional image data based on the identified position of the portion of the tool within the first and second x-ray images and registering the first and second two-dimensional x-ray images to the three-dimensional image data; and An output is generated that indicates the relationship between the position of the portion of the tool relative to the three-dimensional image data and a given position within the three-dimensional image data.

37. The method according to claim 36, wherein: Generating the output to derive the relationship between the position of the portion of the tool relative to the three-dimensional image data and the given position within the three-dimensional image data includes generating the output on a two-dimensional cross-section of the bone portion derived from the three-dimensional image data.

38. The method according to claim 36, wherein: Generating the output to derive the relationship between the position of the portion of the tool relative to the three-dimensional image data and the given position within the three-dimensional image data includes generating the output on a two-dimensional projection of the bone portion derived from the three-dimensional image data.

39. The method according to claim 36, wherein: Deriving the relationship between the position of the portion of the tool relative to the three-dimensional image data and the given position within the three-dimensional image data includes generating the output on a three-dimensional image of the skeletal portion derived from the three-dimensional image data.

40. The method according to any one of claims 36 to 39, characterized in that: Deriving the relationship between the position of the portion of the tool relative to the three-dimensional image data and the given position within the three-dimensional image data includes deriving an expected longitudinal insertion path of the tool relative to the given position within the three-dimensional image data.

41. The method according to any one of claims 36 to 39, characterized in that: Exporting the relationship between the position of the portion of the tool relative to the three-dimensional image data and the given position within the three-dimensional image data includes exporting a relationship between the first position of the portion of the tool relative to the three-dimensional image data and a pre-specified target position within the three-dimensional image data.

42. The method according to any one of claims 36 to 39, characterized in that: Deriving the relationship between the position of the portion of the tool relative to the three-dimensional image data and the given position within the three-dimensional image data includes deriving a relationship between the first position of the portion of the tool relative to the three-dimensional image data and a plurality of corresponding volumes within the three-dimensional image data, the plurality of corresponding volumes specifying a plurality of corresponding acceptable degrees of protrusion of the tool relative to the plurality of corresponding volumes.

43. A method for performing a procedure using a tool configured to be advanced along a longitudinal insertion path into a bony portion within a body of a subject, characterized in that: The method comprises: Acquiring three-dimensional image data of the bone part; specifying a position within the skeletal portion relative to the three-dimensional image data; acquiring a first two-dimensional X-ray image of at least the portion of the tool and the bone portion from a first viewing angle using a two-dimensional X-ray imaging device, the two-dimensional X-ray imaging device not being registered to the subject's body and the two-dimensional X-ray imaging device being positioned in a first posture relative to the subject's body; then moving the two-dimensional X-ray imaging device to a second posture relative to the subject's body; When the two-dimensional X-ray imaging device is in the second posture, acquiring a second two-dimensional X-ray image of at least the portion of the tool and the bone portion from a second viewing angle; Using at least one computer processor to: Registering the first two-dimensional X-ray image and the second two-dimensional X-ray image to the three-dimensional image data, the registering comprising: generating a plurality of two-dimensional projections from the three-dimensional image data; and identifying a first two-dimensional projection and a second two-dimensional projection corresponding to the first two-dimensional X-ray image and the second two-dimensional X-ray image data of the skeletal portion; identifying, by image processing, the position of the portion of the tool relative to the bone portion within the first two-dimensional x-ray image and the second two-dimensional x-ray image; deriving an orientation of the designated location within at least one of the plurality of two-dimensional x-ray images of the bone portion based on the identified position of the portion of the tool within the first two-dimensional x-ray image and the second two-dimensional x-ray image, and registering the first two-dimensional x-ray image and the second two-dimensional x-ray image to the three-dimensional image data, identifying, by image processing, a position of at least a portion of the tool relative to the at least one of the plurality of two-dimensional rays of the skeletal portion; confirming a relationship between an intended longitudinal insertion path of the tool and the designated location within at least one of the plurality of two-dimensional x-rays of the bone portion based on the identified location of the tool portion; and An output is generated at least in part responsive thereto.

44. The method according to claim 43, wherein: Specifying the position within the skeletal component relative to the three-dimensional image data includes specifying a target position within the skeletal component relative to the three-dimensional image data.

45. The method according to claim 43, wherein: Specifying the location within the bone portion relative to the three-dimensional image data includes specifying one or more locations within the bone portion that the tool should avoid relative to the three-dimensional image data.

46. The method according to any one of claims 43 to 45, characterized in that: Specifying the position within the skeletal portion relative to the three-dimensional image data includes specifying the position relative to a cross-section of the skeletal portion derived from the three-dimensional image data.

47. The method according to any one of claims 43 to 45, characterized in that: Specifying the position within the skeletal component relative to the three-dimensional image data includes specifying the position relative to a two-dimensional projection of the skeletal component derived from the three-dimensional image data.

48. The method according to any one of claims 43 to 45, characterized in that: Specifying the position within the skeletal component relative to the three-dimensional image data includes specifying the position relative to a three-dimensional image of the skeletal component derived from the three-dimensional image data.

49. A method for executing a program using a tool, characterized in that: The tool is configured to be advanced along a longitudinal insertion path into a bony portion within a body of a subject, the method comprising: Acquiring three-dimensional image data of the bone part; acquiring a first two-dimensional X-ray image of at least the portion of the tool and the bone portion from a first viewing angle using a two-dimensional X-ray imaging device, the two-dimensional X-ray imaging device not being registered to the subject's body and the two-dimensional X-ray imaging device being positioned in a first posture relative to the subject's body; then moving the two-dimensional X-ray imaging device to a second posture relative to the subject's body; When the two-dimensional X-ray imaging device is in the second posture, acquiring a second two-dimensional X-ray image of at least the portion of the tool and the bone portion from a second viewing angle; Using at least one computer processor to: Registering the first two-dimensional X-ray image and the second two-dimensional X-ray image to the three-dimensional image data, the registering comprising: generating a plurality of two-dimensional projections from the three-dimensional image data; and identifying a first two-dimensional projection and a second two-dimensional projection corresponding to the first two-dimensional X-ray image and the second two-dimensional X-ray image data of the skeletal portion; identifying, by image processing, a position of a tip of the tool relative to the bone portion in the first two-dimensional X-ray image and the second two-dimensional X-ray image; confirming a position of the tip of the tool relative to the tool in the three-dimensional image data based on the identified position of the portion of the tool in the first and second two-dimensional x-ray images and registering the first and second two-dimensional x-ray images to the three-dimensional image data; and In response thereto, a display is driven to display a cross-section of the bone portion, the cross-section being derived from the three-dimensional image data and corresponding to the position of the tip of the tool.

50. The method of claim 49, wherein: The method also includes driving the display to display an indication of a position of the tool on the cross-section of the bone portion.

51. The method according to claim 49 or 50, characterized in that: Driving the display to display the cross-section of the bone portion includes: driving the display to display a cross-sectional view based on a direction in which the tool is inserted, the cross-sectional view being selected from a group consisting of an axial cross-section, a coronal cross-section, a sagittal cross-section, and a cross-sectional view.

52. A method for performing a procedure using a tool configured to be advanced along a longitudinal insertion path into a given vertebra within a body of a subject, characterized in that: The method comprises: acquiring three-dimensional image data of at least a portion of a spinal column of the subject, the spinal column including the given vertebra; acquiring a first two-dimensional x-ray image of at least the portion of the tool and a spine of the subject, the spine including the given vertebra, from a first viewing angle using a two-dimensional x-ray imaging device, the two-dimensional x-ray imaging device not being registered to the subject's body and the two-dimensional x-ray imaging device being positioned in a first posture relative to the subject's body; then moving the two-dimensional X-ray imaging device to a second posture relative to the subject's body; acquiring a second two-dimensional X-ray image of at least the portion of the tool and the subject's spine from a second viewing angle when the two-dimensional X-ray imaging device is in the second posture, the spine including the given vertebra; Using at least one computer processor to: receiving an input indicating a location of the given vertebra within the three-dimensional image data; automatically identifying a position of the given vertebra in the first two-dimensional X-ray image and the second two-dimensional X-ray image through image processing; Registering the given vertebra in the first two-dimensional X-ray image and the second two-dimensional X-ray image to the given vertebra in the three-dimensional image data, the registering comprising: generating a plurality of two-dimensional projections of the given vertebra from the three-dimensional image data; and identifying a first two-dimensional projection and a second two-dimensional projection that match the given vertebra within the first two-dimensional X-ray image and the second two-dimensional X-ray image of the skeletal portion, respectively; identifying, by image processing, a position of the portion of the tool relative to the given vertebra within the first two-dimensional X-ray image and the second two-dimensional X-ray image of the portion of the spine; confirming the position of the portion of the tool relative to the three-dimensional image data based on the identified position of the portion of the tool within the first and second x-ray images and registering the first and second two-dimensional x-ray images to the three-dimensional image data; and In response thereto, an output is generated.

53. A method for use with a radiographic imaging device and an instrument having a straight radiopaque component, characterized in that: The method comprises: using the radiocontrast imaging device, acquiring a radiocontrast image in which at least a portion of the straight radiopaque component of the device is visible; Using at least one computer processor to: identifying, by image processing, the straight radiopaque component of the device within the radiocontrast image; at least partially correcting distortion in at least a portion of the radiographic image by deforming the portion of the radiographic image so that the straight radiopaque portion of the device within the radiographic image appears straight; and An output is generated in response thereto.

54. The method according to claim 53, wherein: The method further includes correcting an additional portion of the image based on the correction applied to a portion of the image that is present within the straight radiopaque component of the instrument.

55. The method according to claim 53 or 54, characterized in that: The method further includes registering the corrected radiographic image to three-dimensional image data using the at least one computer processor.

56. The method according to claim 55, characterized in that: Acquiring the radiocontrast image includes acquiring a radiocontrast image of the straight radiopaque member of the instrument and a body portion of a subject, and registering the corrected radiocontrast image to three-dimensional image data includes: generating a plurality of two-dimensional projections of the portion of the subject's body from the three-dimensional image data; and A two-dimensional projection is identified that matches the portion of the subject's body within the radiographic image.

57. An apparatus for use with a radiographic imaging device, an output device, and an instrument having a straight radiopaque component, characterized in that: The device comprises: At least one computer processor configured to: receiving a radiocontrast image from the radiocontrast imaging device in which at least a portion of the straight radiopaque member of the device is visible; identifying, by image processing, the straight radiopaque component of the device within the radiocontrast image; at least partially correcting distortion in at least a portion of the radiographic image by deforming the portion of the radiographic image such that the straight radiopaque portion of the device within the radiographic image appears straight; and In response thereto, an output is generated on the output device.

58. The apparatus according to claim 57, characterized in that: The at least one computer processor is further configured to correct an additional portion of the image based on the correction applied to a portion of the image that is present within the straight radiopaque component of the instrument.

59. The apparatus according to claim 57 or 58, characterized in that: The at least one computer processor is further configured to register the corrected radiographic image to three-dimensional image data.

60. The apparatus according to claim 59, wherein: The radiographic image includes the straight radiopaque component of the instrument and a body portion of a subject, and the at least one computer processor is configured to register the corrected radiographic image to the three-dimensional image data by: generating a plurality of two-dimensional projections of the portion of the subject's body from the three-dimensional image data, and A two-dimensional projection is identified that matches the portion of the subject's body within the radiographic image.

61. A computer software product for use with a radiographic imaging device, an output device, and an instrument having a straight radiopaque component, characterized in that: The computer software product includes a non-transitory computer-readable medium storing a plurality of program instructions that, when read by a computer, cause the computer to perform the following steps: receiving a radiocontrast image from the radiocontrast imaging device in which at least a portion of the straight radiopaque component of the device is visible; identifying, by image processing, the straight radiopaque portion of the device within the radiocontrast image; at least partially correcting distortion in at least a portion of the radiocontrast image by deforming the portion of the radiocontrast image so that the straight radiopaque portion of the device within the radiocontrast image appears straight, and In response thereto, an output is generated on the output device.

62. A method for performing a plurality of interventional procedures on at least a first vertebra and a second vertebra of a spinal column of a subject, characterized in that: The method comprises: acquiring imaging data of the subject's spine using an imaging device; Using at least one computer processor to: generating on a display a spinal roadmap image of at least a portion of the spine, the spine including the first vertebra and the second vertebra; automatically labeling vertebrae within the spinal roadmap image; confirming that an intervention has been performed on the first vertebra such that an appearance of the first vertebra has been altered; and The spinal roadmap is automatically updated to reflect the change in the appearance of the first vertebra such that when the intervention is performed on the second vertebra, the updated spinal roadmap is displayed.

63. The method according to claim 62, wherein: Acquiring the imaging data includes acquiring three-dimensional imaging data, and generating the spinal path map image includes generating a three-dimensional spinal path map image.

64. The method according to claim 62, wherein: Acquiring the imaging data includes acquiring 2D imaging data, and generating the spinal path map image includes generating a 2D spinal path map image.

65. The method according to any one of claims 62 to 64, characterized in that: Confirming that an intervention has been performed on the first vertebra such that an appearance of the first vertebra has been changed includes confirming that a tool has been inserted into the first vertebra, and updating the spinal roadmap includes updating the spinal roadmap to display the tool within the vertebra.

66. An apparatus for use in performing a procedure for multiple interventions on at least a first vertebra and a second vertebra of a spine of a subject, a display, and an imaging device configured to acquire imaging data of the subject's spine, characterized in that: The device comprises: At least one computer processor configured to: receiving the imaging data from the imaging device; generating on the display a spinal roadmap image of at least a portion of the spinal column, the spinal column including the first vertebra and the second vertebra; automatically labeling vertebrae within the spinal roadmap image; confirming that an intervention has been performed on the first vertebra such that an appearance of the first vertebra has been altered; and The spinal roadmap is automatically updated to reflect the change in the appearance of the first vertebra such that when the intervention is performed on the second vertebra, the updated spinal roadmap is displayed.

67. The apparatus according to claim 66, characterized in that: The imaging device includes an imaging device configured to acquire three-dimensional imaging data, and the at least one computer processor is configured to generate the spinal roadmap image by generating a three-dimensional spinal roadmap image.

68. The apparatus according to claim 66, characterized in that: The imaging device includes an imaging device configured to acquire two-dimensional imaging data, and the at least one computer processor is configured to generate the spinal roadmap image by generating a two-dimensional spinal roadmap image.

69. The apparatus according to any one of claims 66 to 68, characterized in that: The at least one computer processor is configured to confirm that an intervention has been performed on the first vertebra by confirming that a tool has been inserted into the first vertebra such that an appearance of the first vertebra has been changed, and the at least one computer processor is configured to update the spinal roadmap by updating the spinal roadmap to display the tool within the vertebra.

70. A computer software product having a program for performing a plurality of interventions in at least a first vertebra and a second vertebra of a subject's spine, a display, and an imaging device configured to acquire imaging data of the subject's spine, characterized in that: The computer software product includes a non-transitory computer-readable medium storing a plurality of program instructions. When the plurality of instructions are read by a computer, the plurality of instructions cause the computer to execute the following steps: receiving the image data from the imaging device; generating a spinal roadmap image of at least a portion of the spine on the display; automatically labeling vertebrae within the spinal roadmap image; confirming that an intervention has been performed on the first vertebra such that an appearance of the first vertebra has been altered; as well as The spinal roadmap is automatically updated to reflect the change in the appearance of the first vertebra such that when the intervention is performed on the second vertebra, the updated spinal roadmap is displayed.

71. A method for applying a tool to a body part of a subject during a medical intervention, characterized in that: The method comprises: acquiring a plurality of images of the tool and the subject's body during the medical intervention; determining, using at least one computer processor, a position of the tool relative to the subject's body by analyzing a plurality of images; detecting, using a motion detection sensor, that movement of at least a portion of the subject's body has occurred exceeding a threshold amount between acquisition of two or more images, the motion detection sensor being disposed on the subject's body and configured to detect movement of the motion detection sensor itself compared to a previous orientation of the motion detection sensor; and An alarm is generated in response thereto, the alarm indicating that the motion has occurred.

72. The method according to claim 71, characterized in that: Generating the alarm includes generating the alarm by driving an output device of the motion detection sensor to generate an alarm.

73. The method according to claim 71, wherein: Generating the alarm includes: the motion detection sensor at the computer processor receiving a signal indicating that movement of at least a portion of the subject's body has occurred exceeding the threshold amount, and Using the computer processor, an output is generated in response thereto.

74. The method according to claim 71, wherein: Generating the alert includes generating an output indicating that one or more images should be reacquired.

75. An apparatus for applying a tool to a part of a body of a subject during a medical intervention, and an imaging device configured to acquire a plurality of images of the subject's body during the intervention, characterized in that: The device comprises: at least one computer processor configured to determine a position of the tool relative to the subject's body by analyzing a plurality of images; a motion detection sensor configured to be disposed on the subject's body and to detect movement of at least the portion of the subject's body by detecting movement of the motion detection sensor itself compared to a previous position of the motion detection sensor, the motion detection sensor being configured to detect movement of at least the portion of the subject's body that exceeds a threshold amount between acquisition of two or more images; and An output device is configured to generate an alarm in response to the motion detection sensor detecting that movement of at least the portion of the subject's body has occurred exceeding the threshold amount, the alarm indicating that movement has occurred.

76. The apparatus according to claim 75, characterized in that: The motion detection sensor is configured to drive the output device to generate the alarm.

77. The apparatus according to claim 75, characterized in that: The at least one computer processor is configured to receive a signal from the sensor indicating that movement of at least a portion of the subject's body has occurred exceeding the threshold amount, and to drive the output device to generate the alarm in response thereto.

78. The apparatus according to claim 75, characterized in that: The output device is configured to generate an output when the motion detection sensor detects that movement of at least the portion of the subject's body has occurred exceeding the threshold amount, the output indicating that one or more images should be reacquired.

79. A method for performing a procedure using a tool configured to be advanced along a longitudinal insertion path into a bony portion within a body of a subject, characterized in that: The method comprises: Acquiring three-dimensional image data of the bone part; acquiring one or more two-dimensional x-ray images of at least the portion of the tool from a single viewing angle using a two-dimensional x-ray imaging device when a tip of the tool is positioned at a given position along the longitudinal insertion path, the two-dimensional x-ray imaging device not being registered to the subject's body and the two-dimensional x-ray imaging device being positioned in a first posture relative to the subject's body; Using a computer processor to: registering one of the one or more two-dimensional X-ray images acquired from the single view to the three-dimensional image data, the registering comprising: generating a plurality of two-dimensional projections from the three-dimensional image data; and identifying a two-dimensional projection that matches the one of the one or more two-dimensional X-ray images of the skeletal portion acquired from the single view; identifying, by image processing, a position of the tip of the tool relative to the bone portion within the one of the one or more two-dimensional x-ray images; confirming a position of the tip of the tool relative to the two-dimensional projection that matches the one of the one or more two-dimensional x-ray images of the bone portion based on the identified position of the tip of the tool relative to the bone portion within the one of the one or more two-dimensional x-ray images and registering the one of the one or more two-dimensional x-ray images to the three-dimensional image data; and In response thereto, a display is driven to display a cross-section of the bone portion, the cross-section being derived from the three-dimensional image data and corresponding to the position of the tip of the tool of the two-dimensional projection matched to the one or more two-dimensional X-ray images of the bone portion.

80. The method according to claim 79, wherein: The method also includes driving the display to display a line on the cross section, the line indicating that the position of the tip of the tool within the cross section is somewhere on the line.

81. A method for applying a tool to a body part of a subject during a medical intervention, characterized by: The method comprises: Acquiring three-dimensional image data of the bone part; Using at least one computer processor to: calculating a position of the proximal portion of the tool disposed outside the subject's body; deriving a position of the distal portion of the tool relative to the portion of the subject's body relative to the three-dimensional image data based on the calculated position of the proximal portion of the tool; and Based on the derived position, driving a display to display the indication of the distal portion of the tool relative to the portion of the subject's body relative to the three-dimensional image data; sequentially: acquiring a first two-dimensional X-ray image of the distal portion of the tool and the subject's body from a first viewing angle using a two-dimensional X-ray imaging device, the two-dimensional X-ray imaging device not being registered to the subject's body and the two-dimensional X-ray imaging device being disposed in a first posture relative to the subject's body; Next, moving the two-dimensional X-ray imaging device to a second posture relative to the subject's body; and acquiring a second two-dimensional X-ray image of the distal portion of the tool and the subject's body from a second viewing angle when the two-dimensional X-ray imaging device is in the second posture; Using at least one computer processor to: registering the portion of the subject's body within the first and second two-dimensional x-ray images to the portion of the subject's body within the three-dimensional image data; identifying, by image processing, a position of the distal end portion of the tool relative to the bone portion in the first two-dimensional X-ray image and the second two-dimensional X-ray image; confirming the position of the distal portion of the tool relative to the tool in the three-dimensional image data based on registering the portion of the subject's body within the first and second two-dimensional x-ray images to the portion of the subject's body within the three-dimensional image data and the identified position of the distal portion of the tool within the first and second two-dimensional x-ray images; and Based on the confirmed position of the distal portion of the tool, the display is driven to update the indication of the position of the distal portion of the tool relative to the portion of the subject's body relative to the three-dimensional image data.

82. The method according to claim 81, characterized in that: Registering the portion of the subject's body within the first and second two-dimensional X-ray images to the portion of the subject's body within the three-dimensional image data includes: generating a plurality of two-dimensional projections of the portion of the subject's body from the three-dimensional image data; and A first two-dimensional projection and a second two-dimensional projection are identified that match the portion of the subject's body within the first two-dimensional X-ray image and the second two-dimensional X-ray image of the portion of the subject's body, respectively.

83. The method according to claim 81 or 82, wherein: Calculating the position of the proximal portion of the tool disposed outside the body of the subject includes calculating the position of the proximal portion of the tool disposed outside the body of the subject via one or more position sensors coupled to the proximal portion of the tool.

84. The method according to claim 81 or 82, wherein: Calculating the position of the proximal portion of the tool disposed outside the body of the subject includes calculating the position of the proximal portion of the tool disposed outside the body of the subject by tracking the proximal portion of the tool through image tracking.

85. The method according to claim 81 or 82, wherein: Calculating the position of the proximal portion of the tool disposed outside the body of the subject includes calculating the position of the proximal portion of the tool disposed outside the body of the subject by tracking a portion of a robot, the portion of the robot being at a known previous position relative to the portion of the robot to which the proximal portion of the tool is coupled.

Images