Tracking system using 2D image projection and spatial registration of images
Through multi-mode marking equipment and two-dimensional image projection technology, the problems of high radiation energy and unavailability of equipment in image registration are solved, and efficient and low-radiation CT and CBCT image registration are achieved.
Patent Information
- Application Number
- CN202080022003.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-04-04
- Filing Date
- 2020-04-06
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-04-06
AI Technical Summary
The prior art requires a large amount of radiation energy in image registration, and CBCT imaging devices are unavailable from time to time, making it difficult to achieve effective CT and CBCT image registration.
The multi-mode marking device is used to register the tracking system and image space through two-dimensional image projection, and the predetermined part of the multi-mode marking acquires the two-dimensional projected image at different angles, estimates its three-dimensional position, and determines the affine based on the relationship between the sensor and the mark. Transform, reduce radiation exposure and achieve registration.
Effective registration of CT and CBCT images under reduced radiation conditions is achieved, the efficiency and accuracy of image registration are improved, and the dependence on CBCT devices is reduced.
Smart Images

Figure CN113573641B_ABST
Abstract
Description
[0001] Cross references to related applications
[0002] This application claims priority to U.S. Provisional Application No. 62 / 829,394, filed on April 4, 2019, entitled “SPATIAL REGISTRATION OF TRACKING SYSTEM WITH AN IMAGE USING TWO-DIMENSIONAL IMAGE PROJECTIONS,” which is incorporated herein by reference in its entirety. Technical Field
[0003] The present disclosure generally relates to systems and methods for registering a tracking system with an image space using two-dimensional image projections. Background Art
[0004] Image registration is the process of determining a spatial transformation for aligning two or more image coordinate systems to each other. An example is the registration of computed tomography (CT) and cone beam computed tomography (CBCT) images. For example, a CT image can be obtained before a procedure, such as a few days or weeks before a procedure (e.g., a study or treatment). A CBCT image can be obtained just before or at the time of a procedure. The CBCT image set is mathematically processed to align it with a previous CT image space. This CT and CBCT registration is used for a variety of purposes, such as planning, diagnosis, and treatment (e.g., interventional radiology, image-guided therapy, etc.). However, the equipment for acquiring CBCT images may not be available. In addition, CBCT imaging typically involves a large amount of radiation energy for patients and medical personnel. Summary of the invention
[0005] The present disclosure generally relates to systems and methods for registering a tracking system with an image space using one or more two-dimensional image projections.
[0006] As an example, a method includes acquiring a first two-dimensional projection image using a medical imaging modality. The first image includes a two-dimensional field of view that includes a patient and a multimodal marker. The method also includes acquiring a second two-dimensional projection image using the medical imaging modality. The second image includes the patient and the multimodal marker and is along a non-coincident angle relative to the first projection image. A predetermined portion of the multimodal marker is visible in the first projection image and the second projection image and has a known position and orientation relative to at least one sensor detectable by a tracking system. The method also includes estimating a three-dimensional position of the predetermined portion of the multimodal marker relative to a coordinate system of the medical imaging modality based on the position of the predetermined portion in each of the corresponding first and second two-dimensional projection images. The method also includes determining an affine transformation for aligning the three-dimensional coordinate system of the tracking system with the three-dimensional coordinate system of the medical imaging modality based on the estimated position of the corresponding predetermined portion of the multimodal marker and the known relationship between the at least one sensor and the predetermined portion of the multimodal marker.
[0007] As another example, a system includes one or more non-transitory computer readable media to store data and instructions executable by a processor. The data includes previously acquired three-dimensional image data for a patient. The data also includes two-dimensional image data, the two-dimensional image data including at least one two-dimensional image acquired by a medical imaging modality to include the patient and a multimodal marker. A predetermined portion of the multimodal marker is visible in the at least one two-dimensional image and has a known position and orientation relative to at least one tracking sensor detectable by a tracking system. The instructions are programmed to perform a method including:
[0008] estimating a three-dimensional position of a predetermined portion of the multimodal marker relative to a coordinate system of the medical imaging modality based on a corresponding position of the predetermined portion in each of the at least one two-dimensional image; and
[0009] Based on the estimated positions of the corresponding predetermined portions of the multimodal marker and a known relationship between at least one tracking sensor and the predetermined portions of the multimodal marker, an affine transformation for registering a three-dimensional coordinate system of the tracking system with a three-dimensional coordinate system of the medical imaging modality is determined.
[0010] As a further example, a method includes acquiring a first two-dimensional projection image using a medical imaging modality, the first image including a two-dimensional field of view, the two-dimensional field of view including a patient and a multimodal marker. The method also includes acquiring a second two-dimensional projection image using the medical imaging modality, the second image including the patient and the multimodal marker and along a non-coincident angle relative to the first projection image. A predetermined portion of the multimodal marker is visible in the first projection image and the second projection image and has a known position and orientation relative to at least one sensor detectable by a tracking system. The method also includes deriving a corresponding forward projection from a previous three-dimensional image corresponding to each of the first two-dimensional projection image and the second two-dimensional projection image. The method also includes determining an affine transformation from a coordinate system of the medical imaging modality to a coordinate system of the previous three-dimensional image based on registering the first and second projection images with the corresponding forward projection.
[0011] As yet another example, a system may include one or more non-transitory computer readable media to store data and instructions executable by a processor. The data includes: prior three-dimensional image data acquired for a patient. The data also includes two-dimensional image data, the two-dimensional image data including at least one two-dimensional image acquired by a medical imaging modality to include the patient and the multimodal marker. A predetermined portion of the multimodal marker is visible in the at least one two-dimensional image and has a known position and orientation relative to at least one tracking sensor detectable by a tracking system. The instructions are programmed to perform a method including:
[0012] deriving a respective forward projection from a previous three-dimensional image corresponding to each two-dimensional image; and
[0013] Based on registering each two-dimensional image with the corresponding forward projection, an affine transformation between a coordinate system of the medical imaging modality and a coordinate system of the previous three-dimensional image is determined. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 is a flow chart depicting an example of a method of registering sensors of a tracking system into a spatial coordinate system of a medical imaging modality.
[0015] Figure 2 is a flow chart for registration from a spatial coordinate system of a medical imaging modality to a spatial coordinate system of a prior three-dimensional image.
[0016] Figure 3 An example of a mark is depicted.
[0017] Figure 4A and Figure 4B An example of multimodal labeling is depicted.
[0018] Figure 5 Another example of multimodal labeling is depicted.
[0019] Figure 6 An example of a marker pad device is shown.
[0020] Fig. 7A and Figure 7B Schematic examples of medical imaging modalities are depicted.
[0021] Figure 8 An example of a system for generating an affine transformation is depicted.
[0022] Fig. 9 An example of a registration manager is depicted to control the use or correction of one or more affine transformations.
[0023] Fig. 10A and Fig. 10B Depicted is a graphical representation of information registered in a given domain according to an affine transformation. DETAILED DESCRIPTION
[0024] The present disclosure generally relates to methods and systems for spatially registering a tracking system with an interventional image.The methods disclosed herein can be implemented using reduced ionizing radiation compared to many existing methods.
[0025] The method utilizes a marking device (e.g., a multimodal marker) that includes a fiducial marker detectable by more than one modality. For example, the marking device may include a first fiducial marker to provide a pattern visible in an image generated by a medical imaging modality. The medical imaging modality may be fluoroscopy, x-ray, or other modality (such as ultrasound) that is configured to provide an image as a two-dimensional projection that includes a portion of a patient and a first fiducial marker. In some examples, the marking device may be configured to select whether the first pattern is visible in subsequent images acquired by the medical imaging modality. The marking device also includes one or more second fiducial markers (e.g., one or more sensors) that can be detected by a three-dimensional spatial tracking system. The second fiducial marker is arranged at a predetermined spatial position and orientation that is known relative to the spatial position of the first fiducial marker, and the first fiducial marker is discernible from the image acquired by the medical imaging modality.
[0026] As a further example, an imaging modality can be used to acquire two-dimensional image projections and store the images in a memory (e.g., as a DICOM image file). Each image includes a field of view that includes a patient and a marker pattern corresponding to a first fiducial marker of a marking device. Thus, during acquisition, the marker is configured (e.g., radiopaque) so that the marker pattern is visible in the acquired image. Each image is processed to locate and identify a predetermined portion of the pattern in each respective image. The identified portion (e.g., a point or region) of the marker pattern is converted into a corresponding three-dimensional position in a three-dimensional spatial coordinate system of the medical imaging modality. Based on the spatial relationship between each of the first and second fiducial markers of the marking device, an affine transformation is calculated that maps the sensor position to the coordinate system of the medical imaging modality (or maps the marker pattern position in the imaging modality coordinate system to the coordinate system of the tracking system). As mentioned, based on the geometry of the pattern of the first fiducial marker implemented on the tracking device and the sensor of the second fiducial marker, the spatial relationship is already known (and stored in the memory). Thus, the transformation enables the systems and methods to register the position and orientation information provided by the tracking system into the coordinate system of the medical imaging modality (or its inverse) based on a small number (e.g., two or three) of projection images, which significantly reduces ionizing radiation compared to many existing methods (e.g., cone beam computed tomography).
[0027] After determining this transformation, a second affine transformation for mapping from the spatial coordinate system of the medical imaging modality to another three-dimensional spatial coordinate system (such as corresponding to a preoperative three-dimensional image scan) can be determined. For example, the preoperative three-dimensional image scan can be a high-resolution imaging technique, such as computed tomography or magnetic resonance imaging, which can be performed hours, days, or even weeks before the procedure. The second affine transformation can be determined by registering the first and second (or more) projection images acquired using the above-mentioned medical imaging modality with corresponding projections derived from the preoperative three-dimensional image scan (e.g., along the same angle).
[0028] Figure 1 1 is a flow chart depicting an example of a method 100 for registering a coordinate system of a tracking system into a coordinate system of a medical imaging modality (e.g., a fluoroscopic or x-ray system). One or more sensors are integrated into a multimodal marker device that includes an arrangement of fiducial markers having a known spatial relationship relative to each other. The fiducial markers include a set of markers (e.g., sensors) detectable by the tracking system and another set of markers (e.g., radiopaque markers) visible in an image acquired by the imaging modality. The method will therefore be described with respect to the tracking system and the medical imaging modality.
[0029] As an example, a medical imaging modality is configured to provide image data (e.g., a DICOM image file) representing a two-dimensional image projection. For example, the medical imaging modality may be an X-ray system, such as a flat-panel X-ray system, a bi-plane X-ray system, a C-arm fluoroscopic system, an ultrasound imaging system, etc.
[0030] As a further example, the marking device includes one or more sensors configured to indicate a three-dimensional position in a coordinate system of the tracking system. For example, the tracking system is an electromagnetic tracking system that generates an electromagnetic field. Each sensor provides a sensor signal based on the electromagnetic field, which is converted into position and orientation information for each corresponding sensor. An example electromagnetic field tracking system is commercially available from Northern Digital, Inc., Ontario, Canada. The tracking system can provide tracking data for each sensor at an output sampling rate (e.g., sixty samples per second) sufficient to achieve substantially real-time determination of the sensor position (e.g., providing a vector describing the sensor position and orientation). Therefore, the tracking system can process each frame of the tracking data so that the tracking data can also represent real-time tracking data acquired by the tracking system, which can be aligned to the coordinate system of the imaging system, as disclosed herein. In some examples, each sensor can be detected by the tracking system so that the sensor can be tracked in five or six degrees of freedom. Other types of sensors and tracking systems can be used in other examples.
[0031] At 102, method 100 includes acquiring a first two-dimensional projection image using a medical imaging modality. Thus, the first image includes a two-dimensional image projection for a field of view that includes a patient (a region of interest of the patient's body) and a multimodal marker. As disclosed herein, the multimodal marker includes a sensor for determining a three-dimensional position in a tracking system coordinate space and a second marker (e.g., a fiducial pattern) detectable by the medical imaging modality. The sensor and the second marker may be co-located or have other fixed relative positions that are known a priori relative to each other.
[0032] At 104, the method includes acquiring a second two-dimensional projection image using a medical imaging modality. The second image, which also includes the patient and the multimodal marker, is taken at an angle that does not coincide with the angle at which the first image was acquired (e.g., less than or equal to about 90 degrees). There may be multiple total images; although the number is kept low (such as less than 10 (e.g., 5 or 2 images)) to reduce exposure to ionizing radiation compared to other modalities (such as CBCT). As an example, the imaging modality may be configured to acquire (at 102 and 104) images to include a right anterior oblique projection, a left anterior oblique projection, and other projections (e.g., anterior-posterior projections). In each image, a predetermined portion of a radiopaque fiducial marker of the multimodal marker device is visible. There may be one or more radiopaque fiducial markers on the multimodal marker device, each having a known position and orientation relative to one or more corresponding tracking sensors detectable by a tracking system. As used herein, radiopaque means that ionizing electromagnetic radiation cannot pass through sufficiently to make objects visible in a corresponding image obtained by an imaging modality (e.g., a two-dimensional (2D) medical imaging modality 456). Thus, the radiopaque object can be a radio-dense material relative to the imaging modality. In an example where two images are acquired (e.g., anterior-posterior projection and lateral projection), the multi-modal marking device can include three spaced-apart radiopaque markers (e.g., spheres), and the markers are arranged so as not to be co-linear in each image. In another example, a single marker can be used; however, the medical imaging modality will require acquisition of at least three images along different relative projection angles.
[0033] At 106, the three-dimensional position of the respective predetermined portions of the multimodal marker relative to the coordinate system of the medical imaging modality is estimated based on the position of these predetermined portions in each of the respective images. At 106, the position of each predetermined portion of the multimodal marker is also determined in the tracking coordinate system, such as to provide a set of points in the tracking system space. Thus, the same portion of one or more markers is identified in both the intraoperative imaging coordinate system and the tracking system coordinate system (e.g., coordinates in the respective space domains). The number of points will vary depending on the structure and number of marking devices used.
[0034] As an example, the fiducial marker(s) represented in the images acquired from the imaging modalities at 102 and 104 may include a radiopaque material in the form of a rectangular marker boundary having corresponding corners, the edges of which meet at the corners. For example, the radiopaque material is provided on the marking device in the form of a representation of an ArUco marker (see, for example, the open source computer vision library: http: / / opencv.org). In this way, an established image processing algorithm (e.g., the detectMarkers() function of the OpenCV library) may be implemented to detect and identify the corresponding corners from the image projection (the images acquired at 102 and 104) including the representation of the ArUco-type fiducial marker. Other relevant parameters may also be determined by the image processing algorithm. Figure 3 , Figure 4A , Fig. 7A and Figure 7B An example of such an ArUco labeling is shown in . Fig. 7A and Figure 7B And the corresponding description shows an example of how to locate the corresponding corners of such a marker in the three-dimensional coordinates of the intraoperative medical imaging modality to enable alignment between the intraoperative space coordinate system of the medical imaging modality and the preoperative space coordinate system (of the preoperative image space).
[0035] As a second example, which may be an alternative or supplement to an ArUco type marker, the radiopaque material may be in the form of a three-dimensional object on or embedded in the marking device, such as a sphere, cone, etc. A sphere is an example of an object that exhibits the same given shape regardless of the viewing angle of the projected 2D image. In this example, each sphere has a known spatial position relative to the coordinates and orientation of the tracking sensor, which are also fixed relative to the marking device. Therefore, the point of each sphere (or other 3D object) system in the tracking system coordinates can be calculated at 106 by multiplying by a transformation for the sensor, where the transformation defines a fixed, known offset relative to each corresponding sphere. In an example with three marking devices, each marking device contains three spheres, and the estimate at 106 determines three sets of three points in the tracking system coordinate system. Similarly, the coordinate position of each sphere is determined for each (at least two) 2D image acquired (at 102 and 104). For example, a sphere is positioned for each tracking sensor (e.g., using the closest point method disclosed herein), and the sphere positions (e.g., the center of each sphere) are stored in a memory linked to the corresponding sensor. This results in the same number of points (for each sphere) as determined in the tracking coordinate system.
[0036] At 108, an affine transformation for mapping the coordinate system of the tracking system to the coordinate system of the medical imaging modality (also referred to herein as the intraoperative imaging modality) is determined. For example, the transformation is determined by applying a co-registration method to align the set of points provided at 106 for each of the tracking coordinate system and the intraoperative coordinate system. As disclosed herein (see, e.g., Figure 8 ), some examples of co-registration methods include error minimization functions (e.g., single value decomposition), basis transformation functions, and other functions that can be used to determine the transformation at 108.
[0037] In some examples, the medical imaging modality includes a C-arm having a rotation center, so the origin of the coordinate system of the medical imaging modality can be defined as the rotation center of the C-arm. The medical imaging system can be modeled as a pinhole camera with corresponding spatial parameters, which are used to define the spatial coordinate system of the medical imaging modality (e.g., C-arm). For example, the spatial parameters of the C-arm (e.g., the rotation center, the rotation angle of the C-arm, the radius of the C-arm) can be manually measured, provided by the manufacturer and / or derived from the fields of the DICOM image file. The transformation determined at 108 can be based on the estimated position of the predetermined portion of the (one or more) markers determined at 106, and based on the known fixed spatial relationship between the predetermined portion of the (one or more) markers and the (one or more) tracking sensors of the marking device. As an example, the fixed relationship between the predetermined portion of the (one or more) markers and the sensor can be determined during manufacturing and printed on the marker. As another example, the relationship can be measured (e.g., manually) and input into a computer programmed to determine the transformation at 108 (e.g., via a user interface). In other examples, the coordinates of each point may be automatically determined as relative coordinates in the image data based on the pixel location of each marker (eg, at the centroid or center).
[0038] Figure 2 1 is a flow chart of a method 150 for registration from a spatial coordinate system of a medical imaging modality to a spatial coordinate system of a prior three-dimensional image, such as stored in a memory (e.g., as a DICOM or other image file). For example, a prior three-dimensional image may be acquired for a given patient preoperatively via a three-dimensional medical imaging modality. As an example, the preoperative image data may correspond to a preoperative arterial CT scan of a region of interest of the patient, such as may be obtained weeks or months prior to the corresponding surgery. Other imaging modalities may be used to provide three-dimensional image data (such as MRI, ultrasonography, positron emission tomography, etc.). Such scans are a common part of preoperative planning in the surgical workflow to help determine prosthesis size and plan surgical or other interventions.
[0039] In some examples, one or more anatomical structures captured in the preoperative image data can be converted into a corresponding three-dimensional model in the preoperative image coordinate system. As an example, the model is an implicit model that mathematically describes a tubular anatomical structure (e.g., a patient's blood vessel), such as including the centerline and surface of the tubular structure. The implicit model can include a small set of parameters, such as corresponding to a lofted b-spline (basis spline) function of an elongated anatomical structure. As an example, the anatomical model generator can be programmed to calculate implicit model data according to the disclosure of U.S. Patent Publication No. 2011 / 0026793, entitled "Automated Centerline Extraction Method and Generation of Corresponding Analytical Expression and Use Thereof", which is incorporated herein by reference. Another example of generating an implicit model for a tubular anatomical structure is disclosed in "Analytical centerline extract and surface fit using CTscans for aortic aneurysm repair", Goel, Vikash R, Master's Thesis, Cornell University (2005), which is incorporated herein by reference. Other types of geometric representations may also be used to provide an implicit model. For example, parameters representing a lofted ellipse or triangular mesh may be generated to provide anatomical model data representing the anatomical structure of interest of the patient in a three-dimensional coordinate system. The generated three-dimensional mesh may be stored in a memory in addition to or in lieu of a three-dimensional image acquired by a preoperative imaging modality.
[0040] At 152, the method includes deriving a corresponding forward projection from a previous three-dimensional image volume based on the acquired images (e.g., the first and second images acquired at 102 and 104). For example, each acquired 2D image is registered with a corresponding forward 3D projection from a similar angle. As an example, two-dimensional LAO and RAO images from a fluoroscopy system are registered by aligning such images (e.g., by translation and rotation) with corresponding LAO and RAO projections derived from a CT image volume. Registration of projection angles can be achieved by manual alignment and / or automation.
[0041] In an example where a three-dimensional mesh has been generated to model anatomical structures (e.g., blood vessels) in a preoperative three-dimensional image, the mesh can be used to align with respect to an intraoperative image (e.g., as obtained at 102 and 104). A 3D projection matrix can be applied to the mesh generated from the preoperative image. For example, if the appropriate parameters for modeling the intraoperative imaging system as a pinhole camera are known, a "perspective" projection matrix is used. In other examples, a "parallel" projection matrix is used. In an example where the marking device includes a radiopaque sphere, the application of the parallel projection matrix should be sufficiently accurate because the spheres on the marking device are close together relative to the focal length of the intraoperative imaging system. If the C-arm angle for each intraoperative image is known, then one 3D projection of the mesh is performed to match the angle of each intraoperative image. If the angle of the C-arm is unknown, then multiple 3D projections can be generated along different angles, and the "best fit" match between the corresponding 3D projection and the corresponding two-dimensional image can be selected manually or automatically. The final result will be a pair of images, where each image pair includes a 2D intraoperative image and a 2D projection derived from the three-dimensional mesh.
[0042] At 154, the method includes determining an affine transformation from a coordinate system of the medical imaging modality to a coordinate system of the prior three-dimensional image based on registering the first and second projection images (e.g., acquired at 102 and 104) with the respective forward projections determined at 152. The transformation generated at 154 may be stored in a memory.
[0043] As an example, a co-registration method can be implemented (e.g., as instructions executable by a processor) to determine a transformation at 154 for mapping spatial data from an intraoperative 2D image space to a preoperative 3D image space. For example, a transformation is determined at 154 by applying a co-registration method to align a set of points identified for each of the 2D medical image and the previous 3D image coordinate system. In each of the 3D projection images (e.g., derived at 152) and the images acquired by the 2D medical imaging modality (at 102 and 104), one or more sets of common points are identified. In an example, these points can be anatomical landmarks or other identifiable reference points, such as bony landmarks on the spine, calcifications visible in both image set images, or points on a blood vessel (such as when contrast agents are used in both intraoperative and preoperative images). Because the preoperative images are in three-dimensional space, the user can use 3 orthogonal views (axial, coronal, and sagittal) to identify points to directly measure x, y, and z positions in the preoperative 3D coordinate system. Points can be identified manually in each image, or automatic methods of feature extraction can be used.
[0044] Similar to the determination at 108, examples of co-registration methods that may be used to determine the transforms at 154 include error minimization functions (e.g., single value decomposition), basis transformation functions, and other functions. In an example, the same function may be called to generate each of the corresponding transforms to provide more efficient storage and computation.
[0045] For example, each transformation generated at 108 and 154 can be stored as a separate transformation or aggregated together to enable registration of a sensor in the tracking system coordinate system to be represented and visualized in the 3D coordinate system of the image volume. In examples where the transformations (determined at 108 and 154) are kept separate, it provides the ability to change one of the transformations without affecting the other. For example, if the second step (e.g., fluoroscopy to preoperative CT) is changed from manual registration to automatic registration, the method remains the same and does not have to be modified. Figure 1 Transformation generation method 100. In addition, if Figure 1 If the transform generation method of one transform is changed, then the other transform generation method 150 does not have to be modified. In examples where these transforms remain separate, when rendering an object based on tracking sensor data, an output visualization can be generated by multiplying the two transforms together. The details of the multiplication (e.g., order and whether to multiply by transform or inverse transform) depend on whether the view is rendered in the coordinate space of the tracking system space, the imaging modality space, or the previous three-dimensional (e.g., CT) image space.
[0046] As a further example when rendering output visualizations in preoperative CT space, the model of the skeleton and vasculature (e.g., generated in preoperative CT image space) may be rendered without transforms, and two transforms will be applied to anything tracked in EM space (catheters, guidewires, etc.). In an example, when rendering in tracking system space, the inverse of the two transforms will be applied to the model of the skeleton and vasculature (in preoperative CT image space), and no transforms will be applied to anything tracked in tracking system space (e.g., objects with one or more tracking sensors, such as catheters, guidewires, etc.). Additionally, in an example, when rendering in imaging modality space (e.g., fluoroscopy, etc.), the inverse of the imaging modality to preoperative CT transform (e.g., determined at 154) will be applied to the model of the skeleton and vasculature, and the tracking system to imaging modality transform (e.g., the transform determined at 108) will be applied to anything tracked in EM space by a tracking sensor (e.g., catheters, guidewires, etc.).
[0047] Figure 3An example of a fiducial marker 200 is depicted. As shown in this example, the marker includes black and white (e.g., binarized), and includes a thick black rectangular (e.g., square) border 202 along each side of its entire peripheral edge (e.g., having a thickness t, such as one or more pixels thick). The interior of the marker 200 includes symbols 204 and 206, which can be used to define an orientation and / or other identifying features of the marker, such as according to the AcUco library.
[0048] Figure 4A and Figure 4B An example of a multimodal marker device 250 is depicted. The multimodal marker may be fixed relative to the patient (eg, attached to the patient's body) during acquisition of first and second images using a 2D imaging modality (eg, fluoroscopy or X-ray). Figure 4A A side surface 252 of the marking device 250 is shown, which includes a fiducial mark (e.g., Figure 3 The reference mark 254 is provided to provide contrast between the white border of the reference mark and a thick black border 258 (eg, extending between the dotted line and the white border 256). Symbols 260 and 262 are on the reference mark spaced apart from the black border 258.
[0049] like Figure 4B As shown in , as seen from the other side of the surface 268 showing the marking device 250, one or more tracking sensors (e.g., electromagnetic sensors) 270 are attached to the marking device 250 at a known position and orientation relative to the corner 264 of the fiducial marker 254. In one example, the one or more sensors 270 can sense multiple degrees of freedom (DOF) in space respectively. For example, the one or more sensors 270 can be configured to sense six (6) DOF, such as disclosed herein. In one example, an electromagnetic tracking system can be used to locate the sensors 270. The tracking system allows the position and orientation of each sensor 270 to be determined based on the sensor signal provided to the tracking system from the sensor in response to the electromagnetic field. Other types of tracking systems configured to track the position and orientation of each sensor in three-dimensional space can be used in other examples.
[0050] In another example, the multimodal marker can include a cavity between its side surfaces 252 and 268 that is configured to hold a volume of material (e.g., a radiopaque material, such as a radiopaque contrast agent) to render a representative black portion of the marker visible in an image acquired by a medical imaging modality. The cavity extends along and defines a rectangular thick border of the marker to define corresponding corners where its edges meet, which corners can be located in the acquired image, as disclosed herein.
[0051] For example, the marking device 250 includes at least one port (e.g., in one of the surfaces 252, 268 or the edges of the device) to access the chamber, such as to add and remove radiopaque material relative to the chamber. Each chamber can therefore be filled with contrast agent to create a recognizable pattern in a two-dimensional projection image acquired by a medical imaging modality. Thus, when the material is in the chamber, the black portion of the fiducial marker is visible in the medical image, and when the radiopaque is removed, the black portion appears to be substantially the same as the white portion. For example, after registration is completed (e.g., after images are acquired at least at 102 and 104 for registration), the contrast agent material will be drained and possibly flushed with a clear saline solution so that it does not interfere with subsequent imaging during the procedure. In other examples, the radiopaque material remains unchanged throughout the procedure.
[0052] Other examples of multi-mode marking apparatus 300 that do not include AruCo type markings are Figure 5 and Figure 6 In the Figure 5 In an example of the invention, a combined marking device 300 includes a plurality of radiopaque reference objects 302 disposed within a substrate 304. Each radiopaque reference object 302 has a predetermined geometric shape and is arranged relative to each other in a predetermined geometric relationship. For example, the radiopaque objects 302 can be implemented as spheres or other shapes, such as having a predetermined angular orientation and spatial arrangement (e.g., configured as scalene right triangles). Therefore, each radiopaque object 302 can be identified in a corresponding 2D imaging modality (e.g., obtained within a procedure via fluoroscopy, biplane X-ray, etc.).
[0053] As mentioned, the type of material used for the respective object 302 can vary depending on the imaging modality used (e.g., for acquiring 2D images, such as at 102 and 104). The marking device 300 is a multi-modal marker and therefore also includes one or more sensors 306 that can be detected by the tracking system. The size and position of each sensor 306 can be designed to have a predetermined spatial relationship (e.g., distance and angle) relative to the geometry of the respective radiopaque object 302. For example, the sensor 306 can include a sensor coil located at the origin of a pair of axes that can be calculated relative to the geometric relationship of the object 302. In addition, the sensor itself 306 can extend along an axis 308 or parallel to an axis defined by the respective radiopaque object 302.
[0054] As a further example, the marking device 300 defines a coordinate system including an X-axis and a Z-axis located in a virtual plane extending through each object 302 arranged as a triangle (of a page), with the corresponding Y-axis extending perpendicular to the virtual plane (e.g., a page of a display graphic). In the example, the central axis 308 of the sensor 306 extends along the Z-axis of the coordinate system. The geometric center of the body of the sensor 306 can thus define the origin of the X, Y, and Z axes. As mentioned, the sensor 306 can be configured as an elongated coil extending axially along the length of the Z-axis and can be detected by the tracking system. For example, the sensor 306 is implemented as a coil of conductive material within the marking device 300, and the center of the sensor coil is located at the origin of the corresponding coordinate system. The sensor 306 provides a sensor signal (e.g., an induced current signal) transmitted to the tracking system in response to an electromagnetic field generated by a field generator of the tracking system.
[0055] Figure 6 An example of a marking pad device 320 is shown, which can help protect the patient's skin from the hard surface of the combined marking. One or more multi-mode marking devices 300 ( Figure 5 ) can be implemented within the pad device 320 to achieve co-registration between the domain of the tracking system and the domain of a medical imaging modality, such as those disclosed herein (e.g., fluoroscopy, etc.). For example, the pad 320 can include a gel or other soft, flexible material to provide cushioning around each combined marker, and the pad can be attached to the patient's skin. In other examples, the pad 320 is placed adjacent to the patient, such as on a bed next to or below the patient.
[0056] exist Figure 6 In the example of , the pad device 320 includes three multi-mode markers 300, which are distributed in a spaced arrangement relative to each other. The pad device 320 can be configured to hold each combined marker in a substantially fixed spatial relationship while allowing flexibility to accommodate patient movement. Each marker device 300 also includes a corresponding connection 310 that can be coupled to a tracking system. For example, the tracking system can be implemented as an electromagnetic tracking system, such as disclosed herein, and each connection 115 therefore includes an electrical connector to provide an electrical signal to the tracking system, representing an induced current in response to an electromagnetic field generated by a transmitter of the tracking system and detected by a corresponding sensing coil. In other examples, the connection can be wireless, and the sensor can communicate via RF or other wireless technology. The tracking system can convert the sensor signal into corresponding tracking system data, which can be analyzed as disclosed herein. For example, the tracking data can include the position and orientation of each marker device 104 at a point in the three-dimensional coordinate space of the tracking system (also referred to herein as the tracking system space).
[0057] Fig. 7A and 7BA schematic example of a medical imaging modality 350 that can be used to acquire two-dimensional projection images from different viewing angles is depicted. For example, the medical imaging modality 350 is configured to acquire images using ionizing radiation (e.g., a fluoroscopic examination system, a portable X-ray system, etc.). In this example, the imaging modality 350 includes an X-ray source 352 configured to provide ionizing radiation (e.g., X-rays) and a detector 354 attached and maintained at a desired spaced-apart position by a movable C-arm 356. The patient can be positioned between the source 352 and the detector 354 along with one or more marking devices 360. Thus, the acquired image can vary depending on the viewing angle (e.g., axis) of the source and the detector, as adjusted by adjusting the position of the C-arm. For registration purposes, the viewing angle includes the patient's region of interest and the marker 360.
[0058] For example, the registration is performed by modeling the X-ray source 352 as an ideal pinhole camera (i.e., assuming no distortion), where each pixel in the resulting image is formed by projecting a 3D point into the image plane using a perspective transformation such as:
[0059]
[0060] in:
[0061] X, Y, and Z are the coordinates of the 3D point in a common coordinate system;
[0062] u and v are the coordinates of the projected point in the camera image, in pixels;
[0063] fx and fy are focal lengths in pixels;
[0064] cx and cy are the image centers in pixels; and
[0065] r## and t# define the position and orientation of the X-ray detector in the common coordinate system, respectively.
[0066] To create vector v1 or v2, the corners of the ArUco marker 360 are located in the image as u and v. The remaining values of the equation can be filled in based on the known spatial positions and the equation is solved for X and Y at the focal length (e.g., the distance between the detector and the corresponding corner position). The vector is then calculated by subtracting the detector position (p1 or p2) from this new position. For example, points p1 and p2 are defined based on the rotation angle of the C-arm of the two projections, where the center of rotation is defined as the origin and the distance from the center is based on the C-arm radius. The focal length of the camera is calculated from the pixel size stored in the associated DICOM file.
[0067] The 3D position of the corner of the ArUco marker can then be calculated by finding the intersection (or nearest approach) of the two vectors v1 and v2. The position and orientation of the ArUco marker in the common coordinate system is calculated by repeating this process for all 4 corner positions identified for the fiducial marker in each corresponding image. For example, the intersection (or nearest approach) of two vectors can be calculated based on a vector intersection function that implements a nearest point function. As an example, the following pseudo code implements a nearest point function to determine the nearest point between corresponding vectors:
[0068]
[0069]
[0070] exist Fig. 7A and 7B In the example of , the marker 360 is shown as an AruCo type radiopaque marker (e.g., corresponding to the marking devices 200, 250). In other examples, the medical imaging modality can be used with different types of marking devices, as disclosed herein, namely, the multi-modal marking device 300 and the combined marking system 320. Therefore, based on the 2D image data, 3D preoperative image data, and real-time tracking data acquired by the medical imaging modality 350, corresponding transformations (e.g., corresponding transformation matrices) such as those determined at 108 and 154 can be used to enable rendering one or more visualizations.
[0071] Figure 8 An example of a system 450 for generating an affine transformation is depicted. In this example, the affine transformation is shown as transformation matrices 452 and 453 for registering tracking data and image data, as disclosed herein. The system 450 is described in the context of data and instructions, and a processor can access the data and execute the instructions to perform the functions disclosed herein. It should be understood that not all functions may be required to implement the system. For example, each of the different transformation matrices can be generated separately, which provides advantages when the imaging modality is changed or replaced in another embodiment because the entire system does not need to be modified.
[0072] exist Figure 8 In the example of , the system 450 is configured to generate a first transformation matrix (T1) 452. The transformation matrix T1 can be configured to transform from a tracking system coordinate system of a tracking system 454 to a coordinate system of a medical imaging modality 456 (e.g., a 2D imaging system such as fluoroscopy or x-ray), and / or from a coordinate system of a medical imaging modality to a tracking coordinate system. The tracking system 454 is configured to provide tracking data 458 to represent the position and orientation of one or more sensors 466 positioned within a patient's body 460.
[0073] Combination marking system 462 (e.g., including Figure 4A , Figure 4B , Figure 5 or Figure 6 One or more multimodal markers) may be attached to or placed near the patient's body 460. Figure 8 In the example of , the combined marking system 462 may include one or more tracking sensors 464 that provide corresponding sensor signals representing the position of the combined marking within the coordinate system of the tracking system 454 to the tracking system 454. In the example, one or more object sensors 466 may be fixed relative to an object movable within the patient's body 460 for identifying the position of such sensors in the coordinate system of the tracking system. Thus, each such object sensor 466 may also provide a signal to the tracking system 454, based on which the tracking system may calculate corresponding tracking data representing the position and orientation of such sensor in the coordinate system of the tracking system. As mentioned, the tracking data 458 therefore represents the position and orientation of each corresponding object tracking sensor 466 of the multi-modal marking system 462 and the marking tracking sensor 464.
[0074] In some examples, the sensor(s) 466 and corresponding tracking data 458 may be ignored (or omitted), such as for purposes of generating the transformation matrix 452. In other examples, the sensor 466 may be placed at a known location relative to the patient's body 460 (e.g., a known anatomical landmark inside or outside the patient's body) to provide additional data points in both the tracking system spatial domain (e.g., provided by the tracking data 458) and the spatial domain of the imaging modality 456 (e.g., provided by the 2D image data 472 at the known location), which may be used to facilitate generating the transformation matrix T1 452.
[0075] For example, the tracking system 454 may include a transmitter (e.g., an electromagnetic field generator) that provides a non-ionizing field, as shown at 455, which is detected by each sensor 464 and 466 to provide a corresponding sensor signal to the tracking system. An example tracking system 454 is an AURORA spatial measurement system commercially available from Northern Digital, Inc. of Ontario, Canada. The tracking system 454 may provide tracking data 458 for each sensor at an output sampling rate (e.g., sixty samples per second) sufficient to enable substantially real-time determination of the sensor position (e.g., providing a vector describing the sensor position and orientation). The tracking processing subsystem may thus process each frame of the tracking data so that the tracking data may also represent real-time tracking data acquired by the tracking system to facilitate generation of a graphical representation in a given domain, wherein the tracking system may be registered to another coordinate system by applying one or more of the generated transformations 452 and / or 453, as disclosed herein.
[0076] The tracking system 454 can provide an output sampling rate to the tracking data 458 to enable calculation of real-time positioning and visualization of the object to which the sensor is attached and the combined marking system. Since the marking system 462 is attached to the patient's body 460, the tracking system 454 calculates the tracking data 458 to accommodate the movement of the patient's body 460 in the coordinate system of the tracking system 454.
[0077] The sensor transformation 470 is configured to convert the tracking data 458 into the position of the radiopaque object implemented on each corresponding marking device, such as disclosed herein. Each position is a 3D spatial coordinate in the tracking system coordinate space, and can remain fixed if the marking device does not move in the tracking space, or can change over time if the marking device moves in the tracking space. For example, in the tracking coordinate system, each radiopaque marker of a given marking device is at a fixed, known offset (e.g., a 3D vector) from the position of the tracking sensor 464, where the tracking sensor 464 is part of a given marking device of the marking system 462. As mentioned above, the marking system can include multiple multi-mode marking devices, such as AruCo type (e.g., device 250), or other marking configurations as disclosed herein (e.g., device 300).
[0078] The sensor transform 470 is thus configured to calculate a point in the tracking system space (e.g., a 3D coordinate of a marker position) based on the tracking data 458 and a known offset of each tracking sensor relative to a predetermined marker position. For an AruCo type multi-mode marker device, the marker position may be a set of four points (e.g., emPoint_1, emPoint_2, emPoint_3, emPoint_4) at the corners of the marker, such as disclosed herein. For example, a set of marker positions for an AruCo type marker device having a sensor for providing tracking data may be calculated for a given marker device by multiplying a sensor transform (TS) with the corresponding offset, the sensor transform including the tracking sensor 3D coordinates, as follows:
[0079] emPoint_1=mult(Ts,offset_1),
[0080] emPoint_2=mult(Ts,offset_2),
[0081] emPoint_3 = mult(Ts, offset_3), and
[0082] emPoint_4=mult(Ts,offset_4)
[0083] For the example of a marking device including an arrangement of spherical radiopaque markers (e.g., for the marking device 300), there are three spherical markers distributed at known offsets around each tracking sensor. Thus, the sensor transformation will generate three points for each marking device in the marking system 462. For example, the transformation 470 can determine the marker position at a point (e.g., emPoint_1, emPoint_2, emPoint_3) located at the center of each spherical marker based on multiplying the corresponding transformation with the known offset (e.g., 3D offset vector) between the tracking sensor position (e.g., 3D point) and the corresponding radiopaque object, such as the following:
[0084] emPoint_1=mult(Ts,offset_1),
[0085] emPoint_2 = mult(Ts, offset_2), and
[0086] emPoint_3=mult(Ts,offset_3).
[0087] Other deterministic positions with fixed offsets associated with radiopaque markers may be used in other examples.In some examples, the points may be arranged in a set of points for each marking device or as a single set containing all points.
[0088] The medical imaging modality 456 is configured to generate 2D image data 472 that includes at least two images (e.g., radiographs) representing objects within a field of view 475 of the imaging modality 456. For example, the imaging modality may include a fluoroscopic scanner (e.g., Fig. 7A and Figure 7B The system 472 is configured to acquire 2D image data for a small number (e.g., at least two, three, or four) of 2D projection images acquired at different viewing angles relative to the patient body 460. Each image in the image data 472 can be acquired to include a 2D projection of the radiopaque markers in each marking device of the marking system 462 and an area of the patient body 460 within the field of view 475. In some examples, the area of the patient body can be an area of interest (such as a portion of a surgical procedure) in which the object sensor 466 will move.
[0089] The marker recognition function 474 can be configured to locate each radiopaque marker (e.g., AruCo marker and / or other object marker) in each image provided in the image data 472. Since the radiopaque marker is opaque relative to the ionizing radiation emitted by the imaging mode 456, the radiopaque marker will be visible in the image. For the example of a combination mark including an AruCo type marker, the AruCo detection function can be called by the marker recognition function 474 to locate each corresponding marker. For the example combination mark including radiopaque objects other than the AruCo type marker, the periphery of each such marker can therefore be located by image thresholding and other image processing techniques applied to image pixel values. The marker recognition function 474 can be fully automated and / or user-interactive in response to user input of the recognition marker. The identified marker (e.g., pixel position in the corresponding image) can be stored in a memory for further processing.
[0090] The marker point generator 476 is programmed to generate spatial coordinates for each marker identified in the (e.g., two or more) images provided by the image data 472. For the example of a combination marker including a radiopaque AruCo type marker, spatial coordinates can be generated for each corner of each marker, i.e., the coordinates of a set of four points around each tracking sensor. For a spherical-shaped radiopaque marker, the spatial coordinates of each marker are provided as the 2D coordinates of the center of a circular projection (e.g., the periphery identified by the marker identification function 474) in each 2D image for the viewing angle provided by the field of view 475 relative to the marking system 462. In the example of three spherical markers surrounding each tracking sensor of a given marking device, the marker point generator is programmed to provide the coordinates of a set of three points of the given marking device. Regardless of the type and configuration of the radiopaque marker, the marker point generator is, for example, programmed to perform a closest point function, such as disclosed herein, to locate a set of points around each corresponding tracking sensor of the marking device. In this way, each set of points can be linked together and associated with a corresponding one of the tracking sensors to facilitate the generation of the first transformation matrix 452.
[0091] The first transformation calculator 478 is programmed to calculate the first transformation matrix 452 based on the points provided by the marker point generator 476 and the sensor transformation function 470. For example, the transformation calculator 478 should be used to align a set of points that have been measured in a spatial coordinate system. Examples of such co-registration algorithms for co-registering points in corresponding domains (e.g., a tracking system coordinate system and a medical imaging coordinate system) can include an error minimization function or a basis transformation function.
[0092] As an example, the transformation calculator 478 is programmed to implement an error minimization function. Given an ordered set of points, the transformation calculator 478 will determine the unknown transformation T1 that minimizes the distance between the projected position and the measured position. For example, for T1, the calculator 478 is programmed to find a transformation that minimizes the distance between the points, such as the following:
[0093] sum(n=1..i,distance(mult(T1,imPoint_n),emPoint_n)^2)
[0094] in:
[0095] n represents a given one of i points (i is the number of points given a multimodal label);
[0096] imPoint_n is the spatial coordinate of point n in image space; and
[0097] emPoint_n is the spatial coordinate of point n in the tracking space.
[0098] In an example, error minimization may be solved by single value decomposition or a variety of error minimization algorithms.
[0099] As another example, transformation calculator 478 is programmed to implement a base transformation function. If the points used are arranged in a manner that allows transformation calculator 478 to generate a set of base vectors (x, y, and z unit vectors that define a coordinate space), a simpler solution than minimizing the error is possible. For example, instead of minimizing the error, transformation calculator 478 is programmed to find base vectors in two coordinate systems and apply them to common points. This is more efficient in calculation than the error minimization method mentioned above, but requires a specific point arrangement.
[0100] For example, to unambiguously define basis vectors, the required points are 3 points arranged at 90 degree angles, with enough additional information to allow the transform calculator 478 to identify which point is which (eg, the legs of the triangle created by the 3 points are different lengths). Figure 3 and the ArUco type marker of FIG. 4 and Figure 5 and Figure 6 The marking devices have a point arrangement sufficient to enable the use of this basis transformation function, but it should be noted that for Figure 5 and Figure 6 For a marking device, each set of 3 points needs to be processed separately.
[0101] In each coordinate system, the transformation calculator 478 constructs basis vectors from 3 points. For example, given point_1, point_2, and point_3 (e.g., vertices of a right triangle), two line segments are provided, one from point_2 to point_1 and the other from point_2 to point_3, which are the legs of the right triangle. These points and line segments provide the following basis vectors:
[0102] basis_z=normalize(point_1-point_2)
[0103] basis_x=normalize(point_3-point_2)
[0104] basis_y=cross(basis_x,basis_z)
[0105] From the basis vectors, the transformation calculator 478 is programmed to create a matrix (e.g., a 4*4 matrix) defining the position and orientation of point_2 as follows:
[0106] matrix=
[0107] [basis_x.x,basis_y.x,basis_z.x,point_2.x,basis_x.y,basis_y.y,basis_z.y,point_2.y,
[0108] basis_x.z,basis_y.z,basis_z.z,point_2.z,
[0109] 0,0,0,1]
[0110] Using the matrices defined in each coordinate system, the transformation calculator 478 can calculate the transformation 452 between the two coordinate systems. For example, for the transformation matrix T1:
[0111] im_Matrix is a matrix defined in terms of basis vectors in a medical imaging (eg, intraoperative) coordinate system; and
[0112] em_Matrix is a matrix defined in terms of basis vectors in the tracking coordinate system.
[0113] As can be seen above, the transformation calculator 478 can determine the transformation matrix (T1) 452 by multiplying the basis vector tracking matrix (em_Matrix) with the inverse of the basis vector imaging matrix (inv(im_Matrix)), such as the following:
[0114] T1=mult(em_Matrix,inv(im_Matrix))
[0115] The transformation matrix can be stored in a memory and used for transformation from tracking system space to medical imaging space. For example, the position of the object sensor 466 within the patient's body (as represented by the tracking data 458) can be registered into the medical imaging space by applying the transformation T1 to the position and orientation information of the tracking data.
[0116] As mentioned, the system 450 is also configured to generate a second transformation (T2) 453 for transforming between the medical imaging coordinate system and the coordinate system of the previous 3D image data 480. For example, the previous 3D image data 480 may be stored in a memory (e.g., as a DICOM image collection) and include 3D images from a preoperative scan (e.g., a CT scan) of the patient's body 460 performed at a time before the medical imaging modality 456 generated its image data 472 (e.g., intraoperatively, such as corresponding to the images acquired at 102 and 104).
[0117] Projection calculator 482 (eg, with Figure 2 The projection calculator 482 (corresponding to the function 152 of ) is programmed to generate a corresponding projection for each image (e.g., two images) provided in the 2D image data 472 according to the 3D image data 480. The projection calculator 482 implements the function of mapping points from the 3D image space onto a two-dimensional plane. For example, the projection calculator derives a forward projection aligned with the viewing angle of the image in the 2D image data 472. The registration of the projection angle of each 3D projection can be achieved and / or automated by manual alignment. In an example, the alignment can be automated, such as based on image metadata in the image data 472 describing the angle of each 2D image (argumented as included in the arrow from the 2D image data 472 to the projection calculator 482). For example, the metadata includes data specifying the projection angle, such as AP, LAO, RAO, such as can be known from the angle of the C-arm and / or provided in response to user input when the imaging modality 456 acquires the image data 472.
[0118] In some examples, as disclosed herein, the 3D image data may include a model of one or more anatomical structures, such as in the form of a 3D mesh corresponding to a vessel surface. A 3D projection matrix (e.g., a perspective or parallel projection matrix) may be applied to a mesh generated from preoperative images 480, such as disclosed herein. If the C-arm angle for each intraoperative image is known, then one 3D projection of the mesh is performed to match the angle of each intraoperative image. If the angle of the C-arm is unknown, then multiple 3D projections may be generated along different angles, and a "best fit" match between the corresponding 3D projection and the corresponding two-dimensional image may be selected manually or automatically.
[0119] Point generator 484 is programmed to generate spatial points in each of the 2D image (provided by image data 472) and the corresponding projection of the 3D image (provided by projection calculator 482). Instead of working with a marked sphere or corner, the point is selected as a feature that is visible in both the 2D image data 472 and the 3D image data 480. For example, features include structures such as bone landmarks on the spine, calcifications visible in both types of images, or points on blood vessels (in the example when contrast agents are used in both images). Other features or reference points can be used in other examples. In some examples, a common feature set can be located in an automated method (e.g., feature extraction). Additionally or alternatively, one or more such features can be selected in response to user input provided by a user interface 486, such as a graphical user interface that interacts with the corresponding images and projections provided to the point generator. For example, a user can see a common visible structure between different views and select / mark it in each view (e.g., by a mouse, keyboard, gesture, or other input). The point generator 484 thus generates points for each predetermined feature and / or user-selected feature. The point generator thus operates similarly to the marker point generator 476, except that a different set of landmarks is used. Since the image data 480 is 3D, in some examples, a user can use a set of orthogonal views (e.g., axial views, coronal views, and sagittal views) of the 3D image of the image data 480 to identify selected points (via the user interface 486) to directly measure the x, y, and z positions of the image data 480 in the 3D coordinate system. Each of these positions can be converted into two-dimensional coordinates and provided as such in the forward projection provided by the projection calculator 482. The point generator 484 is programmed to locate the same point in the 2D image data, such as by using a vector intersection function (such as the closest point function disclosed herein) applied to the 2D image.
[0120] The generated points in the corresponding images are provided to a second transform calculator 488 for use in generating a transformation matrix 453. The transform calculator 488 is programmed to calculate a transformation matrix to align the image of the second image data with the 3D image data 480 based on the common points provided by the point generator 484. For example, the transform calculator 488 constructs the transformation matrix (T2) 453 by implementing an error minimization function with respect to the set of common points, such as the singular value decomposition described with respect to the first transform calculator 478. Other error minimization functions may be used in other examples.
[0121] In some examples, the system 450 includes a transform correction function 490 that is programmed to implement manual corrections to one or both transform matrices 452 and 453 based on instructions provided via a correction user interface 492. Manual corrections can be applied even if an estimate of a T1 or T2 transform is initially provided. For example, if the image data 480 and / or 472 does not have a clearly defined set of measurement points (e.g., on the spine or other anatomical structure) to work with to perform registration, the system can define an initial estimate for the transform T2 (or in some examples, an arbitrary T2 transform (e.g., an "identification" matrix)) and allow the user to make corrections via the correction function 490 to generate the final T2 transform 453.
[0122] As a further example, see Fig. 9 The registration manager 500 is used to control the user's correction of one or both of the transformations T1 and T2, 452 and 453, respectively. The registration manager can be implemented as Figure 8 450, or implemented as a separate function. Thus, for consistency, Figure 8 The functions and data introduced in Fig. 9 For further information on such functions and data, please refer back to Figure 8 and the corresponding description.
[0123] The registration manager 500 includes a transform correction function 490 and includes first and second transform matrices 452 and 453, respectively. In this example, it is assumed that one or both of the transform matrices 452 and 453 may need correction. The need for correction can be indicated to the user by applying a transform to register two or more domains and providing the resulting visualization on the display 510. For example, the output generator 512 is configured to render the visualization in a selected domain, such as a coordinate system of a tracking system, a coordinate system of a medical imaging modality 456, or a coordinate system of previous 3D image data 480. In the example, the manager 500 includes a domain selector 514 that is programmed to select the domain for which the output visualization is being rendered based on a user input instruction received via the user interface 520. In addition, based on the selected domain, the registration manager applies one or both of the transforms T1 or T2 accordingly. As an example, the following table provides a description of which one or more transforms are applied to the image data 472, 480 or tracking data 458 of each selected domain, where the output generator 512 is rendering the output visualization for the selected domain. The registration manager 500 may also be used to control the application of corresponding transformations to provide visualization of a selected domain, such as by applying one or more transformations or inverses of such transformations, as set forth in the table.
[0124] track Medical Imaging Previous 3D To track: [Identification] inv(T1) inv(T1)inv(T2) To Medical Imaging: T1 [Identification] inv(T2) To Previous 3D: T2 T1 T2 [Identification]
[0125] As a further example, manual correction to the transform 452 or 453 may be provided by multiplying the corresponding transformation matrix T1 or T2 by a correction matrix, such as the following:
[0126] correctedT1=mult(correctionMatrix,T1) or
[0127] correctedT2=mult(correctionMatrix,T2)
[0128] In the example, the supported correction types include translation, rotation and scaling, such as can be applied in the form of a matrix as follows:
[0129] translationMatrix=
[0130] [1,0,0,translation.x,
[0131] 0,1,0,translation.y,
[0132] 0,0,1,translation.z,
[0133] 0,0,0,1]
[0134] scalingMatrix=
[0135] [scale,0,0,0,
[0136] 0,scale,0,0,
[0137] 0,0,scale,0,
[0138] 0,0,0,1]
[0139] rotationMatrix = (depends on the rotation axis)
[0140] As a further example, the user initiates the correction using a mouse down / drag / mouse up action or other action on the user interface 516. The values used in the correction matrix may be set based on the projection matrix used to display the viewport on the display 510. For example, a translation initiated from the AP view will result in the X and Y mouse movements being used to set the translation.x and translation.z values (translation.y will be 0). Thus, such translations allow the user to change the view of a single image or the alignment of multiple images.
[0141] As a further example, such as when a correction to transformation T2 is implemented, the domain registration manager 500 applies transformation T2 to the image data 472, and the output generator 512 provides a visualization of the 2D image registered in the 3D image based on transformation T2. If the landmarks are aligned correctly, as shown on the display 510, then correction may not be required. However, if the location of the landmarks in the 2D image is not aligned with their corresponding locations in the 3D image, then correction of T2 may be required. Therefore, the user can adjust the alignment of the 2D image relative to the 3D image (or its forward projection) through the user interface 516. As mentioned above, the adjustment can include translating, rotating and / or scaling in two dimensions in response to instructions entered through the user interface using an input device (e.g., a mouse or keyboard). The output generator 512 can update the visualization shown in the display to show the image registration in response to each adjustment (e.g., in real time). Once the desired alignment is visualized, the user can use the user interface 516 to apply and store the correction to transformation T2, and the updated T2 can be stored in the memory for subsequent application. Similar types of adjustments may be made with respect to the first transformation matrix 452 .
[0142] Figure 10A-10B Examples of images 600 and 630 that may be generated and visualized in a display (display 510) based on application of one or more transformations (e.g., implemented by correction function 490) before and after manual correction are shown. Fig. 10A and Fig. 10B In both, the images include a common set of anatomical features and marker features. For example, the anatomical features include vertebrae of a spine 602. The marker features of a given marker device 610 include the positions of a radiopaque marker 604 and a tracking sensor 606, which have been mapped by applying a transformation from one image space (e.g., a tracking system coordinate system) to another image space (e.g., a medical imaging coordinate system). The marker features of the marker device 610 also include the position of the radiopaque marker 608 in its original domain (e.g., a medical imaging coordinate system). The image also includes marker features of another marker device 618, including the positions of a radiopaque marker 612 and a tracking sensor 614, which have been mapped by applying a transformation from one image space (e.g., a tracking system coordinate system) to another image space (e.g., a medical imaging coordinate system) and the position of a radiopaque marker 616 in its original domain (e.g., a medical imaging coordinate system).
[0143] like Fig. 10AAs shown in , due to misregistration, the corresponding marker features and anatomical features are not aligned. As disclosed herein, the output display is interactive and includes GUI elements 622, which can be used to adjust (e.g., translate, rotate, or scale) one image relative to another image. In response to such adjustments, the correction function 490 generates one or more correction matrices, such as disclosed herein. Once the desired alignment is achieved, such as Fig. 10B As shown in visualization 630 of , the user may activate a “freeze” GUI element (button) 624 to apply the correction to the appropriate transform(s), and the updated transforms may be stored in memory for further application.
[0144] In some examples, after performing the first step on the AP image, the system should have registration in two dimensions - patient left to right and head to toe. The above process can be repeated with a side view or an oblique view to obtain a third dimension - patient front to back. In a further example, if the user is to advance a catheter or guidewire (e.g., including a tracking sensor) near the aortic bifurcation or other identifiable landmark, then a rough registration in this third dimension can be performed without a second 2D medical image.
[0145] For example, the two-dimensional image data may include a two-dimensional image acquired by a medical imaging modality (e.g., intraoperatively) to include a patient and a multimodal marker, such that a predetermined portion of the multimodal marker is visible in the 2D image and has a known position and orientation relative to at least one tracking sensor detectable by the tracking system. The three-dimensional position of the predetermined portion of the multimodal marker can be estimated based on the corresponding position of the predetermined portion of the marker in each of the one (or more) two-dimensional images relative to the coordinate system of the medical imaging modality. An affine transformation for aligning the three-dimensional coordinate system of the tracking system with the three-dimensional coordinate system of the medical imaging modality is determined based on the estimated position of the corresponding predetermined portion of the multimodal marker and the known (a priori) relationship of at least one tracking sensor (e.g., including a tracking sensor on a guidewire or catheter at a known landmark) to the predetermined portion of the multimodal marker. Therefore, the second 2D image can be omitted and the alignment can still be performed.
[0146] In view of the above structure and function description, it will be appreciated by those skilled in the art that the parts of the systems and methods disclosed herein can be implemented as methods, data processing systems or computer program products, such as non-transitory computer-readable media. Thus, these parts of the methods disclosed herein can take the form of complete hardware embodiments, complete software embodiments (e.g., in one or more non-transitory machine-readable media) or embodiments combining software and hardware. In addition, parts of the systems and methods disclosed herein can be computer program products on computer-usable storage media, with computer-readable program code on the medium. Any suitable computer-readable medium can be used, including but not limited to static and dynamic storage devices, hard disks, optical storage devices, and magnetic storage devices.
[0147] Certain embodiments are also described herein with reference to block diagrams of methods, systems, and computer program products. It will be understood that the illustrated blocks and combinations of the blocks in the illustrated blocks can be implemented by computer executable instructions. These computer executable instructions can be provided to one or more processors of a general-purpose computer, a special-purpose computer, or other programmable data processing device (or a combination of a device and a circuit) to produce a machine, so that the instructions executed by the processor implement the functions specified in one or more blocks.
[0148] These computer executable instructions may also be stored in a computer readable memory, which may direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory result in the product including instructions for implementing the functions specified in one or more flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flowchart blocks.
[0149] What is described above is an example. Of course, it is impossible to describe every conceivable combination of components or methods, but those of ordinary skill in the art will recognize that many further combinations and permutations are possible. Thus, the present invention is intended to include all such changes, modifications and variations that fall within the scope of the present application, including the appended claims. As used herein, the term "includes" means including but not limited to, and the term "includes" means including but not limited to. The term "based on" means at least partially based on. In addition, when the present disclosure or claims record "one", "one", "first" or "another" element or its equivalent, it should be interpreted as including one or more than one such element, neither requiring nor excluding two or more such elements.
Claims
1. A method comprising: acquiring a first two-dimensional projection image using a medical imaging modality, the first two-dimensional projection image comprising a two-dimensional field of view including a patient and a multi-modal marker; acquiring, using a medical imaging modality, a second two-dimensional projection image, the second two-dimensional projection image including the patient and the multimodal marker and along a non-coincident angle relative to the first two-dimensional projection image, a predetermined portion of the multimodal marker being visible in the first two-dimensional projection image and the second two-dimensional projection image and having a known position and orientation relative to at least one sensor detectable by a tracking system, wherein the at least one sensor is attached to the multimodal marker or the multimodal marker includes the at least one sensor; estimating a three-dimensional position of the predetermined portion relative to a coordinate system of the medical imaging modality based on the position of the predetermined portion of the multimodal marker in each of the respective first two-dimensional projection images and the second two-dimensional projection images; Based on the estimated positions of the corresponding predetermined portions of the multimodal marker and the known relationship between the at least one sensor and the predetermined portions of the multimodal marker, an affine transformation for registering a three-dimensional coordinate system of the tracking system with a three-dimensional coordinate system of the medical imaging modality is determined.
2. The method of claim 1, wherein the affine transformation is a first affine transformation, the method further comprising: deriving a respective forward projection from a previous three-dimensional image corresponding to each of the first two-dimensional projection image and the second two-dimensional projection image; as well as Based on registering the first two-dimensional projection image and the second two-dimensional projection image with the corresponding forward projection, a second affine transformation from the coordinate system of the medical imaging modality to the coordinate system of the previous three-dimensional image is determined.
3. The method of claim 2, wherein the three-dimensional mesh model is derived from a previous three-dimensional image to represent the patient's anatomy in the coordinate system of the previous three-dimensional image, and Therein, a corresponding forward projection from a previous three-dimensional image is determined based on the mesh model.
4. The method of claim 2 or 3, wherein determining the second affine transformation further comprises: identifying a common set of features in each of the first two-dimensional projection image and the second two-dimensional projection image and the previous three-dimensional image; determining positions of at least some of the common feature set in a coordinate system of the medical imaging modality and in a coordinate system of the previous three-dimensional image; as well as A second affine transformation is calculated as a corresponding transformation matrix configured to align the positions determined for the common feature set in one of the coordinate system of the medical imaging modality or the coordinate system of the previous three-dimensional image.
5. The method of any one of claims 2, 3 or 4, wherein the first affine transformation is calculated by one of an error minimization function or a basis transformation function, and The second affine transformation is calculated by minimizing the error function.
6. The method of claim 1 or 2, wherein the multimodal tag comprises a chamber and at least one port for accessing the chamber, the chamber having a predetermined spatial position and orientation relative to each sensor, wherein the multimodal marker is fixed relative to the patient during acquisition of the first two-dimensional projection image and the second two-dimensional projection image, and in, During acquisition of the first two-dimensional projection image and the second two-dimensional projection image, the multimodal marking includes a contrast agent within the chamber such that the chamber is visible in each of the first two-dimensional projection image and the second two-dimensional projection image, the method further comprising: adding a volume of contrast media to the cavity via the port prior to acquiring the first two-dimensional projection image and the second two-dimensional projection image to make the cavity visible in an image acquired by the medical imaging modality; and The contrast agent is removed from the cavity after acquiring the first two-dimensional projection image and the second two-dimensional projection image such that the cavity is not visible in subsequent images acquired by the medical imaging modality.
7. The method of any of the preceding claims, wherein the medical imaging modality is configured to acquire images using ionizing radiation.
8. The method of claim 7, wherein the medical imaging modality comprises a C-arm having a center of rotation, the method further comprising defining an origin of a coordinate system of the medical imaging modality relative to the center of rotation of the C-arm.
9. The method of any of the preceding claims, wherein the first two-dimensional projection image and the second two-dimensional projection image are selected from the group consisting of: a right anterior oblique projection, a left anterior oblique projection, and an anterior-posterior projection.
10. The method of any one of the preceding claims, wherein the non-coincidence angle is less than or equal to 90 degrees.
11. The method of any one of claims 2, 3, 4, 5, 6, 7, 8, 9 or 10, wherein the previous three-dimensional image comprises one of a computed tomography image or a magnetic resonance image of the patient.
12. One or more non-transitory computer readable media programmed to perform the method of any preceding claim.
13. A system comprising: One or more non-transitory computer-readable media for storing data and instructions executable by a processor, the data comprising: Prior three-dimensional image data acquired for the patient; two-dimensional image data comprising at least one two-dimensional image acquired by a medical imaging modality to include a patient and a multimodal marker, a predetermined portion of the multimodal marker being visible in the at least one two-dimensional image and having a known position and orientation relative to at least one tracking sensor detectable by a tracking system, wherein the at least one tracking sensor is attached to the multimodal marker or the multimodal marker includes the at least one tracking sensor; The instructions are programmed to perform a method comprising: estimating a three-dimensional position of the predetermined portion of the multimodal marker relative to a coordinate system of the medical imaging modality based on the respective positions of the predetermined portion in each of the at least one two-dimensional image; and Based on the estimated positions of the corresponding predetermined portions of the multimodal marker and the known relationship between the at least one tracking sensor and the predetermined portions of the multimodal marker, an affine transformation for registering a three-dimensional coordinate system of the tracking system with a three-dimensional coordinate system of the medical imaging modality is determined.
14. The system of claim 13, wherein the at least one two-dimensional image is a first two-dimensional image, and The two-dimensional image data further includes a second two-dimensional image acquired by a medical imaging modality, the second two-dimensional image including the patient and the multi-modal marker and along a non-coincident angle relative to the first two-dimensional image.
15. The system of claim 13 or 14, wherein the affine transformation is a first affine transformation, and the method performed by the instructions further comprises: deriving a corresponding forward projection from a previous three-dimensional image corresponding to each two-dimensional image; as well as Based on registering each two-dimensional image with the corresponding forward projection, a second affine transformation between the coordinate system of the medical imaging modality and the coordinate system of the previous three-dimensional image is determined.
16. The system of claim 15, wherein the three-dimensional mesh model is derived from a previous three-dimensional image to represent the patient's anatomy in the coordinate system of the previous three-dimensional image, and Therein the corresponding forward projection from the previous three-dimensional image is determined as a projection based on the mesh model.
17. The system of claim 15 or 16, wherein determining the second affine transformation further comprises: identifying a set of common features in a previous three-dimensional image and each two-dimensional image; determining positions of at least some of the common feature sets in a coordinate system of the medical imaging modality and a coordinate system of the previous three-dimensional image; as well as A second affine transformation is calculated as a corresponding transformation matrix configured to align the positions determined for the common feature set in one of the coordinate system of the medical imaging modality or the coordinate system of the previous three-dimensional image.
18. The system of any one of claims 15, 16 or 17, wherein the first affine transformation is calculated by one of an error minimization function or a basis transformation function, and The second affine transformation is calculated by minimizing the error function.
19. The system of any of claims 13, 14 or 15, further comprising a multimodal marker comprising a chamber and at least one port into the chamber, the chamber having a predetermined spatial position and orientation relative to the at least one tracking sensor, wherein the multimodal marker is fixed relative to the patient during acquisition of the first image and the second image, and in, During acquisition of the first image and the second image, the multimodal marker includes a volume of liquid contrast agent within the chamber such that the chamber is visible in each of the first image and the second image.
20. The system of claim 13, further comprising a medical imaging modality, wherein the medical imaging modality is configured to acquire each of the at least one two-dimensional image using ionizing radiation.
21. The system of claim 20, wherein the medical imaging modality comprises a C-arm having a center of rotation, wherein the origin of a coordinate system of the medical imaging modality is defined relative to the center of rotation of the C-arm.
22. The system of any one of claims 13, 14, 15, 16, 17, 18, 19, 20 or 21, wherein the at least one two-dimensional image comprises two images selected from the group consisting of: a right anterior oblique projection, a left anterior oblique projection, and an anterior-posterior projection.
23. The system of any one of claims 13, 14, 15, 16, 17, 18, 19, 20, 21 or 22, wherein the non-coincidence angle is less than or equal to 90 degrees.
24. A method comprising: acquiring a first two-dimensional projection image using a medical imaging modality, the first two-dimensional projection image comprising a two-dimensional field of view including a patient and a multi-modal marker; acquiring, using a medical imaging modality, a second two-dimensional projection image including the patient and the multimodal marker and along a non-coincident angle relative to the first two-dimensional projection image, a predetermined portion of the multimodal marker being visible in the first two-dimensional projection image and the second two-dimensional projection image and having a known position and orientation relative to at least one sensor detectable by a tracking system, wherein the at least one sensor is attached to or includes the multimodal marker; deriving a respective forward projection from a previous three-dimensional image corresponding to each of the first two-dimensional projection image and the second two-dimensional projection image; as well as An affine transformation from a coordinate system of the medical imaging modality to a coordinate system of the previous three-dimensional image is determined based on registering the first two-dimensional projection image and the second two-dimensional projection image with the corresponding forward projection.
25. The method of claim 24, further comprising: estimating a three-dimensional position of the predetermined portion relative to a coordinate system of the medical imaging modality based on the position of the predetermined portion of the multimodal marker in each of the respective first two-dimensional projection images and the second two-dimensional projection images; as well as Based on the estimated positions of the corresponding predetermined portions of the multimodal marker and the known relationship of the at least one sensor to the predetermined portions of the multimodal marker, a second affine transformation is determined for aligning the three-dimensional coordinate system of the tracking system with the three-dimensional coordinate system of the medical imaging modality.
26. A system comprising: One or more non-transitory computer-readable media for storing data and instructions executable by a processor, the data comprising: Prior three-dimensional image data acquired for the patient; two-dimensional image data, including at least one two-dimensional image acquired by a medical imaging modality to include a patient and a multimodal marker, a predetermined portion of the multimodal marker being visible in the at least one two-dimensional image and having a known position and orientation relative to at least one tracking sensor detectable by a tracking system, wherein the at least one tracking sensor is attached to the multimodal marker or the multimodal marker includes the at least one tracking sensor; The instructions are programmed to perform a method comprising: deriving a respective forward projection from a previous three-dimensional image corresponding to each two-dimensional image; and Based on registering each two-dimensional image with the corresponding forward projection, an affine transformation between a coordinate system of the medical imaging modality and a coordinate system of the previous three-dimensional image is determined.
27. The system of claim 26, wherein the affine transformation is a first affine transformation, and the method performed by the instructions further comprises: estimating a three-dimensional position of the predetermined portion of the multimodal marker relative to a coordinate system of the medical imaging modality based on a corresponding position of the predetermined portion in each of the at least one two-dimensional image; as well as Based on the estimated positions of the corresponding predetermined portions of the multimodal marker and the known relationship of the at least one tracking sensor to the predetermined portions of the multimodal marker, a second affine transformation is determined for aligning the three-dimensional coordinate system of the tracking system with the three-dimensional coordinate system of the medical imaging modality.
28. The system of claim 26 or 27, wherein the at least one two-dimensional image is a first two-dimensional image, and The two-dimensional image data further includes a second two-dimensional image acquired by a medical imaging modality, the second two-dimensional image including the patient and the multi-modal marker and along a non-coincident angle relative to the first two-dimensional image.
