Systems and methods for tracking surgical devices

Abstract

To provide streamlined systems and methods for registering a CT image with a 3D camera image.SOLUTION: Systems, methods and devices for registering a 3D image of a patient with magnetic coordinates are disclosed. A tri-axial sensor (TAS) may be added to the 3D camera. The location and orientation of the TAS sensor may be determined based on the known magnetic fields that are applied by a magnetic field transmitter. The camera coordinate system may then be transferred to the magnetic coordinate system. After completing the registration of the 3D image with a CT image and the 3D image with the magnetic coordinates, the CT image may then be registered with the magnetic coordinates.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Application No. 63 / 059,805, which is incorporated by reference as if fully set forth.

[0002] FIELD OF THE INVENTION The present invention relates generally to the registration of different coordinate systems, and more particularly to the registration of different coordinate systems for surgical procedures. [Background technology]

[0003] Invasive surgical procedures, including minimally invasive procedures, typically require tracking of surgical devices, such as catheters, within a patient's body that are not directly visible to the physician performing the procedure. Typically, computed tomography (CT) images, such as from fluoroscopy or magnetic resonance imaging (MRI), of the patient are available to the physician.

[0004] One method used to track surgical devices within a patient's body is the TruDi™ electromagnetic image-guided navigation system manufactured by Acclarent, Inc. (33 Technology Drive, Irvine, CA 92618 USA). In this system, an alternating magnetic field is transmitted from a fixed transmitter external to the patient and passes through the patient's body. When a procedure is performed on the patient's head, such as an ear, nose, and throat (ENT) procedure, a fixed magnetic field transmitter may be placed around the patient's head. As part of the procedure, the surgical device is placed in the center of a calibration chamber and three orthogonal magnetic fields may be applied. A sensor, typically a single or multiple axial coil, is attached to the surgical device and inserted into the patient to perform voltage measurements. A processor records the current generated by the magnetic field crossing the sensor. The processor analyzes the current to determine both the position and orientation of the sensor within the electromagnetic reference frame defined by the fixed transmitter.

[0005] Galvanomagnetic-based position detection systems may also utilize flexible sensors located on the surgical device in combination with an algorithm or processor to estimate the position, shape, and size of the surgical device based on voltage measurements taken by the sensors. Flexible sensors typically consist of a single-axis sensor (SAS) with nine transmitters and one sensor for a total of nine measurements taken at a single point. A triaxial sensor (TAS) with nine transmitters and three sensors with two coils collects a total of 27 voltage measurements at a single point. As the surgical device navigates or advances, additional points may be collected within a short period of time.

[0006] Registration of CT images with three-dimensional (3D) camera images is a preliminary step in the TruDi™ navigation procedure. Registration involves determining the position of a surgical device relative to the registered CT image. In other words, by registering or aligning the 3D image and the magnetic coordinates with the CT image, the position of a surgical device tracked by an electromagnetic tracking system can be accurately determined and displayed on the 3D camera image, the CT image, or a combination thereof. In some cases, it may be beneficial to provide the operator with a 3D view of the surface of the anatomical structures within the patient's head. After completing the registration of the 3D image with the CT image and the 3D image with the magnetic coordinates, the magnetic coordinates can then be registered with the CT image. Current systems for registering 3D camera images with CT images require a device, such as a patient tracking device, attached to the patient's head, and specific measures must be taken to account for the patient tracking device during registration. Summary of the Invention [Problem to be solved by the invention]

[0007] It would be desirable to have a streamlined system and method for registering 3D camera images and CT images. [Means for solving the problem]

[0008] A system, method, and device for registering magnetic coordinates and a 3D image of a patient are disclosed. A triaxial sensor (TAS) may be added to a 3D camera. The position and orientation of the TAS sensor may be determined based on a known magnetic field applied by a magnetic field transmitter. The camera coordinate system can then be transferred to a magnetic coordinate system. After completing the registration of the 3D image using the CT image and the 3D image using the magnetic coordinates, the magnetic coordinates and the CT image can then be registered. By registering the magnetic coordinates and the CT image, the position of a catheter tracked by an electromagnetic tracking system can be accurately shown on a display of a CT image, an optical image, or a combination thereof. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic diagram of an alignment system, according to one embodiment. [Figure 2A] FIG. 2 is a schematic diagram illustrating a patient tracking device used in the system of FIG. 1, according to one embodiment. [Figure 2B] FIG. 2 is a schematic diagram illustrating a patient tracking device used in the system of FIG. 1, according to one embodiment. [Figure 3] FIG. 1 is a flowchart diagram of a method for performing a registration algorithm, according to one embodiment. [Figure 4] FIG. 10 is a flowchart diagram of a method for performing a registration algorithm, according to another embodiment. [Figure 5] 1 is a schematic diagram of a 3D scatter plot corresponding to an optical image of a patient positioned in a registration system, according to one embodiment. [Figure 6A] FIG. 1 is a schematic diagram showing the mapping of a 3D scatter plot corresponding to an optical image to a CT image. [Figure 6B] FIG. 1 is a schematic diagram showing the mapping of a 3D scatter plot corresponding to an optical image to a CT image. [Figure 6C] FIG. 1 is a schematic diagram showing the mapping of a 3D scatter plot corresponding to an optical image to a CT image. DETAILED DESCRIPTION OF THE INVENTION

[0010] FIG. 1 is a schematic diagram of an alignment system 20 according to one embodiment of the present disclosure.

[0011] It is assumed that the medical procedure the patient undergoes includes tracking of a surgical device, such as a catheter, inserted into the patient by a medical professional 25. Tracking is provided by an electromagnetic tracking system 24, which is described in more detail below.

[0012] The electromagnetic tracking system includes a magnetic emitter assembly 26 positioned around the patient's head. The assembly 26 includes a magnetic field transmitter 28 that is fixed in place and transmits a sinusoidal alternating magnetic field into a region 30 in which the head of the patient 22 is located. By way of example, the magnetic field transmitters 28 of the assembly 24 are arranged in a generally horseshoe shape around the head of the patient 22. However, other configurations of the emitters of the assembly 26 will be apparent to those skilled in the art, and all such configurations are contemplated as being within the scope of the present invention.

[0013] A magnetic sensor, assumed to be a coil herein, is attached to a surgical device being tracked within the body of patient 22. The attached coil generates electrical signals in response to alternating magnetic fields traversing the coil, and these signals are transferred to a system processor 40. The processor 40 is configured to process the signals to derive position and orientation values ​​for the sensor. Other elements of system 20, including the magnetic transmitter 28, are controlled by the system processor 40.

[0014] The TruDi™ system described above uses a tracking system similar to that described herein to find the position and orientation of the coil within the region illuminated by the magnetic field.

[0015] The processor 40 operates the system 20 using software stored in the memory 42. The software may be downloaded to the processor 40 in electronic form, for example, over a network, or the software may additionally or alternatively be provided and / or stored on a non-transitory tangible medium, such as magnetic, optical, or electronic memory. The processor 40 uses the software to analyze signals received from the magnetic sensors. Software for an alignment algorithm 60 executed by the processor 40 in implementing the alignment system 20 is also stored in the memory 42. The alignment algorithm 60 is described in more detail below.

[0016] The processor 40 may be mounted within a console 50 with motion controls 58, typically including a keypad and / or a pointing device such as a mouse or trackball. The console 50 connects to the emitter via a cable 92 and / or wirelessly. The medical professional 25 may use the motion controls 58 to interact with the processor while performing the medical procedure described above. While performing the procedure, the processor may present the results of the procedure on a screen 56. Presenting the results of the procedure on the screen 56 allows the medical professional 25 using the system to visualize the precise position of a surgical device, such as a catheter, relative to the patient's CT images.

[0017] As described above, the electromagnetic tracking system 24 is able to track the position and orientation of the magnetic sensors within the region 30 by means of the magnetic fields transmitted into the region from the magnetic transmitters 28. It will be understood that the derived position and orientation for the system 24 is relative to the frame of reference (FOR) of the magnetic system, as defined by the position of the magnetic transmitters 28. For the tracking of the sensors to be useful, the FOR of the magnetic system needs to be aligned with the FOR of the images of the patient 22 stored in memory 42. Subsets 66 and 68 of the images 64, described further below, are also stored in memory 42.

[0018] CT images may typically include magnetic resonance imaging (MRI) images or fluoroscopic images, but in the description herein it is assumed that the images include, by way of example, fluoroscopic CT images.

[0019] Medical professional 25 uses a three-dimensional (3D) camera 70 to capture a 3D optical image of patient 22's face. In some embodiments, camera 70 is a RealSense 3D camera manufactured by Intel Corporation of Santa Clara, California. 3D camera 70 may include at least one optical sensor. In some embodiments, 3D camera 70 may include two separate optical sensors. The 3D optical image includes a set of optical voxels, each having three Cartesian coordinates and a color, typically red, green, and blue (RGB) values. The set of optical voxels is also referred to herein as a 3D scatter plot 74, and the optical voxels of scatter plot 74 are stored in memory 42.

[0020] For registration to be performed by system 20, a patient tracker 78 is positioned on patient 22. Patient tracker 78 is described below with reference to Figures 2A and 2B.

[0021] Figures 2A and 2B are schematic diagrams illustrating a patient tracker 78, according to one embodiment. The patient tracker 78 is formed as a substantially planar sheet, with Figure 2A showing a view of the tracker from the camera 70, i.e., after the tracker has been positioned on the patient 22. Figure 2B is an exploded view of the tracker.

[0022] In some embodiments, the patient tracking device 78 is comprised of five layered sheets 80A, 80B, 80C, 80D, and 80E, all of which have substantially the same shape and are bonded together. Sheet 80A is the top sheet, also shown in FIG. 2A, and has multiple optically identifiable indicia 82 incorporated therein. By way of example, sheet 80A includes three optical indicia 82; however, other embodiments may include other numbers of indicia.

[0023] Sheet 80C is typically a layered intermediate sheet formed from a flexible insulating material, which, when formed (typically by printing), forms conductive spiral-shaped planar conductive coils 84. The coils 84 act as electromagnetic sensors. There are as many coils 84 as there are indicia 82, with each coil positioned on sheet 80C in a known spatial relationship with its respective indicia 82. As an example, each coil 84 is positioned so that it is directly aligned with its respective indicia 82 when the sheets of tracking device are joined together. However, other embodiments may be formed using different known spatial relationships between the coils and indicia. For example, the coils and indicia may be offset by a known spatial amount.

[0024] A cable 90 (shown in Figure 1) connects the coil 84 to the processor 40. The connection of the coil 84 to the cable is not shown in Figures 2A and 2B for simplicity.

[0025] Sheet 80E is a bottom layer sheet formed from a biocompatible adhesive, and it is this sheet that comes into contact with patient 22 during operation of system 20.

[0026] Sheets 80B and 80D are intermediate layered sheets formed of a conductive material to act as an electrical shield for coil 84. Within sheet 80B is a non-conductive region 86 aligned with coil 84. The presence of non-conductive region 86 allows the coil to operate properly. In some embodiments, non-conductive region 86 is an opening.

[0027] 3 is a flowchart diagram of a method 300 for executing the registration algorithm 60, according to one embodiment. The method 300 may be performed on the registration system 20 shown in FIG.

[0028] At 301, the electromagnetic tracking system 24 is activated and the patient 22's head is positioned within the system's field 30. The patient tracking device 78 is attached to the patient's forehead using a biocompatible adhesive sheet 80E, so that the optical landmarks 82 are uppermost and visible. A cable 90 is connected between the patient tracking device and the processor 40, which may be activated to acquire signals transmitted by the cable from the coil 84. The processor analyzes the signals and calculates the position of the coil within the FOR defined by the magnetic transmitter 28. If the calculated position is found to be within the expected portion of the field 30, the processor 40 may provide an indication to the medical professional 25 that the electromagnetic tracking system 24 is operating correctly. An exemplary indication that the electromagnetic tracking system 24 is operating correctly is the processor sending a notification that is displayed on the screen 56.

[0029] At 302, processor 40 may analyze a CT image of the patient's head stored in memory 42. In some embodiments, processor 40 may analyze the image to identify a subset of CT voxels of the stored image that correspond to surface features of the patient's head and store this subset as surface subset 66.

[0030] At 303, the medical professional 25 activates the 3D camera 70 to capture 3D optical images of the patient's 22 face and stores the captured images in memory 42 as a scatter plot 74. It will be appreciated that the images captured by the 3D camera 70 include an image of a patient tracker 78 positioned on the patient's face.

[0031] Any suitable algorithm can be used to find a transformation that best maps the surface subset of CT voxels 66 to the optical voxels of the 3D scatter plot 74. Any cloud point matching algorithm can be used, such as robust point matching and kernel correlation. In some embodiments, an iterative closest point (ICP) algorithm can be used. However, by 303, there is a known difference between the two voxel sets. This is because the patient tracker image is present in the scatter plot 74 but not in the CT voxel subset 66.

[0032] At 304, the absence of the patient tracker image in the CT voxel subset 66 is compensated for by adding the patient tracker image to the CT voxel subset. This addition may be performed by presenting the CT voxel subset image on the screen 56 to the medical professional 25, allowing the professional to overlay the patient tracker image on the presented image, and storing the combined image as the adjusted CT voxel subset 68.

[0033] Alternatively, at 304, an adjusted subset 68 is extracted from the CT voxel subset 66 by the medical expert 25 selecting a portion of the subset 66 that does not include an image of the patient tracker. The medical expert 25 may perform the selection on an image of the subset 66 presented on the screen 56, and the selected portion is stored as the adjusted CT voxel subset 68.

[0034] At 305, processor 40 maps adjusted CT voxel subset 68 to voxels of scatter plot 74. If adjusted CT subset 68 includes patient tracker images at 304, mapping may be performed for all voxels in the two sets. Alternatively, if 304 is performed by selecting a portion of subset 66 that does not include patient tracker images, processor 40 makes a corresponding selection in voxels of scatter plot 74, and mapping is performed between the selected voxel sets.

[0035] The mapping results in alignment between the FOR of the CT images of the patient 22 and the FOR of the optical images of the patient. The processor 40 may quantify the alignment as a first transformation matrix M[CT-OPT], which may be used to transform entities contained in one of the reference frames to the other FOR.

[0036] At 306, processor 40 uses the known spatial relationship between optical landmarks 82 and coils 84 to perform a mapping between the locations of the landmarks in optical 3D scatter plot 74 and the locations of the coils of the FOR of electromagnetic tracking system 24, as found at 301. The mapping results in an alignment between the FOR of the electromagnetic tracking system and the FOR of the optical image, which can be quantified as a second transformation matrix M[MAGN-OPT].

[0037] At 307, the processor 40 combines the two registrations generated at 305 and 306 to generate a third registration between the FOR of the electromagnetic tracking system 24 and the FOR of the CT images. The resulting registration may be quantified as a third transformation matrix M[CT-MAGN], and it will be appreciated that the matrix M]CT-MAGN] can be generated from the matrices M[MAGN-OPT] and M[CT-OPT].

[0038] As mentioned above, the method of Figure 3 may be performed on the registration system 20 of Figure 1, which includes a patient tracker 78. In some embodiments, the registration system 20 does not include a patient tracker 78. Instead, additional sensors may be added to the 3D camera 70 to align the 3D image of the head of the patient 22 with the magnetic coordinates of the electromagnetic tracking system 24. The additional sensors may determine the position and orientation of the 3D camera 70. The position and orientation of the 3D camera 70 may be used to find the alignment between the electromagnetic system and the CT images, as described in more detail below.

[0039] In some embodiments, the additional sensor is a triaxial sensor (TAS). The TAS includes three sensors with nine transmitters and two coils, collecting a total of 27 (3 × 9 = 27) voltage measurements at a single point. The three sensors of the TAS can provide simultaneous measurements in three orthogonal directions. The 3D camera may be tracked and navigated to match the actual position (magnetic position and orientation) of the TAS. The position and orientation of the TAS sensor may be determined based on known magnetic fields applied by the magnetic field transmitters. The 3D camera 70 may be tracked and navigated using the TAS sensor, which reads the magnetic field configuration to retrieve the position and orientation of the 3D camera 70. The addition of the TAS sensor to the 3D camera 70 provides the ability to register the 3D camera image with magnetic coordinates, thereby improving the accuracy of registering CT images using magnetic coordinates.

[0040] In other embodiments, the additional sensors are single-axis sensors (SAS) or dual-axis sensors (DAS). The SAS collects a total of nine voltage measurements (1 x 9 = 9). The DAS collects eighteen voltage measurements (2 x 9 = 18).

[0041] FIG. 4 is a flowchart diagram of a method 400 for performing a registration algorithm according to one embodiment. Method 400 may be performed by the registration system 20 shown in FIG. 1 , but method 400 does not require the use of a patient tracker 78. Instead of using a patient tracker 78, the 3D camera 70 further includes at least one magnetic sensor in addition to at least one optical sensor, which can be used to determine the position and orientation of the 3D camera and to register the CT image and the 3D image. In some embodiments, the magnetic sensor is a TAS, as described above. A TAS can provide simultaneous measurements in three orthogonal directions. In other embodiments, a SAS or DAS may be used instead of a TAS.

[0042] As mentioned above, the 3D camera 70 also includes at least one optical sensor. In some embodiments, the 3D camera 70 may include two separate optical sensors. The 3D optical image includes a set of optical voxels, each having three Cartesian coordinates and a color, typically a red, green, and blue (RGB) value. The set of optical voxels is also referred to herein as a 3D scatter plot 74, and the optical voxels of the scatter plot 74 are stored in memory 42.

[0043] At 401, the electromagnetic tracking system 24 is activated and the head of the patient 22 is positioned within the system's field 30. When the electromagnetic tracking system 24 is activated, the magnetic field transmitter 28 is fixed in place and transmits a sinusoidal alternating magnetic field within the field 30, positioned where the head of the patient 22 will be located.

[0044] At 402, medical professional 25 activates camera 70 to capture a 3D optical image of the face of patient 22. Processor 40 stores the captured 3D image as a 3D scatter plot 74 in memory 42.

[0045] At 403, processor 40 may be activated to analyze the voltage measurements of the TAS. Processor 40 is configured to process the signals to derive position and orientation values ​​of the sensor. Processor 40 is configured to process the signals to derive position and orientation values ​​of the sensor. Other elements of system 20, including magnetic transmitter 28, are controlled by system processor 40. Other elements of system 20, including magnetic transmitter 28, are controlled by system processor 40. Processor 40 is configured to process the signals to derive position and orientation values ​​of the sensor, and thereby the camera. Other elements of system 20, including magnetic transmitter 28, are controlled by system processor 40.

[0046] In some embodiments, 402 and 403 are performed simultaneously. In further embodiments, timestamps may be taken at both 402 and 403 and cross-referenced to ensure they match (i.e., to confirm that the 3D image of the patient acquired at 402 is simultaneous with the time the camera position and orientation was determined at 403).

[0047] At 404, the processor 40 transfers the optical 3D scatter plot 74 to the magnetic coordinate system of the electromagnetic tracking system. The position and orientation of the 3D camera 70 determined at 403 may be used to map the 3D scatter plot 74 to magnetic coordinates. Any suitable position algorithm may be used to map the 3D scatter plot 74 to magnetic coordinates. For example, a cloud point matching algorithm such as ICP or robust point matching may be used. The mapping may provide a registration between the optical image and the magnetic coordinates.

[0048] At 405, the processor may register the CT image and the 3D image of the patient 22. The CT image of the head of the patient 22 may be retrieved from memory 42. The processor 40 may analyze the image to identify CT voxels in the stored image that correspond to surface features of the head of the patient 22. The CT voxels may be mapped to optical voxels of the 3D scatter plot 74. Any suitable algorithm may be used to find a transformation that best maps the optical voxels of the 3D scatter plot 74 to the surface of the CT voxels. In some embodiments, an ICP algorithm may be used. This mapping may provide registration between the optical and CT images.

[0049] In some embodiments, after completing the registration of the 3D image using the CT image and the 3D image using the magnetic coordinates, the magnetic coordinates and the CT image may then be registered. For example, the processor 40 may combine the registration between the optical image and the magnetic coordinates and the registration between the optical image and the CT image to generate another registration between the magnetic coordinates and the CT image. By first registering the magnetic coordinates and the 3D image, the accuracy of the registration of the CT image using the magnetic coordinates may be improved.

[0050] Figure 5 is a schematic illustration of a 3D scatter plot 510 corresponding to a 3D optical image of a patient 22 positioned in region 30 of a registration system, according to one embodiment. The registration system may include registration system 20 shown in Figure 1, with patient tracker 78 being optional. In Figure 5, the 3D scatter plot 510 of the acquired 3D image is superimposed on the face of patient 22 for illustrative purposes.

[0051] 6A-6C are schematic diagrams of mapping a 3D scatter plot 510 of an acquired 3D image to a CT coordinate system 520 of an electromagnetic tracking system, according to one embodiment. This mapping may occur at 405 of the method described with respect to FIG. 4. FIG. 6A illustrates when the 3D scatter plot 510 and the CT voxels 520 are distinct. FIGS. 6B and 6C are schematic diagrams of the 3D scatter plot 510 being mapped to the CT voxels 520. In some embodiments, the position and orientation of the 3D camera 70 when the 3D image was acquired is used in the algorithm for mapping the 3D scatter plot to magnetic coordinates, as described above.

[0052] It will be understood that the above-described embodiments are given by way of example, and that the present disclosure is not limited to what is particularly shown and described hereinabove. Rather, the scope of the present disclosure includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof not disclosed in the prior art that would occur to one skilled in the art upon reading the foregoing description.

[0053] [Embodiment] (1) A method comprising: activating an electromagnetic tracking system on the patient's head; receiving three-dimensional (3D) images of the patient's head and storing a scatter plot of the 3D images in a memory, the 3D images being acquired using a 3D camera including at least one optical sensor and at least one magnetic sensor; determining a position and orientation of the 3D camera when the 3D image is acquired via a magnetic sensor based on known magnetic fields applied by one or more transmitters of the electromagnetic tracking system; and aligning the scatter plot to magnetic coordinates of the electromagnetic tracking system using the determined position and orientation of the 3D camera, wherein the aligned scatter plot is stored in the memory. (2) The method of embodiment 1, wherein the at least one magnetic sensor is a three-axis sensor (TAS). (3) The method of embodiment 1, wherein the at least one magnetic sensor is a single-axis sensor (SAS) or a dual-axis sensor (DAS). (4) The method of claim 1, wherein the scatter plot is a reference fixed to two separate optical sensors on the 3D camera. (5) The method of embodiment 1, further comprising registering the received 3D image of the subject with a computed tomography (CT) image of the subject.

[0054] (6) The method of embodiment 5, wherein the CT image is registered with the magnetic coordinates using an iterative closest point (ICP) algorithm. (7) The method of embodiment 5, further comprising registering the magnetic coordinates with a CT image of the patient. (8) A system comprising: an electromagnetic tracking system comprising one or more transmitters configured to apply a magnetic field; a 3D camera including at least one optical sensor and at least one magnetic sensor; a memory configured to store 3D images of the subject acquired by the 3D camera as a scatter plot; 1. A processor, comprising: activating the electromagnetic tracking system; Retrieving the scatter plot from the memory; determining a position and orientation of the 3D camera when the 3D camera acquired the 3D image via the magnetic sensor based on a known magnetic field applied by a transmitter of the electromagnetic tracking system; a processor configured to align the scatter plot to magnetic coordinates of the electromagnetic tracking system; The system, wherein the memory is further configured to store the aligned scatter plots. (9) The system of embodiment 8, wherein the at least one magnetic sensor is a three-axis sensor (TAS). (10) The system described in embodiment 8, wherein the at least one magnetic sensor is a single-axis sensor (SAS) or a dual-axis sensor (DAS).

[0055] (11) The system of embodiment 8, wherein the scatter plot is a reference fixed to the at least one optical sensor of the 3D camera. (12) The system of embodiment 8, wherein the processor is further configured to register the 3D image with a computed tomography (CT) image. (13) The system of embodiment 12, wherein the CT image is registered with the magnetic coordinates using an ICP algorithm. (14) The system of embodiment 12, wherein the processor is further configured to register the magnetic coordinates with the CT image of the subject. (15) A non-transitory computer-readable storage medium having stored thereon program instructions, the instructions, when read by a processor, causing the processor to: activating an electromagnetic tracking system; acquiring a scatter plot of a 3D image of the subject, the 3D image being acquired by a 3D camera including at least one optical sensor and at least one magnetic sensor; determining the position and orientation of the 3D camera when the 3D image was acquired via the magnetic sensor based on a known magnetic field applied by a transmitter of the electromagnetic tracking system; aligning the scatter plot to the magnetic coordinates of the electromagnetic tracking system; A non-transitory computer-readable storage medium that causes

[0056] (16) The non-transitory computer-readable storage medium of embodiment 15, wherein the at least one magnetic sensor is a triaxial sensor (TAS). (17) The non-transitory computer-readable storage medium of embodiment 15, wherein the at least one magnetic sensor is a single-axis sensor (SAS) or a dual-axis sensor (DAS). (18) The non-transitory computer-readable storage medium of embodiment 15, wherein the scatter plot is a reference fixed to the at least one optical sensor. (19) A non-transitory computer-readable storage medium as described in embodiment 15, wherein the stored program instructions, when read by a processor, further cause the processor to register the 3D image with a computed tomography (CT) image. (20) The non-transitory computer-readable storage medium of embodiment 19, wherein the stored program instructions, when read by a processor, further cause the processor to register the magnetic coordinates with the CT image.

Claims

1. 1. A system comprising: an electromagnetic tracking system comprising one or more transmitters configured to apply a magnetic field; a 3D camera including at least one optical sensor and at least one magnetic sensor; a memory configured to store 3D images of a subject acquired by the 3D camera as a scatter plot; 1. A processor, comprising: activating the electromagnetic tracking system; Retrieving the scatter plot from the memory; determining a position and orientation of the 3D camera when the 3D camera captured the 3D image via the at least one magnetic sensor based on known magnetic fields applied by the one or more transmitters of the electromagnetic tracking system; a processor configured to align the scatter plot to magnetic coordinates of the electromagnetic tracking system; The system, wherein the memory is further configured to store the aligned scatter plots.

2. The system of claim 1 , wherein the at least one magnetic sensor is a triaxial sensor (TAS).

3. The system of claim 1 , wherein the at least one magnetic sensor is a single-axis sensor (SAS) or a dual-axis sensor (DAS).

4. The system of claim 1 , wherein the 3D image is a 3D optical image of the subject's head.

5. The system of claim 1 , wherein the processor is further configured to register the 3D image with a computed tomography (CT) image.

6. The system of claim 5 , wherein the CT image is registered with the magnetic coordinates using an ICP algorithm.

7. The system of claim 5 , wherein the processor is further configured to register the magnetic coordinates with the CT image of the subject.

8. The system of claim 4 , wherein the one or more transmitters are positioned around the head of the subject.

9. The processor: obtaining a CT image of the head of the subject; registering the scatter plot to CT coordinates; The system of claim 8 , further configured to align the magnetic coordinates to the CT coordinates based on an alignment of the scatter plot with the magnetic coordinates and an alignment of the scatter plot with the CT coordinates.

10. The processor: obtaining a CT image of the head of the subject; analyzing the CT images to identify CT coordinates in the stored images that correspond to surface features of the subject's head; registering the scatter plot to the CT coordinates; The system of claim 8 , further configured to align the magnetic coordinates to the CT coordinates based on an alignment of the scatter plot with the magnetic coordinates and an alignment of the scatter plot with the CT coordinates.

11. 10. The system of claim 9, wherein the processor is configured to use the determined position and orientation of the 3D camera to align the scatter plot to the magnetic coordinates of the electromagnetic tracking system.

12. A non-transitory computer-readable storage medium having stored thereon program instructions, the program instructions, when read by a processor, causing the processor to: activating an electromagnetic tracking system; acquiring a scatter plot of a 3D image of a subject, the 3D image being acquired by a 3D camera including at least one optical sensor and at least one magnetic sensor; determining a position and orientation of the 3D camera when the 3D image was acquired via the at least one magnetic sensor based on a known magnetic field applied by a transmitter of the electromagnetic tracking system; aligning the scatter plot to the magnetic coordinates of the electromagnetic tracking system; A non-transitory computer-readable storage medium that causes

13. 13. The non-transitory computer-readable storage medium of claim 12, wherein the at least one magnetic sensor is a triaxial sensor (TAS).

14. 13. The non-transitory computer-readable storage medium of claim 12, wherein the at least one magnetic sensor is a single-axis sensor (SAS) or a dual-axis sensor (DAS).

15. 13. The non-transitory computer-readable storage medium of claim 12, wherein the 3D image is a 3D optical image of the subject's head.

16. 13. The non-transitory computer-readable storage medium of claim 12, wherein the stored program instructions, when read by the processor, further cause the processor to register the 3D image with a computed tomography (CT) image.

17. 17. The non-transitory computer-readable storage medium of claim 16, wherein the stored program instructions, when read by the processor, further cause the processor to register the magnetic coordinates with the CT image.

18. 16. The non-transitory computer-readable storage medium of claim 15, wherein the electromagnetic tracking system includes one or more of the transmitters positioned around the head of the subject.

19. The stored program instructions, when read by the processor, further cause the processor to: obtaining a CT image of the head of the subject; aligning the scatter plot to the identified CT coordinates by analyzing the CT image; 20. The non-transitory computer-readable storage medium of claim 18, wherein the magnetic coordinates are aligned to the CT coordinates based on an alignment of the scatter plot with the magnetic coordinates and an alignment of the scatter plot with the CT coordinates.

20. 1. A method of operating a system, the system comprising: an electromagnetic tracking system including one or more transmitters; a 3D camera including at least one optical sensor and at least one magnetic sensor; Memory and a processor, and the method of operation includes: activating the electromagnetic tracking system positioned around the patient's head; receiving three-dimensional (3D) images of the patient's head acquired using the 3D camera and storing a scatter plot of the 3D images in the memory; determining a position and orientation of the 3D camera when the 3D image was acquired via the at least one magnetic sensor based on a known magnetic field applied by the one or more transmitters of the electromagnetic tracking system; aligning the scatter plot to magnetic coordinates of the electromagnetic tracking system using the determined position and orientation of the 3D camera, wherein the aligned scatter plot is stored in the memory; and The method includes the processor performing the following.

21. 21. The method of claim 20, wherein the at least one magnetic sensor is a triaxial sensor (TAS).

22. 21. The method of claim 20, wherein the at least one magnetic sensor is a single axis sensor (SAS) or a dual axis sensor (DAS).

23. 21. The method of claim 20, wherein the 3D image is a 3D optical image.

24. 21. The method of claim 20, further comprising the processor registering the received 3D image of the patient with a computed tomography (CT) image of the patient.

25. 25. The method of claim 24, wherein the CT image is registered with the magnetic coordinates using an iterative closest point (ICP) algorithm.

26. 25. The method of claim 24, further comprising the processor registering the magnetic coordinates with a CT image of the patient.

27. obtaining a CT image of the head of the patient; registering the scatter plot to CT coordinates; aligning the magnetic coordinates to the CT coordinates based on the alignment of the scatter plot with the magnetic coordinates and the alignment of the scatter plot with the CT coordinates; The method of claim 23 , further comprising the processor:

Images