Registration of Spatial Tracking Systems with Augmented Reality Displays

Abstract

An example method includes acquiring images from cameras, each camera having a known position and orientation relative to a spatial coordinate system of an augmented reality (AR) device. The acquired images include a portion of a multimodal marker device, the multimodal marker device including at least one tracking sensor having a three-dimensional position detectable in a coordinate system of a tracking system. Based on each of the corresponding acquired images and the known position and orientation of the camera relative to the AR device, a three-dimensional position of the portion of the multimodal marker device relative to the spatial coordinate system of the AR device is estimated. The method includes calculating an affine transformation configured to align the coordinate system of the tracking system with the visual space of a display located in the spatial coordinate system of the AR device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 62 / 838,027, filed on April 24, 2019, entitled “REGISTRATION OF SPATIALTRACKING SYSTEM WITH AUGMENTED REALITY DISPLAY,” and U.S. Provisional Application No. 62 / 829,394, filed on April 4, 2019, entitled “SPATIAL REGISTRATION OF TRACKING SYSTEM WITH ANIMAGE USING TWO-DIMENSIONAL IMAGE PROJECTIONS,” each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to systems and methods for registering a tracking system with an augmented reality system. Background Art

[0004] Augmented (or mixed) reality is an interactive experience of a real-world environment in which objects residing in the real world are "augmented" by computer-generated sensory information, such as by overlaying constructive sensory information or destructive sensory information. One example of a constructive sensory information example is the use of an augmented reality headset to overlay computer-generated graphics on a real physical view of an environment so that it is perceived as an immersive aspect of the real environment. However, because the headset is fixed to the user, the computer-generated graphics need to be properly registered to the real physical view of the environment on the fly. This becomes more complicated when the registered graphics being registered do not represent objects visible in the environment. Summary of the invention

[0005] The present disclosure relates to systems and methods for registering a tracking system with an augmented reality system.

[0006] As an example, a method includes acquiring images from cameras, each camera having a known position and orientation relative to a spatial coordinate system of an augmented reality device. The acquired images may include a predetermined portion of a multi-modal marker device having a fixed known spatial position relative to at least one tracking sensor of the multi-modal marker device. At least one tracking sensor has a three-dimensional position detectable in a coordinate system of a tracking system. The method also includes estimating a three-dimensional position of the predetermined portion of the multi-modal marker device relative to the spatial coordinate system of the augmented reality device based on each of the corresponding acquired images and the known position and orientation of the camera relative to the spatial coordinate system of the augmented reality device. The method also includes calculating an affine transformation, which is configured to align the coordinate system of the tracking system with the visual space of a display located in the spatial coordinate system of the augmented reality device based on the estimated three-dimensional position of the corresponding predetermined portion of the multi-modal marker device and the known spatial position of the predetermined portion of the multi-modal marker device relative to at least one tracking sensor.

[0007] As another example, a system includes an augmented reality device including a camera for acquiring an image of a corresponding field of view. One or more non-transitory computer-readable media are configured to store data and instructions executable by a processor. The data includes augmented reality image data for an image acquired by a camera, each camera having a known position and orientation relative to a spatial coordinate system of the augmented reality device. The augmented reality image data may include a predetermined portion of a multi-mode marker device having a fixed known spatial position relative to at least one tracking sensor of the multi-mode marker device, and at least one tracking sensor having a three-dimensional position detectable in a coordinate system of a tracking system. The instructions include code for generating a three-dimensional position of a predetermined portion of the multi-mode marker device relative to the spatial coordinate system of the augmented reality device based on the acquired augmented reality image data and the known position and orientation of the camera relative to the spatial coordinate system of the augmented reality device. The instructions also include code for calculating an affine transformation, which is used to align the coordinate system of the tracking system with the visual space of a display located in the spatial coordinate system of the augmented reality device based on the three-dimensional position of the corresponding predetermined portion of the multi-mode marker device and the known spatial position and orientation of the predetermined portion of the multi-mode marker device relative to at least one tracking sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a flow chart depicting an example of a method of registering sensors of a tracking system into a spatial coordinate system of an augmented reality display.

[0009] Figure 2 An example of a marking device is depicted.

[0010] Figure 3A and Figure 3B An example of a multimodal marking device is depicted.

[0011] Figure 4 An example of an augmented reality device is depicted that includes a camera for acquiring a two-dimensional image of a visualization space.

[0012] Figure 5 An example of a system for generating an affine transformation is depicted.

[0013] Figure 6 An example of a registration manager for controlling use or correction of one or more affine transformations is depicted.

[0014] Figure 7 and Figure 8 is an image from a corresponding camera of an augmented reality device comprising a multimodal marker adjacent to the co-registered model of the anatomical structure.

[0015] Fig. 9 Describes the Figure 1 Example of augmented reality visualization generated by the registration performed by the method. DETAILED DESCRIPTION

[0016] The present disclosure generally relates to methods and systems for registering a tracking system and a set of one or more models with an augmented reality (AR) field of view rendered on an AR display device (such as a head-mounted display). The method utilizes a marking device (e.g., a multi-modal marking) that includes a fiducial marker that can be detected by more than one modality. For example, the marking device includes a first fiducial marker to provide a pattern visible in an image generated by a set of cameras with fixed positions relative to a visualization space (e.g., an AR field of view), and another set of one or more markers that can be detected by a three-dimensional space tracking system.

[0017] As an example, an arrangement of two or more cameras (e.g., digital grayscale cameras) are mounted as forward-facing cameras spaced apart from each other along the frame of the AR device. Thus, the cameras are configured to provide two-dimensional images for overlapping fields of view. In this way, the field of view of the camera includes the field of view of the AR device and may include one or more fiducial markers of the multi-modal marking device. In addition to one or more fiducial markers visible to the spectrum of the camera (which may not be visible to the human eye), the marking device also includes one or more second fiducial markers (e.g., one or more tracking sensors) detectable by a three-dimensional spatial tracking system. Each second fiducial marker is arranged at a predetermined spatial position and orientation relative to the first fiducial marker, and the first fiducial marker is discernible in a corresponding image (e.g., a real-time image) acquired by the camera.

[0018] As a further example, each camera acquires an image including a field of view including a marker pattern corresponding to a first fiducial marker of a marker device. Each image is processed to locate and identify a predetermined portion of the pattern in each respective image (e.g., a corner of a rectangular printed marker). Using a known position (e.g., a fixed position) of each camera relative to the AR device, the identified portion (e.g., a point or region) of the marker pattern is converted to a corresponding three-dimensional position in a three-dimensional spatial coordinate system of the AR system (i.e., the AR field of view).

[0019] The position and orientation of one or more tracking sensors relative to (one or more) fiducial markers are also stored in the memory as tracking position data. In addition, one or more affine transformations for aligning (one or more) tracking sensors with the coordinate system can be pre-calculated and stored in the memory (e.g., as a transformation of the tracking to the model system). In an example, the pre-calculated transformation is a set of one or more affine transformations that are pre-calculated to align the tracking coordinate system with a previous three-dimensional (3D) image scan (e.g., a pre-procedural scan). The previous 3D image scan can be a high-resolution imaging technology, such as computed tomography (CT), magnetic resonance imaging (MRI), which can be performed hours, days, or even weeks before the procedure. One or more models (such as a centerline model and / or a mesh model of a tubular anatomical structure) can be derived from the previous 3D image scan and are therefore spatially registered in the coordinate system of the previous 3D image. As disclosed herein, (one or more) pre-calculated affine transformations can be calculated to align the position and orientation of each tracking sensor with the previous 3D image in a common coordinate system.

[0020] Another affine transformation (also referred to herein as an AR alignment transformation or a zero transformation matrix) is calculated to align the coordinate system of the tracking system with the AR coordinate system. For example, the AR alignment transformation is determined based on the tracking position data, the AR image data, and the tracking sensor transformation. The tracking sensor transformation can define a predetermined spatial relationship between the tracking sensor and one or more benchmarks, which are integrated into the multi-modal marking device and have a fixed spatial offset, and enable the determination of a predetermined spatial position portion of the marker in the coordinate space of the tracking system. Therefore, the AR alignment transformation enables the system and method to align the position and orientation information of each tracking sensor (as provided by the tracking system) with the coordinate system of the AR system modality. The additional transformations disclosed herein can also be used to transform from other spatial domains to the AR coordinate system for concurrent rendering in the AR display. As disclosed herein, the AR display device and the tracking sensor can move relative to the patient's body, and the system can continuously (e.g., in real time) recalculate the transformation based on such AR image data and tracking sensor data that change over time.

[0021] Figure 11 is a flowchart depicting an example of a method 100 for aligning a three-dimensional coordinate system with a coordinate system of an AR visual display of an AR device. In the example, the method 100 is a collection of machine-readable instructions that can be executed by a processor device to perform the method based on data stored in a memory 101. As background, the method 100 is used to align one or more objects (physical and / or virtual objects) having a known spatial position and orientation in another coordinate system with the coordinate system of an AR display. The objects may include objects that are not visible within the field of view of the AR device (e.g., sensors and / or models representing internal anatomical structures). For example, one or more sensors have a position and orientation that can be detected by a three-dimensional tracking system. The sensors can be hidden from view, including being located within the patient's body and as part of a marking device (e.g., embedded in a marker). The AR display may also include objects that are visible within the field of view of the AR device.

[0022] One or more transformations 114 for aligning the tracking sensor(s) with the model coordinate system may be pre-computed and stored (e.g., as a sensor-to-model space transform) in the memory 101, as shown at 114. For example, the transform 114 may be a sensor-to-model space affine transform programmed to register the tracking coordinate system in a common coordinate system with the three-dimensional space coordinate system of a prior 3D medical image (e.g., a preoperative CT scan). One or more anatomical models of the region of interest may be generated from the preoperative medical image and thus registered within the common coordinate system of the prior 3D image. As disclosed herein, the models may include centerline models and surface models for the vasculature as well as other anatomical structures of interest.

[0023] As a further example, a preoperative CT scan is performed to generate three-dimensional image data of a patient's region of interest (e.g., the patient's torso). The image data can be stored in a memory as a DICOM image or other known format. The image data can be processed (e.g., segmented and extracted) to provide a segmented image volume, which includes (one or more) regions of interest for which one or more models (such as the models disclosed herein) can be generated. For example, a previous three-dimensional image can be acquired for a given patient preoperatively by a three-dimensional medical imaging modality. As an example, the preoperative image data can correspond to a preoperative arterial CT scan of the patient's region of interest, such as can be obtained weeks or months before the corresponding operation. Other imaging modalities can be used to provide three-dimensional image data (such as MRI, ultrasonography, positron emission tomography, etc.). Such scans are common parts of preoperative planning in surgical workflows to help determine the size of the prosthesis and plan surgical or other interventions.

[0024] In some examples, one or more anatomical structures captured in the preoperative image data can be converted into a corresponding three-dimensional model in the preoperative image coordinate system. As an example, the model is an implicit model that mathematically describes a tubular anatomical structure (e.g., a patient's blood vessel), such as including the centerline and surface of the tubular structure. The implicit model can include a small set of parameters, such as corresponding to a lofted b-spline (basis spline) function of an elongated anatomical structure. As an example, the anatomical model generator can be programmed to calculate implicit model data according to the disclosure of U.S. Patent Publication No. 2011 / 0026793, entitled "Automated Centerline Extraction Method and Generation of Corresponding Analytical Expression and Use Thereof", which is incorporated herein by reference. Another example of generating an implicit model for a tubular anatomical structure is disclosed in "Analytical centerline extract and surface fit using CTscans for aortic aneurysm repair", Goel, Vikash R, Master's Thesis, Cornell University (2005), which is incorporated herein by reference. Other types of geometric representations may also be used to provide implicit models. For example, parameters representing a lofted elliptical or triangular mesh may be generated to provide anatomical model data representing an anatomical structure of interest to the patient in a three-dimensional coordinate system. The generated three-dimensional mesh (based on three-dimensional prior image data acquired by a preoperative medical imaging modality) may be stored in memory 101 as a supplement or alternative to the three-dimensional image acquired by the preoperative image modality. The mesh may be a static (e.g., fixed) mesh or may vary over time (e.g., as the subject's heartbeat or breathing varies). For example, a mesh model is generated as a four-dimensional model (in model space) to have a three-dimensional configuration that varies over time, such as a gated biological function, such as respiration or heart rate (e.g., detected in an EKG).

[0025] An intraoperative registration phase is performed based on acquired intraoperative image data. The intraoperative data may be acquired prior to or during the procedure and may include 3D image data or 2D image data, such as from an intraoperative cone beam CT (CBCT) scan or another intraoperative radiographic scan (e.g., the non-CBCT registration method disclosed in U.S. Application No. 62 / 829,394 incorporated above). Intraoperative registration (e.g., CBCT registration or non-CBCT registration) is performed when a marking device (e.g., a tracking plate) is attached to the patient, such as just prior to or during the procedure. For example, the marking device includes one or more radiopaque objects in a tracking plate having a known position and orientation relative to one or more tracking sensors, which can be used to determine the tracking sensor positions in the registration space. That is, the marking device enables determination of a transformation (e.g., a tracking system to intraoperative transformation—also referred to herein as a first transformation matrix) for spatially aligning the space of the tracking system with the intraoperative registration space. The intraoperative registration space is the coordinate system that the patient is located in during the procedure and is used to concurrently acquire AR and tracking data by the AR device and the tracking system, respectively, during the procedure.

[0026] Another transformation (e.g., an intraoperative to preoperative transformation—also referred to herein as a second transformation matrix) is determined to spatially align the coordinate system of the intraoperative image with the preoperative CT scan. For example, a manual registration is performed to align the bones in the CBCT scan with the bones in the preoperative CT scan. Alternatively, an automatic or semi-automatic registration process can be performed. Thus, the intraoperative to preoperative transformation enables spatial mapping between the intraoperative image space and the preoperative CT coordinate space. The intraoperative to preoperative transformation can be combined with the tracking system to intraoperative transformation (e.g., by matrix multiplication) to provide a tracking system to preoperative transformation 114 that implements spatial registration from the tracking system coordinate system to the preoperative image coordinate system. For example, the position and orientation of any sensor in the tracking system space (e.g., tracking sensor data 120 from the tracking system) can first be mapped from the tracking system space to the intraoperative space (e.g., using the tracking system to intraoperative transformation), and then mapped from the intraoperative space to the preoperative space (using the intraoperative to preoperative transformation). As mentioned above, the tracking system to intra-operative transformation and the intra-operative to pre-operative transformation may be combined to provide the tracking system to pre-operative transformation 114 .

[0027] As disclosed herein, a multi-modal marking device includes one or more visible fiducial markers (see, e.g., Figure 2 ) and one or more tracking sensors integrated into a common fixed structure (see e.g. Figure 3A and Figure 3B). The fixed structure of the marking device provides the fiducial marker(s) and the tracking sensor(s) with a known spatial relationship and orientation (e.g., a fixed spatial offset) relative to each other in three-dimensional space, which relationship can be stored in memory as tracking sensor data demonstrated at 112. The fiducial markers on the multi-modal marker include one or more marker patterns (see, e.g., Figure 2 ), which is captured by a corresponding camera having a fixed position relative to the AR device (see e.g. Figure 4 ) in an image captured by the camera. The image may be in the visible spectrum or another spectrum outside the visible spectrum (e.g., infrared), which may be captured in an image acquired by the camera at 102. The position and orientation of each camera relative to the coordinates of the AR device may be stored in the memory 101 as camera position data presented at 108. As an example, the cameras may be implemented as a collection of forward-facing cameras mounted at respective fixed locations on a frame of a display of the AR device (e.g., at locations spaced apart along the front of a head-mounted display).

[0028] As yet another example, the marking device includes one or more sensors configured to indicate a three-dimensional position in a coordinate system of a tracking system. For example, the tracking system is an electromagnetic tracking system that generates an electromagnetic field. Each sensor provides a sensor signal based on the electromagnetic field, which is converted into position and orientation information for each corresponding sensor. An example electromagnetic field tracking system is commercially available from Northern Digital, Inc., Ontario, Canada. The tracking system can provide tracking data for each sensor at an output sampling rate (e.g., sixty samples per second) sufficient to achieve substantially real-time determination of the sensor position (e.g., providing a vector describing the sensor position and orientation). The tracking system can therefore process each frame of the tracking data so that the tracking data can also represent real-time tracking data acquired by the tracking system, which can be aligned to the coordinate system of the imaging system, as disclosed herein. In some examples, each sensor can be detectable by the tracking system to enable tracking of the sensor in five or six degrees of freedom. Other types of sensors and tracking systems can be used in other examples.

[0029] In this example context, at 102, the method includes acquiring images from each camera mounted to an AR device (e.g., an AR headset 308). Each camera can be configured to acquire corresponding images for fields of view that overlap each other. For example, where the AR device includes two cameras, first and second images are acquired. The images can be acquired at an imaging sampling rate that can correspond to the native sampling rate of the camera or a multiple thereof and are continuously updated over time. For purposes of this example, it is assumed that the image acquired at 102 includes at least one fiducial marker of the multimodal marker, and such marker is placed near or attached to the patient's body.

[0030] At 104, image processing is performed (e.g., by a marker identification function 444) to identify the fiducial marker(s) in each image acquired at 102. There may be multiple total images for each sampling time - one for each camera. As an example, visible fiducial markers are provided on the surface of a marking device in the form of ArUco markers (see, e.g., Open Source Computer Vision Library: http: / / opencv.org). Figure 2 , Figure 3A and Figure 4 An example of such a fiducial marker is shown in . In this manner, an image processing algorithm (e.g., the detectMarkers() function of the OpenCV library) can be implemented at 104 to detect and identify each such fiducial marker. In examples with different types of markers, other image processing techniques can be used to locate the markers. The marker recognition at 104 can be fully automated and / or user interactive in response to user input of identifying the marker. The identified markers (e.g., pixel locations in the corresponding image) can be stored in the memory 101 for further processing.

[0031] At 106, a three-dimensional position is estimated for a corresponding predetermined portion of the fiducial marker relative to the coordinate system of the AR device (e.g., by a marker point generator 446). The three-dimensional position is determined based on the position of such predetermined portion in each of the corresponding images (determined at 104) and based on the AR camera position data 108. The (one or more) fiducial markers represented in the image acquired from the camera at 102 may include a pattern of marker boundaries having a rectangular shape (or other recognizable shape) having corresponding corners where their edges intersect. For the example of a combined marker including an ArUco type marker visible to the camera, spatial coordinates may be generated for each corner of each marker, i.e., coordinates for a set of four points surrounding each tracking sensor. In addition, the position of the corresponding corners of each image from the representation including the ArUco type fiducial marker may be determined, such as disclosed herein (see, e.g., with respect to Figure 4Related description). Figure 4 and the corresponding description show examples of how the corresponding corners of such a fiducial marker can be located in three-dimensional coordinates of the AR space.

[0032] At 110, an affine transformation is calculated (e.g., by a zero transformation calculator 462) for aligning the coordinate system of the tracking system with the AR coordinate system. The transformation calculated at 110 can be stored in the memory 101 (e.g., corresponding to the zero transformation matrix 410). Therefore, the affine transformation generated at 110 can be directly applied to align the tracking data from the tracking system space to the AR coordinate space and / or to align the AR data from the AR coordinate space to the tracking system space. The affine transformation determined at 110 can be derived based on the estimated position of the predetermined portion of the (one or more) markers determined at 106 and the tracking sensor data 112. As mentioned above, the tracking sensor data 112 can represent a known, fixed three-dimensional spatial relationship of the predetermined portion of the (one or more) markers and the (one or more) tracking sensors of the marker device. As an example, the fixed relationship of the predetermined portion of the (one or more) markers to the sensor can be determined during manufacturing and printed on the marker. As another example, the relationship can be measured and entered into a computer programmed to determine the transformation at 110 (e.g., via a user interface).

[0033] At 116, the affine transformation determined at 110 and one or more other transformations 114 are applied to one or more models (e.g., 3D mesh structures) and tracking position data (for one or more sensors) to place such models and sensors in the coordinate system of the AR display. For example, when generating models from a high-resolution preoperative CT scan, each model to be used by the AR device (e.g., centerline model, surface mesh model) is naturally expressed in the preoperative coordinate space. In order to place such models in the appropriate position so that they overlap with real-world objects in the AR display, the affine transformation determined at 110 is combined with one or more other transformations 114 to map into the AR coordinate system currently being used by the AR device. The other transformations 114 may include a first transformation (e.g., a first transformation matrix 412) programmed to align between the intraoperative image coordinate system and the tracking system coordinate space. Additionally or alternatively, the other transformations 114 may include a second transformation (e.g., a second transformation matrix 414) programmed to align between the intraoperative image coordinate system and the coordinate system of the previous 3D image (e.g., the preoperative image space). The specific manner in which method 100 applies each of transformations 110 and 114 (or their inverses) at 116 depends on the final visualization space and the domain of the data that is commonly configured in such visualization space. The domain can be automatically identified, such as based on the type of data or metadata describing the domain, and / or it can be specified by the user in response to user input. In the following examples, it is assumed that the visualization space is an AR coordinate system.

[0034] At 118, the AR field of view is displayed on the AR display, which may include a computer-generated model at a location where a real-world object is superimposed (e.g., graphically superimposed) at a 3D spatial location determined by applying method 100 to the model and other input data. From 118, the method returns to 102 and repeats to update the affine transformation at 110 based on changes in the image acquired at 102. In this way, the AR field of view (e.g., hologram) is continuously updated in real time so that the hologram generated on the AR display is aligned in space and time with the internal anatomical structures of the patient's body, even if these structures are not actually visible. For example, as disclosed herein, method 100 operates to align internal anatomical structures (not visible in the real world) with the patient's body in the spatial coordinate system of the AR display, and the AR display can be moved relative to the patient's body. Advantageously, by implementing method 100, the transformation calculated at 110 changes in response to changing information in the image acquired at 102; however, other transformations (including transformation 114) can remain unchanged, so that associated calculations can be performed more efficiently in real time.

[0035] For example, when rendering the output visualization at 118 in the AR spatial domain, the model for the skeleton and vasculature (e.g., generated from the previous 3D image space) can be rendered in the AR display by applying multiple transformations (e.g., inv(T0)*inv(T1)*inv(T2)), and a single transformation (e.g., inv(T0)) will be applied to anything tracked in the EM space (catheter, guidewire, etc.). In the example, when rendering the visualization in the previous 3D image space, no transformation is required to be applied to the model for the skeleton and vasculature (in the preoperative CT image space), while two transformations (e.g., T1*T2) will be applied to anything tracked in the tracking system space (e.g., an object with one or more tracking sensors, such as a catheter, guidewire, etc.). For example, as disclosed herein, the transformations can be applied by matrix multiplication to map data from one spatial domain to another spatial domain.

[0036] As a further example, an AR device (e.g., AR device 308) can be implemented as an AR headset (e.g., Hololens or Hololens 2 from Microsoft or other smart glasses). In such an AR headset, the AR device continuously refines the map of its surroundings. Therefore, the hologram generated in the AR field of view has a tendency to "drift" from its original position. When precise alignment is required (such as for medical applications), "drift" becomes a problem. Therefore, based on the image provided as an image stream from the front camera of the AR headset acquired at 102, the method 100 continuously updates the transformation at 110. In addition, by using two non-parallel cameras, the position of the corners of the marker can be accurately estimated by computationally efficient triangulation (reducing CPU load) and continuous updating. This allows "drift" to be corrected without re-alignment.

[0037] Figure 2 An example of a fiducial marker 200 is depicted. As shown in this example, the marker includes black and white (e.g., binarized) and includes a thick black rectangular (e.g., square) border 202 along each side of its entire peripheral edge (e.g., having a thickness "t", such as one or more pixels thick). The interior of the marker 200 includes white symbols 204 and 206, which can be used to define an orientation and / or other identification features that can be associated with the marker, such as according to the ArUco library.

[0038] Figure 3A and Figure 3BAn example of a multi-modal marking device 250 is depicted. During acquisition of the first and second images (e.g., at 102), the multi-modal marking device 250 can be placed in proximity to (e.g., next to or on) the patient. For example, the multi-modal marking device 250 can be placed in a visibly unobstructed surface (e.g., a hospital bed) or attached to the patient's body during a procedure. Figure 3A One side surface 252 of the marker 250 is shown, which includes a fiducial marker (e.g., Figure 2 The reference mark 254 is provided to provide contrast between the white border and the thick black border 258 of the fiducial mark (eg, extending between the dotted line and the white border 256). Symbols 260 and 262 are on the fiducial mark spaced apart from the black border 258.

[0039] Figure 3B The example is a view from the same reference numeral 250 showing another side surface 268. Figure 3B In one embodiment, one or more tracking sensors (e.g., electromagnetic sensors) 270 are attached to the marking device 250 at a known position and orientation relative to a corner 264 of a fiducial marker 254. In one example, the one or more sensors 270 can each sense multiple degrees of freedom (DOF) in space. For example, the one or more sensors 270 can be configured to sense six (6) DOF. In one example, the sensors 270 can be positioned using an electromagnetic tracking system, such as disclosed herein. The tracking system allows the position and orientation of each sensor 270 to be determined based on sensor signals, such as provided from the sensor to the tracking system in response to an electromagnetic field.

[0040] Figure 4 A schematic example of a camera system 300 that can be used to acquire a two-dimensional image of a fiducial marker 302 from multiple non-parallel perspectives is depicted. For example, the camera system 300 includes a pair of front cameras 304 and 306 integrated into the front panel of an AR headset 308. For example, the cameras 304 and 306 can be implemented as digital grayscale cameras to acquire images of objects in the visible part of the spectrum. In other examples, the cameras 304 and 306 can acquire images outside the visible spectrum, and the fiducial marker 302 can be invisible to the user's eyes. Because the cameras 304 and 306 are attached to the headset or other portable device 308, the images acquired by each camera can change over time based on the user's movement. For example, when the user moves his head while wearing the AR headset 308, the perspective will also move accordingly, thereby changing the position of the fiducial marker in each image.

[0041] For example, the registration is performed by modeling each of cameras 304 and 306 as an ideal pinhole camera (e.g., assuming no distortion), where each pixel in the resulting image is formed by projecting a 3D point into the image plane using a perspective transformation such as:

[0042]

[0043] in:

[0044] X, Y, and Z are the coordinates of the 3D point in a common coordinate system;

[0045] u and v are the coordinates of the projected point in the camera image, in pixels;

[0046] fx and fy are focal lengths in pixels;

[0047] cx and cy are the image centers, in pixels; and

[0048] r## and t# define the position and orientation of the camera in the public coordinate system respectively.

[0049] To create vector v1 or v2, the corners of the fiducial marker 302 (e.g., an ArUco type marker) are located in the image as u and v. The rest of the equation can be filled in based on the known spatial positions and the equation is solved for X and Y at the focal length (e.g., the distance between the camera and the corresponding corner position). The vector is then calculated by subtracting the position of the camera (p1 or p2) from this new position. For example, points p1 and p2 are defined based on the position of the headset 308. The focal length of the camera is measured during device calibration.

[0050] The 3D position of the corner of the marker 302 can then be calculated by finding the intersection (or nearest approximation) of the two vectors v1 and v2. The position and orientation of the ArUco marker in the common coordinate system is calculated by repeating this process for all four corner positions identified for the fiducial marker in each corresponding image. For example, the intersection (or nearest approximation) of two vectors can be calculated according to the following pseudo code:

[0051]

[0052] Thus, the estimated positions of the corners of the marker (eg, determined at 106 ) and the corresponding transformations (eg, determined at 110 ) may be used to enable rendering of one or more visualizations in the AR field of view.

[0053] As an example, the generated transformations disclosed herein can be implemented by a registration engine (e.g., registration manager 494) to register tracking data from one or more tracking sensors into an AR visual coordinate system to provide registered tracking data. An output generator (e.g., output generator 512) executed on an AR device or a computer to which the AR device is linked can utilize the registered tracking data and model data to provide a corresponding output visualization that is graphically rendered on a display (e.g., display 510), where the model is visualized as a holographic overlay in an AR visual space above the patient's body.

[0054] Figure 5 An example of a system 400 for generating an affine transformation is depicted. In this example, the affine transformations are shown as transformation matrices 410, 412, and 414 for registering tracking data, models, and image data as disclosed herein. The system 400 is described in the context of data and instructions, and a processor can access the data and execute the instructions to perform the functions disclosed herein. It should be understood that not all functions may be required to implement the system. For example, each of the different transformation matrices can be generated separately, which provides advantages when the imaging modality is changed or replaced in another embodiment because the entire system does not need to be modified.

[0055] exist Figure 5 In an example of , the system 400 is configured to execute program code to generate a zero transformation matrix (T0) 410. The transformation matrix T0 can be configured to transform from the tracking system coordinate system of the tracking system 424 to the AR coordinate system of the AR device and / or from the AR coordinate system to the tracking coordinate system. As disclosed herein, the AR device includes two or more AR cameras 440. As an example, the AR device can be implemented as an AR headset, a smart phone, a tablet computer, or other mobile device. The camera 440 can be integrated into or otherwise coupled to a known position relative to the AR device. Each camera 440 provides AR image data for a corresponding field of view 443, such as at least one AR marker 436 that can be a multi-modal marker device (e.g., the marker device 250). The tracking system 424 is configured to provide tracking data 426 to represent the position and orientation of one or more marker tracking sensors 434 and / or object tracking sensors 438.

[0056] For example, a combination marking system 432 (e.g., including Figure 3A , Figure 3B or Figure 4 One or more multimodal marking devices) can be attached to or placed near the patient's body 430. Figure 5In the example of, the combined marking system 432 may include one or more marker tracking sensors 434 that provide marker tracking data representing the position and orientation of each marking device in the coordinate system of the tracking system 424. In the example, one or more object sensors 438 may be fixed relative to an object movable within the patient's body 430 for identifying the position of such sensor in the coordinate system of the tracking system. For example, each marker tracking sensor 434 provides a signal (e.g., an induced current) in response to an electromagnetic field generated by a field generator of the tracking system 424. Each such object sensor 438 may be fixed to an object (e.g., a guidewire, a catheter, etc.) movable within the patient's body 430. Therefore, the object tracking sensor 438 may also provide a signal to the tracking system 424, based on which the tracking system 424 may calculate corresponding tracking data representing the position and orientation of such sensor (and the object to which it is attached) in the coordinate system of the tracking system. As mentioned above, the tracking data 426 therefore represents the position and orientation of each corresponding object tracking sensor 438 and the marker tracking sensor 434 of the multi-mode marking system 432.

[0057] For example, the tracking system 424 may include a transmitter (e.g., an electromagnetic field generator) shown at 428 that provides a non-ionizing field that is detected by each sensor 434 and 438 to provide a corresponding sensor signal to the tracking system. An example tracking system 424 is an AURORA spatial measurement system commercially available from Northern Digital, Inc. of Ontario, Canada. The tracking system 424 may provide tracking data 426 for each sensor at an output sampling rate (e.g., sixty samples per second) sufficient to enable substantially real-time determination of the sensor's position (e.g., providing a vector describing the sensor's position and orientation). Thus, the tracking processing subsystem of the system 424 may process each frame of the tracking data so that the tracking data may also represent real-time tracking data acquired by the tracking system to enable generation of a graphical representation in a given spatial domain, wherein the tracking system may be registered to another coordinate system by applying one or more generated transformations 410, 412, and / or 414, as disclosed herein. Tracking system 424 may provide tracking data 426 at a calculated output sampling rate that enables real-time localization and visualization of the object to which the sensor is attached and the combined tagging system.

[0058] The zero sensor transformation 460 is configured to convert the tracking data 426 into the position of the AR marker 436 implemented on each corresponding marker device, such as disclosed herein. The transformation 460 provides each position as a 3D spatial coordinate in the tracking system coordinate space, and can remain fixed if the marker device is not moving in the tracking space, or can change over time if the marker device is moving in the tracking space. For example, in the tracking coordinate system, each AR marker of a given marker device is at a fixed, known offset (e.g., a 3D vector) from the position of the marker tracking sensor 434 that is part of the given marker device of the marking system 432. As mentioned above, the marking system can include multiple multi-mode marking devices, such as ArUco type (e.g., device 250), or other marking configurations as disclosed herein.

[0059] As an example, the sensor transform 460 is thus configured to calculate a point in the tracking system space (e.g., a 3D coordinate of a marker position) based on the tracking data 426 and a known offset of each tracking sensor relative to a predetermined marker position. For the example of an ArUco type multi-modal marker device, the marker position may be a set of four points (e.g., emPoint_1, emPoint_2, emPoint_3, emPoint_4) at the corners of the marker, such as disclosed herein. For example, a point in the tracking system space of a set of marker positions of an ArUco type marker device having a sensor providing the tracking data 426 may be calculated for a given marker device by multiplying by a sensor transform (TS), which includes the tracking sensor 3D coordinates and the corresponding offsets, as follows:

[0060] emPoint_1=mult(TS,offset_1),

[0061] emPoint_2=mult(TS,offset_2),

[0062] emPoint_3 = mult(TS, offset_3), and

[0063] emPoint_4=mult(TS,offset_4)

[0064] The points determined by the sensor transform 460 for the AR marker 436 may be arranged in sets of points for each respective marker device (if more than one marker device) or as a single set containing all points.

[0065] As mentioned above, each AR camera 440 provides AR camera data 442 for an AR field of view 443. For example, the AR field of view 443 may include one or more AR markers 436, such as on an exposed surface of a multi-modal marker device that also includes one or more marker tracking sensors 434. Thus, the sensor transformation 460 provides 3D spatial coordinates in a tracking coordinate system for points on the same AR marker visible in the image represented by the AR camera data 442.

[0066] As a further example, the system 400 includes a marker recognition function 444 (e.g., executable instructions, such as corresponding to the recognition at 104), which is configured to locate each marker (e.g., ArUco marker or other type of marker) in each image frame provided in the AR image data 442. For the example of a combination of markers including ArUco type markers, the function 444 can call the ArUco detection function to locate each corresponding marker. For the example combination of markers including markers other than ArUco type markers, the periphery or other features of such markers can therefore be located by image thresholding and other image processing techniques (e.g., feature extraction) applied to image pixels in the AR image 442. The marker recognition function 444 can be fully automated. The identified markers (e.g., pixel locations in the corresponding image) can be stored in a memory for further processing.

[0067] The marker point generator 446 is programmed to generate spatial coordinates of the portion of each marker identified in the (e.g., two or more) images provided by the image data 442. For the example of a marker device including an ArUco type marker, spatial coordinates may be generated for the corners of each marker, i.e., coordinates of a set of four points (e.g., surrounding a tracking sensor or otherwise having a known relative position to a tracking sensor). As an example, the marker point generator is, for example, programmed to execute a closest point function (e.g., a ClosestPoint() function), such as disclosed herein, to locate a set of points around each respective tracking sensor of the marker device. Each set of points for a given AR marker 436 may be linked and associated with a respective marker tracking sensor 434 to facilitate generating a transformation matrix 410.

[0068] The zero transform calculator 462 is programmed to calculate the zero transform matrix 410 based on the points (spatial coordinates) provided by the marker point generator 446 in the AR space domain and the points (spatial coordinates) provided by the zero sensor transform function 460 in the tracking space domain. Therefore, these points represent the same part of the AR marker in different coordinate systems. For example, the transform calculator 462 is programmed to align (e.g., co-register) a set of points that have been measured in each spatial coordinate system. Examples of such registration algorithms implemented by the transform calculator 462 to co-register points in corresponding domains (e.g., tracking system coordinate system and AR coordinate system) can include an error minimization function or a basis transformation function.

[0069] As an example, the transformation calculator 462 is programmed to implement an error minimization function. Given an ordered set of points, the transformation calculator 478 will determine the unknown transformation T0 that minimizes the distance between the projected AR position and the measured position. For example, for T1, the transformation calculator 462 is programmed to find a transformation that minimizes the distance between the points, such as the following:

[0070] sum(n=1..i,distance(mult(T1,imPoint_n),emPoint_n)^2)

[0071] in:

[0072] n represents a given one of i points (i is the number of points of a given multimodal label);

[0073] arPoint_n is the spatial coordinate of point n in the AR image space; and

[0074] emPoint_n is the spatial coordinate of point n in the tracking space.

[0075] In an example, error minimization may be solved by single value decomposition or a variety of error minimization algorithms.

[0076] As another example, the transformation calculator 462 is programmed to implement a basis transformation function to derive the zero transformation matrix 410. In an example where the AR marker is an ArUco marker, the corners of the AR marker are arranged in a manner that enables the generation of a set of basis vectors (x, y, and z unit vectors that define a coordinate space). For example, instead of minimizing the error, the transformation calculator 462 is programmed to find basis vectors in two coordinate systems and apply them to common points. This is computationally more efficient than the error minimization method mentioned above, but requires a specific point arrangement.

[0077] For example, to unambiguously define basis vectors, the desired arrangement is 3 points that form a 90 degree angle, with enough additional information to allow us to identify which point is which (e.g., having the legs of the triangle created by the 3 points be of different lengths). Figure 2 , Figure 3A and Figure 4 The ArUco type signature shown in has a point arrangement sufficient to enable the use of such a basis transformation function.

[0078] In each coordinate system, the transformation calculator 462 constructs basis vectors from 3 points. For example, given point_1, point_2, and point_3 (e.g., vertices of a right triangle), two line segments are provided, one from point_2 to point_1 and the other from point_2 to point_3, which are the legs of the right triangle. These points and line segments provide the following basis vectors:

[0079] basis_z=normalize(point_1-point_2)

[0080] basis_x=normalize(point_3-point_2)

[0081] basis_y=cross(basis_x,basis_z)

[0082] Based on the basis vectors, the transformation calculator 162 is programmed to create a matrix (eg, a 4*4 matrix) defining the position and orientation of point_2 as follows:

[0083] Matrix (point_2)=[basis_x.x,basis_y.x,basis_z.x,point_2.x,basis_x.y,basis_y.y,basis_z.y,point_2.y,basis_x.z,basis_y.z,basis_z.z,point_2.z,0,0,0,1]

[0084] Using the matrices defined in each coordinate system, the transformation calculator 462 can calculate the transformation matrix 410 between the two coordinate systems. For example, for the transformation matrix T0:

[0085] ar_Matrix is ​​a matrix defined in terms of basis vectors in the AR coordinate system; and

[0086] em_Matrix is ​​a matrix defined in terms of basis vectors in the tracking coordinate system.

[0087] As can be seen above, the transformation calculator 462 can determine the transformation matrix (T0) 410 by multiplying the basis vector tracking matrix (em_Matrix) with the inverse matrix of the basis vector AR matrix (inv(ar_Matrix)), such as as follows:

[0088] T0=mult(em_Matrix,inv(im_Matrix))

[0089] The transformation matrix 410 can be stored in a memory and used for transformation from the tracking system space to the AR display space. For example, the position of the object sensor 438 within the patient's body (as represented by the tracking data 426) can be registered in the AR space by applying the transformation T0 to the position and orientation information of the tracking data. As mentioned above, the transformation T0 can be continuously updated in real time, such as to compensate for the movement of the field of view of the AR camera relative to the AR marker and / or if the multimodal marker is moved (e.g., relative to the patient's body or AR camera). In some examples, the system 400 can be configured to generate additional transformation matrices 412 and / or 414 to achieve co-registration of additional data and visualization in the coordinate system of the AR display and in other coordinate systems. In other examples, other transformation matrices 412 and / or 414 may be pre-calculated or not generated.

[0090] exist Figure 5 In an example, the system is also configured to generate a first transformation matrix (T1) 412. The transformation matrix T1 can be configured to transform from a tracking system coordinate system of the tracking system 424 to a coordinate system of a medical imaging modality 456 (e.g., a 2D imaging system such as fluoroscopy or x-ray) and / or from a coordinate system of a medical imaging modality to a tracking coordinate system. In an example, the marking system 432 includes one or more marking devices, including a marking tracking sensor 434, which can be attached to the patient's body 430, so that the tracking system 424 calculates tracking data 426 of such tracking sensors to accommodate movement of the patient's body 430 in the coordinate system of the tracking system 424.

[0091] In some examples, the object tracking sensor(s) 438 and corresponding tracking data 426 may be ignored (or omitted), such as for purposes of generating the transformation matrix 410 and / or the transformation matrix 412. In other examples, the object tracking sensor 438 may be placed at a known location relative to the patient's body 430 (e.g., a known anatomical landmark inside or outside the patient's body) to provide additional data points in the tracking system spatial domain (e.g., provided by the tracking data 426) and in the spatial domain of one or more imaging modalities (e.g., in intraoperative image data 472), as long as the location at which it is placed is visible in the generated image provided by the modality that generated such data. In an example, an intraoperative medical imaging modality (e.g., fluoroscopy or other x-ray) provides image data 472 (e.g., including the known location of the object tracking sensor 438) that can be used to facilitate generating the transformation matrix (T1) 412.

[0092] The first sensor transformation 470 is configured to convert the tracking data 426 into the position of the radiopaque object implemented on each corresponding marking device, such as disclosed herein. Each position is a 3D spatial coordinate in the tracking system coordinate space, and can remain fixed if the marking device does not move in the tracking space, or can change over time if the marking device moves in the tracking space. For example, in the tracking coordinate system, each radiopaque marker of a given marking device is at a fixed, known offset (e.g., a 3D vector) from the position of the tracking sensor 434, where the tracking sensor 434 is part of a given marking device of the marking system 432. As mentioned above, the marking system can include multiple multi-modal marking devices, such as ArUco type (e.g., device 250), or other marking configurations as disclosed herein (e.g., AR device 308). The multi-modal marking device can therefore include a radiopaque element visible in the image data 472, an AR element visible in the AR image data 442, and (one or more) tracking sensors detectable by the tracking system. The radiopaque element may be in the form of a radiopaque ArUco type marker and / or as a radiopaque sphere 272 such as Figure 3B As shown in .

[0093] The sensor transform 470 is thus configured to calculate a point in the tracking system space (e.g., a 3D coordinate of a marker position) based on the tracking data 426 and the known offset of each tracking sensor relative to a predetermined marker position. For an ArUco type multi-modal marker device, the marker position may be a set of four points (e.g., emPoint_1, emPoint_2, emPoint_3, emPoint_4) at the corners of the marker, such as disclosed herein with respect to the sensor transform 460.

[0094] For a marking device including an arrangement of spherical radiopaque markers (e.g., for Figure 3B In the example of the marker device 250 of FIG. 4 , there are three spherical markers distributed at known offsets around each tracking sensor 270. Thus, the sensor transformation 470 will generate three points for each marker device in the marking system 432. For example, the transformation 470 can determine the marker position at a point (e.g., emPoint_1, emPoint_2, emPoint_3) located at the center of each spherical marker based on multiplying the corresponding transformation with the known offset (e.g., 3D offset vector) between the tracking sensor position (e.g., 3D point) and the corresponding radiopaque object, such as the following:

[0095] emPoint_1=mult(Ts,offset_1),

[0096] emPoint_2 = mult(Ts, offset_2), and

[0097] emPoint_3=mult(Ts,offset_3).

[0098] Other deterministic positions with fixed offsets associated with radiopaque markers may be used in other examples.In some examples, the points may be arranged in a set of points for each marking device or as a single set containing all points.

[0099] The image data 472 may be generated as 2D or 3D data representing an object within a field of view 475 of an imaging modality. For example, the imaging modality may include cone beam CT, a fluoroscopic scanner, or other medical imaging modality. In one example, the image data 472 is 2D image data of a small number (e.g., at least two, three, or four) of 2D projection images acquired at different viewing angles relative to the patient's body 430. In some examples, the area of ​​the patient's body may be an area of ​​interest in which the object sensor 438 will move, such as part of a surgical procedure.

[0100] The marker recognition function 474 can be configured to locate each radiopaque marker (e.g., ArUco marker and / or other object marker) in the image data 472. Since the radiopaque marker is opaque relative to the ionizing radiation emitted by the imaging mode 456, the radiopaque marker will be visible in the image. For the example of a combination of markers including ArUco type markers, the marker recognition function 474 can call the ArUco detection function to locate each corresponding marker. For the example combination of markers including radiopaque objects other than ArUco type markers, the periphery of each such marker can therefore be located by image thresholding and other image processing techniques applied to image pixel values. The marker recognition function 474 can be fully automated and / or user-interactive in response to user input of the recognition marker. The identified markers (e.g., pixel positions in the corresponding images) can be stored in a memory for further processing.

[0101] The marker point generator 476 is programmed to generate spatial coordinates for each marker that the marker identification function 474 has identified in the image data 472. For the example of a combination of markers including radiopaque ArUco type markers, spatial coordinates can be generated for each corner of each marker, i.e., the coordinates of a set of four points around each tracking sensor. For radiopaque markers of spherical shape, the spatial coordinates of each marker are provided as the 2D coordinates of the center of a circular projection (e.g., the periphery identified by the marker identification function 474) in each 2D image for the viewing angle provided by the field of view 475 relative to the marking system 432. In the example of three spherical markers surrounding each tracking sensor of a given marking device, the marker point generator 476 is programmed to provide the coordinates of a set of three points of the given marking device. Regardless of the type and configuration of the radiopaque marker, the marker point generator 476 is, for example, programmed to perform a closest point function (such as disclosed herein) to locate a set of points around each corresponding tracking sensor of the marking device. In this way, each set of points can be linked together and associated with a corresponding one of the tracking sensors to facilitate the generation of the first transformation matrix 412.

[0102] The first transformation calculator 478 is programmed to calculate the first transformation matrix 412 based on the points provided by the marker point generator 476 and the points provided by the sensor transformation function 470. For example, the transformation calculator 478 is applied to align a set of points that have been measured in a spatial coordinate system. Examples of such co-registration algorithms for co-registering points in corresponding domains (e.g., a tracking system coordinate system and a medical imaging coordinate system) may include an error minimization function or a basis transformation function, such as disclosed herein.

[0103] As an example, the transformation calculator 478 is programmed to implement an error minimization function. Given an ordered set of points, the transformation calculator 478 will determine the unknown transformation T1 that minimizes the distance between the projected position and the measured position. For example, for T1, we want to find a transformation that minimizes the distance between the points, such as the following:

[0104] sum(n=1..i,distance(mult(T1,imPoint_n),emPoint_n)^2)

[0105] in:

[0106] n represents a given one of i points (i is the number of points of a given multimodal label);

[0107] arPoint_n is the spatial coordinate of point n in image space; and

[0108] emPoint_n is the spatial coordinate of point n in the tracking space.

[0109] In an example, error minimization may be solved by single value decomposition or a variety of error minimization algorithms.

[0110] As another example, transform calculator 478 is programmed to implement a basis transform function, such as disclosed herein with respect to transform calculator 462. As mentioned above, where applicable, transform calculator 478 is programmed to implement a basis transform function, which is computationally more efficient than the error minimization method mentioned above. Figure 3A and Figure 3B The ArUco type markers have a point arrangement sufficient to enable the use of this basis transformation function, but it should be noted that for Figure 3B Each set of 3 points of each marking device is processed separately. Using the matrix defined in each coordinate system, the transformation calculator 478 can calculate the transformation 412 between the two coordinate systems. For example, for the transformation matrix T1:

[0111] im_Matrix is ​​a matrix defined in terms of basis vectors in a medical imaging (eg, intraoperative) coordinate system; and

[0112] em_Matrix is ​​a matrix defined in terms of basis vectors in the tracking coordinate system.

[0113] As can be seen above, the transformation calculator 478 can determine the transformation matrix (T1) 412 by multiplying the basis vector tracking matrix (em_Matrix) with the inverse matrix of the basis vector imaging matrix (inv(im_Matrix)), such as the following:

[0114] T1=mult(em_Matrix,inv(im_Matrix))

[0115] The transformation matrix can be stored in a memory and used for transformation from tracking system space to medical imaging space. For example, the position of the object sensor 438 within the patient's body (as represented by the tracking data 426) can be registered into the medical imaging space by applying the transformation T1 to the position and orientation information of the tracking data.

[0116] As mentioned above, the system 400 is also configured to generate a second transformation (T2) 414 for transforming between the medical imaging coordinate system of the intraoperative image data 472 and the coordinate system of the previous 3D image data 480. For example, the previous 3D image data 480 may be stored in a memory (e.g., as a DICOM image set) and include 3D images from a preoperative scan (e.g., a CT scan) of the patient's body 430 performed at a time before the medical imaging modality 456 generated its image data 472 (e.g., intraoperatively, such as corresponding to the images acquired at 102 and 104).

[0117] In some examples, such as an example where the intraoperative image data is provided as a small number of 2D image projections, the system includes a projection calculator 482. The projection calculator 482 is programmed to generate a corresponding projection for each image (e.g., two images) provided in the 2D image data 472 from the 3D image data 480. The projection calculator 482 implements the function of mapping points from the 3D image space to a two-dimensional plane. For example, the projection calculator derives a forward projection aligned with the viewing angle of the image in the 2D image data 472. The registration of the projection angle of each 3D projection can be achieved and / or automated by manual alignment. In an example, the alignment can be automated, such as based on image metadata in the image data 472 describing the angle of each 2D image (argumented as included in the arrow from the 2D image data 472 to the projection calculator 482). For example, the metadata includes data specifying the projection angle, such as AP, LAO, RAO, such as can be known from the angle of the C-arm and / or provided in response to user input when the imaging modality 456 acquires the image data 472.

[0118] In some examples, as disclosed herein, the 3D image data may include a model of one or more anatomical structures, such as in the form of a 3D mesh corresponding to a vessel surface. A 3D projection matrix (e.g., a perspective or parallel projection matrix) may be applied to a mesh generated from preoperative images 480, such as disclosed herein. If the C-arm angle for each intraoperative image is known, then one 3D projection of the mesh is performed to match the angle of each intraoperative image. If the angle of the C-arm is unknown, then multiple 3D projections may be generated along different angles, and a "best fit" match between the corresponding 3D projection and the corresponding two-dimensional image may be selected manually or automatically.

[0119] Point generator 484 is programmed to generate spatial points in each of the 2D image (provided by image data 472) and the corresponding projection of the 3D image (provided by projection calculator 482). Instead of working with a marked sphere or corner, the points are selected as features visible in both the 2D image data 472 and the 3D image data 480. In other examples, the intraoperative image data 472 can be acquired as 3D data, such as by cone beam CT or other intraoperative 3D imaging modalities. In such an example, the projection calculator can be omitted to enable point generator 484 to identify and generate a corresponding set of points in 3D space provided by both image data sets 472 and 480.

[0120] As a further example, features include structures such as bone landmarks on the spine, calcifications visible in both types of images, or points on blood vessels (in the example of using contrast agents in both images). Other features or reference points can be used in other examples. In some examples, a common feature set can be located in an automated method (e.g., feature extraction). Additionally or alternatively, one or more such features can be selected in response to user input provided by a user interface 486, such as a graphical user interface that interacts with the corresponding images and projections provided to the point generator. For example, a user can see a common visible structure between different views and select / mark it in each view (e.g., by a mouse, keyboard, gesture, or other input). Therefore, the point generator 484 generates points for each predetermined feature and / or user-selected feature. Therefore, the point generator operates similarly to the marker point generator 476, but uses a different landmark set. Since the image data 480 is 3D, in some examples, a user can use a collection of orthogonal views (e.g., axial, coronal, and sagittal views) of the 3D image of the image data 480 to identify selected points (via the user interface 486) to directly measure the x, y, and z positions of the image data 480 in a 3D coordinate system. In examples where the intraoperative image data is in 2D space, each of these positions can be converted into two-dimensional coordinates and provided as such in a forward projection provided by the projection calculator 482. The point generator 484 is programmed to locate the same point in the 2D image data, such as by using a vector intersection function (such as the closest point function disclosed herein) applied to the 2D image. In other examples where the intraoperative image data is in 3D space, the point generator 484 can locate points in the 3D coordinates of the two image sets, such as automatically or with the assistance of user input through the user interface 486.

[0121] The generated points in the corresponding images are provided to the second transform calculator 488 for use in generating the transformation matrix 414. The transform calculator 488 is programmed to calculate the transformation matrix to align the image of the image data 472 with the 3D image data 480 based on the common points provided by the point generator 484. For example, the transform calculator 488 constructs the transformation matrix (T2) 414 by implementing an error minimization function with respect to the set of common points, such as the singular value decomposition described with respect to the first transform calculator 478. Other error minimization functions may be used in other examples.

[0122] In some examples, the system 400 includes a transform correction function 490 that is programmed to implement manual corrections to one or more transform matrices based on instructions provided via a correction user interface 492. Manual corrections can be applied even if a T1 or T2 transform has already been estimated. For example, if the image data 480 and / or 472 does not have a clearly defined set of measurement points (e.g., on the spine or other anatomical structure) to work with to perform registration, the system can define an initial estimate for the transform T2 (or in some examples, an arbitrary T2 transform (e.g., an "identification" matrix)) and allow the user to make corrections via the correction function 490 to generate a final T2 transform 414.

[0123] As a further example, the registration manager 494 is programmed to select and control the application of corresponding transformation matrices 410, 412, and 414. For example, the spatial domain for one or more output visualization spaces can be set automatically or in response to user input. For each output visualization space, the registration manager can define a set of one or more transformations to apply so that images and models can be correctly rendered in each corresponding output space. For example, the output space may include an AR display, a display of a mobile device, or a computer. Each display may also include multiple windows (e.g., screen partitions), each of which can display a different visualization, including the spatial domain of any one of a tracking system, intraoperative image data, an AR display, or a previous 3D image. Therefore, the registration manager 494 can define a set of transformation matrices and apply them to render the correct output image in the desired spatial domain.

[0124] As a further example, see Figure 6 , the registration manager 494 can be used to control the application of one or more of the transformations 410, 412, and 414 and to control user corrections to one or more of such transformations. The registration manager 494 can be implemented as Figure 5 400, as shown, or implemented as separate functions. Thus, for consistency, Figure 5 The functions and data introduced in Figure 6 For further information on such functions and data, please refer to Figure 5 and the corresponding description.

[0125] The registration manager 494 includes a transform correction function 490 and includes first and second transform matrices 412 and 414, respectively. In this example, it is assumed that one or both of the transform matrices 412 and 414 may require correction. The need for correction may be indicated to the user by applying a transform to register two or more domains and providing the resulting visualization on the display 510. For example, the output generator 512 is configured to present a visualization in a selected domain, such as may be a coordinate system of the AR device 440, a coordinate system of the tracking system 424, a coordinate system of the intraoperative image data 472, or a coordinate system of the prior 3D image data 480.

[0126] In an example, the manager 494 includes a domain selector 514 that is programmed to select the domain in which the output visualization is being rendered based on a user input instruction received via the user interface 520. In addition, based on the selected domain, the registration manager applies one or more of the transformations T0, T1, or T2 accordingly. As an example, the following table provides a description of which one or more transformations are applied to the image data 472, 480 or the tracking data 426, and a description of the model that may have been generated in the corresponding coordinate system for each selected domain, wherein the output visualization is rendered by the output generator 512 for each selected domain. The registration manager 494 can also be used to control the application of the corresponding transformations to provide visualization in the selected domain, such as by applying one or more transformations or the inverse of such transformations via matrix multiplication, such as set forth in the table.

[0127] AR track Medical Imaging Previous 3D To AR: [Identification] inv(T0) inv(T0)*inv(T1) inv(T0)*inv(T1)*inv(T2) To track: T0 [Identification] inv(T1) inv(T1)*inv(T2) To Medical Imaging: T1*T0 T1 [Identification] inv(T2) To Previous 3D: T2*T1*T0 T2*T1 T2 [Identification]

[0128] As a further example, manual correction to the transform 412 or 414 may be provided by multiplying the corresponding transformation matrix T0, T1 or T2 by a correction matrix, such as the following:

[0129] correctedT0=mult(correctionMatrix,T0),

[0130] correctedT1=mult(correctionMatrix,T1) or

[0131] correctedT2=mult(correctionMatrix,T2)

[0132] In the example, the supported correction types include translation, rotation and scaling, such as can be applied in the form of a matrix as follows:

[0133] translationMatrix=[1,0,0,translation.x,0,1,0,translation.y,0,0,1,translation.z,0,0,0,1]

[0134] scalingMatrix=[scale,0,0,0,0,scale,0,0,0,0,scale,0,0,0,0,1]

[0135] rotationMatrix = (depends on the rotation axis)

[0136] As a further example, the user initiates correction using a mouse down / drag / mouse up action or other action through the user interface 516. The values ​​used in the correction matrix can be set based on the projection matrix used to display the viewport on the display 510. For example, a translation initiated from the AP view will result in the X and Y mouse movements being used to set the translation.x and translation.z values ​​(translation.y will be 0). Thus, such translations allow the user to change the view of a single image or the alignment of multiple images.

[0137] As a further example, such as when a correction to transformation T2 is implemented, the domain registration manager 494 applies transformation T2 to the image data 472 and the output generator 512 provides a visualization of the 2D image registered in the 3D image based on transformation T2. ​​If the landmarks are aligned correctly, as shown on the display 510, then correction may not be required. However, if the location of the landmarks in the 2D image is inconsistent with their corresponding locations in the 3D image, then correction of T2 may be required. Therefore, the user can adjust the alignment of the 2D image relative to the 3D image (or its forward projection) through the user interface 516. As mentioned above, the adjustment can include translation, rotation and / or scaling in two dimensions in response to instructions entered through the user interface using an input device (e.g., a mouse or keyboard). The output generator 512 can update the visualization shown in the display to show the image registration in response to each adjustment (e.g., in real time). Once the desired alignment is visualized, the user can use the user interface 516 to apply and store the correction to transformation T2, and the updated T2 can be stored in the memory for subsequent application. Similar types of adjustments may be made with respect to the first transformation matrix 412 .

[0138] Figure 7 and Figure 8An example of images 600 and 602 acquired from respective front-facing cameras of an AR headset is depicted. In this example, a multimodal marker (e.g., corresponding markers 250, 302) 604 is positioned on a table 606 adjacent to a physical model of a patient's body 608 containing simulated organs 610. In real life, it should be understood that the organs within the body will not be visible, but are shown to help demonstrate the accuracy of the transformations generated based on the systems and methods disclosed herein. Figure 8 , image 602 is from a slightly different perspective and includes an AR marker 604 and a hand 612 of a user (eg, a person using an AR device).

[0139] As shown in images 600 and 602, marker 604 includes a portion (e.g., a corner) that is identified (e.g., by functions 444 and 446) in the coordinate system of the AR display. Based on sensor data generated by a marker tracking sensor (e.g., sensor 434), the same point of marker 604 is located in the tracking coordinate system to enable generation of a time-varying transformation matrix (e.g., matrix 410), as disclosed herein. Other transformation matrices (e.g., matrices 412 and 414) can also be generated as disclosed herein to align other coordinate systems and images and / or models that may have been generated in such other coordinate systems.

[0140] Fig. 9 Describes something like Figure 7 and Figure 8 The AR image 650 includes a holographic overlay of a mesh model 652 superimposed on the simulated organ 610. Figure 7 and Figure 8 The same reference numerals used in Fig. 9 Based on applying a set of transformations to the mesh model 652 (e.g., according to Figure 1 1 and 4), the overlay is aligned with the patient's anatomical structure (organ 610) in the AR display image 650. For example, where the mesh model 652 is generated in the coordinate system of the 3D previous image, the model can be co-registered into the AR coordinate system by applying the inverse of each of the transformations T0, T1, and T2 to the mesh model (e.g., inv(T0)*inv(T1)*inv(T2), such as shown in the table herein).

[0141] In some examples, annotations 654 are shown in the output visualization to provide additional information to the user, such as the distance and projection angle from the object (e.g., the object to which the object tracking sensor 438 is attached) to the target site. The view can be further modified (e.g., enhanced) in response to user input (e.g., on a user input device, a voice command or a gesture command). For example, an output engine that generates a holographic visualization on an AR display can scale or enlarge the current view overlaid on the patient's body in a real field of view. Additionally or alternatively, the user can enter a command to change the viewing angle. In some examples, such as when enabled, the corners of the marker 604 (or other parts thereof) can be illuminated or otherwise distinguished in the output visualization to confirm that such parts of the marker are correctly aligned. Other image enhancements are also possible.

[0142] In view of the above structure and function description, it will be appreciated by those skilled in the art that the parts of the systems and methods disclosed herein can be implemented as methods, data processing systems or computer program products, such as non-transitory computer-readable media. Thus, these parts of the methods disclosed herein can take the form of complete hardware embodiments, complete software embodiments (e.g., in one or more non-transitory machine-readable media) or embodiments combining software and hardware. In addition, parts of the systems and methods disclosed herein can be computer program products on computer-usable storage media, with computer-readable program code on the medium. Any suitable computer-readable medium can be used, including but not limited to static and dynamic storage devices, hard disks, optical storage devices, and magnetic storage devices.

[0143] Certain embodiments are also described herein with reference to block diagrams of methods, systems, and computer program products. It will be understood that the illustrated blocks and combinations of the blocks in the illustrated blocks can be implemented by computer executable instructions. These computer executable instructions can be provided to one or more processors of a general-purpose computer, a special-purpose computer, or other programmable data processing device (or a combination of a device and a circuit) to produce a machine, so that the instructions executed by the processor implement the functions specified in one or more blocks.

[0144] These computer executable instructions may also be stored in a computer readable memory, which may direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory result in the product including instructions for implementing the functions specified in one or more flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flowchart blocks.

[0145] What is described above is an example. Of course, it is impossible to describe every conceivable combination of components or methods, but those of ordinary skill in the art will recognize that many further combinations and permutations are possible. Thus, the present invention is intended to include all such changes, modifications and variations that fall within the scope of the present application, including the appended claims. As used herein, the term "comprising" means including but not limited to, and the term "comprising" means including but not limited to. The term "based on" means at least partially based on. In addition, when the present disclosure or claims record "one", "one", "first" or "another" element or its equivalent, it should be interpreted as including one or more than one such element, neither requiring nor excluding two or more such elements.

Claims

1. A registration method, comprising: Acquire images from a plurality of cameras, each camera being spaced apart from one another and each camera having a fixed known position and orientation relative to a spatial coordinate system of the augmented reality device, the acquired images comprising predetermined portions of the multi-modal marker device, each predetermined portion having a corresponding fixed known spatial position relative to at least one first tracking sensor of the multi-modal marker device, and the at least one first tracking sensor having a three-dimensional position detectable in a coordinate system of the tracking system; estimating a three-dimensional position of each predetermined portion of the multimodal marker device in a spatial coordinate system of the augmented reality device based on each of the corresponding images acquired from the camera and a known position and orientation of the camera relative to the spatial coordinate system of the augmented reality device; computing an affine transformation configured to register a coordinate system of the tracking system with a visual space of the display in a spatial coordinate system of the augmented reality device based on an estimated three-dimensional position of each respective predetermined portion of the multimodal marker device and a known spatial position of each predetermined portion of the multimodal marker device relative to the at least one first tracking sensor of the multimodal marker device; receiving tracking data of at least one second tracking sensor attached to the object and movable relative to the patient's body, the at least one second tracking sensor having a three-dimensional position detectable in a coordinate system of the tracking system; applying an affine transformation to tracking data of the at least one second tracking sensor to register a position and orientation of the at least one second tracking sensor in a visual space of the display in a spatial coordinate system of the augmented reality device; as well as A graphical output is displayed in a visual space of the display, wherein the graphical output includes a graphical representation of the object superimposed on the visual space of the display based on applying an affine transformation to the tracking data acquired for the at least one second tracking sensor.

2. The registration method of claim 1, wherein the affine transformation is a first affine transformation, the method further comprising: storing in the memory a second affine transformation configured to register the three-dimensional coordinate system of the model space with the coordinate system of the tracking system; storing anatomical model data in a memory to represent at least one three-dimensional model of the patient's anatomy for the internal anatomical structure in a model space; as well as applying a first affine transformation and a second affine transformation to the anatomical model data to align the at least one three-dimensional model of the patient's anatomy for the internal anatomical structure in a visual space of the display, Therein, the graphical output includes at least one graphical representation of the model and a graphical representation of the object, both of which are superimposed on the visual space of the display based on the applied respective first and second affine transformations.

3. A registration method as described in claim 2, wherein the at least one three-dimensional model of the patient's anatomy for the internal anatomical structure includes a grid structure derived in the model space based on three-dimensional previous image data acquired by a pre-operative medical imaging modality. 4 . The registration method of claim 1 , further comprising correcting the affine transformation in response to a user input applying a correction matrix to perform at least one of a translation, a rotation, and a scaling of the affine transformation.

5. A registration method as claimed in any one of the preceding claims, wherein the coordinates of each predetermined portion of the multimodal marking device are determined based on the positions of pixels in the predetermined portion in each of the corresponding acquired images.

6. A registration method as described in claim 5, wherein the multi-modal marking device includes reference markers visible in at least some of the images acquired by the camera, wherein the reference markers are identified in the images acquired by the camera and the coordinates of predetermined portions of the multi-modal marking device are determined for each identified reference marker.

7. The registration method of claim 6, wherein the fiducial marker comprises a rectangular-shaped border having corresponding corners where edges thereof intersect, wherein the predetermined portions of the marker correspond to the corresponding corners of the border.

8. A registration method as described in any of the preceding claims 1-4, wherein the camera is at a known position relative to the display of the augmented reality device and is configured to acquire images to include non-parallel images with overlapping fields of view.

9. The registration method of claim 8, wherein the augmented reality device comprises a headset, the headset comprising a camera and a display thereof, the display being a head mounted display configured to overlay a holographic image on the display within a field of view of the user.

10. The registration method of claim 8, wherein the augmented reality device comprises a smart phone or a tablet computer.

11. The registration method of any one of claims 2 or 3, wherein calculating the first affine transformation is repeatedly performed on the image frames acquired by the camera to update the first affine transformation to accommodate movement of the camera relative to the predetermined portion of the multi-modal marking device; and Graphic output in the visual space of the display is updated in real time based on the updated first affine transformation and the stored second affine transformation.

12. The registration method of any one of claims 2 or 3, wherein the model space comprises a three-dimensional spatial coordinate system of a previous three-dimensional preoperative medical imaging modality, and wherein determining further affine transformations further comprises: calculating a first transformation for registering a coordinate system of the intraoperative medical imaging modality with a coordinate system of the three-dimensional medical imaging modality defining the model space; as well as A second affine transformation for aligning a three-dimensional coordinate system of the tracking system with a three-dimensional coordinate system of the intraoperative medical imaging modality is calculated based on the estimated position of the corresponding predetermined portion of the multimodal marking device and the known relationship between the at least one tracking sensor and the corresponding predetermined portion of the multimodal marking device.

13. One or more non-transitory computer readable media programmed to perform the registration method of any of the preceding claims 1-12.

14. An augmented reality registration system, comprising: An augmented reality device including a camera for acquiring an image of a corresponding field of view; a tracking system configured to provide sensor tracking data indicating a position and orientation of a corresponding tracking sensor; One or more non-transitory computer-readable media for storing data and instructions executable by a processor, the data comprising: augmented reality image data for an image acquired by cameras, each camera having a known position and orientation relative to a spatial coordinate system of the augmented reality device, the augmented reality image data comprising a predetermined portion of the multimodal marker device having a fixed known spatial position relative to at least one first tracking sensor of the multimodal marker device, the at least one first tracking sensor having a three-dimensional position detectable in a coordinate system of the tracking system; The instructions include: generating a code for a three-dimensional position of a predetermined portion of the multimodal marking device relative to a spatial coordinate system of the augmented reality device based on the acquired augmented reality image data and a known position and orientation of the camera relative to a spatial coordinate system of the augmented reality device; and code to compute an affine transformation for registering a coordinate system of the tracking system with a visual space of a display in a spatial coordinate system of the augmented reality device based on the three-dimensional position of the corresponding predetermined portion of the multimodal marker device and the known spatial position and orientation of the predetermined portion of the multimodal marker device relative to the at least one first tracking sensor; code to apply an affine transformation to tracking data of at least one second tracking sensor to register a position and orientation of the at least one second tracking sensor in a visual space of a display in a spatial coordinate system of an augmented reality device, the at least one second tracking sensor being attached to an object capable of movement, the at least one second tracking sensor having a three-dimensional position detectable in a coordinate system of a tracking system; and Code for displaying a graphical output in a visual space of the display, wherein the graphical output includes a graphical representation of the object superimposed on the visual space of the display based on applying an affine transformation to the tracking data acquired for the at least one second tracking sensor.

15. The augmented reality registration system of claim 14, wherein the affine transformation is a first affine transformation, and the instructions further include code for calculating at least one second affine transformation for registering the three-dimensional coordinate system of the model space with the coordinate system of the tracking system.

16. The augmented reality registration system of claim 15, wherein the data further comprises anatomical model data stored to represent at least one three-dimensional model of the patient's anatomy for the internal anatomical structure in the model space, the instructions further comprising: Code for applying a first affine transformation and a second affine transformation to anatomical model data to co-register the at least one three-dimensional model of the patient's anatomy for the internal anatomical structure to a visual space of a display, the visual space being in a spatial coordinate system of an augmented reality device, wherein a graphical output includes a graphical representation of at least one model and a graphical representation of the object, both superimposed on the visual space of the display based on the respective first affine transformation and the second affine transformation.

17. The augmented reality registration system of claim 16, wherein the at least one three-dimensional model of the patient's anatomy for the internal anatomical structure comprises a mesh structure derived in a model space based on three-dimensional prior image data acquired via a pre-operative medical imaging modality.

18. The augmented reality registration system of any one of claims 15, 16 or 17, wherein the code for calculating the second affine transformation further comprises: code for calculating a third transformation for registering a coordinate system of the intraoperative medical imaging modality with a coordinate system of the three-dimensional medical imaging modality defining the model space; as well as Code for calculating a fourth affine transformation for aligning a three-dimensional coordinate system of the tracking system with a three-dimensional coordinate system of the medical imaging modality based on an estimated position of a corresponding predetermined portion for the multimodal marking device and a known relationship between the at least one tracking sensor and the corresponding predetermined portion of the multimodal marking device.

19. The augmented reality registration system of claim 14, wherein the instructions further comprise code to correct the affine transformation in response to user input applying a correction matrix to perform at least one of a translation, rotation, and scaling of the affine transformation.

20. An augmented reality registration system as described in any of claims 14, 15, 16 or 17, wherein the code for generating the three-dimensional position of each predetermined portion of the multi-modal marker device is also programmed to determine the three-dimensional position of each predetermined portion of the multi-modal marker device based on the position of pixels in the predetermined portion in each of the corresponding acquired images acquired by the camera.

21. An augmented reality alignment system as described in claim 20, wherein the multi-modal marker device includes reference markers that are visible in at least some of the images acquired by the camera, and wherein the instructions also include code for identifying the reference markers in the images acquired by the camera, and determining the position of a predetermined portion of the multi-modal marker device for each identified reference marker.

22. The augmented reality registration system of claim 21, wherein the fiducial marker comprises a rectangular shaped boundary having corresponding corners where edges thereof intersect, wherein the predetermined portions of the marker correspond to the corresponding corners of the boundary.

23. The augmented reality registration system of any of claims 14, 15, 16 or 17, wherein the camera is located at a known position relative to the display of the augmented reality device and is configured to acquire images as non-parallel images with overlapping fields of view.

24. The augmented reality registration system of claim 23, wherein the augmented reality device comprises a headset including a camera and a display thereof, the display being a head mounted display configured to overlay a holographic image on the display within a field of view of the user.

25. The augmented reality registration system of claim 23, wherein the augmented reality device comprises a smart phone or a tablet computer.

26. The augmented reality registration system of any one of claims 15, 16 or 17, wherein the code for calculating the first affine transformation is programmed to repeatedly recalculate the first affine transformation of the image frame acquired by the camera to update the first affine transformation to accommodate movement of the camera relative to the predetermined portion of the multi-modal marker device, and The instructions also include code for updating, in real time, a graphics output in a visual space of the display based on the recalculated first affine transformation and second affine transformation.

Images