Systems for integrated imaging and multi-view augmented reality for surgical intervention

Abstract

Systems for an augmented reality system including a plurality of cameras positioned in orthogonal relation to each other in operative association with an image scanner and a processor operable for producing a stereoscopic view in which a virtual surgical target and a planned insertion path for a surgical device are overlaid in the stereoscopic image produced by the first and second cameras at different angles by the processor to provide real-time image guidance to the user.

Description

SYSTEMS FOR INTEGRATED IMAGING AND MULTI-VIEW AUGMENTED REALITY FOR SURGICAL INTERVENTIONFIELD

[0001] The present disclosure generally relates interventional radiology, and in particular systems and methods for integrated imaging and multi-view augmented reality for imager guided procedures.BACKGROUND

[0002] Interventional procedures such as biopsies and ablations rely on the experience and skill of the performing physician to navigate complex anatomy where accuracy defines outcomes. Following and estimating a planned angle may remain challenging and fraught with error, dependent on expertise and experience of the operator. Translating two-dimensional ultrasound images into a three- dimensional understanding of the patient’s anatomy or CT relies upon cognitive estimation, thus deviating from the planned angle is common and may increase risk for complications, procedure time, and radiation exposure, when confirmatory CTs are required.

[0003] Augmented reality (AR) presents a transformative technology for percutaneous needle procedures such as biopsies and ablations. AR overlays virtual information onto the real world through different media such as smart goggles, tables, smartphones and monitors. In the context of biopsies and ablations, AR systems can integrate related pre-procedural imaging data, such as CT scans or ultrasound, needle pathway planning, and the body’s anatomy onto the visible monitor or the patient. For a clinically valuable integration into medical procedures, high accuracy and reliable registration are needed. Standard AR systems commonly integrate either an external optical or electromagnetic tracking system to further improve accuracy and track or combine additional hardware. However, the registration process is complex and challenging to accurately superimpose anatomy, targets and needle paths in relation to patient anatomy. Often, multiple fiducials on hardware and skin markers are required to ensure reliability and steadiness of the visual projections. Fiducial markers are typically disposable and must be sterilized in a clinical environment, which adds to cost and preparation time. Additionally, patientmounted fiducial markers can sometimes obstruct or be displaced by the ultrasound probe. When vision-based tracking is employed, the fiducial markers often need tobe large enough for the camera to detect, register, and track, leading to non- ergonomic attachments on the patient, operator, and medical devices.

[0004] The registration process frequently introduces additional time to the procedure. In most AR systems, registration is performed at the start of the procedure using pre-operative imaging. If the target moves during the procedure, it is often impractical to repeat the registration process with intraoperative CT scans to account for target motion, which can result in a loss of accuracy in image guidance. Furthermore, current AR displays may also have their own limitations. Headmounted displays may lead to eye fatigue and cyber sickness, smartphones occupy at least one additional hand and typically have a small interface. A seamless integration of AR with existing imaging systems and user-friendly interfaces are essential for a widespread adoption in medicine.

[0005] It is with these observations in mind, among others, that various aspects of the present disclosure were conceived and developed.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1 is a simplified illustration showing a computer tomography (CT) mounted dual-camera navigation system for augmented reality (AR).

[0007] FIG. 2 is an image of checkerboard templates with CT visible fiducial markers for camera calibration using computer tomography.

[0008] FIG. 3 is an image of an AR navigation setup showing two orthogonal cameras of the dual camera AR system mounted on a CT gantry.

[0009] FIGS. 4A and 4B are a pair of images showing a validation procedure that demonstrates the accuracy of the dual camera AR system.

[0010] FIGS. 5A and 5B are a pair of images showing a camera navigation feature of the AR system using an AR guided needle placement by visually aligning a needle with the target (red ball) in the axial and sagittal views.

[0011] FIGS. 6A and 6B are a pair images showing a path navigation feature of the AR system using AR needle navigation by aligning the segmented needle trajectory (marked in yellow arrows) with the target (red ball) in both axial and sagittal views in which a segmentation algorithm also detects other straight lines in the background.

[0012] FIG. 7 is a graphical representation showing needle placement accuracy using the camera navigation feature, path navigation feature, and freehand without use of the dual camera AR system.

[0013] FIG. 8 is a graphical representation showing angle error using the camera navigation feature, path navigation feature, and freehand without use of the dual camera AR system.

[0014] FIG. 9 is a graphical representation showing placement time using the camera navigation feature, path navigation feature, and freehand without use of the dual camera AR system.

[0015] FIGS. 10A and 10B are images showing needle placement (marked in yellow arrows) using the needle track (marked in green arrows) of the previous scan as a reference in augmented reality.

[0016] FIG. 11 is an exemplary computer system for effectuating the functionalities of the dual camera AR system.

[0017] FIG. 12 is a simplified illustration showing the operation of the processor in overlaying the augmented reality images with the real-time video images along two different perspectives.

[0018] Corresponding reference characters indicate corresponding elements among the view of the drawings. The headings used in the figures do not limit the scope of the claims.DETAILED DESCRIPTION

[0019] This present disclosure discloses an easy-to-use dual-camera augmented reality (AR) navigation system 100 that eliminates the registration step from the clinical workflow without losing the advantages inherent in a conventional AR system when performing medical procedures such as computer tomography (CT) guided percutaneous needle placement. As shown in FIGS. 1-12, the dual-camera AR navigation system 100 may include two cameras 102 and 104 positioned at an angle relative to each other such that two different perspective views of a body 106 being scanned are presented to the user and allows for imaging and visualization of the scanned body 106 and needle 10 without the need for subsequent registrations or use of fiducials or skin markers. Moreover, the AR navigation system 100 includes a processor 220 that overlays virtual reality images over the real-time video images of the cameras 102 and 104. In particular, the processor 220 in the virtual realityimages may generate a virtual target 122 within the anatomy of the body 106 and a virtual trajectory line 120 defined between the needle 10 and the virtual target 122 along first and second perspective views of the cameras 102 and 104 shown on a display 230, thereby enabling adjustment of the needle path in real time by the user according to each camera’s 102 and 104 perspective. In one embodiment, the cameras 102 and 104 may be positioned in substantially orthogonal relation to each other to provide an optimal stereoscopic view using two different perspective views when performing a medical procedure.OVERVIEW

[0020] As illustrated in FIG. 1 , in one embodiment the dual-camera AR guidance system 100 includes a first camera 102 and a second camera 104 positioned in substantially orthogonal relation relative to each other and in visual communication with a body 106 which may be positioned on a CT gantry 108 of a CT scanner 110 for imaging the body 106. In one aspect, the first camera 102 may be aligned along the sagittal plane of the body 106 and the second camera 104 may be aligned along the axial plane of the body 106.

[0021] In one aspect, the first camera 102 generates real-time video images along the sagittal plane in a first perspective view and the second camera 104 generates real-time video images along the axial plane in a second perspective view in which the real-time video images are communicated to a processor 220 in operative communication with a display 230. The display 230 is operative for displaying the real-time video images along the sagittal plane side-by-side with the real-time videos along the axial plane so that a stereoscopic view is displayed to the user.

[0022] In a further aspect shown in FIG. 12, the processor 220 may overlay the augmented reality images along the sagittal plane on the display with the real-time video images along the sagittal plane. The processor 220 may also overlay the augmented reality images along the axial plane on the display with the real-time video images along the axial plane. As shown in FIG. 6A and 6B, the AR guidance processes / services 214 is operable for generating in the augmented reality images along the sagittal and axial planes a virtual trajectory line 122 for defining a trajectory between a virtual surgical target 120 and a needle 10 held by the user. In operation, the virtual trajectory line 122 is continually adjusted by the processor 220 based onthe orientation and position of the needle 10 relative to the virtual surgical target 120. This allows the user to manually adjust the position of the needle 10 along an optimal pathway by following the virtual trajectory line 122 when viewing the two different perspective views shown side-by-side on the display 230. This side-by-side arrangement of the two perspective views on the display 230 collectively provide the user with a stereoscopic view of the body 106 including the needle 10, virtual trajectory line 122, and virtual surgical target 120 to better manipulate the needle 10 during the surgical procedure.TESTINGDual Camera AR Guidance System

[0023] In one example, the dual-camera AR guidance system 100 may include a pair of first and second cameras 102 and 104 (Logitech C920x HD Pro Webcams) mounted on the CT scanner 110 with first camera 102 positioned on one side of the CT gantry 108 that captures the sagittal plane of the body 106 being imaged and the second camera 104 positioned on another side substantially orthogonal to the first camera 102 that captures the axial plane of the body 106. While it is preferable that the first and second cameras 102 and 104 be positioned in orthogonal relation to each other, it is not required that they be precisely orthogonal. As shown in FIG. 2, a custom checkerboard calibration template was designed with multiple CT-visible fiducial markers (n > 3) positioned at known locations along the checkerboard. By analyzing the images from the first and second cameras 102 and 104 and conducting a three-dimensional scan of the fiducial markers, the relative positions of the first and second cameras 102 and 104 can be calculated for calibration purposes. Calibration of the AR system 100 need only be performed once, provided the first and second cameras 102 and 104 remain fixed to the CT gantry 108 or another stable surface on or proximate to the CT scanner 110.

[0024] In one embodiment, the AR navigation system 100 includes a processor 112 (FIG. 11) for operating an AR image guidance module 214 that interfaces with the first and second cameras 102 and 104 and the CT scanner 110, thereby enabling the most recent CT scan from an interventional procedure to be automatically overlaid onto the real-time video feeds from the first and second cameras 102 and 104. In one aspect, the AR navigation system 100 can account for and compensates for the motion of the CT table to enhance the accuracy of the ARimage overlays. In one example, angle views from each of the first and second cameras 102 and 104 along with the augmented reality images overlays are live streamed next to each other on one mobile monitor that is placed opposite of the user’s position at the CT table (FIG. 3). Using the perspectives of the sagittal and axial cameras 102 and 104 as well as the augmented reality projection of the segmented target and anatomy over a phantom body, the desired angulation of a biopsy needle (17 G, 11-17 cm) can be achieved by aligning the biopsy needle with the target within the body 106 on the respective angles of the both the first and second cameras 102 and 104. In one step, a customized, automatic needle segmentation was included which automatically identified the needle shaft and shape and extended the needle trajectory virtually in the augmented reality overlay which is superimposed upon both real time perspective images of the first and second cameras 102 and 104, thereby facilitating proper alignment of the needle with the virtual surgical target shown in the augmented reality overlay with the two real time video images from the first and second cameras 102 and 104.Phantom Model

[0025] During testing, an anthropomorphic abdominal phantom (Model 057A, SIRS, Inc., Norfolk, VA) with six pre-existing, CT-visible target lesions having diameters ranging from 6.5 mm to 15.0 mm was used. Corresponding skin entry points were predetermined to force out-of-plane needle insertions with a variety of insertion depths having a range of 5.7 cm to 14.5 cm and a variety of needle angulations. The six paired targets and skin insertion sites were identical for all users and insertion techniques. Pre-procedurally, the anatomy of the abdominal phantom and target lesions were segmented using a three-dimensional slicer and uploaded to a customized software.Baseline System Accuracy

[0026] To establish a baseline system accuracy of the AR navigation system 100 without accounting for human errors, five needles were randomly inserted into the abdominal phantom. Each needle was identified in the axial and sagittal feeds of the sagittal and axial cameras 102 and 104, and each needle’s 3D position relative to the CT gantry was triangulated based on camera calibrations of the sagittal and axial cameras 102 and 104. A CT scan of the abdominal phantomwas then performed to serve as the ground truth, thereby validating the positions of the needles by the sagittal and axial cameras 102 and 104.Freehand

[0027] For freehand needle insertions, users individually planned needle trajectories and depth for each target lesion on a preprocedural CT phantom images. Interval CT imaging during needle placement was not permitted to equalize radiation exposure among cohorts. Needle placement was then performed cognitively as a single pass. It is important to note that the AR navigation system 100 enables the physician to select the entry point on the fly, without prior planning. In experiments that were carried out, the same entry points were used for both freehand and AR-guided approaches to ensure a fair comparison.Phantom Procedure

[0028] During testing, four interventional radiologists each targeted all six pre-determined target lesions in the abdominal phantom in three cohorts: dual camera-based navigation (FIGS. 5A and SB, “Cam-Nav” hereafter) (n=6), dual camera-based navigation with automatic needle trajectory definition (FIGS. GA and 6B, “Path-Nav” hereafter) (n=6), and CT-guided freehand (n=6) for a total of 18 needle insertions per operator. The order of target insertions within each cohort as well as the order of the cohorts was randomized among operators. After 3 consecutive needle placements, a CT scan was acquired, followed by removal of the three needles from the abdominal phantom and subsequent needle insertions. The primary endpoints were placement accuracy (defined as the shortest distance between the needle and the center of the spherical target), angle error (defined as the deviation from the ideal angle to the target), and placement time (time from needle path planning until final position after needle insertion). Placement accuracy and angle error were determined using the 3D Slicer. The x, y, and z coordinates of each target center and needle trajectory were recorded and the closest distance calculated using the python console in 3D Slicer.Statistical Analysis

[0029] All statistical tests were computed in the open-source software R (R Foundation for Statistical Computing, Vienna, Austria. URL https: / / www.R-project.org / ). The Shapiro-Wilk normality test revealed a non-gaussian distribution of placement accuracy, angle error, and placement time. Descriptive statistics were presented as median and interquartile range (IQR). The Kruskal-Wallis test and Wilcoxon rank sum test were performed to compare operators (n=4) and study cohorts (n=3) reported with p-values. A post hoc Dunn’s test was used to adjust the p-values in multiple comparisons using Bonferroni-Holm correction criteria.Baseline Accuracy

[0030] By comparing the needle positions determined by the first and second cameras 102 and 104 with those from the CT scan, the baseline accuracy of the dual-camera AR system 100 was assessed, yielding an angular error of 0.64° ± 0.23° and a positional error of 1 .3 ± 2.1 mm.Placement Accuracy

[0031] Median needle placement accuracy using dual camera-based navigation (Cam-Nav), dual camera navigation with automatic needle trajectory definition (Path-Nav), and freehand were 5.3 mm (3.8, 8.3), 5.1 (2.3, 6.1), and 16.7 (7.7, 26.6), respectively (FIG. 7, Table 1). Post hoc Bonferroni-Holm correction for multiple comparisons demonstrated no difference in needle placement accuracy between Cam-Nav and Path-Nav (p=0.35). Placement accuracy of Cam-Nav and Path-Nav were greater compared to freehand (both p<0.001 ).Angle Error

[0032] Median angle errors for Cam-Nav, Path-Nav, and freehand were 3.3 degrees (2.3, 4.5), 2.6 (2.3, 3.8), and 9.7 (6.1 , 15.3), respectively (FIG. 8, Table 1). Post hoc Bonferroni-Holm correction for multiple comparisons demonstrated no difference in angle error between Cam-Nav and Path-Nav (p=0.43). Angle error of Cam-Nav and Path-Nav were greater compared to freehand (both p<0.001 ).Placement Time

[0033] Median needle placement times using Cam-Nav, Path-Nav, and freehand were 32 sec (21 , 45), 36 sec (26, 60), and 86 sec (65, 114), respectively (FIG. 9, Table 1). Post hoc Bonferroni-Holm correction for multiple comparisons demonstrated no difference in placement times between Cam-Nav and Path-Nav(p=0.29). Angle error of Cam-Nav and Path-Nav were greater compared to freehand (both p<0.001).Discussion

[0034] As noted above, the dual-camera AR navigation system 100 does not require fiducials or electromagnetic tracking. In testing, Cam-Nav and Path- Nav solutions both were more accurate than the freehand single pass estimation technique. Although not statistically different, the median accuracy was also enhanced by the addition of an automatically detected and defined needle trajectory such that the operator could adjust iteratively during insertion to maintain a satisfactory trajectory in real time, with continuous adjustment during insertion and dual plane feedback. The dual-camera AR navigation system 100 accomplished steady navigation in a fashion similar to cone beam computed tomography (CBCT) having two orthogonal views with the first view being down the shaft of the needle and the second view being oblique to needle; however, CBCT paths require movement of the detector for each perspective, thereby viewing only one perspective and planned needle path at a time.

[0035] Once the dual-camera AR navigation system 100 and AR image guidance software are fully integrated with a CT scanner 110, the dual-camera AR navigation system 100 functions as a convenient accessory on demand to assist the operator when needed. The dual-camera AR navigation system 100 does not significantly alter the standard workflow, modify interventional devices, or introduce disposable costs. The transition between the AR navigation system and conventional methods is instantaneous and seamless.

[0036] The first and second cameras 102 and 104 of the AR navigation system 100 provide camera perspectives familiar to radiologists performing interventions. The position of the first camera 102 depends on which side of the CT table the physician chooses to operate from. Therefore, mounting a first camera 102 on each side of the CT gantry 108 could expand the applications of the AR navigation system 100. While the axial camera 104 could be mounted on the CT table instead of the CT gantry 108, this setup might complicate adjustments to the distance between the second camera 104 and the worksite.

[0037] The evaluation was conducted using a stationary phantom. Variables not studied here, such as patient movement, respiratory motion, and organ deformation are important factors that can affect navigational performance.

[0038] Although the dual-camera AR navigation system 100 does not directly address these motion-related issues, its registration-free design can provide image guidance in augmented reality based on the most recent intraoperative scan. This scan likely more accurately reflects the target's current position compared to the initial planning scan, thereby helping to mitigate the effects of target motion on image guidance of the needle. If the needle is off target due to motion or other factors, an intraoperative CT scan can confirm the target's position relative to the needle. The AR guidance processes / services 214 can then overlay the previous needle track of the confirmation scan onto the real-time video feed of each camera 102 and 104, thereby providing the physician with immediate feedback to adjust the needle trajectory using the old track as a reference in augmented reality (FIGS. 10A and 10B).

[0039] In conclusion, dual-camera needle navigation of the AR navigation system 100 provides an ergonomically friendly and low-cost tool in comparison to other conventional AR systems, such as goggle-based tools. Improved accuracy leads to improved outcomes for needle-based CT-guided procedures.Table 1« / x . / . x Placement timeAccuracy (mm) Angle error (degrees)Median (IQR) Median (IQR) Median'lQR)Cam-Nav 5.3 (3.8, 8.3) 3.3 (2.3, 4.5) 32 (21, 45)Path-Nav 5.1 (2.3, 6.1) 2.6 (2.3, 3.8) 36 (26, 60)Freehand 16.7 (7.7, 26.6) 9.7 (6.1 , 15.3) 86 (65, 114)Cam-Nav = camera-based augmented reality; Path-Nav = camera-based augmented reality with automatic needle segmentation, Freehand = single, cognitive needle insertion without navigation system, IQR = interquartile range.

[0040] FIG. 11 is a schematic block diagram of an example computer device 200 that may be used with one or more embodiments described herein, e.g.,as a component of dual-camera AR navigation system 100 for executing its functionalities including the AR image guidance processes / services 214.

[0041] Device 200 comprises one or more network interfaces 210 (e.g., wired, wireless, PLC, etc.), at least one processor 220, and a memory 240 interconnected by a system bus 250, as well as a power supply 260 (e.g., battery, plug-in, etc. Device 200 can also include or otherwise communicate with a display interface device 230 which can include one or more input / output devices that enable a user to input data, and to view or otherwise access output data. Input / output devices can include but are not limited to a monitor, a touch-screen, a speaker, a keyboard, a mouse, and the like.

[0042] Network interface(s) 210 include the mechanical, electrical, and signaling circuitry for communicating data over the communication links coupled to a communication network. Network interfaces 210 are configured to transmit and / or receive data using a variety of different communication protocols. As illustrated, the box representing network interfaces 210 is shown for simplicity, and it is appreciated that such interfaces may represent different types of network connections such as wireless and wired (physical) connections. Network interfaces 210 are shown separately from power supply 260, however it is appreciated that the interfaces that support PLC protocols may communicate through power supply 260 and / or may be an integral component coupled to power supply 260.

[0043] Memory 240 includes a plurality of storage locations that are addressable by processor 220 and network interfaces 210 for storing software programs and data structures associated with the embodiments described herein. In some embodiments, device 200 may have limited memory or no memory (e.g., no memory for storage other than for programs / processes operating on the device and associated caches).

[0044] Processor 220 comprises hardware elements or logic adapted to execute the software programs (e.g., instructions) and manipulate data structures 245. An operating system 242, portions of which are typically resident in memory 240 and executed by the processor, functionally organizes device 200 by, inter alia, invoking operations in support of software processes and / or services executing on the device. These software processes and / or services may include AR image guidance processes / services 214 described herein. Note that while AR image guidance processes / services 214 is illustrated in centralized memory 240, alternativeembodiments provide for the process to be operated within the network interfaces 210, such as a component of a MAC layer, and / or as part of a distributed computing network environment.

[0045] It will be apparent to those skilled in the art that other processor and memory types, including various computer-readable media, may be used to store and execute program instructions pertaining to the techniques described herein. Also, while the description illustrates various processes, it is expressly contemplated that various processes may be embodied as modules or engines configured to operate in accordance with the techniques herein (e.g., according to the functionality of a similar process). In this context, the term module and engine may be interchangeable. In general, the term module or engine refers to model or an organization of interrelated software components / functions. Further, while the AR image guidance processes / services 214 is shown as a standalone process, those skilled in the art will appreciate that this process may be executed as a routine or module within other processes.

[0046] It should be understood from the foregoing that, while particular embodiments have been illustrated and described, various modifications can be made thereto without departing from the spirit and scope of the invention as will be apparent to those skilled in the art. Such changes and modifications are within the scope and teachings of this invention as defined in the claims appended hereto.

Claims

CLAIMSWhat is claimed is:

1. An augmented reality system comprising: a first camera and a second camera in operative association with a computer tomography (CT) scanner and in visual communication with a body, the first camera being positioned at an angle relative to the second camera with the first and second cameras being operable for providing first and second real-time video images along first and second perspectives views, respectively; a processor in operative communication with the first camera, the second camera, and the CT scanner, the processor in communication with a memory for executing instructions in memory for: receiving the first real-time video images of the body from the first camera and receiving the second real-time video images of the body from the second camera; overlaying augmented reality images from the first perspective view with the first real time video images received from the first camera and overlaying augmented reality images along the second perspective view with the second real-time video images from the second camera; and displaying the overlaid augmented reality images along the first perspective view with the first real-time video images and displaying the overlaid augmented reality images along the second perspective view with the second real-time video images to generate a stereoscopic view.

2. The augmented reality system of claim 1 , wherein the first and second realtime video images provide an image of a needle viewed along the first and second perspective views, respectively, and wherein the overlaid augmentedreality images viewed along the first and second perspectives views provide a virtual trajectory line of the needle.

3. The augmented reality system of claim 2, wherein the virtual trajectory line of the needle is adjusted in the augmented reality images along the first and second perspective views in relation to the movement of the needle in the first and second real-time video images along the first and second perspective views.

4. The augmented reality system of claim 1 , wherein the first camera is aligned along the sagittal plane of the body and the second camera is aligned along the axial plane of the body.

5. The augmented reality system of claim 1 , wherein the first camera is positioned in orthogonal relation to the second camera.

6. The augmented reality system of claim 1 , wherein the augmented reality video images along the first and second perspective views provide an image of a virtual surgical target and a virtual trajectory line defined between the needle shown in the first and second real-time video images and the virtual surgical target shown in the overlaid augmented reality images along the first and second perspective views.

7. The augmented reality system of claim 6, wherein movement of the needle in the first and second real-time video images causes an adjustment of the virtual trajectory line relative to the virtual surgical target in the overlaid augmented reality images along the first and second perspective views.

8. The augmented reality system of claim 7, wherein the first and second realtime video images and the overlaid augmented reality images from the first and second perspective views are displayed side by side to provide the stereoscopic view of the body to the user.The augmented reality system of claim 7, wherein the virtual trajectory line is represented by a straight line defined between the needle and the virtual surgical target.

Images