Live 3D holographic guidance and navigation for performing interventional procedures

Through a live 3D holographic guidance and navigation system, the complexity of interventional instrument tracking and alignment in interventional surgery is solved using head-mounted devices and augmented reality technology, and the manipulation and accuracy of interventional surgery are improved.

CN112423692BActive Publication Date: 2024-12-06THE CLEVELAND CLINIC FOUND
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980045359.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-05-07
Filing Date
2019-05-10
Publication Date
2024-12-06
Estimated Expiration
2039-05-10

AI Technical Summary

Technical Problem

In interventional surgery, tracking interventional instruments using ultrasound is complicated, especially the difficulty of alignment with the target position after the interventional instrument enters the patient's body increases, and traditional 2D displays reduce the manipulation of the interventional instrument.

Method used

Using a live 3D holographic guidance and navigation system, ultrasonic image stream and preoperative CT image data are received through a head-mounted device, combined with a head tracking mechanism, the position and path of the interventional instrument in the 3D space is displayed in real time, and augmented reality technology will be used to intervene. The instrument is projected consistently with the anatomy for alignment and navigation.

Benefits of technology

It reduces the alignment complexity between the interventional instrument and the target position, improves the manipulation and accuracy of the interventional instrument, adapts to changes in anatomical structure, and simplifies the interventional surgical process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112423692B_ABST
    Figure CN112423692B_ABST
Patent Text Reader

Abstract

Interventional procedures can be performed less invasively using live 3D holographic guidance and navigation. Advantageously, 3D holographic guidance and navigation overcomes the limitations of conventional display of medical images on standard 2D flat screens. Live 3D holographic guidance can provide a complete holographic view of a portion of a patient's body using 3D holographic visualization derived from complementary imaging modalities (e.g., ultrasound and computed tomography) to enable navigation of tracked interventional instruments / devices.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 667,687, filed May 7, 2018, entitled “REAL-TIME (LIVE) 3D HOLOGRAPHIC GUIDANCE AND NAVIGATION FOR PERFORMING INTERVENTIONAL PROCEDURES”. This application also claims the benefit of U.S. Patent Application No. 16 / 405,875, filed May 7, 2019, entitled “LIVE 3D HOLOGRAPHIC GUIDANCE AND NAVIGATION FOR PERFORMING INTERVENTIONAL PROCEDURES”. The entireties of these applications are hereby incorporated by reference in their entirety for all purposes. Technical Field

[0003] The present disclosure relates generally to interventional procedures and, more particularly, to systems and methods that provide live three-dimensional (3D) holographic guidance and navigation for performing interventional procedures. Background Art

[0004] Image guidance generally refers to the tracking of the interventional instrument / device used for medical surgery through the patient's anatomical structure to the target position during the interventional surgery. The patient's anatomical structure is represented by preoperative and / or intraoperative images, and the tracked interventional instrument / device is registered to the preoperative and / or postoperative images. For a wide range of diagnostic and therapeutic operations, ultrasound examination can be used to track the interventional instrument / device. However, using ultrasound examination to track the interventional instrument / device complicates image guidance. First, ultrasound examination can only be used to track the interventional instrument / device in the patient's body, and cannot track the interventional instrument / device before the interventional instrument / device enters the patient's body. Once entering the patient's body, the relationship between the ultrasound probe, the interventional device / device, the target position and the imaging plane of the ultrasound probe may be unclear, which complicates the alignment of the interventional instrument / device with the target position. In addition, traditionally, the image of the target position and the interventional instrument / device is displayed on a flat 2D monitor on the side of the table, which further complicates the maneuverability of the interventional instrument. For example, a 2D display enables a medical professional to translate the position of an instrument / device and the trajectory of the instrument / device relative to a target position into the physical trajectory adjustments required to correct the path of the instrument; in medical surgery, this mental translation from 2D to 3D is quite difficult. Summary of the invention

[0005] The present disclosure relates to systems and methods that provide live three-dimensional (3D) holographic guidance and navigation for performing interventional procedures.

[0006] In one aspect, the present disclosure may include a method for providing live 3D holographic guidance for performing an interventional procedure. The method may be performed by a head-mounted device including a processor and a head tracking mechanism, which head-mounted device may receive live tracking data represented in 3D Cartesian coordinates of a navigation system. The live tracking data may include a tracked physical ultrasound transducer / probe connected to an ultrasound system, a tracked physical interventional device / instrument, and the position and orientation of a physical reference position sensor at a specific anatomical location on a physical patient. The physical head-mounted device may receive a live image stream acquired by a physical ultrasound transducer / probe connected to the ultrasound system; transform the live tracking data into a head-mounted viewer coordinate system; and display a live holographic projection of the live image stream in the head-mounted viewer coordinate system. The ultrasound image stream extends from the tracked physical ultrasound probe / transducer at the tracked position and tracked orientation of the probe / transducer. The live holographic projection is aligned with an operator's view evaluated using the head tracking mechanism. Figure 1 The physical head mounted device can then retrieve a digital anatomical object derived from preoperative 3D computed tomography (CT) image data of the physical patient; transform the digital anatomical object from the 3D CT coordinate system to the head mounted view coordinate system; translate the anatomical object in the head mounted view coordinate system by a 3D vector calculated based on the 3D point position on the live holographic projection of the live image stream and the corresponding point within the preoperative CT image data stored in the head mounted view coordinate system to correct for anatomical changes in the physical patient; and display a holographic visualization including a holographic representation of the tracked physical interventional device / instrument consistent with the registered holographic projection of the live ultrasound image stream and the holographic anatomical object derived from CT. From the viewpoint determined by the head tracking mechanism, the three holographic projections are used to navigate the tracked physical ultrasound probe and guide the physical interventional device / instrument to the treatment target.

[0007] In another aspect, the present disclosure may include a system for providing live 3D holographic guidance for performing an interventional procedure. The system includes a computing device that includes a memory that stores a digital anatomical object derived from preoperative 3D computed tomography (CT) image data of a physical patient. The CT image data is represented in 3D coordinates of a CT coordinate system. The system also includes a head-mounted device, which includes a processor and a head tracking mechanism to: receive live tracking data represented by 3D Cartesian coordinates of a navigation system, wherein the live tracking data includes the position and orientation of a tracked physical ultrasound transducer / probe connected to the ultrasound system, a tracked physical interventional device / instrument, and a physical reference position sensor at a specific anatomical location on a physical patient; receive a live image stream acquired by the physical ultrasound transducer / probe connected to the ultrasound system; transform the live tracking data into a head-mounted viewer coordinate system; display a live holographic projection of the live image stream in the head-mounted viewer coordinate system, wherein the ultrasound image stream extends from the tracked physical ultrasound probe / transducer at the tracked position and tracked orientation of the probe / transducer, and wherein the live holographic projection is consistent with an operator's view evaluated using the head tracking mechanism. Figure 1 The invention relates to a method of providing a holographic visualization of a patient's condition and a holographic image of a live ultrasound image stream, wherein the holographic visualization comprises a holographic representation of a tracked physical interventional device / instrument consistent with the registered holographic projection of the live ultrasound image stream and the holographic anatomical object derived from the CT, wherein the three holographic projections are used to position the tracked physical ultrasound probe and guide the physical interventional device / instrument to a treatment target from a viewpoint determined by a head tracking mechanism. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The foregoing and other features of the present disclosure will become apparent to those skilled in the art to which the present disclosure relates when reading the following description with reference to the accompanying drawings, in which:

[0009] Figure 1 is a block diagram illustrating an example of a system providing live three-dimensional (3D) holographic guidance and navigation for performing interventional procedures according to one aspect of the present disclosure;

[0010] Figure 2 It shows Figure 1 A block diagram of an example of a head mounted device;

[0011] Figure 3 and Figure 4is a process flow diagram of an example method for providing live 3D holographic guidance and navigation for performing interventional procedures according to another aspect of the present disclosure;

[0012] Figure 5 is an image of a component used for live 3D holographic guidance;

[0013] Figure 6 is an image showing an example use of live 3D holographic guidance; and

[0014] Figure 7 is an image of an example head mounted device. DETAILED DESCRIPTION

[0015] I. definition

[0016] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs.

[0017] In the context of the present disclosure, the singular forms "a", "an" and "the" may also include plural forms, unless the context clearly indicates otherwise.

[0018] As used herein, the term “comprising” may specify the presence of stated features, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups.

[0019] As used herein, the term "and / or" may include any and all combinations of one or more of the associated listed items.

[0020] In addition, although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. Thus, without departing from the teachings of the present disclosure, the "first" element discussed below may also be referred to as the "second" element. Unless otherwise specifically noted, the order of operations (or actions / steps) is not limited to the order presented in the claims or drawings.

[0021] As used herein, the term "interventional procedure" refers to a medical procedure for diagnosis or treatment that involves entering the patient's body. An interventional procedure can be a percutaneous nonvascular procedure (e.g., nerve block, biopsy, tumor ablation, etc.), a percutaneous vascular procedure (e.g., stenting, virtual histology, blood flow reserve fraction, etc.), or an open surgical procedure (e.g., mitral valve replacement, tricuspid valve replacement, minimally invasive surgery, etc.).

[0022] As used herein, the terms "interventional instrument / device" and "interventional device / instrument" refer to any tool used within a patient's body to facilitate an interventional procedure.

[0023] As used herein, the term "tracking data" refers to information related to observations of one or more objects measured by a navigation system in a tracking coordinate system, and may or may not be experiencing motion. The objects may include a tracked physical ultrasound transducer / probe connected to an ultrasound system, a tracked physical interventional device / instrument, a physical fiducial position sensor, etc.

[0024] As used herein, the term "tracking coordinate system" refers to a 3D Cartesian coordinate system that uses one or more numbers to determine the location of points or other geometric elements that are unique to a particular tracking system. For example, the tracking coordinate system can be rotated, scaled, etc. from a standard 3D Cartesian coordinate system.

[0025] As used herein, the term "head mounted device" or "head mounted viewer" refers to a display device configured to be worn on the head, having one or more display optics (including lenses) in front of one or more eyes. An example of a head mounted device is Microsoft HoloLens.

[0026] As used herein, the term "head mounted view coordinate system" or "world coordinate system" refers to a 3D Cartesian coordinate system that uses one or more numbers to determine the location of points or other geometric elements that are unique to a particular head mounted device system. For example, the head mounted view coordinate system can be rotated, scaled, etc. from a standard 3D Cartesian coordinate system.

[0027] As used herein, the term "imaging stream" refers to a real-time ultrasound image of a portion of a patient's body.

[0028] As used herein, the term "anatomical object" refers to a discrete portion of a preoperative CT image of a portion of a patient's body. In some cases, the anatomical object has been obtained from the original preoperative CT image.

[0029] As used herein, the term "CT coordinate system" refers to a 3D Cartesian coordinate system that uses one or more numbers to determine the location of points or other geometric elements that are unique to a particular CT imaging system. For example, the imaging coordinate system can be rotated, scaled, etc. from a standard 3D Cartesian coordinate system.

[0030] As used herein, the term "hologram", "holographic projection" or "holographic representation" refers to a computer-generated image projected into the lenses of a head-mounted viewer. Typically, a hologram can be synthetically generated (in augmented reality (AR)) and has no relation to physical reality.

[0031] As used herein, the term "physical" refers to real things. Physical things are not holographic (or not computer-generated).

[0032] As used herein, the term "two-dimensional" or "2D" refers to things that are represented in two physical dimensions.

[0033] As used herein, the term "three-dimensional" or "3D" refers to things that are represented in three physical dimensions. The definition of three-dimensional or 3D will include elements of "4D" (eg, 3D plus the dimension of time).

[0034] As used herein, the terms "real time" and "live" refer to the actual time at which a process or event occurs. In other words, a real-time event is performed live (within milliseconds, so that the results are immediately available as feedback). For example, a real-time event can be represented within 100 milliseconds of the event occurring.

[0035] As used herein, the term "depth" may refer to an indication of how deep within a patient's body an image is (eg, in centimeters). Depth may be related to scale.

[0036] As used herein, the terms "subject" and "patient" are used interchangeably and refer to any vertebrate animal.

[0037] II. Overview

[0038] The present disclosure generally relates to interventional procedures in which interventional instruments / devices are guided or navigated through a patient's body. Traditionally, the use of ultrasound examination (also referred to herein as "ultrasound", "ultrasound scanning", "echo" and similar terms) to track interventional instruments / devices complicates image guidance, at least because ultrasound examination can only be used to track interventional instruments / devices within a patient's body, and cannot track interventional instruments / devices before entering the patient's body; once entering the patient's body, the relationship between the ultrasound probe, the interventional instrument / device, the target position, and the imaging plane of the ultrasound probe may be unclear, which complicates the alignment of the interventional instrument / device with the target position; and, traditionally, the target position and the image of the interventional instrument / device are displayed on a flat 2D monitor on the side of the table, which further complicates the maneuverability of the interventional instrument (e.g., requiring medical professionals to translate the position of the instrument / device and the trajectory of the instrument / device relative to the target position into physical trajectory adjustments required to correct the path of the instrument). The present disclosure reduces the traditional complexity caused by the use of ultrasound examination to track interventional instruments / devices through a patient's body. In fact, the present disclosure describes a system and method for providing live three-dimensional (3D) holographic guidance and navigation for performing ultrasound-guided interventional procedures.

[0039] The 3D holographic visualization may include a holographic representation of the tracked interventional instrument / device displayed in unison with the holographic projection of the live ultrasound image stream and the holographic anatomical objects derived from the preoperative images (all data transformed into a common holographic coordinate system). Notably, the anatomical objects may be translated to accommodate anatomical changes (e.g., breathing) of the patient by 3D vectors calculated based on 3D point positions on the live holographic projection of the live ultrasound image stream (also referred to as the "live image stream") and corresponding points within the stored preoperative CT image data in the head mounted view coordinate system.

[0040] III. system

[0041] One aspect of the present disclosure may include a system 10 ( Figure 1 ) that provides live three-dimensional (3D) holographic guidance and navigation for interventional procedures performed on a patient (also referred to as a "physical" patient). An interventional procedure can be any procedure that uses an imaging modality (e.g., ultrasonography, computed tomography, etc.) to provide guidance. In particular, 3D holographic guidance and navigation of interventional instruments can involve the fusion of two imaging modalities - the fusion of live ultrasound images and pre-operative computed tomography imaging. For example, ultrasonography can be used to guide any interventional procedure for diagnosis or treatment that involves entering the patient's body, including percutaneous non-vascular procedures (e.g., nerve blocks, biopsies, tumor ablations, etc.), percutaneous vascular procedures (e.g., stenting, virtual histology, fractional flow reserve, etc.), and open surgical procedures (e.g., mitral valve replacement, tricuspid valve replacement, minimally invasive surgery, etc.). Live 3D holographic guidance and navigation can be provided for any procedure that uses ultrasonography to guide an interventional device.

[0042] The system 10 may provide live 3D holographic guidance and navigation using augmented reality 3D holographic displays, which may replace or otherwise enhance traditional 2D guidance. The system 10 may include a head mounted device 1, which may be configured to generate an augmented reality 3D holographic display based on patient-specific and interventional procedure-specific data received from computing devices 13 and 14. It should be noted that computing devices 13 and 14 may be separate devices (both local to the head mounted device 1, one local to the head mounted device 1 and the other remote from the head mounted device 1, both remote from the head mounted device 1, etc.), but may be part of the same device. As an example, the head mounted device may be an optical see-through device.

[0043] The computing device 13 may receive data from the ultrasound system 12 (live image streaming data) and the navigation system 11 (tracking data). The computing device 13 may be coupled to the ultrasound system 12 and the navigation system 11 according to a wired connection and / or a wireless connection. The computing device 13 may also be coupled to the head mounted device 1 according to a wired connection and / or a wireless connection. It should be understood that the connections between the computing device 13 and the navigation system 11, the ultrasound system 12, and the head mounted device may be independent of each other.

[0044] The ultrasound system 12 can send ultrasound signals to an ultrasound transducer / probe (also referred to as a "physical" ultrasound transducer / probe) and receive a live image stream from the ultrasound transducer / probe T / P during an interventional procedure, and provide the live image stream to the computing device 13. For example, the ultrasound transducer / probe T / P can be a B-type probe, a linear probe, an intravascular ultrasound probe, or any other type of ultrasound transducer or probe. The live image stream can be 2D. The computing device 13 can be coupled to the ultrasound system 12 according to a wired connection and / or a wireless connection configured for real-time data transmission of the live image stream. For example, the live data stream can be transmitted between the ultrasound system 12 and the computing device 13 according to an HDMI connection.

[0045] The navigation system 11 can receive a signal including live tracking data associated with the tracking devices T / PS, IDS, and LS (also referred to as "physical" tracking devices) and send the live tracking data to the computing device 13. One or more tracking devices T / PS can be on and / or within the ultrasound transducer / probe T / P. One or more tracking devices IDS can be on and / or within the interventional device ID used during the interventional procedure. One or more tracking devices LS can be located at a fixed point on and / or near the patient's body. For example, one or more tracking devices LS can be at a reference position on the patient's body. As another example, one or more tracking devices LS can be at a position outside the patient's body. As another example, one or more tracking devices LS can be at a reference position on the patient's body and at a position outside the patient's body. The tracking data can include position information and orientation information (each of which is three-dimensional) from each of the multiple tracking devices T / PS, IDS, and LS. The navigation system 11 can include a component for generating a signal for tracking (e.g., based on the signal from the component, tracking data can be generated). For example, the component may include an electromagnetic (EM) field generator that can generate an electromagnetic field, and the tracking devices T / PS, IDS, and LS, which may be sensors (e.g., coil-type sensors), respond by generating tracking data. As another example, the component may include an optical generator, and the tracking devices T / PS, IDS, and LS, which may be reflective markers, may be optically tracked to provide tracking data. The computing device 13 may be coupled to the navigation system 11 according to a wired connection and / or a wireless connection configured for real-time data transmission of tracking data. For example, the tracking data may be transmitted between the navigation system 11 and the computing device 13 according to a serial connection. It should be noted that the navigation system 11 described herein may utilize tracking using a fiducial marker. It should be understood that other navigation and tracking mechanisms may be used without departing from the spirit of the present disclosure.

[0046] The computing device 14 can provide preoperative data related to the patient's anatomical structure (e.g., anatomical objects AO derived from preoperative computed tomography (PCT) images and / or preoperative computed tomography (PCT) images by image segmentation) to the head-mounted device 1. For example, the anatomical object AO can be a CT-based hologram of a part of the patient's body. The computing device 14 can be connected to the head-mounted device according to a wired connection and / or a wireless connection. In some cases, the computing device 14 (or another computing device) can provide information related to a treatment plan to the head-mounted device 1, which treatment plan is related to an interventional procedure. In response to the treatment plan, the head-mounted device 1 can provide a projection to the surgical site, a dynamically aligned, planned and tracked HLR, and a holographic area for the occurrence of the interventional procedure.

[0047] The head mounted device 1 is shown in Figure 2 . The head-mounted device 1 may include a non-transitory memory 2 and a processing unit 3 (which may include one or more hardware processors), which may access the non-transitory memory 2 and execute instructions stored thereon to assist in the generation and display of a holographic display. For example, the non-transitory memory 2 may store instructions such as transformations 7, corrective translations 8 (or adjustments), and holographic projections 9, which may be executed by the processing unit 3. The head-mounted device 1 may also include an input / output (I / O) 4 for bidirectional communication with computing devices 13 and 14. In addition, the head-mounted device 1 may include a projector, which may facilitate the display of 3D holograms. Further, the head-mounted device 1 may also include a head tracking mechanism 6, which may help perform various actions based on the movement of the head of a person wearing the head-mounted device 1. The head-mounted device 1 may also include other components, such as cameras and / or other visualization and / or recording elements.

[0048] An example of the operation of the head mounted device 1 is described below. Through I / O 4 (which may be a wireless transmitter and / or receiver), the head mounted device 1 may receive live tracking data and a live image stream from a computing device 13, and preoperative data (e.g., anatomical object AO derived from a preoperative computed tomography PCT image and / or a preoperative computed tomography scan PCT and / or a treatment plan) from a computing device 14. The live tracking data is in the form of coordinates of a tracking coordinate system. At least a portion of the preoperative data is typically in the form of coordinates of a CT coordinate system.

[0049] The transformation 7 instructions stored in the non-transitory memory 2 and executed by the processing unit 3 can transform all received data into a common coordinate system (head mounted view coordinate system). For example, live tracking data can be transformed from the coordinates of the tracking coordinate system to the common coordinates of the head mounted view coordinate system; preoperative data can be transformed from the coordinates of the CT coordinate system to the common coordinates of the head mounted view coordinate system.

[0050] The holographic projection 9 instructions stored in the non-transitory memory 2 and executed by the processing unit 3 can be used to generate an augmented reality 3D holographic display. The augmented reality 3D holographic display can include a live holographic projection of a live holographic image stream in the head-mounted viewer coordinate system. The live image stream can be pre-calibrated using local rotation, translation, and scaling, and automatically recalibrated to maintain consistency with the patient's physical anatomy. For example, a live holographic projection of the live image stream can be extended from a tracked physical ultrasound probe / transducer with the tracked position and tracking orientation of the probe / transducer, and the live holographic projection can be scaled to match the operator's view evaluated using a head tracking mechanism. Figure 1The augmented reality 3D holographic display may also include a holographic projection of an anatomical object registered with the live holographic projection. The anatomical object may be projected to enable visualization without obstructing the holographic ultrasound plane. The augmented reality 3D holographic display may include a holographic representation of a tracked physical interventional device / instrument that is consistent with the registered holographic projection of the live ultrasound image stream, which may also be consistent with the registered holographic projection of the anatomical object.

[0051] A holographic representation of a tracked physical intervention device can be displayed using a reference graphic (which can be based on a treatment plan) and other guidance control graphics related to a surgical site corresponding to at least a portion of the patient's anatomical structure. The reference graphic and guidance control graphics can provide guidance (e.g., visual guidance (illustration, type, annotation, etc.) and / or auditory guidance) to track the physical intervention device through the patient's anatomical structure using holographic guidance (using a 3D anatomical holographic projection and a 3D holographic representation of the intervention device). For example, when a line (or graphic) associated with a reference graphic and a line (or graphic) associated with a guidance control graphic intersect, the physical intervention device can be aligned with a trajectory that will facilitate the placement of the physical intervention device within the vascular system. This can be accompanied by a holographic annotation that reports the distance and / or angle deviation from the target position or orientation. The reference graphic and guidance control graphics can be used to provide event-driven guidance. For example, when a stent is within the patient's vascular system, the reference graphic and guidance control graphics can provide auditory and / or visual guidance as the stent moves. As the stent moves through the patient's vascular tree, a beep can be used to indicate that the stent is close to the target position. Similarly, the graphic may provide real-time annotation of the position and orientation of the support and / or show intersections with target locations. In other words, event-driven guidance may use one or more event-driven signals to notify the user when they are on the right track.

[0052] The projector 5 subsystem can display an augmented reality 3D holographic display. For example, the 3D holographic display can be stereoscopically projected onto the patient in accordance with the patient's physical anatomy as a visualization. The visualization can be scaled and / or moved based on input and / or auditory input from the head tracking mechanism 6. In addition, the viewpoint can be determined based on input from the head tracking mechanism 6 (which can track the head in the head-mounted display 1 based on an accelerometer, a gyroscope, and / or a magnetometer). Upon receiving operator input, the visualization can be displayed as inconsistent with the patient's physical anatomy and / or physical ultrasound transducer / probe and / or physical interventional device through the head-mounted device projection, while maintaining the co-registration of the holographic projection. Upon receiving another operator input, the visualization can be continuously realigned based on tracking data from the head tracking mechanism 6 to redirect the virtual display so that it is always facing the operator. Upon receiving another operator input, the visualization can be translated, rotated, and / or scaled relative to the physical anatomy, and the elements can be co-registered. Rotation of the holographic scene can be limited or prevented to maintain the operator's hand-eye coordination when navigating the ultrasound transducer / probe (T / P) and interventional device (ID). Based on the input, the tracked physical ultrasound probe (T / P) can be used to track and guide the physical interventional device / instrument (ID) to the treatment target (determined according to preoperative planning information).

[0053] The correction translation 8 instructions stored in the non-transitory memory 2 and executed by the processing unit 3 can be used on the preoperative data to correct for anatomical changes of the physical patient (e.g., respiratory motion, total patient motion, etc.). The correction translation 8 instructions assist in performing 3D translation and adjustment for improving the registration between CT and ultrasound. As an example, the translation 8 instructions can be used to increase the registration between the preoperative CT image and the live ultrasound image. The preoperative anatomical object (AO) in the head-mounted view coordinate system can be translated by a 3D vector, which is calculated based on the 3D point position on the live holographic projection of the live image stream in the head-mounted view coordinate system and the corresponding point in the preoperative CT image data in the head-mounted view coordinate system. The point position can be identified and located on the holographic projection of the live ultrasound image based on operator input. For example, the point can be the center of the imaged tumor or blood vessel cross section.

[0054] IV. method

[0055] Another aspect of the present disclosure may include a method for providing live 3D holographic guidance and navigation for performing ultrasound-guided interventional procedures. Figure 3 and Figure 4 The method 30 and the method 40 can be executed by hardware, for example, by the above-mentioned Figure 1 and Figure 2The head mounted device 1 shown performs.

[0056] Methods 30 and 40 are illustrated as process flow diagrams with flowchart illustrations. For simplicity, methods 30 and 40 are shown and described as being performed sequentially; however, it should be understood and appreciated that the present disclosure is not limited by the illustrated order, as some steps can be performed in a different order and / or simultaneously with other steps shown and described herein. Moreover, not all illustrated aspects are required to implement methods 30 and 40. In addition, one or more elements of implementing methods 30 and 40, such as Figure 1 and 2 The head-mounted device 1 may include a head tracking mechanism that can facilitate holographic image guidance, a non-transitory memory, and one or more processors.

[0057] Now refer to Figure 3 , method 30 is shown, which is part of a larger method for providing live 3D holographic guidance and navigation for performing ultrasound-guided interventional procedures. In step 32, live data represented by 3D Cartesian coordinates of the navigation system (e.g., live tracking data from sensors on the ultrasound transducer probe T / PS, sensors on / inside the interventional device IDS, and sensors at the reference position LS provided to the head-mounted device 1 by the navigation system 11) can be received. It should be noted that the navigation system 11 described herein can utilize tracking using reference markers. It should be understood that other navigation and tracking mechanisms can be used without departing from the spirit of the present disclosure. In step 34, the live tracking data can be transformed into a head-mounted viewer coordinate system (by the head-mounted device 1). In step 36, a live image stream (acquired by the ultrasound transducer / probe T / P) can be received (by the head-mounted device 1). At step 38, a live holographic projection of the live image stream in the head mounted view coordinate system (determined using at least a portion of the live tracking data) may be displayed (by the head mounted device 1) together with another live holographic representation of the interventional device in the head mounted view coordinate system.

[0058] exist Figure 4The method 40 of the present invention continues with a method for providing live 3D holographic guidance and navigation for performing ultrasound-guided interventional procedures. In step 42, a digital anatomical object (e.g., AO) derived from preoperative computed tomography (e.g., PCT) image data of a patient (e.g., by image segmentation) may be received (by the head mounted device 1). In step 44, the digital anatomical object (by the head mounted device 1) may be transformed to a head mounted view coordinate system. In step 46, the digital anatomical object in the head mounted view coordinate system may be translated by a 3D vector (by the head mounted device 1) to correct for anatomical changes in the patient. The 3D vector may be based on a 3D point selected by a user on the holographic ultrasound and associated with the holographic anatomical object. In step 48, a 3D holographic visualization may be displayed (by the head mounted device 1) and used for holographic guidance. The 3D holographic visualization may include a 3D representation of an interventional device / instrument registered to a 3D holographic projection of a live ultrasound stream and a 3D holographic projection of a digital anatomical object. The visualization may be stereoscopically projected onto the patient in conformity with the patient's physical anatomy as a visualization. The visualization may be scaled and / or moved based on input from the head tracking mechanism 6 and / or auditory input. The visualization may also include guidance control graphics and / or reference graphics to facilitate guiding the interventional device through the patient's anatomy.

[0059] V. Example Setup

[0060] Figures 5 to 7 Shows Figure 1 The system 10 is described for benchtop experimental use. Figure 5 A mannequin body with abdomen for surgical operation is shown. Four position sensors (LS) are distributed at reference positions on the patient's body. An ultrasound probe and surgical instruments are also provided, each with tracking sensors (T / PS and IDS). Figure 6 Shown in combination with holographic display Figure 5 The use of components. Figure 7 Microsoft HoloLens is shown, which is used as a head mounted device 1 to provide Figure 6 The following discusses the techniques that can be used in the examples.

[0061] VI. Example Technology

[0062] User-specified 3D point location on the holographic ultrasound plane

[0063] The operator can locate a point on the holographic ultrasound plane. The cursor position can be adjusted by the position on the "gaze" plane. If the operator gazes at the same position for a while, a small ball can be placed at that position. For example, the position can be the center of a tumor projected on the holographic ultrasound plane (HUP).

[0064] Point position P on HUP hup It can be used for tissue targeting validation testing as follows: Assuming that the 3D point is also located at the intersection of the interventional instrument HLR - a line segment extrapolated from the physical tip along the trajectory of the instrument - and the HUP, the distance between these two points can be calculated and reported in holographic text.

[0065] CT registration of live ultrasound at homologous points

[0066] Point position P hup It can also be used to spatially register two holographic imaging types, such as ultrasound (real-time planar) and CT (preoperative 3D volume). The center of the tumor P TCT It can be positioned during the preoperative treatment phase. Since both CT and HUP are transformed into HoloLens ( Figure 7 ) coordinates, so the CT-derived hologram (which may contain not only the segmented tumor but also organs and blood vessels) can be calculated as P TCT -P hup The vector P hup to translate. The translation distance of the CT image can be recorded. If the CT data set has more than one tumor, each with an associated center point, then a corrective registration can be performed in a tumor-specific manner. More generally, homologous points and features identified in ultrasound and CT holograms can be used to improve the registration between CT and US image holograms based on rigid affine transformations, and even elastic (non-rigid) transformations can be used.

[0067] Scaling up of fused (CT / US / HLR) holographic Cartesian volumes

[0068] For augmented and mixed reality (AR / MR), patient-specific holograms and device holograms are typically registered to each other (in the form of holographic coordinates) and then projected onto the patient stereoscopically (registered to the patient); however, conventional image-guided systems establish a correspondence between physical and displayed anatomical and device images, but displayed on a 2D monitor instead of on the patient. In order for the holograms to be consistent with the physical space (patient coordinates), they must be at a unified (i.e., identical) scale. Projecting device and anatomical (including HUP) holograms onto the patient provides the following advantages: 1) optimal eye-hand coordination for navigating the device to the target tissue, and 2) no need to observe the image on a separate 2D monitor, and 3) consistency of augmented reality and physical reality. For some applications, this can provide improved 3D visualization of holographic content, particularly for optical see-through devices (OSDs). For OSDs, the quality used to visualize the hologram depends on its context, and in some cases projection onto the patient may not be optimal.

[0069] The hologram projected onto the patient can also be translated (e.g., to the space in front of / above the patient) and scaled (by a factor of 2) to a position that has an improved background or is more comfortable while maintaining 1) alignment between the device and the multimodal hologram, and 2) eye-hand coordination for the device to navigate to the target. Eye-hand coordination is maintained when the hologram is scaled and translated, but is not as relevant as when the hologram is projected onto the patient. The hologram can also be rotated relative to the physical anatomy, but this will further make eye-hand coordination irrelevant. For example, the holographic ultrasound plane can rotate to follow the operator's viewing direction, i.e., always facing the operator. This will decouple eye-hand coordination, but prevent the operator from looking back and forth at the display if the 2D display is in a fixed position.

[0070] The operator can optionally switch back and forth between the scaled / panned / rotated view and the projection to the patient. This can be implemented using voice commands or other inputs on a head mounted display such as a holographic dashboard.

[0071] Depth adjustment for non-scale holographic ultrasound planes

[0072] The depth adjustment knob on the ultrasound device changes the scale of the image. Increasing the depth (e.g., to 8 cm) will scale the ultrasound image and display it on the 2D monitor so that the vertical length of the image (e.g., 1024 pixels) will correspond to 8 cm. Therefore, the captured frame image used in this article will also correspond to a depth of 8 cm. In this case, the HUP must also be scaled according to the depth adjustment.

[0073] This can be done based on voice commands on the head mounted display. The operator can specify the depth of other words (e.g., "eight" or "ten") to specify the depth setting. This will maintain the scale and physics of the HUP (assuming the same scale factor) when the hologram is projected onto the patient.

[0074] Calibration of the holographic image plane to the physical transducer

[0075] In order to position the holographic ultrasound plane HUP relative to the ultrasound transducer, an adjustable 6-DOF transformation can be provided in manufacturing so that the HUP extends accurately from the physical transducer. Six slider inputs can be used to interactively determine the transformation. The transformation from the slider is used to position the center tip of the ultrasound array. The HUP is depicted with holographic lines to facilitate adjustment of 3-axis rotation and 3-axis translation. The 6-DOF transformation of the tracked transducer can be stored for each transducer type and manufacturer. The ultrasound image is transformed to correspond to the ultrasound probe.

[0076] Methods for holographic CT and ultrasound visualization

[0077] When combining HUP and 3D CT, the CT hologram (based on segmentation) results can potentially obscure echogenic lesions on HUP (VIDEO: SCALE UP ADJ CT). This can be prevented by using transparent materials for CT-based holograms (e.g. tumors, organs, and vessels). Another approach is to project the surface as a wireframe shader so that it can be compared to the CT hologram. Figure 1 Another approach is to turn on and off the CT-based hologram to observe the echogenic features on the HUP.

[0078] From the above description, those skilled in the art will recognize improvements, changes and modifications. Such improvements, changes and modifications are within the skill of those skilled in the art and are intended to be covered by the appended claims. All patents, patent applications and publications cited herein are incorporated herein by reference in their entirety.

Claims

1. A system comprising: A computing device comprising a memory storing a digital anatomical object derived from preoperative 3D computed tomography (CT) image data of a physical patient, wherein the CT image data is represented in 3D coordinates of a CT coordinate system; A head-mounted device comprising a processor, a head-mounted display and a head tracking mechanism for: receiving live tracking data from the surgery in 3D Cartesian coordinates from the navigation system, The live tracking data includes the position and orientation of: A tracked physical ultrasound transducer / probe connected to the ultrasound system, The physical interventional device / instrument being tracked, and a physical fiducial position sensor at a specific anatomical location on the physical patient; receiving a live ultrasound image stream acquired by the physical ultrasound transducer / probe connected to the ultrasound system; transforming the live ultrasound image stream to a head mounted viewer coordinate system; transforming the live tracking data to a head mounted viewer coordinate system; displaying in the head mounted display a live holographic projection of the live ultrasound image stream in the head mounted viewer coordinate system, wherein the live holographic projection of the ultrasound image stream is based on the tracked position and orientation of the transducer / probe, wherein the live holographic projection is scaled consistently with an operator view assessed using a head tracking mechanism; retrieving the digital anatomical object; transforming the 3D coordinates of the digital anatomical object from the CT coordinate system to the head mounted viewer coordinate system; translating the anatomical object in the head mounted view coordinate system by a 3D vector calculated based on 3D point locations on the live holographic projection of the live image stream and corresponding points within pre-operative CT image data stored in the head mounted view coordinate system to correct for live anatomical movement of the physical patient during the surgery; and A holographic visualization is displayed in the head-mounted display, the holographic visualization including a holographic representation of the tracked physical interventional device / instrument, the holographic representation being consistent with the registered holographic projection of the live ultrasound image stream and the holographic anatomical object derived from CT, wherein the holographic visualization is utilized to position the tracked physical ultrasound transducer / probe and guide the physical interventional device / instrument to a treatment target.

2. The system according to claim 1, wherein: The live tracking data is based on electromagnetic tracking of electromagnetic sensor coils and / or optical tracking of reflective markers, and Wherein, the electromagnetic sensor coil and / or the reflective marker are located in and / or on the tracked physical ultrasound transducer / probe, the tracked physical intervention device / instrument and the physical reference position sensor.

3. The system according to claim 1, wherein: The holographic visualization is stereoscopically projected onto the patient in concert with the patient's physical anatomy, the ultrasound transducer / probe, and the physical interventional device / instrument.

4. The system according to claim 3, wherein: Also includes: Upon receiving operator input, a visualization that is inconsistent with the patient's physical anatomy and / or the physical ultrasound transducer / probe and / or the physical interventional device / instrument is projected through the head-mounted device while maintaining co-registration of the holographic projection.

5. The system according to claim 3, wherein: Also includes: Upon receiving operator input, the visualization is continuously realigned by the head mounted device based on tracking data from the head mounted device to always face the operator.

6. The system according to claim 3, wherein: Also includes: Co-registration of holographic rays, the registered holographic projection of the live ultrasound image stream, and the holographic anatomical object derived from CT is maintained when the visualization is translated, rotated and / or scaled relative to the physical anatomy.

7. The system according to claim 6, wherein: The intersection of the holographic rays with the registered holographic projection of the live ultrasound image stream is displayed as a spherical intersection.