Augmented reality system with improved alignment method and multiple therapeutic application methods

Abstract

A method for aligning a patient's anatomical features using augmented reality is provided. This includes a system for aligning a patient's anatomical features, comprising an imaging system, a tracking device, an augmented reality system, and a computer system. The imaging system can be configured to image the patient and generate an imaging dataset. The tracking device can be configured to generate a tracking device dataset. The augmented reality system can be configured to display an imaging volume hologram in augmented reality. The computer system can communicate with the augmented reality system, the imaging system, and the tracking device. The computer system can generate an imaging volume dataset based on a first and second axis plane, render the imaging volume dataset into an imaging volume hologram, and project the imaging volume hologram into augmented reality.

Description

Technical Field

[0001] <Cross - References to Related Applications> This application claims the priority of U.S. Provisional Application No. 63 / 504,675, filed on May 26, 2023. The disclosure of the above application is incorporated herein by reference.

[0002] This technology relates to holographic augmented reality applications, more specifically, medical applications that employ holographic augmented reality.

Background Art

[0003] This section provides background information related to the present disclosure, which is not necessarily prior art.

[0004] Image - guided surgery has become a standard approach in various procedures. Image - guided surgery can visually associate intraoperative data with preoperative and postoperative data to assist the operator. The use of image - guided surgery has been shown to improve the safety and success rate of various surgical procedures. Image - guided surgery can be further enhanced by the use of augmented reality (AR). The use of AR can provide an interactive experience in the real - world environment, where one or more features existing in the real world may be enhanced (sometimes across multiple sensory modalities) by computer - generated perceptual information. In the medical field, AR can be useful for enhancing the real - world environment related to patients and operating rooms. For example, during the performance of a medical procedure, an operator can view information corresponding to a target within the same field of view as the patient without shifting their line of sight.

[0005] However, there are certain problems that can arise before, during, and / or after image-guided surgery. For example, a patient's anatomical structure is not always static. Various internal movements, such as respiration and heartbeat, can cause rhythmic changes in the patient's internal anatomical structure. Undesirably, these internal movements can cause displacement of the surgical site, potentially interfering with the use of augmented reality during the procedure. This problem can be further exacerbated by the fact that these internal movements are not linear. For example, at certain stages of the respiratory cycle, expansion and contraction can cause significant changes in lung deformation and airflow volume.

[0006] Furthermore, one of the standard methods for creating three-dimensional (3D) medical images today is to use computed tomography (CT) scans that generate a series of images called DICOM datasets. These DICOM datasets can be further processed using software to segment body structures and generate 3D images of these structures that can be used for further research or augmented reality (AR) applications. These DICOM datasets must be carefully examined one by one and then processed using software segmentation techniques to outline and identify each structure of interest within individual slice images.

[0007] In other words, a CT scan produces two-dimensional (2D) image slices of varying thickness. These individual 2D-segmented DICOM slices must be reconstructed into a 3D model, rendered, and then smoothed. Processing 2D image slices from a CT scan may involve numerous image transfer and processing steps to generate anatomical volumes suitable for viewing in augmented reality. Because this process involves a large number of steps and the acquisition and operation of CT scans are costly, this scanning method may not be feasible in certain situations. Furthermore, the high cost of CT scans limits the number of available scans, meaning they may not be readily accessible to all patients who need them. Additionally, patients may be exposed to undesirable radiation doses during a CT scan. This radiation exposure is harmful to human tissue and can endanger both the patient and their caregiver. This radiation exposure can also have long-term adverse effects.

[0008] In certain augmented reality systems, recent alignment techniques are cumbersome and not ideal for all surgical applications. Therefore, there is a continuing need for improved alignment techniques in holographic augmented reality systems and their application areas. [Overview of the Initiative]

[0009] Consistent with this disclosure, an augmented reality system has been remarkably discovered that can provide real-time two-plane / multi-view perspective fused holograms rendered near the patient for use during treatment.

[0010] This disclosure provides a system for aligning a patient's anatomical features using augmented reality. The system may include an imaging system, a tracking device, augmented reality, and a computer system. The imaging system may be configured to image the patient's anatomical features and generate an imaging dataset. The tracking device may be configured to generate a tracking device dataset. The augmented reality system may be configured to display an imaging volume hologram of the patient's anatomical features in augmented reality. The computer system may communicate with the augmented reality system, the imaging system, and the tracking device. The computer system may generate an imaging volume dataset based on a first axis plane and a second axis plane, render the imaging volume dataset into an imaging volume hologram, and project the imaging volume hologram into the augmented reality environment of the augmented reality system.

[0011] This disclosure further provides a method for aligning a patient's anatomical structures using augmented reality for patient treatment. This method may include steps of providing a system of this disclosure, including an augmented reality system, an imaging system, a tracking device, and a computer system. The method may include steps of positioning the tracking device in a first axis plane and collecting a tracking device dataset from the tracking device. The method may include steps of positioning the imaging system in a second axis plane and collecting an imaging dataset from the imaging system. The method may include steps of using both the tracking device and the imaging system to identify internal landmarks of the patient. The first and second axis planes can be aligned based on the identified internal landmarks. An imaging volume dataset based on the first and second axis planes can be generated in a 3D coordinate system. The method may include steps of aligning the imaging volume dataset to the augmented reality system by the computer system. The imaging volume dataset can be rendered into an imaging volume hologram, and the imaging volume hologram can be projected by the augmented reality system.

[0012] Further areas of application will become apparent from the descriptions provided herein. The descriptions and examples in this abstract are for illustrative purposes only and do not limit the scope of this disclosure. [Brief explanation of the drawing]

[0013] The drawings described herein are for illustrative purposes only of selected embodiments and do not represent all possible embodiments, nor are they intended to limit the scope of this disclosure.

[0014] [Figure 1] This is a schematic diagram of a system that uses augmented reality to align the anatomical features of a patient.

[0015] [Figure 2A] A flowchart showing a method of aligning a patient's anatomical structure using augmented reality for treatment of the patient. [Figure 2B] A flowchart showing a method of aligning a patient's anatomical structure using augmented reality for treatment of the patient. [Figure 2C] A flowchart showing a method of aligning a patient's anatomical structure using augmented reality for treatment of the patient.

[0016] [Figure 3] A flowchart showing a procedure for utilizing a combination of anatomical landmarks and the patient's midline.

[0017] [Figure 4A] A flowchart showing a procedure for aligning and calibrating an augmented reality system and a surgical robot. [Figure 4B] A flowchart showing a procedure for aligning and calibrating an augmented reality system and a surgical robot.

[0018] [Figure 5] A flowchart showing a method of aligning a pre-treatment image dataset with an augmented reality system. [Figure 6] .

[0019] [Figure 7] A flowchart showing a procedure for determining an in-vivo ablation region.

Mode for Carrying Out the Invention

[0020] The following technical descriptions are merely illustrative of the subject matter, manufacture, and use of one or more inventions and are not intended to limit the scope, application, or use of any particular invention claimed in this application or any other application that may be filed claiming priority to this application, or in any patents issued therefrom. With respect to the disclosed methods, the order of the steps presented is illustrative, and therefore, unless otherwise specified, the order of the steps may differ in various embodiments, including when certain steps can be performed simultaneously. In this specification, the singular forms ("a" and "an") indicate that there is "at least one" of the subject matter. Where possible, there may be more than one. Unless otherwise specified, all numerical values ​​in this specification are understood to be modified by the word "about," and all geometric and spatial descriptions are understood to be modified by the word "substantially," which best describes the scope of the art. When applied to numerical values, "about" indicates that some degree of inaccuracy is permissible in the value in the calculation or measurement (meaning it approximates the exact value to some extent, is roughly or reasonably close, or nearly). If for any reason the inaccuracy provided by “approximately” and / or “substantially” is not otherwise understood in the art in this ordinary sense, then as used herein, “approximately” and / or “substantially” shall at least indicate the variation that may arise from the ordinary methods of measuring or using such parameters.

[0021] All documents cited herein, including patents, patent applications, and scientific literature, are incorporated herein by reference unless otherwise specified. In the event of any inconsistency or ambiguity between the documents incorporated by reference and this specification, this specification shall prevail.

[0022] To describe embodiments of the technology and claim the patent rights, the open-ended term "comprising" is used herein as a synonym for non-limiting terms such as "including", "containing", "having", etc., but embodiments can alternatively be described using more limiting terms such as "consisting of" or "consisting essentially of". Thus, for any embodiment that describes a material, component, or process step, the technology excludes additional materials, components, or processes (if consisting of) and excludes additional materials, components, or processes that affect the important characteristics of the embodiment (if consisting essentially of), even if such additional materials, components, or processes are not explicitly described in this application. The technology particularly includes embodiments consisting of or consisting essentially of such materials, components, or process steps. For example, the description of a composition or process that describes elements A, B, and C assumes, in particular, embodiments consisting of or consisting essentially of A, B, and C that exclude element D, even if element D is not explicitly described as excluded herein, and element D is a possible element described in the technical field.

[0023] In this specification, a disclosure of a range, unless otherwise specified, includes the endpoints and encompasses all individual values ​​and further subdivided ranges within the entire range. Thus, for example, a range "from A to B" or "about A to about B" includes A and B. Disclosure of values ​​and ranges of values ​​for a particular parameter (quantity, mass percentage, etc.) does not preclude other values ​​and ranges of values ​​that are useful in this specification. It is assumed that two or more specific exemplary values ​​for a particular parameter may define the endpoints of the range of values ​​that can be claimed for that parameter. For example, if parameter X is exemplified in this specification as having the value A and also exemplifying it as having the value Z, it is assumed that parameter X may have a range of values ​​from about A to about Z. Similarly, if two or more ranges of values ​​(whether nested, overlapping, or distinct) are disclosed for a parameter, it is assumed that all possible combinations of ranges of values ​​that can be claimed using the endpoints of the disclosed ranges are encompassed. For example, if parameter X is exemplified in this specification as having values ​​in the range of 1 to 10, 2 to 9, or 3 to 8, it is also conceivable that parameter X may include other ranges of values ​​such as 1 to 9, 1 to 8, 1 to 3, 1 to 2, 2 to 10, 2 to 8, 2 to 3, 3 to 10, and 3 to 9.

[0024] When an element or layer is described as “on,” “engaged to,” “connected to,” or “coupled to” another element or layer, it may be directly on, engaged to, connected to, or coupled to that other element or layer, or there may be an intervening element or layer. Conversely, when an element is described as “directly on,” “directly engaged,” “directly connected,” or “directly coupled” to another element or layer, there may be no intervening element or layer. Other terms used to describe relationships between elements (e.g., “directly between” versus “between,” “directly adjacent” versus “adjacent,” etc.) should be interpreted similarly. In this specification, the term “and / or” encompasses any combination that includes one or more of the relevant enumerated items.

[0025] In this specification, terms such as "first," "second," and "third" may be used to describe various elements, components, regions, layers, and / or sections, but these elements, components, regions, layers, and / or sections should not be limited by these terms. These terms may be used only to distinguish one element, component, region, layer, or section from another region, layer, or section. Numerical terms such as "first" and "second" as used herein do not imply sequence or order unless clearly indicated by the context. Accordingly, the first element, component, region, layer, or section described below may be referred to as the second element, component, region, layer, or section without departing from the teachings of the exemplary embodiments.

[0026] In this specification, spatially relative terms such as “inner,” “outer,” “beneath,” “below,” “lower,” “above,” and “upper” may be used for convenience to describe the relationship between one element or feature and another, as shown in the drawings. Spatially relative terms may be intended to encompass different orientations of the device in use or operation, in addition to the orientation shown in the drawings. For example, if the device in the drawing is inverted, an element described as “below” or “below” another element or feature will be located “above” that other element or feature. Thus, the exemplary term “below” can encompass both up and down orientations. The device can also be positioned in other orientations (90-degree rotation, or other orientations), and the spatially relative descriptors used herein will be interpreted accordingly.

[0027] In this specification, the terms “intervention device” or “tracking device” refer to medical devices used during medical procedures.

[0028] In this specification, the term “tracking system” refers to a system used to observe one or more objects in motion and to supply a time-ordered sequence of tracking data (e.g., positional data, orientation data, etc.) in a tracking coordinate system for further processing. As an example, a tracking system may be an electromagnetic or optical (e.g., fiber optic) tracking system in which an intervention device equipped with sensor coils can be observed as it enters, moves through, and exits a patient’s body, as well as while it is outside the body.

[0029] In this specification, the term "tracking data" refers to information recorded by a tracking system related to the observation of one or more moving objects.

[0030] In this specification, the terms “head-mounted device,” “headset,” or “HMD” refer to a display device configured to be worn on the head and having one or more display optics (including lenses) in front of one or more eyes. These terms may also be referred to more generally as “augmented reality system,” but it should be noted that the term “augmented reality system” is not limited to a display device configured to be worn on the head. In some cases, a head-mounted device may also include non-temporary memory and processing units. An example of a suitable head-mounted device is the Microsoft HoloLens® head-mounted device (Microsoft, Redmond, Washington).

[0031] In this specification, terms such as “imaging system,” “image acquisition device,” and “image acquisition system” refer to technologies that create visual representations of the inside of a patient’s body. For example, imaging systems include computed tomography (CT) systems, fluoroscopy systems, positron emission tomography (POST) systems, magnetic resonance imaging (MRI) systems, and ultrasound (US) systems including contrast agents and color flow Doppler.

[0032] In this specification, the terms “coordinate system,” “augmented reality system coordinate system,” or “augmented reality system coordinates” refer to a three-dimensional Cartesian coordinate system that uses one or more numerical values ​​to determine the position of a point or other geometric element specific to the particular augmented reality system or image acquisition system to which it relates. For example, a 3D point in a headset coordinate system may be translated, rotated, scaled, etc., from a standard three-dimensional Cartesian coordinate system.

[0033] In this specification, the terms “image data,” “imaging dataset,” or “imaging data” refer to information recorded in 3D by an imaging system related to the observation of the inside of a patient’s body. For example, “image data” or “imaging dataset” may include processed two-dimensional or three-dimensional images or models (e.g., imaging images represented in data formats compliant with the DICOM (Digital Imaging and Communications in Medicine) standard or other relevant imaging standards).

[0034] In this specification, the terms “imaging coordinate system” or “image acquisition system coordinate system” refer to a 3D Cartesian coordinate system that uses one or more numerical values ​​to determine the position of points or other geometric elements specific to a particular imaging system. For example, 3D points or vectors in an imaging coordinate system may be translated, rotated, scaled, etc., relative to the 3D Cartesian coordinate system of an augmented reality system (head-mounted display).

[0035] In this specification, the terms “hologram,” “holographic,” “holographic projection,” or “holographic representation” mean a computer-generated image projected three-dimensionally through the lenses of a headset. Generally, a hologram is synthetically generated (in augmented reality (AR)) and is not a real thing (a physical entity).

[0036] In this specification, the term "physical" refers to something real. A physical object is not holographic (or computer-generated).

[0037] In this specification, the terms "two-dimensional" or "2D" refer to something that is represented in two physical dimensions.

[0038] As used herein, the terms “three-dimensional” or “3D” refer to something that can be represented in three actual dimensions. Elements that are “4D” (for example, 3D plus the dimensions of time and / or motion) are included in the definition of three-dimensional or 3D.

[0039] In this specification, the term “integrated” may refer to two or more things that are linked or coordinated. For example, a coil sensor can be integrated with an intervention device.

[0040] As used herein, the terms “real-time,” “near real-time,” or “live” refer to the actual time during which a process or event is occurring. That is, a real-time event is performed live (within milliseconds, so that results are immediately available as feedback). For example, a real-time event is represented within 100 milliseconds of the event occurring.

[0041] In this specification, the terms “subject” and “patient” are used interchangeably and may refer to any vertebrate.

[0042] In this specification, the term spatial “alignment” refers to the step of transforming tracking datasets and imaging datasets, as well as additional body image data, related to the virtual representation of the tracked device (including holographic guides, applicators, and ultrasound image streams) in order to align and associate the virtual device with the image data in a head-mounted display coordinate system. This makes it possible to project images and information stereoscopically onto the actual patient's body during treatment. These are further described, for example, in U.S. Patent Application No. 10,895,906 (West et al.), U.S. Patent Application No. 17 / 110,991 (Black et al.), co-owned by the applicant, and U.S. Patent Application No. 11,701,183 (Martin III et al.), all of which are incorporated herein by reference.

[0043] This disclosure relates to a system 100 for anatomical structure alignment using augmented reality, schematically shown in Figure 1. The system 100 may comprise an augmented reality system 102, an imaging system 104, a tracking device 106, and a computer system 108. The augmented reality system 102 may be configured to display an augmented representation 110 in the form of a hologram. The augmented representation 110 may include a two-dimensional (2D) or three-dimensional (3D) depiction of information related to a medical procedure. Examples of related information include pre- and / or intra-operative data, such as a three-dimensional depiction of the patient's anatomical features. Anatomical features may be the patient's biological tissue and / or region that is the focus of the current procedure. Further examples of anatomical features include organs, parts of organs, tissues, joints, bones, tumors, blood vessels, implants, etc. The augmented representation 110 has many applications and uses, such as pre-procedure planning, procedure guidance, and training. Those skilled in the art should understand that other information can be selected to be depicted in the augmented representation 110. Furthermore, it should be understood that anatomical or physiological / functional features can include any part of the patient's anatomical structure.

[0044] The augmented reality system 102 can be configured to display the augmented representation 110 via an augmented reality display 112, such as a head-mounted display, in an augmented reality environment. Preferably, this allows the practitioner to view the augmented representation 110 within the same field of view as the patient. The augmented reality system 102 can be configured to display the augmented representation 110 on top of a part of the patient in an augmented reality environment. The part of the patient may be an anatomical feature of the patient. This allows the augmented representation 110 to be displayed directly on top of the anatomical feature, providing relevant feedback in the context of the location of the anatomical feature. For example, the augmented representation 110 may be an anatomical feature scanned during surgery and can be superimposed on the anatomical feature for the practitioner to view. In other examples, the patient's body part may be adjacent to the patient's anatomical feature. Preferably, this allows the practitioner to observe the augmented representation 110 and the anatomical feature simultaneously within the same field of view.

[0045] The augmented reality display 112, using the augmented reality display 112, can be displayed on the approximate location of the patient's anatomical features. For example, the computer system 108 can use algorithms, machine learning, artificial intelligence, and / or a combination thereof to estimate the approximate location of the patient's anatomical features based on medically approved tolerances. The computer system 108 can operate manually or as a hybrid of manual and automatic control. However, it should be noted that the augmented reality display 110 can also be displayed on other surfaces and / or on the augmented reality display 110 as needed.

[0046] In certain embodiments, the augmented reality system 102 may also include one or more image targets 114 for position determination. The image targets 114 of the augmented reality system 102 may be configured to determine and generate positional data of the augmented reality system 102 (e.g., approximate position in three-dimensional (3D) space, orientation of the augmented reality system 102, angular velocity, acceleration, etc.). For example, this should be understood to enable accurate display of a holographic image within the practitioner's field of view during an operation. Examples of image targets 114 include accelerometers, gyroscopes, electromagnetic sensors, and / or optical tracking sensors. Furthermore, those skilled in the art should understand that the type and number of image targets 114 of the augmented reality system 102 can be changed depending on the procedure or situation in which the augmented reality system 102 is used. It should be noted that system 100 does not require the use of image targets 114 for alignment.

[0047] The imaging system 104 can be configured to image the patient's anatomical features and generate an imaging dataset 116. The imaging dataset 116 may include information and / or media related to the structure, rotation, and / or position of the anatomical features relative to the patient. Those skilled in the art should understand that the types of data to be included in the imaging dataset 116 can be selected. Preferably, the imaging system 104 can be used to image the patient's anatomical features and generate the imaging dataset 116 before, during, and / or in combination with the procedure.

[0048] As described later, the imaging dataset 116 can be used by the computer system 108 to generate an augmented representation 110. In other words, the imaging system 104 can be used to perform scans and / or other imaging processes to generate the imaging dataset 116 used to generate the augmented representation 110. For example, the imaging system 104 may include an ultrasound system having at least one active ultrasound probe. The operator can move the ultrasound probe over the patient's anatomical features to acquire an imaging dataset 116 which may include 2D images. The ultrasound probe can move along a path along the patient to generate 2D images. The 2D images are then converted into an augmented representation 110 by the computer system 108. Other examples of the imaging system 104 include computed tomography (CT) systems, electromagnetic systems, cone-beam computed tomography (CBCT) systems, blood gas exchange systems, mechanical ventilation systems, spirometry (pulmonary function measurement) systems, electrocardiogram (ECG) systems, magnetic resonance imaging (MRI) systems, electromechanical wave propagation systems, transesophageal echocardiography (TEE) systems, and combinations thereof. However, those skilled in the art should understand that other imaging procedures and systems can be employed in the imaging system 104 within the scope of this disclosure.

[0049] The tracking device 106 can also be configured to image the patient's anatomical features and generate a tracking device dataset 118. The tracking device dataset 118 may include information and / or media related to the structure, rotation, and / or position of the anatomical features relative to the patient. Since the tracking device 106 can include various instruments such as needles and electromagnetically tracked ultrasound, the tracking device dataset 118 may include ultrasound data. It should be understood that the tracking device dataset 118 may also include electromagnetic data. Those skilled in the art can select the types of data to include in the tracking device dataset 118. Preferably, the tracking device 106 can be used to image another anatomical feature of the patient during the procedure and generate the tracking device dataset 118.

[0050] The computer system 108 can generate an imaging volume dataset 120 and be configured to align the imaging volume dataset 120 with the augmented reality system 102. The imaging volume dataset 120 can be generated from a tracking device dataset 118 and an imaging dataset 116. As described herein, the tracking device 106 can image the patient's first axis plane 122 to collect the tracking device dataset 118, and the imaging system 104 can image the second axis plane 124 to collect the imaging dataset 116. The first axis plane 122 and the second axis plane 124 are aligned to generate the imaging volume dataset 120, from which an imaging volume hologram 126 is rendered and projected.

[0051] Generally, alignment involves converting tracking devices 106 such as needles, electromagnetically tracked ultrasound, or electromagnetic data from an imaging system 104 into coordinates for the augmented reality system. Additional imaging data (such as MRI or CT images) can also be aligned to the coordinates of the augmented reality system. In certain embodiments, both tracking data and imaging data from one or more different devices, including imaging devices, can be aligned to the actual patient. In certain embodiments, additional imaging data can be aligned using fiducial sensors placed at one or more locations on the patient's body.

[0052] [Imaging systems without segmentation, tracking ultrasound, and instrument positioning]

[0053] Instead of, or in combination with, the segmentation of anatomical structures and fiducial markers from datasets, which can involve complex workflows, reconstructed images can be computed in system 100 based on the import of an imaging volume dataset 120. The coordinate system of the reconstructed or reformatted images is aligned to be coplanar and orientably aligned with the images of the tracking instrument 106 and can be used with a holographic representation of the tracking instrument 106. Referring to Figures 2A to 2C, a method 200 is provided that utilizes the positions of the tracking instrument 106 and the imaging system 104 to locate the anatomical structures of a patient. Thus, method 200 includes step 202, which provides a system for anatomical structure locatement using augmented reality as described herein. In step 204, the tracking instrument 106 can be positioned in a first axial plane 122 corresponding to the patient. In step 206, the tracking instrument 106 can collect a tracking instrument dataset 118 representing images indicated by the tracking instrument 106, such as an ultrasound probe. In step 208, the imaging system 104 can be positioned in the second axis plane 124. In step 210, the imaging system 104 can collect an imaging dataset 116 representing the images displayed by the imaging system 104 (e.g., computed tomography (CT) or magnetic resonance imaging (MRI)). While the tracking instrument 106 is held stationary in the second axis plane 124, the corresponding axial images in the imaging dataset can be identified. In another embodiment, the second axis plane 124 of the imaging system 104 can be selected first, and the tracking instrument 106 can be navigated to identify an image of the tracking instrument 106 that matches an image of a reference imaging system 104.

[0054] Method 200 may include step 212 of identifying internal anatomical landmarks corresponding to both the imaging system 104 and the tracking instrument 106. In step 214, the first axis plane and the second axis plane can be aligned based on the identified anatomical landmarks. In step 216, the computer system 108 can generate an imaging volume dataset 120 based on the tracking instrument dataset 118 in the first axis plane 122 and the imaging volume dataset 120 in the second axis plane 124. The imaging volume dataset 120 can be reformatted into orthogonal or oblique images, including images reformatted to lie coplanar with the tracking instrument images. In step 218, the computer system 108 can align the imaging volume dataset 120 with the augmented reality system 102. The imaging volume dataset 120 can be used by the computer system 108 to render an imaging volume hologram 126 in step 220. Method 200 may include step 222 of projecting the imaging volume hologram 126 using the augmented reality system 102.

[0055] Referring to Figure 3, in certain embodiments, Method 200 can utilize a combination of the patient's anatomical landmarks and midline. In step 230, the patient's midline can be identified. In step 214, the imaging volume hologram 126 is positioned so that the physical anatomical landmarks and holographic anatomical landmarks coincide. Then, in step 232, the imaging volume hologram 126 can be rotated manually or semi-automatically along the anatomical anterior-posterior axis until the holographic midline of the imaging volume hologram 126 coincides with the actual midline. This additional positional adjustment provides a highly calibrated alignment between the actual patient and the imaging volume hologram 126.

[0056] In a particular embodiment of Method 200, step 234 may include automatically segmenting skin surface images 128 from imaging dataset 116 to create skin imaging image data 130. In step 236, a medical professional can align (position together with) the skin surface image data 130 with a real patient using a tracking sensor 132 in the coordinates of the augmented reality system. The tracking sensor 132 may include a 6-degree-of-freedom (DoF) tracking sensor. Method 200 may include step 238, which aligns the skin imaging image data 130 with the imaging dataset 116 and the tracking instrument dataset 118, as these are also aligned with a real patient by performing an electromagnetic-to-augmented reality system 102 transformation by locating the image target and the electromagnetic sensor in their respective coordinate systems. Note that the location of the tracking sensor 132 can be determined using a locationable camera mounted on the head-mounted display of the augmented reality system 102. In this way, the imaging system 104, the tracking instrument 106, and the patient's skin surface are aligned with each other in the coordinates of the augmented reality system. After alignment and co-projection of augmented reality, methods are used to manage occlusion and mitigate depth ambiguity in relation to the actual patient. Alternatively, occlusion and depth ambiguity can be prevented by projecting the aligned holographic entity near the patient, ensuring that the line of sight does not intersect the holographic projection with the actual target tissue.

[0057] [Robot integration and alignment for augmented reality or extended reality (XR) visualization]

[0058] The multi-arm surgical robot 134 can be used in conjunction with system 100 and can be operated via hand controls from a display console. This improves precise motion control when manipulating endoscopic tools and surgical instruments through small incision (keyhole) surgery. The display console can provide 3D information based on virtual reality (VR) or XR, but the image content is not intended to facilitate eye-hand coordination (visual-motor axis). Therefore, manipulating surgical instruments with an endoscopic camera may require considerable training.

[0059] Referring to Figures 4A and 4B, in order to align the robot kinematic chain to the coordinates of the head-mounted display of system 100, method 200 may include step 240 of mounting the robot image target 136 in a predetermined position and orientation on a part of the surgical robot 134 (e.g., an end effector). In step 242, the robot image target 136 can be aligned (positioned) to the three-dimensional Cartesian coordinate system of the robot system via the kinematic chain (e.g., a mechanically fixed base, shoulder, elbow, forearm, and a 6-degree-of-freedom wrist / end effector). The robot image target 136 is also localizable in the coordinates of the augmented reality system via a localizable RGB camera or depth camera. The coordinate system of the augmented reality system is determined during application initialization and is subsequently tracked by the inertial measurement unit (IMU) of the head-mounted display of the augmented reality system 102.

[0060] Method 200 may further include a process for calibrating and aligning the surgical robot 134 for the procedure. In step 248, the surgical robot 134 can be programmed to move the robot image target 136 in a predetermined pattern with periodic stops across the surgical field. In step 250, at each stop, the orientation of the portion of the surgical robot 134 is adjusted so that the robot image target 136 can be imaged by the camera 138 of the augmented reality system 102. In step 252, the camera 138 of the augmented reality system 102 can image the robot image target 136. In step 254, each of a plurality of predetermined periodic stops can be associated with robot coordinates. In step 256, for each image of the robot image target 136, the corresponding stop of the robot image target is measured in the coordinates of the surgical robot 134 and the augmented reality system. These corresponding measurements can be used in step 258 to determine the conversion from robot coordinates to augmented reality system coordinates. This process can be automatically repeated for multiple subspaces within the reachable volume of the surgical robot 134. If the surgical robot 134 is equipped with rigid or flexible tracking sensors, the coordinate systems of these sensors can be defined relative to the alignment. For example, in the case of flexible endobronchial ultrasound (EBUS) / transrectal ultrasound (TRUS) and catheters tracked by electromagnetic (EM) or fiber optic sensors, the coordinates can be displayed as relative coordinates to both the robot's kinematic linkage mechanism and the augmented reality system, in a 3D projection with virtual 2D live image display using a HUD (head-up display) or HDD (head-down display, described later).

[0061] Referring to Figure 5, in certain embodiments, the preoperative image dataset 116 can also be aligned from the imaging system 104 to the augmented reality system coordinates. In step 260, one or more fiducial markers can be placed on the skin surface for localization during imaging. Holographic projections based on imaging can be used as a reference for positioning the endoscope. During the procedure, each fiducial marker can be localized in step 262 by positioning the tip of a calibration instrument attached to a part of the surgical robot 134 under operator control. Alternatively, the imaging dataset 116 can first be aligned to robot coordinates in step 264, and then both the imaging dataset 116 and the part of the surgical robot 134 can be aligned to head-mounted display coordinates in step 266.

[0062] After calibration and alignment, both the endoscopic arm and / or surgical instrument arm of the surgical robot 134 can be positioned within the intervention area. Holographic projections of the endoscopic field of view by the augmented reality system 102 and holographic representations of the tracking instruments 106 can be positioned and oriented to facilitate eye-hand coordination. The medical professional's console can be positioned tableside relative to the projections of the augmented reality system 102 to promote medical professional safety and eye-hand coordination. For example, a down-the-barrel view (a field of view coaxial with the endoscopic line of sight) can support lateral and anterior-posterior views of the surgical field and be associated with the corresponding movements of surgical instruments.

[0063] [XR Head-Down Display (HDD) for Live Image-Guided Therapy]

[0064] Live images from imaging systems 104, such as CT and fluoroscopic 2D images, can be used for interventional procedures. However, live 2D images displayed on a monitor may not be positioned or oriented in a way that facilitates eye-hand coordination when navigating imaging instruments to the target tissue. The head-mounted display of the augmented reality system 102 can project the live imaging stream holographically and be used as a virtual 2D display monitor that roughly coincides with the image acquisition plane, i.e., to display it in a position that roughly coincides with the actual patient. However, in certain situations, the image projection may be affected by occlusion and depth ambiguity. HDDs are similar to head-up displays (HUDs) but allow for specific positions and orientations, as well as managing occlusion and mitigating depth ambiguity. While much of the value proposition of HDDs relates to interventional procedures, it should be noted that they are also applicable to certain aspects of minimally invasive and conventional surgical applications when live imaging is used.

[0065] It should be noted that HDD projection can project a live image relative to the actual body, synchronized with the movement of surgical instruments along anatomical axes (e.g., left-right lateral, anterior-posterior, and caudal-cranial directions) as the instruments are advanced from the skin to the target site, in order to avoid occlusion and depth ambiguity. To achieve this, the imaging system coordinates (ISC) can be aligned with the augmented reality system coordinates.

[0066] Therefore, live imaging 2D data of the body acquisition area can be transmitted to or streamed to a head-mounted device of the augmented reality system 102 using a display adapter (e.g., frame grabbing). Coordinate alignment allows the head-mounted display to project live imaging of the body acquisition area in roughly alignment with the actual patient during minimally invasive surgical procedures. The live imaging can typically be used to guide the advance of an imaged tracking instrument 106 within the image acquisition plane. This is because, in in-plane procedures (e.g., fluoroscopic procedures), both the intervention field and the tracking instrument 106 can be imaged simultaneously. For occlusion management, additional image targets 114 can be placed on the patient's skin. The HMD camera identifies the position of the Vuforia® (Parametric Technology Corporation, Boston, Massachusetts) image targets 114 and uses this positional information to adjust the placement of holographic projections of live 2D CT and fluoroscopic images onto the HDD to meet the aforementioned criteria for eye-hand coordination and occlusion management.

[0067] In the case of in-plane guidance, the HDD is projected so as not to appear in the line of sight between the medical professional and the actual target tissue. If the HDD is projected in the line of sight, it is suggested that the 2D virtual display can be used without imaging untracked surgical instruments advancing toward the actual target tissue. Thus, the HDD can be adjusted with minimal displacement (e.g., translation or floating) so that the projection does not appear in the line of sight of the target tissue. The live image can be rotated by a small angle (e.g., less than 30°) while maintaining almost the anatomical orientation of the projection to facilitate eye-hand coordination. In this case, to avoid depth ambiguity, the stream of the live imaging system 104 can be projected above the transducer so as not to co-project with the actual patient. The method of positioning and adjusting the orientation of the HDD to prevent occlusion and depth ambiguity can also be applied to stereoscopic 3D projection of 3D preoperative or intraoperative imaging data in real coordinate space (rather than a virtual 2D display of the live image stream). The 3D floating projection is positioned so as not to intersect with the line of sight of the actual patient, indicated by the image target 114 on the skin surface.

[0068] [CT data alignment and gross motion correction of patients without EM tracking]

[0069] An imaging volume dataset 120, including a multi-detector CT or cone-beam CT in a particular embodiment, can be used as a reference in combination with a live imaging modality such as ultrasound, endoscopy, or CT fluoroscopy, and can be aligned in augmented reality system coordinates by locating the position of one or more image targets 114 in CT coordinates on the skin surface and then determining the transformation from CT coordinates to HMD coordinates. The image targets 114 have a three-dimensional position and three-dimensional orientation in augmented reality system coordinates and can have up to six degrees of freedom.

number

number

[0070] During the alignment process, an augmented reality system, such as Vuforia®, can use the camera of a HoloLens® head-mounted device to locate the actual image target 114 within the augmented reality system coordinates. The image target 114 must have clear angles that can be manually or automatically located during the pre-operative processing of CT images. To accommodate large patient movements during the procedure, the position and orientation of one or more image targets can be re-acquired. JPEG2026518134000004.jpg515 has been updated and applied to the virtual CT object.

[0071] [Aligning 3D printed models using holographic guidance and navigation systems]

[0072] Referring to Figure 6, surgically relevant structures segmented from the imaging dataset 116 can be used to create actual 3D printed models and / or (simultaneously or alternatively) projected stereoscopically as holograms onto a head-mounted display. In step 270, the 3D module 140 of the imaging volume dataset 120 can be aligned to the computer system 108. In step 272, an actual 3D model of the 3D module 140 is created, and in step 274, the actual 3D model is converted to coordinates in an augmented reality system. This can be projected aligned with the 3D printed model and / or roughly aligned with the actual patient before or during a surgical procedure.

[0073] Segmented structures can be sent to a 3D printer in STL file format. 3D models can be printed in a variety of materials, including biocompatible materials such as extracellular matrix, and actual printed cell materials such as those cultured from stem cells in a gel matrix, which can simulate the deformability of tissues. 3D printed models can include one or more touchable or sensored fiducial marker locations that can be used to position the actual model in a head-mounted display coordinate system (see tracking systems described herein). Alternatively, image targets that can be positioned in the coordinates of an augmented reality system can be printed onto the 3D model. Holograms can also be projected onto actual patients using the same STL file used to create the 3D model. Patient-specific physiological information and simulations can also be projected onto the 3D model.

[0074] 3D models can be used to simulate surgical procedures at the tableside, either pre-operatively or intraoperatively. Surgical instrument tracking systems can also be aligned to actual patients, CT-based holograms, and 3D-printed models. Components of 3D-printed models, such as skin and blood vessels, can be disassembled during surgical simulation while maintaining the anatomical structure of the remaining components. It should be noted that these 3D models reflect actual structures both anatomically and physiologically. Therefore, implantable sensors or fiducial markers printed on devices of actual orthopedic tissues (e.g., face, hip, knee) or soft tissues (e.g., bladder, kidney, heart) can be combined with guidance systems for planning and implantation.

[0075] [Placement of holographic projection on an actual patient (dual-alignment portal)]

[0076] The sense of aligning a holographic projection to the real world can be influenced by the field of view and the head-mounted display's ability to stabilize the hologram in response to head movements. When it is useful to position a hologram to align with the real world, methods are used to control the field of view while providing the operator with guidance and visualization to complete the task. One example is projecting a deformable skin surface hologram onto an actual skin surface. Tissue deformation can be adjusted to match the static holographic projection obtained through imaging, but this should be done with a reproducible field of view, and that field of view should also be used when completing the procedure. Controlling and limiting the field of view when projecting a hologram onto a corresponding real structure is called a (single) alignment portal.

[0077] Actual instruments that are not tracked have depth ambiguity in holographic projection onto the actual body. This can be addressed by positioning the central axis of the surgical instrument so that it aligns with the optical axis of the stereoscopic projection while the line of sight is aligned with the optical axis of the stereoscopic projection. By using a cylindrical hologram, this positioning can be planned based on structures of anatomical interest in order to systematically align the central axis of the surgical instrument. Thus, the system can provide feedback when the operator's line of sight is aligned with the cylindrical hologram (e.g., down-the-barrel view) without relying on depth cues that may be ambiguous when projecting the hologram using a transmissive lens. The holographic cylinder can be positioned to coincide with the operator's visual axis. Criteria for establishing a down-the-barrel view include line of sight tracking or the line of sight of a head-mounted display at a sufficiently low angle to the alignment cylinder. One or more points of surgical interest on a directly exposed anatomical surface can be localized by one or more holographic projections. The image target can be positioned relative to one or more anatomical structures, and the holographic projection can be positioned in relation to said structures. The calibration step ensures that the virtual representation of the image target matches the position of the actual target, thereby aligning the position of nearby structures. Controlling and limiting the projection field of view while projecting the hologram onto the corresponding actual structure, and while projecting untracked surgical devices, is called a dual alignment portal.

[0078] [A method for determining the conversion from an electromagnetic system to an augmented reality system that can track electromagnetic devices on actual patients]

[0079] The systems and methods of this disclosure can be used to determine the conversion from an electromagnetic field to an augmented reality system using an electromagnetic device tracked at the patient's actual location. For example, a fixed electromagnetic generator can be used to align electromagnetic data with the patient's actual location. In other embodiments, for example, a composite marker or a 6-degree-of-freedom sensor can be used to perform the electromagnetic-to-augmented reality system conversion. Preferably, these embodiments allow practitioners to use the system even when fiducial markers are not readily available.

[0080] [Treatment using ablation]

[0081] Referring to Figure 7, in a particular embodiment, method 200 may include a step 280 of providing a predetermined ablation area. The predetermined ablation area can be selected and determined preoperatively by a medical professional. In step 282, the predetermined ablation area can be compared with the in vivo ablation area during the procedure. The method may include a step 284 of modifying the predetermined area based on the in vivo ablation area, enabling more accurate ablation in real time. Thus, the method may include a step 286 of rendering a feedback hologram by an augmented reality system 102 to determine whether additional intervention is needed after the ablation has been performed. [example]

[0082] Examples of applications of this technology will be explained with reference to the attached figures. The operation of the system described herein can be explained illustratively. In particular, this system is useful in applications where it can be expanded and linked according to the situation to enable more accurate decision-making by the user. Some of these examples are provided in the following application examples.

[0083] [Application Example 1: Urology]

[0084] Prostate cancer is the second leading cause of cancer death among American men, with one in eight men being diagnosed in their lifetime. More than one million prostate biopsies are performed annually in the United States. A clinical diagnosis cannot be made until a biopsy is performed and a pathologist evaluates the histological results. However, repeated biopsies are often necessary if sufficient sample is not obtained or if the sample is not adequately distributed across different areas of the prostate. The false-negative rate for the initial biopsy is high, ranging from 20% to 30%. Furthermore, the complication rate for 24-sample biopsies is high, at 57%.

[0085] Typically, the prostate can be visualized by inserting a transrectal ultrasound (TRUS) probe into the rectum. Alternatively, a perineal approach may be used. Furthermore, pre-procedural MRI may be acquired and fused with ultrasound images to guide the biopsy procedure. The combined use of MRI and TRUS is becoming common in prostate biopsies. During a prostate biopsy, adding an EM-tracked TRUS probe along with an EM-tracked biopsy needle and aligning them allows for improved navigation accuracy and depth perception while visualizing the live ultrasound feed within the spatial context of the procedural workspace. Pre-operative MRI images can be segmented and aligned with the two tracking devices, utilizing the system's unique alignment capabilities.

[0086] In recent years, increasing the number of biopsy cores has become a common practice to improve detection rates. However, without integrating MRI, there is a higher chance of detecting small, indolent cancers that are not clinically relevant. Planned biopsy cores can be spatially aligned with the prostate, which has been segmented by MRI preoperatively. These planned cores can be referenced and adjusted to anatomical structures imaged with TRUS by adjusting the MRI alignment to correlate with the prostate boundaries. Typically, each core is aligned and positioned at 5mm intervals, and the alignment and optimization are performed by an algorithm and projected into the HMD coordinate space. These planned trajectories can be fine-tuned to ensure coverage integrity and reduce the likelihood of repeat biopsies. Each biopsy core can be labeled, and the final histological data obtained is uploaded to the patient's medical record, associated with the volume and coordinates of the specific core from which it was taken. This data is stored for future surgical interventions, such as resection, if deemed necessary based on the biopsy findings.

[0087] The data can be replayed in 3D space as "ghost" feedback during the procedure, allowing the physician to replay past procedures holographically in real time during the procedure. Segmented MRI, ultrasound images, and spatially aligned 3D holographic representations of the tracking needle can be projected onto the appropriate location of the prostate in an orientation that facilitates eye-hand coordination when deploying the needle via either a transrectal or transperineal approach. MRI data can be textured to indicate prostate areas that are likely to be malignant, thereby improving the diagnostic accuracy of biopsies. This AR system for the prostate can also be used in treatments such as the placement of radioactive brachytherapy seeds and during prostate tumor resection (rumpectomy). The system can reconstruct MRI images on the same plane as transrectal ultrasound images.

[0088] These planned biopsy cores can also be reviewed retrospectively to assess placement errors and their correlation with successful biopsies, thereby reducing unnecessary repeat biopsies. Postoperatively, the results of the biopsy cores can be used to map the prostate into healthy tissue areas and cancerous tissue areas.

[0089] Any of the 3D printing-related alignment methods described herein can also be applied to urological treatment. For example, a donor bladder was cultured using pluripotent stem cells and transplanted into a human, where it functioned for several years.

[0090] [Application Example 2: Breast]

[0091] In breast cancer treatment, due to the high flexibility of the tissue, markers may be implanted before rampectomy or ablation. These markers can be aligned in a holographic coordinate system and tracked during the procedure. In minimally invasive surgery, the tumor may or may not be excised. In the cavity, navigation can be performed relative to the cavity and / or markers (if present) under ultrasound guidance displayed on the HMD. Subsequently, the pre-planned ablation area can be aligned with the tracked ablation margin with an ice ball using ultrasound, MRI, or similar devices. This can be analyzed and addressed during or after the procedure. Staining agents can be used to track cancer cells that metastasize to lymph nodes, and these may be visualized in an aligned hologram. If the tumor is removed, the circumstances of removal, the actual morphology, and the location of the tumor can be tagged in this same coordinate system for later tracking. It should be understood that in any ablation technique discussed herein, multiple probes can be aligned relative to each other. The system can be displayed with the addition of overlapping ellipsoids. However, by arranging these ablation regions in close proximity to each other and setting energy, thermal, or electric field gradients, the superposition of ablation regions can cover different volumes. Furthermore, these multiple probes can be combined individually with the generated monpolar electric field, but they can also be intentionally combined via bipolar and / or biphase means, thereby allowing the phases to coordinate to form a kill zone exceeding the "sum of each component".

[0092] The rampectomy device is trackable within the previously existing tumor space, and the placement and irradiation of the radioactive seed are trackable based on location. Irradiation data is visualized in 3D, compared to expected margins, and adjusted as needed during the procedure. This procedure can be performed outside the hospital, and therefore information including preoperative, intraoperative, and postoperative procedures, augmented reality (AR) elements, and interaction with other specialists can be shared remotely and on-site.

[0093] When using a 3D dataset for image guidance acquired before a procedure, a common problem is that deformation of anatomical structures and target tissues occurs during surgery compared to the static preoperative dataset. If tissue deformation occurs, misregistration between the actual deformed tissue and the static dataset may lead to an inaccurate combination of the static dataset and the virtual representation of the tracking device. One way to control target tissue deformation is to acquire imaging datasets, such as magnetic resonance imaging, in a controlled manner so that the target tissue has approximately the same shape at the time of image acquisition (e.g., supine position) and at the time of procedure (e.g., supine position). If the patient's position and orientation are similar at the time of image acquisition and surgical procedure, deformation of soft tissue structures can be reduced compared to when the patient is positioned in a different orientation (e.g., prone position) at the time of image acquisition.

[0094] Additional methods can be used to control the shape of the target tissue relative to the image acquisition. The segmented skin surface from the imaging dataset can be projected onto the actual tissue to verify and adjust the shape of the target tissue to closely resemble the shape at the time of imaging data acquisition. During shape adjustment, the projection of the segmented skin hologram onto the actual surface can be performed using a controlled field of view and alignment portal, as described above. This suppresses the perceived variability in the alignment of the skin hologram relative to the actual surface based on changes in the field of view. The ultrasound probe pressed against the actual skin surface can also cause deformation of the target tissue. Methods can be used to minimize this deformation, such as applying additional ultrasound gel to the skin surface or optionally using an ultrasound probe holder. This deformation can be detected and mitigated by comparing the position of the tracked actual ultrasound probe surface with the segmented holographic skin surface from the imaging dataset. This aligns the tracked probe and the holographic skin surface to HMD coordinates. Feedback can be provided to the operator regarding the extent to which the tracked actual probe is located below or inside the segmented skin surface.

[0095] [Application Example 3: Respiratory Medicine / Lungs]

[0096] Endobronchial ultrasound (EBUS) bronchoscopy is a procedure used to diagnose various types of lung diseases, including inflammation, infection, and cancer. EBUS bronchoscopy is performed by a pulmonologist and uses a flexible tube inserted through the patient's mouth into the trachea and lungs. Similar to the devices used in colonoscopy, but smaller, EBUS scopes can feature a video camera with an ultrasound probe, creating localized images of the lungs and surrounding lymph nodes, allowing for precise identification and evaluation of areas requiring further examination identified by pre-procedure X-rays or CT scans. As is evident from the procedure overview, EBUS and / or robotic EBUS are aligned to a holographic coordinate system (i.e., a head-mounted display or world) and can be tracked by ultrasound (US), EM, fiber optics, or other sensors. EBUS-specific characteristics, such as balloon shape and echo markers, can be modeled and aligned in holographic coordinates. The flexibility of HMD / HDD positioning allows physicians to use multiple preferred positions and move during the procedure as needed.

[0097] Adjacent structures, such as the vocal cords, can be visualized and avoided by the planned and defined 3D trajectory. By planning the target trajectory and tracking and visualizing it relatively, the EBUS can be optimally bent and rotated. The needle tip can be tracked during the 3D approach and Doppler view, allowing confirmation of the target's position. The needle tip can be tracked during multiple operations to ensure optimal biopsy. It should also be noted that a flexible catheter can be used to ablate the target in the same manner. This can be tracked by EM or fiber optics. The sector of the EBUS transducer can be demarcated by holographic lines. Furthermore, referring to Figure 1 and the method described above, both respiratory phase motion and robotics are generally considered.

[0098] CT-to-body divergence is a well-known problem in CT-guided bronchoscopy, where, for example, the pre-procedure multislice CT used to plan procedures such as peripheral pulmonary nodule biopsy does not match the actual lung structure.

[0099] Using augmented reality, the current static patient position and shape can be evaluated intraoperatively using cone-beam CT, and then compared to preoperative multi-slice detector CT to assess the discrepancy. To compare multi-slice detector CT with intraoperative cone-beam CT, the two imaging results can be co-registered and co-projected onto the actual patient by co-registering them to anatomical structures (e.g., the sternum, which is relatively invariant to ventilation and respiratory movements) using volume rendering or segmentation results. Vuforia® image targets can be accurately placed on the actual sternum, which exists both in reality and in both imaging datasets, based on anatomical landmarks such as the jugular notch and xiphoid process. Nodules can also be located in both imaging datasets. This makes it possible to calculate the Euclidean distance between the centers of virtual target nodules in the two imaging modalities. Subsequently, the cone-beam CT can be repeated to adjust ventilation and / or patient position and reduce the discrepancy with the planned multi-slice detector CT. Live fluoroscopy, which uses lower radiation doses compared to CBCT, can be streamed as a virtual display monitor on the HMD and combined with the reprojection of the preoperative MDCT dataset to adjust ventilation and patient positioning. In this case, the MDCT reprojection is performed using the geometric structure of the C-arm camera (distance from X-ray source to detector, etc.) and the pose of the C-arm.

[0100] [Application Example 4: Pancreas / Large Intestine]

[0101] As specified herein, applications are not limited to percutaneous procedures; this technology can actually be applied to minimally invasive or surgical procedures. The following example of pancreatic cancer is an example where HDD can greatly benefit from planning adjacent targets and visualizing the treatment boundary, and the anatomical structure of the pancreas illustrates an example of a complex procedure involving adjacent structures and a planned treatment boundary. Other ablation therapies, such as radiofrequency ablation (RFA), microwave ablation (MWA), and pulsed electric field (PEF) by irreversible electroporation, can also benefit from this technology when combined with 3D printing-based positioning and visualization techniques. It should be noted that in Whipple surgery, reconstruction after resection requires an essential understanding of angiogenesis caused by cancer. Because the pancreas is located deep within the body cavity, all layers, connective tissue, and adjacent structures that require navigation are important.

[0102] Some practitioners prefer microwave ablation (MWA) because it is less affected by heat sinks and the differences can be planned within a holographic model. For example, if a physician plans to use cryoablation, the ablation area can be modeled and aligned in different ways, and the formation of the frozen area can be monitored with ultrasound or CT. When using PEF, the technique models and adjusts the electric field accordingly, and because PEF can be performed in a much shorter time, the data can be displayed in different ways. Another treatment option is ultrasound-guided alcohol injection / ablation, which can also be applied using this technique. In this way, the practitioner can plan and track subsequent procedures, verify the therapeutic effect on the center of the tumor, and add external radiation therapy if necessary. Depending on the needs of adjacent structures, it is possible to consider combining multiple modalities.

[0103] During the planning phase of a procedure, complex anatomical structures can be projected holographically in 3D and displayed floating near or above the patient on the HMD's head-down display, as described above. Material, transparency or opacity, and 3D lighting can be adjusted to appropriately visualize overlapping and enclosed structures, assisting the surgeon during the procedure. Multiple imaging datasets, such as pre- and post-radiotherapy images, can be displayed side-by-side (vertically or horizontally). Vascular structures can be segmented from high-resolution imaging images, and a holographic angiography (vascular) roadmap can be projected. This roadmap can be zoomed and rotated in conjunction with the actual patient to assist during complex surgeries.

[0104] [Application Example 5: Liver]

[0105] Radiofrequency ablation (RFA), microwave ablation (MWA), pulsed electric field ablation (PEF) with irreversible electroporation, percutaneous ethanol injection (PEI), and embolization therapy are all available for liver treatment. Unlike MWA antennas, some RFA probes have multiple small teeth or wires that advance and retract relative to the needle, and these can be tracked, similar to how transcatheter cardiac valves or stents can be aligned at multiple points. Furthermore, it is possible to integrate data such as 3D ultrasound modeling, alignment, holographic alignment, and display for planning, and visualize and display ablation margin data overlaid on the patient. Ablation also has transient effects; for example, MWA ablation expands after 24 hours, so this system can be used and compared at multiple time points, requiring alignment for accurate comparison at those points in time. The planned ablation area and the resulting ablation area are recorded before and after ablation using axial images of the tumor, and are aligned to compare and quantify the degree of agreement in terms of (1) volume, (2) surface area, (3) the percentage of volume exceeding the tumor margin of 5 mm, (4) the percentage of volume not included in the tumor, and (5) the IoU (intersection-over-union) index.

[0106] [Application 6: Planning of Ablation Areas]

[0107] Ablation techniques such as microwave, cryogenic, and pulsed electric field ablation are empirically estimated by applying energy to animals or physical models, but this does not necessarily provide an accurate estimate of the ablation area in human tissue. Recording both the planned and resulting ablation areas allows for a comparison of their volume and shape. The planned and resulting areas are recorded against an imaged tumor and can be compared by aligning the image target tumor. Comparing the planned and resulting ablation areas provides input for updating the estimation of the planned ablation area. For example, if the resulting ablation area is consistently larger than the estimated and planned ablation area, the size of the planned ablation area can be increased in future cases. In the long term, the planned ablation area will better align with the resulting ablation area, providing advantages in 3D guidance and navigation for probe and antenna placement. As a result, the tumor can be better covered, reducing secondary damage to adjacent structures and improving treatment outcomes.

[0108] The alignment and visualization methods described herein can be combined in application. For example, by attaching a skin hologram to a tracking sensor, the segmented skin hologram can be mapped to actual skin from the image, enabling alignment without the use of fiducial markers. This alignment can be applied to breast and liver applications by combining electromagnetic-to-HMD (EM-to-HMD) methods and variations of the alignment portal.

[0109] As another example, it is possible to combine the 3D stereoscopic projection of a 3D hologram floating above an actual patient with the use of an HDD for live imaging. In this case, both projections can benefit from techniques that enhance eye-hand coordination and prevent occlusion and depth blurring.

[0110] As another example, after aligning tracking and imaging data to HMD coordinates, CT or MRI multiplanar images reconstructed from the DICOM data volume can be placed coplanar with the tracked ultrasound images in HMD coordinates, and tumor images including the planned ablation area can be output and compared with the resulting ablation area. The output images can be displayed in standard anatomical directions such as axial, sagittal, and coronal. The system can provide angular feedback to assist the operator in generating anatomical planes.

[0111] To thoroughly explain this disclosure and fully convey its scope to those skilled in the art, exemplary applications are provided. Numerous specific details, including examples of applications of particular components, devices, and methods, are described to ensure a full understanding of the embodiments of this disclosure. It will be clear to those skilled in the art that specific details do not necessarily need to be adopted, the applied embodiments can be embodied in various forms, and nothing should be construed as limiting the scope of this disclosure. Some applied embodiments do not describe in detail well-known processes, well-known device structures, and well-known techniques. Some embodiments, materials, compositions, and methods can be modified, adapted, and transformed to substantially the same results within the scope of the Art.

Claims

1. A method of using augmented reality to align the anatomical structures of a patient for treatment purposes, A step of providing a system, wherein the system is An augmented reality system configured to display an augmented representation of the patient's anatomical features in augmented reality, An imaging system configured to image the anatomical features of the patient and generate an imaging dataset without using a fiducial sensor, A tracking device used during the aforementioned procedure and configured to generate a tracking device dataset, A step comprising the augmented reality system, the imaging system, and the computer system capable of communicating with the tracking device, The steps include: arranging the tracking device in the first axis plane and collecting the tracking device dataset from the tracking device; The steps include: positioning the imaging system in the second axis plane and collecting the imaging dataset from the imaging system; The steps include identifying internal landmarks of the patient using both the tracking device and the imaging system, A step of aligning the first axis plane and the second axis plane based on the identified internal landmark, The computer system generates an imaging volume dataset based on the first axis plane and the second axis plane in a 3D coordinate system. The computer system performs the steps of aligning the imaging volume dataset with the augmented reality system, The steps include rendering the aforementioned imaging volume dataset into an imaging volume hologram, A method comprising the steps of projecting the imaging volume hologram using the augmented reality system.

2. In the method according to claim 1, A method by which the first axis plane and the second axis plane are aligned so as to be orthogonal.

3. In the method according to claim 1, A method by which the first axis plane and the second axis plane are aligned at an oblique angle.

4. In the method according to claim 1, A method for aligning the first axis plane and the second axis plane so that they lie on the same plane.

5. In the method of claim 1, further, The steps include identifying the midline on the patient, A method comprising the steps of rotating the imaging volume hologram so that the holographic midline of the imaging volume hologram coincides with the midline on the patient.

6. The method according to claim 1, further, The steps include creating skin surface imaging data by segmenting skin surface images from the aforementioned imaging dataset, A step of aligning the skin surface imaging data with the patient using a tracking sensor, A method comprising the step of aligning the skin surface imaging data with the imaging dataset and the tracking device dataset.

7. In the method according to claim 1, The method wherein the aforementioned tracking device dataset includes an electromagnetic dataset.

8. The method according to claim 1, further, A step of positioning a surgical robot for the aforementioned procedure, The steps include: attaching a robot image target to a part of the surgical robot in a predetermined position and orientation; The steps include aligning the robot image target in a 3D Cartesian coordinate system, A method comprising the step of determining the position of the robot image target using an augmented reality system coordinate system.

9. The method according to claim 8, further, A step of calibrating and positioning the surgical robot for the procedure, The steps include: programming the surgical robot to move the robot image target in a predetermined pattern that includes a plurality of predetermined periodic stops; In each of the plurality of predetermined periodic stops, the step of adjusting the orientation of the portion of the surgical robot having the robot image target, The steps include capturing an image of the robot image target with the camera of the augmented reality system at each of the plurality of predetermined periodic stops, The steps include associating each of the aforementioned plurality of predetermined periodic stops with the robot coordinate system, The steps include: measuring the position of the robot image target in the robot coordinate system; A method comprising the step of determining a transformation from the robot coordinate system to the augmented reality system coordinate system.

10. In the method according to claim 9, The step of collecting the aforementioned imaging dataset further includes: The steps include placing an image target on the skin surface of the patient, A method comprising the step of identifying the image target using a calibration chip connected to the part of the surgical robot.

11. In the method according to claim 10, The step of aligning the imaging volume dataset with the augmented reality system further includes: The steps include aligning the aforementioned imaging volume dataset with the robot coordinate system, A method comprising the step of transforming the imaging volume dataset from the robot coordinate system to the augmented reality system coordinate system.

12. In the method according to claim 1, The augmented reality system includes a head-down display for projecting the imaging volume hologram.

13. The method according to claim 1, further, A step of providing a predetermined ablation area, A method comprising the step of modifying a predetermined ablation region based on the imaging volume hologram.

14. The method according to claim 1, further, A method comprising the step of aligning the 3D modules of the aforementioned imaging volume dataset.

15. In the method according to claim 14, A method comprising the step of aligning 3D modules of the aforementioned imaging volume dataset, the step of creating an actual 3D model.

16. In the method according to claim 15, The method includes an actual 3D model, an image target position for aligning the actual 3D model to an augmented reality coordinate system.

17. A system that uses augmented reality to align the anatomical features of a patient, An imaging system configured to image the anatomical features of the patient and generate an imaging dataset, A tracking device configured to generate a tracking device dataset, An augmented reality system configured to display an imaging volume hologram of the anatomical features of the patient in augmented reality, The system comprises the augmented reality system, the imaging system, and the computer system capable of communicating with the tracking device. The aforementioned computer system, An imaging volume dataset is generated based on the first axis plane and the second axis plane. The aforementioned imaging volume dataset is rendered as the aforementioned imaging volume hologram. A system configured to project the aforementioned imaging volume hologram into the augmented reality environment of the augmented reality system.

18. In the system described in claim 17, The tracking device is a system comprising at least one of a needle and an electromagnetically tracked ultrasonic wave.

19. In the system described in claim 17, The imaging system includes at least one of a computed tomography (CT) system, a fluoroscopy system, a positron emission tomography (PET) system, a magnetic resonance imaging (MRI) system, and an ultrasound (US) system.

20. In the system described in claim 17, The augmented reality system includes a head-down display.

Images