Device and method for aligning an image model with an augmented reality system before or during surgery

Abstract

A system and method for improving the alignment of an AR (Augmented Reality) unit in a surgical scene is provided. The system includes an Augmented Reality (AR) unit that includes and / or communicates with a head mounted display (HMD) used by a user, and a physical device made of a sterilizable biocompatible material and configured as an alignment template. The AR unit and / or its associated segmentation software can be configured to align a device representation of the physical device on an image model of a patient, and the AR unit and / or the HMD can be configured to align the device representation on the HMD with the aligned image model on the physical device positioned relative to the patient and viewed through the HMD to display the image model or a portion thereof in a corresponding spatial relationship to the patient. The alignment using the physical device simplifies the coordination between the real and virtual devices.

Description

Technical Field

[0001] Background of the Invention 1. Technical Field

[0001] The present invention relates to the field of AR (augmented reality) - assisted surgery, and more particularly, to an improvement in alignment in an AR surgical system.

Background Art

[0002] 2. Description of Related Art

[0002] The prior art includes 3D and 2D (3D / 2D) alignment methods for image modalities, number of image dimensions, alignment basis, geometric transformation, user interaction, optimization procedures, objects, and objects to be aligned.

[0003]

[0003] The prior art includes an AR - based surgical navigation system (AR - SNS) that uses an optical see - through type HMD (head - mounted display). During the intraoperative motion tracking process, calibration of the device, alignment, and calibration of the HMD are used to align the 3D virtual important anatomical structures of the head - mounted display with the actual structures of the patient in the real - world scenario.

[0004]

[0004] The prior art includes projecting a computed tomography (CT) scan using Microsoft®'s Hololens® (Hololens) and then aligning the projected image with a set of fiducial markers.

[0005]

[0005] The prior art includes an evaluation of the surgical accuracy of holographic pedicle screw navigation by a head - mounted device using 3D intraoperative fluoroscopy.

Summary of the Invention

Means for Solving the Problems

[0006] Summary of the Invention

[0006] The following is a simplified summary for providing an initial understanding of the present invention. This summary is not necessarily to identify the main elements, nor to limit the scope of the present invention, but merely serves as an introduction to the following description.

[0007]

[0007] One aspect of an embodiment of the present invention includes an augmented reality (AR) unit that is used by a user and / or communicates with a head-mounted display (HMD), and a physical device made of a sterilizable biocompatible material and configured as an alignment template. The AR unit and / or the segmented software associated therewith is configured to align the device representation of the physical device on the patient's image model. The AR unit and / or the HMD is configured to, on the HMD, align the device representation with the aligned image model on the physical device positioned relative to the patient and viewable through the HMD, and display the image model or a part thereof in a corresponding spatial relationship to the patient.

[0008]

[0008] One aspect of an embodiment of the present invention provides a method including aligning a device representation of a physical device on a patient's image model, and on the HMD, aligning the device representation with the aligned image model on the physical device positioned relative to the patient and viewable through the HMD, and displaying the image model or a part thereof on the HMD in a corresponding spatial relationship to the patient.

[0009]

[0009] These aspects and / or advantages of the present invention, additional aspects and / or advantages, and / or other aspects and / or advantages are described in the following detailed description, may be inferred from the detailed description in some cases, and / or may be learned by practicing the present invention.

[0010] Brief Description of the Drawings

[0010] To better understand the embodiments of the present invention and to show how it can be implemented, reference is now made, purely by way of example, to the accompanying drawings, in which, throughout, like numerals indicate corresponding elements or sections.

[0011]

[0011] The accompanying drawings are as follows.

Brief Description of the Drawings

[0012]

Figure 1A

[0012] A high-level schematic view of a surgical scene using an alignment device according to some embodiments of the present invention.

Figure 1B

[0013] A high-level schematic view of the use of an alignment device according to some embodiments of the present invention.

Figure 1C

[0014] A high-level schematic block diagram of an alignment system according to some embodiments of the present invention.

Figure 1D

[0015] A high-level schematic block diagram of an AR unit and associated HMD according to some embodiments of the present invention.

Figure 1E

[0015] A high-level schematic block diagram of an AR unit and associated HMD according to some embodiments of the present invention.

Figure 1F

[0015] A high-level schematic block diagram of an AR unit and associated HMD according to some embodiments of the present invention.

Figure 2A

[0016] Includes a high-level schematic view of a physical device according to some embodiments of the present invention.

Figure 2B

[0016] Includes a high-level schematic view of a physical device according to some embodiments of the present invention.

Figure 2C

[0016] Includes a high-level schematic view of a physical device according to some embodiments of the present invention.

Figure 2D

[0016] Includes a high-level schematic view of a physical device according to some embodiments of the present invention.

Figure 2E

[0016] Includes a high-level schematic diagram of a physical device according to some embodiments of the present invention.

Figure 2F

[0016] Includes a high-level schematic diagram of a physical device according to some embodiments of the present invention.

Figure 3A

[0017] A high-level flowchart illustrating a method according to some embodiments of the present invention.

Figure 3B

[0017] A high-level flowchart illustrating a method according to some embodiments of the present invention.

Figure 4

[0017] A high-level flowchart illustrating a method according to some embodiments of the present invention.

Figure 5

[0018] A high-level block diagram of an exemplary computing device that can be used with embodiments of the present invention.

Mode for Carrying Out the Invention

[0013] Detailed Description of the Invention

[0019] In the following description, various aspects of the present invention will be described. For the purpose of explanation, specific configurations and details are set forth to enable a full understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be practiced without the specific details presented herein. Furthermore, well-known features may be omitted or simplified so as not to obscure the present invention. In particular with respect to the drawings, the details shown are for illustrative purposes only to exemplify the present invention and are presented to provide the most useful and easily understood examples of the principles and conceptual aspects of the present invention. It is emphasized that in this regard, rather than showing the structural details of the present invention in more detail than is necessary for a basic understanding of the present invention, the description is presented together with the drawings to make it clear to those skilled in the art how the various forms of the present invention can be actually embodied.

[0014]

[0020] Before describing at least one embodiment of the present invention in detail, it is to be understood that the present invention is not limited to the details of the structures and arrangements of components described in the following description or illustrated in the drawings in its application. The present invention is applicable not only to other embodiments that can be practiced or implemented in various ways, but also to combinations of the disclosed embodiments. Further, it is to be understood that the phrases and terms used herein are for the purpose of description and should not be regarded as limiting.

[0015]

[0021] Unless otherwise specifically stated, as will be apparent from the following description, throughout this specification, descriptions using terms such as "process," "electronically process," "calculate," "obtain," "improve," or "derive" refer to the operation and / or process of a computer, computing system, or similar electronic computing device that manipulates and / or transforms data represented as a physical quantity, such as an electronic quantity, in a register and / or memory of a computing system into other data represented as a similar physical quantity in a memory, register, or other such information storage, transmission device, or display device of the computing system.

[0016]

[0022] Embodiments of the present invention provide an efficient and economical method and mechanism for aligning a patient's image model in an augmented reality (AR) surgical system, thereby providing an improvement in the technical field of AR-assisted surgery. Various embodiments include using an alignment template to align a patient's image model in an AR surgical system before and / or during surgery. Embodiments may include a system and method for improving the alignment of an AR (augmented reality) unit in a surgical scene. Some embodiments include a head-mounted display (HMD) used by a user and / or an AR unit communicating therewith, and a physical device made of a sterilizable biocompatible material and configured as an alignment template. The AR unit and / or associated segmentation software may be configured to align a device representation of the physical device onto a patient's image model, and the AR unit and / or HMD may, on the HMD, align the device representation with an aligned image model on the physical device positioned relative to the patient and viewable through the HMD to display the image model or a portion thereof in a corresponding spatial relationship to the patient. Embodiments may include ways to coordinate the design of the physical device to spatially synchronize a virtual model with the actual patient and additional displays relevant to the surgery, for example based on image data. Alignment using the physical device simplifies the coordination between the real and virtual devices.

[0017]

[0023] In current AR-assisted surgical systems, a patient's image model (derived, for example, from past images or recent CT or magnetic resonance imaging (MRI) models) is used to plan and perform surgical procedures. However, precise alignment of the patient's image model in an AR system to provide a highly reliable AR interface to the surgeon is challenging for several reasons. That is, (i) during surgery, a large portion of the patient's surface may be covered and thus not available for use in alignment, (ii) markers or stickers used in prior art for triangulation are not accurate enough, especially for deep and precise surgical procedures, (iii) the use of markers (e.g., fiducial markers) in image processing may require a recent patient scan and may not be able to use past scans, and is often not accurate enough, and further, there is a possibility that markers may remain on the patient's body from the scan to the procedure, and (iv) changes in the patient's precise position and orientation can negatively affect the AR alignment.

[0018]

[0024] In contrast, the disclosed embodiments provide a device that can function as an alignment template that can be placed on a patient before or during surgery when AR realignment is recognized as necessary. The AR system can be configured to identify the device and use its position to align the image model on the AR model. The device can be rigid and made of a sterilizable biocompatible material (e.g., sterilizable plastic) and can have features related to a particular patient, features related to general patient anatomy, and / or features related to the particular surgical procedure being performed. The device can be 3D printed in situ or used as a template for a particular type of surgery where the anatomy is relatively similar among patients.

[0019]

[0025] The specific shape of the device can be designed using the image model (and / or general anatomical data) to achieve maximum accuracy with respect to the geometric characteristics of the surgical scene in light of the type of surgery in which the device is used.

[0020]

[0026] Advantageously, the disclosed embodiments bridge the gap between the real surgical scene and the AR model using physical devices not used in the prior art. Further, these techniques typically require a significant portion of the patient to be exposed, which is usually a condition that cannot be obtained during surgery since the patient's body is mostly covered except at the exact location of the surgery. Advantageously, the disclosed devices can provide patient-specific alignment. This is in contrast to general alignment that directly associates the real world as taught by the prior art to the virtual environment, without the intervention of the disclosed alignment by physical devices related to the patient. Additional advantages of the disclosed embodiments may include not having to use fiducial markers and overcoming the limitations associated with the methods of using fiducial markers included in the prior art.

[0021]

[0027] Advantageously, embodiments of the disclosed devices can be non-invasive and, in fact, enable AR visualization that replaces the direct use of invasive surgical guides taught by the prior art. In the disclosed embodiments, the alignment of the device can be preparation for the actual surgery and may eliminate the need to use a surgical guide and align with a head-mounted device during surgery, as taught by the prior art.

[0022]

[0028] Figure 1A is a high-level schematic view of a surgical scene using an alignment device 110 according to some embodiments of the present invention, Figure 1B is a high-level schematic view of the use of the alignment device 110 according to some embodiments of the present invention, and Figure 1C is a high-level schematic block diagram of an alignment system 100 according to some embodiments of the present invention. The system 100 may include an AR unit 120 that is used by a user and / or communicates with a head-mounted display (HMD) 130, and a physical device 110 made of a sterilizable biocompatible material and configured as an alignment template. The AR unit 120 and / or associated segmentation software 150 may be configured to align a device representation 115 of the physical device 110 onto a patient image model 122, and the AR unit 120 and / or HMD 130 may, on the HMD 130, align the device representation 115 with an aligned image model 122 on the physical device 110 that is positioned relative to the patient and viewed through the HMD 130, and be configured to display the image model 122 or a portion of the image model in a corresponding spatial relationship to the patient.

[0023]

[0029] As schematically illustrated in Figure 1A, one or more treating physicians and / or other personnel treating a patient and using the HMD 130 can use the device 110 as a reference for associating spatial information related to the patient with the actual morphological characteristics of the patient being treated, and align spatial information related to the patient (e.g., the position of the patient or a part of the patient) with the actual surgical scene.

[0024]

[0030] As schematically illustrated in FIG. 1B, for example, the initial alignment step 154 on the virtual display associated with the AR unit 120 and / or the segmentation software 150 can be performed by aligning the device representation 115 onto the patient's image model 122 (e.g., spatial data derived from computer tomography (CT), magnetic resonance imaging (MRI), and / or ultrasound (US) images, etc.). The AR unit 120 can then spatially associate the device representation 115 with the image model 122, for example, in the form of a 3D combined model (e.g., as a mesh or point cloud). The segmentation software 150 associated with the AR unit 120 can be configured to perform any data conversion involved, such as adjusting the data representation method and format as needed. In various embodiments, spatially associating the device representation 115 with the image model 122 can be performed automatically, partially automatically (e.g., with manual confirmation), or manually (e.g., by the segmentation software 150 and / or the AR unit 120). In various embodiments, spatially associating the device representation 115 with the image model 122 can be performed automatically, partially automatically, or manually by the HMD 130 itself.

[0025]

[0031] During the operation, the user may only be required to perform an alignment step 210 that includes moving (virtually) a combined model including a device representation 115 spatially associated with the image model 122 so that the device representation 115 overlaps the actual physical device 110 viewable through the user's HMD 130. The alignment step 210 may be performed with respect to the shape of the physical device 110, with respect to the device representation 115 on the physical device 110, and / or with respect to markings, patterns, and / or codes (e.g., QR codes). For example, the HMD 130 may include software or software that causes a software or processor configured to morphologically align the device representation to the physical device 110, such as by applying a transformation matrix to identify the position of the virtual scene relative to the physical device 110. Since the image model 122 is aligned with the device representation 115 in step 154, when the device representation 115 is aligned on the physical device 110, the image model 122 is also correctly aligned over the actual patient (e.g., displayed by the contour 109 as viewable through the HMD 130). For example, spatial content such as internal structures can be correctly aligned with the anatomical features of the patient (e.g., where the device 110 is placed on top). In various embodiments, the alignment 210 can be performed automatically, partially automatically (e.g., with manual confirmation), or manually (e.g., by the AR unit 120 and / or the HMD 130).

[0026]

[0032] If there are multiple users with multiple HMDs 130 or other devices, any of those users can perform the alignment step 210 independently, so that the image model 122 can be aligned in parallel and individually over the patient from each perspective without the need for further coordination and communication between the HMDs 130 and between them and the AR unit 120.

[0027]

[0033] As schematically illustrated in FIG. 1C, various configurations of the system 100 and the AR unit 120 can be implemented. The AR unit 120 can be integrated with the HMD 130, and either the AR unit 120 and / or the HMD 130 can include a camera for capturing the device 110 over the patient. For example, the AR unit 120 and / or the HMD 130 can include any type of AR device such as an HMD and / or Microsoft® Hololens®. Any portion of the image model 122 and / or the associated medical information can be presented on the HMD 130 during surgery and according to the user's preference as needed. An additional HMD and / or display 130A can communicate with the AR unit 120 via, for example, a communication link 99 (e.g., WiFi, Bluetooth, or any other communication protocol). In certain embodiments, the additional HMD and / or display 130A can be mutually aligned with the HMD 130 using only the device 110 as a common alignment target without using communication between any displays. Examples of the additional HMD and / or display 130A can include additional HMDs (e.g., Hololens® devices) used by another doctor in the operating room and / or experts, advisors, interns, trainees, etc. located at a remote location. Examples of the additional HMD and / or display 130A can include other devices such as smartphones and / or remote displays that are used in real time for purposes of consultation, monitoring, or teaching. In certain embodiments, the additional HMD and / or display 130A can include an internal representation of the surgical system that can be used by a robotic surgical system to derive data related to assisting with or analyzing a procedure being performed.

[0028]

[0034] The AR unit 120 typically includes a display surface or a projection module that provides additional content superimposed over the user's field of view and / or over content presented from different sources (e.g., simulations or models). For example, using AR glasses, content such as surgical - related data or images, which are used for the expansion of a surgical scene observed by a doctor, can be added over the user's field of view. The AR unit 120 integrates additional content with respect to the user's field of view and / or content from other sources by alignment, for example, providing common spatial coordinates for the added content and the content from the field of view and / or other sources (e.g., simulations or models). The AR unit 120 may include various corresponding sensors and communication modules to assist in the integration. Examples of the AR unit 120 include not only display or projection units with glasses (e.g., virtual retinal display - VRD or EyeTap), contact lenses, head - mounted displays (HMD), such as optical head - mounted displays (OHMD), head - up displays (HUD), similar to Microsoft®'s Hololens® and related devices, but also any smartphone display configured to provide AR content (e.g., content superimposed over content from the device's camera and / or content from other sources). Although the AR unit 120 and the HMD 130 are shown separately, they can be integrated as one device, optionally assisted by a connection processor.

[0029]

[0035] Figures 1D - 1F are high - level schematic block diagrams of an AR unit 120 and related HMD 130 according to some embodiments of the present invention. Various embodiments of the AR unit 120 and HMD 130 or other HMDs and / or display 130A can be used in various embodiments. For example, the AR unit 120 can be separate from the HMD 130 or integrated with the HMD within an AR device such as, for example, the Microsoft® Hololens® system. The computing processing power of the system 100 (see, for example, the schematic example of FIG. 1E) can be improved by using the Hololens® system as an independent device without an external AR unit 120 (see, for example, the schematic stand - alone type example of FIG. 1D) or together with an AR unit 120 implemented by, for example, one or more processors. In some embodiments, the AR unit 120 is used as a main processor that streams AR content to the HMD 130 and / or Hololens® 130, and its computing processing load can be reduced. In some embodiments, multiple displays and / or HMDs 130, 130A are used, and the AR unit 120 can be configured to perform alignment for all user displays using the same physical device 110 (see, for example, the schematic example of FIG. 1E). In some embodiments, an additional computing processing module 120A (e.g., segmentation software 150) can be used to perform at least a portion of the computing processing effort. For example, the 3D modeling module 120A can be implemented separately or integrated with the AR unit 120 to perform a combination of the device representation 115 and an anatomical image model 122 derived, for example, from an image module or directly through, for example, the HMD 130 (see, for example, the schematic example of FIG. 1F).

[0030]

[0036] For example, the 3D modeling module 120A may include segmentation software 150 such as D2P™ (DICOM-to-PRINT, where DICOM is short for Digital Imaging and Communications in Medicine) to convert various types of medical data into a 3D digital model (e.g., by converting slice data into volumetric measurement data while applying AI algorithms for image processing and / or feature identification, etc.). The segmentation software 150 such as D2P™ may be configured to convert medical image data, such as CT, MRI, and / or US images, into any type of 3D model that can be digitally processed. For example, the segmentation software 150 can segment the image, aggregate the segmentation, and bring about a digital file that can be used in 3D printers, VR devices, surgical planning software, and CAD software, thereby converting the imported DICOM images into their respective 3D models. The AR unit 120 and / or the segmentation software 150 may optionally be configured to apply image processing related to the specific procedure that is about to be performed, for example, to display only specific portions of the image model 122 on the HMD 130 according to the user's preference. The device representation 115 can be imported into the segmentation software 150 as a virtual scan aligned with the 3D model and / or the image model 122. The alignment 154 can be performed via an interface to the segmentation software 150 by moving and rotating the device representation 115 and / or the image model 122 until they are aligned with identifiable anatomical features within the image model 122, if any. In certain embodiments, the alignment 154 is at least partially automatically performed by the segmentation software 150 and can be adjusted manually, if necessary (in a virtual environment, see, for example, FIG. 1B).

[0031]

[0037] Alternatively or additionally, the HMD 130 can be used to directly align the device 110 with the patient's anatomical structure as viewed through the HMD 130. In various embodiments, the alignment of the device 110 can be performed by either the AR unit 120, the HMD 130, or another device that communicates with the AR unit 120 and / or the HMD 130.

[0032]

[0038] In various embodiments, the physical device 110 can be used to synchronize multiple HMDs 130, such as multiple HoloLenses and / or multiple computer displays, that utilize simple hardware (the device 110) rather than a communication link. This approach can synchronize the HMDs 130 by a common alignment of the device 110 instead of communicating composite 3D data, and can be particularly beneficial when the data stream and communication load are large. For example, the image model 122 can include CT data, additional information related to the surgery, and models for the surgery, which can be computationally very intensive. Instead of the prior art that requires streaming data between the HMDs 130, the alignment using the device 110 eliminates this need while providing complete spatial synchronization between the HMDs 130 and between related units and modules. If the physical device 110 is needed during the surgery (e.g., when the patient moves, when the surgery involves multiple stages, etc.), a simple repetition of the alignment is achieved by placing the physical device 110 in the appropriate position and spatially synchronizing the HMDs 130 accordingly. In a robotic system, spatial synchronization can also be achieved through device alignment to replace or extend a composite user interface for this purpose.

[0033]

[0039] The image model 122 can be constructed from one or more sources such as CT (Computed Tomography), MR (Magnetic Resonance Data such as MRI - Magnetic Resonance Imaging), US (Ultrasound, for example when performing surgery on stones such as urinary stones or gallstones), PET (Positron Emission Tomography), etc. The image model 122 can be directly constructed using a 3D scanner that generates a point cloud model capable of directly aligning the device 110, regardless of the presence or absence of intermediate segmentation and mesh model generation. In certain embodiments, the device 110 can be aligned using only the morphological structure of the outer surface (for example, when anatomical features are prominent) without requiring image data. In such cases, the 3D scanner can be used directly without using an intermediate mesh model for the patient's anatomical structure and / or the device 110. For example, for surgical procedure planning, or for non-invasive procedures, or as a baseline for invasive procedures or (for example, in emergency cases or to reduce the radiation dose to which the patient is exposed) to review previous image data where additional imaging is not required, only an outer surface scan can be used.

[0034]

[0040] In certain embodiments, the image model 122 can at least partially include a virtual model, for example, with respect to the peripheral region or when no image data is available for the patient. One advantage of the disclosed device 110 is emphasized that they can be used for alignment even in the absence of specific image data, and thus it is not necessarily required to perform an imaging procedure or have recent image data to perform the alignment. This advantage is important in cases of emergency surgery, when it is recommended to avoid radiation, or when old image data that may serve as a basis for a virtual model is available.

[0035]

[0041] In various embodiments, the disclosed alignment can be used to perform any of non-invasive procedures such as focused ultrasound irradiation, minimally invasive procedures, and fully invasive procedures.

[0036]

[0042] In certain embodiments, an additional HMD and / or display 130A may include a simulation model used to check possible techniques for the current surgery. For example, an expert located remotely can use a simulation model spatially synchronized with the actual patient via alignment of the device 110 to check or verify different surgical techniques, and the actual surgeon can update the suitable technique. Optionally, the physical device 110 can be modified to provide a more appropriate alignment with respect to the recommended technique and generated in real time in the operating room. During the surgery, the physician can keep the model 110 displayed on the HMD 130 (even when the physical device 110 is removed) and use it for orientation or navigation throughout the surgery as needed, and optionally maintain communication with an expert regarding the physical device 110 located remotely. The device 110 can be generated to include instructive annotations to assist the surgery. In certain embodiments, the actual intraoperative collaboration can be enabled by the physical device 110 to improve the simulation specific to the current patient (such as in the Procedure Rehearsal Studio™ system, PRS).

[0037]

[0043] FIGS. 2A-2F include high-level schematic views of a physical device 110 according to some embodiments of the present invention. The physical device 110 as an alignment template can be shaped to fit a particular patient, designated anatomical features, and / or a particular surgical procedure.

[0038]

[0044] Figures 2A and 2B schematically illustrate (in side view and top view, respectively) a physical device 110 placed above the patient's sacrum for surgery on the patient's sacral region. As schematically illustrated, device 110 can be designed to have protrusions 111 that contact at selected patient anatomical points, such as the posterior pelvis and sacral bones. Device 110 can further include a marker 113 or other indicia (e.g., a coded chart, sticker, tracker, etc.) to assist with alignment and / or can have a particular shape or portion.

[0039]

[0045] Figures 2C and 2D schematically illustrate (in side view and top view, respectively) a physical device 110 placed above the patient's face for surgery on the patient's facial region. As schematically illustrated, device 110 can be designed to have protrusions 111 that contact at selected patient anatomical points, such as the forehead, nose, and cheekbones as schematically illustrated. Device 110 can further include a marker 113 or other indicia (e.g., a coded chart, sticker, tracker, etc.) to assist with alignment and / or can have a particular shape or portion. Various embodiments of device 110 can be adjusted for use with any particular patient anatomical feature or landmark.

[0040]

[0046] Figures 2E and 2F are high-level schematic diagrams of a physical device 110 according to some embodiments of the present invention. Figure 2E provides a schematic example of a device 110 having two or more portions with different characteristics, as disclosed herein, and Figure 2F provides a schematic example of a device 110 having adjustable features for conforming a template of the device to a particular patient, as disclosed herein.

[0041]

[0047] As schematically shown in FIGS. 2A - 2D, it should be noted that the image model 122 can be configured to include portions corresponding to anatomical features of a patient on which the device 110 may be placed, such as facial bones of the face (e.g., cheeks, forehead), nose or optionally teeth, or bony prominences of the pelvis or sacrum, or any other anatomical features.

[0042]

[0048] The corresponding marker 113 can be used to specify the template of an individual device and / or portions of the device template so that they can be distinguished. In certain embodiments, the AR unit 120 can be configured to detect unintentional changes or deviations of the device 110 and provide corresponding warnings to prevent misalignment inaccuracies.

[0043]

[0049] In certain embodiments, the physical device 110 can have a first portion 112 used for alignment and a second portion 114 adjustable for particular patient characteristics and / or changes in the patient's anatomical structure. For example, the second portion 114 can include at least a portion of the periphery of the physical device 110 that contacts the patient. The second portion 114 can be flexible and / or mechanically modifiable to provide adjustments for particular patient characteristics. For example, a physical device 110 for facial surgery can have a rigid first portion 112 and a flexible periphery 114 (e.g., the portion 114 in FIG. 2E is spread out to form a complete circle or a part of a complete circle or ellipse or other peripheral shape). In certain embodiments, the device 110 can include a fixed upper geometry and an adjustable lower geometry. For example, the second portion 114 can be flexible and / or malleable (see, e.g., FIG. 2E). The device 110 can be generated from a template, e.g., a library, and digitally adjusted for the patient's anatomical structure and / or changes in the patient's anatomical structure. Alternatively or additionally, the device 110 can be configurable at specific locations such as joints 116 (see, e.g., FIG. 2F). For example, the joint 116 can be cylindrical. The degree of deformation of one or more second portions 114 can be visually detected, e.g., by the AR unit 120 and / or the HMD 130, and optionally using scale marks 117 to directly display the degree of modification (e.g., rotation) applied to the portion of the device 110 when deforming or conforming to the particular patient's anatomical features.

[0044]

[0050] In various embodiments, device 110 may include various color usages and / or patterns to provide or assist with efficient and accurate optical alignment. In various embodiments, device 110 may include multiple separate (or interconnected) portions as multiple devices 110 that are used simultaneously for alignment. For example, in the case of one large (e.g., tens of cm in width) device 110, while inconvenient for use, smaller, and in some cases multiple, devices 110 can be placed on top of anatomical features and alignment can be performed for the multiple devices 110. Alternatively or additionally, alignment adjustments can be made, for example, by using or adding a small device 110 during surgery to improve alignment accuracy. In certain embodiments, to provide alignment with fewer obstacles and / or more prominent landmarks than can be obtained in the immediate vicinity of where the surgery is being performed, device 110 can be placed outside the direct area of the surgery, and in some cases, further away anatomical landmarks or even next to the patient. Optionally, using device 110 without contacting the patient can reduce the frequency of sterilization and / or can be sterilized together with other operating room devices or reused during surgery without additional sterilization. Different types of device 110 can be used under different circumstances such as stages of surgery and the required and achieved alignment accuracy, and can also be switched to handle changing circumstances.

[0045]

[0051] Device 110 can be generated as a modifiable template for different patient characteristics such as size, age, anatomical features, etc., and can be generated by modifying it to fit a specific patient as needed. In certain embodiments, along with the device template, modifications to a specific device can be provided as instructions for adjusting the template for a specific situation, for example, suggesting a specific rotation angle for a specific use. The adjustable device 110 can be 3D printed as one piece and / or multiple pieces that can be assembled to form the device 110. The AR unit 120 and / or the HMD 130 can be configured to identify the scale marks 117 and then determine the precise structure of the modified device template 110 therefrom.

[0046]

[0052] In certain embodiments, the system 100 can include one or more libraries of device representations 115 selected prior to a specific surgery as a specific device and / or template. For example, codes can be used to specify the device representations 115 and associate them with a specific surgery, anatomical region, patient characteristics, and / or a specific patient. Correspondingly, it can enable simulation and / or reconstruction of a specific procedure for the alignment device used therein.

[0047]

[0053] In certain embodiments, the device 110 can include one or more large reusable parts (e.g., part 112) and small adjustable disposable parts (e.g., part 114) to reduce the time and materials required for printing in a specific case.

[0048]

[0054] Device 110 can be customized according to a specific patient, a specific surgery, and / or specific anatomical features. In certain embodiments, the shape of device 110 is defined using the patient's image model 122 and can be configured to conform to a body part of the patient that is close to the surgical site (e.g., in oral and maxillofacial surgeries). The corresponding device 110 can be printed (in one or more copies) by 3D printer 140 as part of the surgical preparation procedure or during the surgery if, for example, it is found that a modification is needed or a copy is required, provided that a sufficiently fast 3D printer is available. Alternatively or additionally, a template 110 of one or more adjustable devices can be used to provide and / or supplement device 110.

[0049]

[0055] Alternatively or additionally, device 110 can be pre - prepared according to specified anatomical features such as the sacral anatomical region, the sternum, or other relatively stable structures. Device 110 can include finely adjustable parts that can be adjusted to fit the precise anatomical structure of the patient, either by physical manipulation of device 110 (e.g., partial removal, pressing of flexible parts, etc.) or by modifying the device template when actually printing the device as part of the surgical preparation for a specific patient. Device 110 can also be shaped for internal use, for example, in the case of major surgical interventions to improve the accuracy of alignment of internal organs or implants.

[0050]

[0056] In certain embodiments, AR unit 120 can be further configured to derive the shape of physical device 110 from the image model 122 and / or specific patient characteristics of a specific patient. In certain embodiments, AR unit 120 can be configured to adjust a given template of physical device 110 according to specific patient characteristics and send the adjusted template to 3D printer 140 to print a personalized physical device 110.

[0051]

[0057] In any of the embodiments, for example, the 3D printer 140 adjacent to the operating room may be configured to print the physical device 110 in preparation for and / or during the surgery according to a given template and / or an adjusted template provided by the AR unit 120.

[0052]

[0058] Figures 3A, 3B, and 4 are high-level flowcharts illustrating method 200 according to some embodiments of the present invention. Method 200 can be implemented at least in part in AR unit 120 (and optionally also within HMD 130), and / or can include using image data 90 as an initial input to segmented software 150 that can be implemented at least in part in processing unit 155. Method 200, when implemented in segmented software 150, includes importing or receiving (step 152) image data 90, such as data of a patient or a part of a patient or something describing the patient or a part of the patient, and creating (step 122A) a patient anatomical structure model 122 by applying additional image processing using segmented software such as D2P™ described above and / or optionally assisted by, for example, artificial intelligence (AI) or deep learning methodologies to extract specified features, and importing (or generating) (step 115A) a virtual alignment device model of physical device 110 that can correspond to device representation 115 from a library 162 of alignment devices specific to the patient and / or specific to the procedure, aligning (step 154) the virtual device model to the patient's anatomical structure, optionally subsequently creating 205 the physical alignment device 110 (e.g., by 3D printing) and sterilizing 207 the device, and applying (step 156) morphological adjustments to the virtual alignment device based on the patient's anatomical structure as needed, and exporting (step 158) the patient anatomical structure model including the aligned alignment device to the AR unit / device 130 / 120, respectively. When the device model is adjusted and generated for the patient and / or the procedure, the physical device can be placed at the patient's anatomical landmarks (step 209) and aligned with the virtual device and the anatomical structure within AR unit 120 and / or HMD 130.Note that, if necessary, the AR unit 120 and / or the HMD 130 can be further used to adjust the placement of the physical device 110, for example, to improve alignment and enhance accessibility to the region of interest.

[0053]

[0059] As schematically illustrated in FIG. 3B, method 200 can include using a template for a physical alignment device, such as a template for an adjustable, flexible, and / or malleable device, at one or more sizes adjustable for a particular patient by deforming, relatively moving parts, and / or peeling or removing parts of the template according to certain patient characteristics disclosed herein (step 204). Following the import of the template of the virtual alignment device model (step 115B) and the alignment of the template of the virtual device model to the patient's anatomical structure (step 154A), the morphological structure of the template of the virtual alignment device can be adjusted based on the patient's anatomical structure (step 157), and the physical template for the alignment device can be adjusted accordingly (step 206), followed by sterilization 207, and then exporting to the AR unit a model of the patient's anatomical structure including the aligned and adjusted alignment device (template) (step 158A). The device template and its adjustment can be performed on a template of one or more devices and / or parts of a template of one or more devices (e.g., having parts of different sizes and / or dimensions). Corresponding markers 113 can be used to designate the individual device templates and / or parts of the device template so that they can be distinguished.

[0054]

[0060] Exemplary method 200 may include, as schematically illustrated in FIG. 4, aligning a device representation of a physical device on a patient's image model (step 154), and on the HMD, aligning the device representation with the aligned image model on the physical device positioned with respect to the patient and viewable through the HMD (step 210), and displaying the image model or a portion thereof on the HMD in a corresponding spatial relationship with respect to the patient (step 212).

[0055]

[0061] Alignment 210 may be performed, according to various embodiments, to reliably spatially correspond data from different sources, such as the image model 122 and / or a portion thereof, with the actual patient viewable, for example, through the HMD 130, in a virtual environment. Alignment 210 includes a spatial transformation between spatially represented data sets, for example, a spatial transformation from the device representation 115 to the physical device 110 as imaged on the HMD 130, and it can be verified that they match. The corresponding spatial transformation can be applied to the image model 122 as that required to transform the device representation 115 to the physical device 110, whereby the image model 122 is aligned over the patient. The spatial transformation can relate different coordinate systems and can be determined by the format and spatial characteristics of the data sets and the realizable modifications or adjustments of the physical device 110 as disclosed herein. For example, the spatial transformation can be related to the change in the viewing angle and distance to the physical device 110 and / or the spatial characteristics of the image model 122. In various embodiments, alignment 210 can be performed using an alignment algorithm that is part of the application programming interface (API) of the HMD 130 (e.g., Hololens (registered trademark)).

[0056]

[0062] In various embodiments, method 200 may include, for example, the following steps: shaping a physical device into a shape that conforms to a particular patient, specified anatomical features, and / or a particular surgical procedure (step 220), deriving the shape of the physical device from an image model and / or particular patient characteristics of the patient (step 222) and / or adjusting a given template for the physical device according to particular patient characteristics (step 224), and 3D printing the adjusted template as a personalized physical device (step 232). In certain embodiments, method 200 may include configuring a first portion of the physical device to perform alignment and configuring a second portion of the physical device to be adjustable with respect to particular patient characteristics (step 226).

[0057]

[0063] In any of the embodiments, method 200 may further include aligning multiple AR displays using the same physical device (step 240) and / or coordinating multiple different types of proximal and / or distal AR displays using alignment (step 242).

[0058]

[0064] The physical device 110 can be made of biocompatible materials within the sterilizable range. In some embodiments, the physical device 110 can be produced by various procedures other than 3D printing, and can be made of any sterilizable biocompatible material including various polymers (e.g., nylon), plastics or metals (e.g., titanium). In certain embodiments, the physical device 110 can be generated using 3D printing, and optionally using a 3D printer adjacent to the operating room to provide the device 110 as part of the surgical preparation or during the surgery as needed. When the physical device 110 is 3D printed, it can also be sterilizable and made from a corresponding compatible material that is biocompatible. Non-limiting examples include materials such as Figure 4® MED-WHT 10, a rigid white material, and Figure 4® MED-AMB 10, a rigid translucent material, both of which are biocompatible and sterilizable UV curable polymers that can be used with the Figure 4® printer from 3DSystems®. Other materials can include polymers such as ABS (acrylonitrile butadiene styrene) or its modified forms, or any other plastic material as long as it is biocompatible and sterilizable. Additional examples of materials include DuraForm PA (SLS), a durable thermoplastic with balanced mechanical properties and high definition surface resolution, or other nylon-like and / or polypropylene-like thermoplastics that are biocompatible and sterilizable. Any of these materials can be cured, for example, by UV or laser as part of the device production process. Metals such as titanium or its alloys can also be used to generate the physical device 110.

[0059]

[0065] The inventors have noted that using 3D printable materials is advantageous in terms of the time required to prepare the device 110 prior to surgery, avoiding waste of time in the operating room. For example, by using a pre-prepared template with automatic segmentation, it may be possible to 3D print the device 110 within 1 to 1.5 hours, including sterilization, as required, and it can be used during surgical planning without a time penalty.

[0060]

[0066] The device 110 may include a flexible material, a combination of a rigid material and a flexible material, or a rigid material, and may also include markers (e.g., titanium markers) and / or stickers for assisting alignment. Portions of the device 110 may be adjustable to the patient's surface features (e.g., by being flexible or modifiable, such as by cutting or curving an edge or using Boolean operations for precise adjustment), while certain portions of the device 110 may be configured to simplify alignment. In relation to CAD (computer-aided design) operations, Boolean operations such as subtraction, intersection, addition, union, etc. can be applied to subtract, for example, scanned or modeled features from regions of a template of a virtual device (thereby resulting in an adjusted virtual device representation 115), so that a part of the template of the device can be adjusted to precise patient features derived from the patient's scanning and / or image model 122. Following the adjustment, the device 110 can be 3D printed, for example, to conform to specific patient features as part of surgical preparation. Certain device features can be left unchanged for alignment purposes, and / or the AR unit 120 can align the physical device 110 according to the adjusted virtual device representation 115.

[0061]

[0067] In certain embodiments, the plurality of devices 110 can be prepared, as needed, to verify or re - establish alignment as alternative designs or auxiliary devices for different stages of the surgery. Note that the 3D printer enables the simultaneous preparation of the plurality of devices 110 for multiple surgeries.

[0062]

[0068] Advantageously, the simple device 110 and its simple use can save valuable time during the surgical procedure and enable achieving maximum alignment accuracy before and during the surgery. For example, using a simple physical model device 110 for alignment and adjustment is easier than using gestures to place a virtual model in the correct position relative to the patient. The ease of use allows for re - applying the physical device 110 and appropriately adjusting the alignment to accommodate changes during the surgery. Thus, the physical device 110 provides a physical user interface for the surgeon to adjust the AR alignment during the surgery, as needed (e.g., following patient movement or re - positioning).

[0063]

[0069] FIG. 5 is a high - level block diagram of an exemplary computing device 170 that can be used with an embodiment of the present invention. For example, the computing device 170 can be used at least in part to implement at least one of the derivation and processing of the AR unit 120, the HMD 130, and / or the image model 122 and / or the device representation 115. Additionally or alternatively, the processing unit 155 and / or the segmentation software 150 can be at least partially implemented by the computing device 170 or a part thereof.

[0064]

[0070] Computing device 170 may include, for example, one or more central processing unit processors (CPUs), one or more graphics processing units (GPUs or general-purpose GPUs, i.e., GPGPUs), a chip, or any suitable computing device or a controller or processor 173 that can be or include it, an operating system 171, a memory 172, a storage 175, an input device 176, and an output device 177.

[0065]

[0071] The operating system 171 may be or include any code segment designed and / or configured to perform tasks involving coordination, scheduling, mediation, supervision, control, or other management of the operation of the computing device 170, such as scheduling the execution of programs. The memory 172 may be, for example, random access memory (RAM), read-only memory (ROM), dynamic RAM (DRAM), synchronous DRAM (SD-RAM), double data rate (DDR) memory chips, flash memory, volatile memory, non-volatile memory, cache memory, buffers, short-term memory units, long-term memory units, or other suitable memory units or storage units, and may be or include multiple, possibly different, memory units. The memory 172 can store, for example, instructions (e.g., code 174) for performing methods and / or data such as user responses, interrupts, etc.

[0066]

[0072] The executable code 174 can be any executable code, such as an application, program, process, task, or script. The executable code 174 can be executed by the controller 173, optionally under the control of the operating system 171. For example, when executed, the executable code 174 can cause application execution such as the generation or compilation of computer code or VR execution or speculation according to embodiments of the present invention. The executable code 174 can be code generated by the method embodiments described herein. For the various modules and functions described herein, one or more computing devices 170 or components of the computing device 170 can be used. Devices including components similar to or different from those included in the computing device 170 can be used, connected to a network, and used as a system. One or more processors 173 can be configured to perform embodiments of the present invention, for example, by executing software or code.

[0067]

[0073] The storage 175 can be or include, for example, a hard disk drive, a floppy disk drive, a compact disc (CD) drive, a writeable CD (CD-R) drive, a universal serial bus (USB) device, or other suitable removable and / or fixed storage unit. Data such as instructions, code, VR model data, parameters, etc. can be stored in the storage 175, loaded from the storage 175 into the memory 172, and processed there by the controller 173. In some embodiments, some of the components shown in FIG. 5 may be omitted.

[0068]

[0074] The input device 176 can be or include, for example, a mouse, a keyboard, a touch screen or pad, or any suitable input device. As indicated by block 176, it will be recognized that any suitable number of input devices can be operably connected to the computing device 170. The output device 177 can include one or more displays, speakers, and / or any other suitable output device. As indicated by block 177, it will be recognized that any suitable number of output devices can be operably connected to the computing device 170. Any applicable input / output (I / O) device, such as a wired or wireless network interface card (NIC), a modem, a printer or fax machine, a universal serial bus (USB) device, can be connected to the computing device 170, or an external hard drive can be included in the input device 176 and / or the output device 177.

[0069]

[0075] Embodiments of the present invention can include one or more articles (e.g., memory 172 or storage 175) such as a non-transitory readable medium of a computer or processor that encodes, includes, or stores computer-executable instructions that, when executed by a processor or controller, for example, perform the methods disclosed herein, or a non-transitory storage medium of a computer or processor such as a memory, a disk drive, or a USB flash memory.

[0070]

[0076] Aspects of the invention have been described above with reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram or portions thereof.

[0071]

[0077] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram or portions thereof.

[0072]

[0078] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other devices to produce a computer-implemented process, such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram or portions thereof.

[0073]

[0079] The foregoing flowcharts and diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each part in a flowchart or a partial diagram may represent a module, segment, or portion of code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions shown in that part may occur out of the order shown in the diagram. For example, two consecutive parts shown may actually be executed substantially simultaneously, or these parts may sometimes be executed in the reverse order depending on the functionality involved. It should also be noted that each part of the partial diagram and / or flowchart diagram, as well as combinations of parts of the partial diagram and / or flowchart diagram, can be implemented by a dedicated hardware-based system for performing the specified function or action, or a combination of dedicated hardware and computer instructions.

[0074]

[0080] In the above description, one embodiment is an example or an implementation form of the present invention. Various occurrences of "one embodiment", "an embodiment", "a specific embodiment", or "some embodiments" do not necessarily all refer to the same embodiment. Although various features of the present invention may be described in relation to a single embodiment, those features can also be provided separately or in any suitable combination. Conversely, although the present invention may be described herein in relation to separate embodiments for clarity, the present invention can also be implemented in a single embodiment. A specific embodiment of the present invention may include features from different embodiments disclosed above, and a specific embodiment may incorporate elements from other embodiments disclosed above. The disclosure of elements of the present invention in relation to a specific embodiment shall not be construed as limiting their use to only that specific embodiment. Furthermore, it should be understood that the present invention can be executed or practiced in various ways, and the present invention can be implemented in specific embodiments other than those outlined in the above description.

[0075]

[0081] The present invention is not limited to those figures or corresponding descriptions. For example, the flow does not have to move through each box or state shown, nor does it have to move in exactly the same order as shown and described. The meanings of the technical and scientific terms used herein are as commonly understood by those skilled in the technical field to which the present invention pertains, unless otherwise defined. Although the present invention has been described with respect to a limited number of embodiments, these are to be construed as illustrative examples of some of the preferred embodiments rather than as limitations on the scope of the present invention. Other possible variations, modifications, and applications are also within the scope of the present invention. Therefore, the scope of the present invention is not limited by what has been described so far, but is to be limited by the appended claims and their legal equivalents.

Claims

1. A system for aligning a patient's image model with an AR system for augmented reality (AR) assisted surgery, comprising: An augmented reality (AR) unit including a head-mounted display (HMD) used by a user; A physical device made of a sterilizable biocompatible material, configured to be non-invasively placed on the patient as an alignment template with respect to anatomical landmarks; The AR unit or associated software is configured to align a device representation of the physical device on the patient's image model, the patient's image model being acquired preoperatively; Without additional imaging, the AR unit or the HMD is configured to: Display the image model or a portion thereof in a corresponding spatial relationship to the patient; Align, on the HMD, a device representation of the physical device, on which the image model is aligned, with the patient's anatomical landmarks and position it with respect to the physical device viewable through the HMD.

2. The system according to claim 1, wherein the physical device is shaped to fit a specific patient, designated anatomical features, and / or a specific surgical procedure.

3. The system according to claim 1, wherein the AR unit is further configured to derive the shape of the physical device from the image model and / or specific patient characteristics of the patient.

4. The system according to claim 1, wherein the AR unit is further configured to adjust a given template for the physical device according to specific patient characteristics and send the adjusted template to a 3D printer to print a personalized physical device.

5. The system according to claim 1, wherein the physical device has a first portion used for alignment and a second portion adjustable for specific patient characteristics.

6. The system according to claim 5, wherein the second portion includes at least a part of the periphery of the physical device in contact with the patient.

7. The system according to claim 6, wherein the second portion is flexible and / or mechanically modified to provide adjustment for the specific patient characteristics.

8. The system according to any one of claims 1 to 7, further comprising a 3D printer configured to print the physical device during the preparation for surgery and / or during the surgery according to a given template and / or the adjusted template provided by the AR unit.

9. The system according to claim 1, further comprising a plurality of user displays, wherein the AR unit is configured to perform the alignment for all the user displays using the physical device.

10. The system according to claim 9, wherein the user display is independently selected from an AR device, different HMDs, smartphones, and remote displays.

11. A method of aligning an image model of a patient to an AR system for augmented reality (AR) assisted surgery, comprising: non-invasively placing a physical device on the patient with respect to the anatomical landmarks of the patient; aligning a device representation of the physical device on an image model of the patient acquired before surgery; aligning, on the HMD, the device representation on which the image model is aligned to the physical device positioned with respect to the patient and viewable through the HMD in a corresponding spatial relationship to the patient without additional scanning, for displaying the image model or a part thereof on the HMD. A method comprising the above steps.

12. The method according to claim 11, further comprising 3D printing the physical device using a sterilizable biocompatible material.

13. The method according to claim 11, further comprising shaping the physical device to fit a specific patient, designated anatomical features, and / or a specific surgical procedure.

14. The method according to claim 11, further comprising deriving the shape of the physical device from the image model and / or specific patient characteristics of the patient.

15. The method according to claim 11, further comprising adjusting a given template for the physical device according to specific patient characteristics and 3D printing the adjusted template as a personalized physical device.

16. ​ The method of claim 11, further comprising configuring a first portion of the physical device to perform the alignment and configuring a second portion of the physical device to be adjustable for a particular patient characteristic. **Claim 17** The method according to any one of claims 11 to 16, further comprising performing the alignment for a plurality of AR displays using the physical device. **Claim 18** The method of claim 17, further comprising using the alignment to coordinate a plurality of different types of proximal and / or remote AR displays.

Images