Dynamic registration of anatomical structures using augmented reality

Abstract

This disclosure relates to a system (100) that may include an augmented reality system (102), an imaging system (104), a measurement system (106), and a computer system (108). The augmented reality system (102) is configured to display an augmented representation (110). The imaging system (104) is configured to image anatomical features (112) and generate anatomical imaging data (142). The measurement system (106) is configured to measure anatomical motion and generate anatomical motion data (144). The computer system (108) is configured to: receive anatomical imaging data (142) and anatomical motion data (144), generate an augmented representation (110) based on the anatomical imaging data (142), associate the augmented representation (110) with the anatomical motion data (144), render the augmented representation (110) on the augmented reality system (102), and selectively update the augmented representation (110) based on the anatomical motion data (144).

Description

[0001] Priority document

[0002] This application claims the benefits of U.S. Provisional Application Serial No. 63 / 025,436 and U.S. Provisional Application Serial No. 63 / 025,604, both filed on May 15, 2020. The full disclosures of the above applications are incorporated herein by reference. Technical Field

[0003] This disclosure relates to augmented reality applications, and more specifically to medical applications employing augmented reality. Background Technology

[0004] This section provides background information related to this disclosure, but it is not necessarily prior art.

[0005] Image-guided surgery has become standard practice in many different medical procedures. It assists physicians by visually correlating intraoperative data with preoperative data. The use of image-guided surgery has been proven to improve the safety and success rate of these procedures. Image-guided surgery can be further enhanced by utilizing augmented reality (AR) technology. AR is an interactive experience of the real-world environment in which computer-generated perceptual information enhances one or more features present in the real world, sometimes across multiple sensory modalities. In a medical setting, these AR technologies can be used to enhance the realistic environment of patient care. For example, during a medical procedure, physicians can view content-specific information within the same field of vision of the patient without shifting their gaze.

[0006] However, even current image-guided surgery presents numerous challenges in medical procedures. For example, a patient's anatomy is not necessarily static. Various internal movements, such as breathing or heartbeat, can cause rhythmic changes in the patient's internal anatomy. These internal movements can undesirably cause displacement of the surgical site, hindering the use of augmented reality during surgery. This problem is further exacerbated by the non-linearity of such internal movements. For instance, inspiration and expiration, as specific phases of the respiratory cycle, can lead to significant changes in lung deformation and airflow.

[0007] More importantly, one of the standard methods for creating three-dimensional (3D) medical images today is using computed tomography (CT), resulting in image series known as DICOM datasets. DICOM datasets can be further processed using software to segment body structures and create 3D images of these structures for further research or augmented reality applications. These DICOM datasets must be carefully examined one by one and then processed using software segmentation methods, where each structure of interest within each individual scan slice must be identified.

[0008] In other words, CT scans produce 2D (two-dimensional) image slices of varying thicknesses. These individual 2D DICOM slices must then be reassembled into a 3D model, rendered, and smoothed. Processing 2D image slices from CT scans involves numerous image transfer and processing steps to produce an anatomical structure suitable for viewing in augmented reality. Due to the numerous steps involved and the high cost of acquiring and operating CT scans, this method may be impractical for the general public. The high cost of CT scans limits the number of available scans, and not all patients in urgent need can easily access them. Furthermore, patients are inevitably exposed to a certain dose of radiation during a CT scan. This radiation exposure can damage human tissues, putting patients and caregivers at risk. It can also lead to long-term negative effects.

[0009] Therefore, a system and method for dynamically registering anatomical structures using augmented reality is still needed. Ideally, such a system and method could be adapted to various body movements. Summary of the Invention

[0010] According to this disclosure, a system and method for dynamically registering anatomical structures using augmented reality and adapting them to in vivo motion are unexpectedly revealed.

[0011] The aforementioned systems and methods may include an augmented reality environment that can be fused with anatomical data and represented as at least one augmented representation. Ideally, physicians can utilize the augmented reality environment to confirm the positional gating accuracy of anatomical structures and / or adjust the augmented representation of active registration. The computer system can provide translation, rotation, and / or deformation tools based on the desired refinement of the active registration augmented representation. Furthermore, the aforementioned systems and methods can employ closed-loop feedback to alert clinicians to the optimal timing of treatment, to perform surgical procedures targeting anatomical features, or to relay pulses or time-segmented gated treatment delivery based on augmented feedback. Rolling averages of vital capacity data can also be used to refine the augmented reality environment animation.

[0012] This technology addresses the need for dynamic registration of human or animal anatomical structures using augmented reality or other digital rendering visualization methods. During surgery, robotic surgery, and / or minimally invasive surgery, patient anatomy may shift due to respiratory cycles, cardiac cycles, and / or normal movement and physiological processes. The systems and methods described above can be applied to measure patient body movements and gate those movements in the form of visual feedback and / or animation.

[0013] In some embodiments, a system for dynamically registering anatomical structures using augmented reality may include an augmented reality system, an imaging system, a measurement system, and a computer system. The augmented reality system may be configured to display an augmented representation in an augmented reality environment. The imaging system may be configured to image patient anatomical features and generate anatomical imaging data. The measurement system may be configured to measure patient anatomical motion and generate anatomical motion data. The computer system may communicate with the imaging system, the measurement system, and the augmented reality system. The computer system may be configured to receive anatomical imaging and positional data from the imaging system and anatomical motion data from the measurement system. The computer system may also be configured to generate an augmented representation based on the anatomical imaging data. Furthermore, the computer system may be configured to associate the augmented representation with the anatomical motion data. Additionally, the computer system may be configured to correlate the augmented representation with the anatomical motion data. The computer system may also be configured to render the augmented representation in the augmented reality environment on the augmented reality system. The computer system may also be configured to selectively update the augmented representation based on the anatomical motion data.

[0014] In some embodiments, a method for dynamically registering anatomical structures using augmented reality may include the steps of: providing a system for dynamically registering anatomical structures using augmented reality. The system may include an imaging system, a measurement system, and a computer system. The imaging system may image patient anatomical features. The imaging system may generate anatomical imaging data by imaging patient anatomical features. The measurement system may measure patient anatomical motion. The measurement system may generate anatomical motion data by measuring patient anatomical motion. The computer system may receive anatomical imaging data and anatomical motion data. The computer system may generate an augmented representation based on the anatomical imaging data. The computer system may associate the augmented representation with patient anatomical features. The computer system may associate the augmented representation with patient anatomical motion data. The computer system may render the augmented representation in the augmented reality environment on an augmented reality system. The computer system may selectively update the augmented representation based on the anatomical motion data.

[0015] The aforementioned systems and methods offer the following clinical advantages: (i) reduced need for high-cost CT scanning equipment, thereby improving surgical mixed reality guidance and navigation; (ii) reduced image transmission and processing steps to produce anatomical structures for viewing in augmented reality, thereby shortening surgical time; (iii) reduced need for harmful radiation to patients and caregivers; and (iv) improved imaging capabilities for non-static anatomical structures.

[0016] It should be understood that although this disclosure is primarily aimed at augmented reality (AR) technology and related applications, it can also be applied to virtual reality (VR) technology and related applications, including mixed reality (MR) applications.

[0017] From the description provided herein, many more applicable fields will become apparent. It should be understood that the summary and specific examples are for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0018] The accompanying drawings are provided only to illustrate the selected embodiments and not all possible technical solutions, and are not intended to limit the scope of this disclosure.

[0019] Figure 1 The schematic diagram illustrates an augmented reality system, an imaging system, a measurement system, and a computer system according to certain embodiments;

[0020] Figure 2 The system in operation according to certain embodiments is schematically depicted, and the process of acquiring 2D images is further illustrated, which can be incorporated into at least one of imaging and position data and anatomical motion data by moving the ultrasound probe over the patient's anatomical features and pausing for a predetermined time at set pause intervals.

[0021] Figure 3 An enhanced representation of an anatomical feature according to certain embodiments is schematically depicted, which may be generated by a computer system using imaging and location data and has jagged edges that may be derived from the generation process;

[0022] Figure 4 schematically depicted Figure 3 The enhanced representation shown uses post-processing to smooth the jagged edges, and further illustrates the dashed outline indicating the boundaries to which the enhanced representation can expand, reflecting how anatomical features expand based on anatomical motion.

[0023] Figure 5 The diagram illustrates vital capacity data with multiple reference points according to certain embodiments, which can be incorporated into anatomical motion data and used by a computer system to determine how to update the augmented representation.

[0024] Figure 6 The system in operation according to certain embodiments is schematically depicted, showing an augmented reality environment with an enhanced representation presented on a portion of a patient, and further showing changes in the position of the enhanced representation reflecting how anatomical features move based on anatomical motion;

[0025] Figure 7 The system in operation according to certain embodiments is schematically depicted, illustrating an augmented reality environment displayed using a flash display method, and further illustrating augmented representation of location migration based on anatomical motion;

[0026] Figure 8 The system in operation according to certain embodiments is schematically depicted, showing an augmented reality environment displayed using a flash display method, and further showing that the augmented representation includes areas representing blood flow in anatomical features toward the ultrasound probe (displayed as striped patterns) and areas depicting blood flow in anatomical features toward the ultrasound probe (displayed as square patterns).

[0027] Figure 9 An enhanced representation according to certain embodiments is schematically depicted, and further illustrates an ECG waveform with designated segments that can be used to determine the minimum and maximum range of movement of the enhanced representation based on how it moves within the patient's cardiac cycle according to anatomical features.

[0028] Figure 10a and Figure 10b A flowchart illustrating a method of using the system described above according to certain embodiments;

[0029] Figure 11a and Figure 11b To illustrate a flowchart of a method using the system described above according to certain embodiments, a step of applying post-processing to enhance the representation is further shown. Detailed Implementation

[0030] The following technical description is essentially only an example of the subject matter, manufacture, and use of one or more inventions, and is not intended to limit the scope, application, or use of any particular invention claimed in this application or in other such applications or patents granted therefrom that claim priority to this application. Regarding the methods of this disclosure, the presented sequence of steps is essentially only illustrative, and therefore the sequence of steps may vary in different embodiments, including cases where certain steps may be performed simultaneously. As used herein, “a” and “an” indicate the presence of “at least one”; where permissible, several such items may be present. All numerical quantities in this specification should be understood to be modified by the word “about”, and all geometric and spatial descriptors should be understood to be modified by the word “substantially” when describing the broadest scope of the technology of this disclosure, unless otherwise expressly stated. When “about” is applied to numerical values, it indicates that the calculation or measurement allows for a slight inaccuracy in the value (achieved through some method; approximated or reasonably close to the value; nearly). If, for some reason, the inaccuracy provided by “about” and / or “substantially” is not understood in this ordinary sense in the art, “about” and / or “substantially” as used herein at least indicates variations that may arise from ordinary methods of measuring or using these parameters.

[0031] While the open-ended term "comprising" is used herein as a synonym for non-limiting terms (such as including, comprising, or having) to describe and claim embodiments of the technology disclosed herein, alternatively, more restrictive terms (such as "consisting of" or "substantially consisting of") may be used to describe embodiments. Therefore, for any given embodiment listing materials, components, or process steps, the technology disclosed herein also specifically includes embodiments that consist of or substantially consist of such materials, components, or process steps, excluding any additional materials, components, or processes (in relation to "consisting of") or any additional materials, components, or processes that affect a material characteristic of the embodiment (in relation to "substantially consisting of"), even if such additional materials, components, or processes are not expressly described herein. For example, a description of a composition or method listing elements A, B, and C specifically contemplates embodiments that consist of and substantially consist of A, B, and C, excluding element D, which may be listed in the art, even if not expressly described herein as an excluded element.

[0032] As indicated herein, the scope of disclosure includes endpoints and all distinct values ​​and further subdivisions throughout the entire scope, unless otherwise specified. Thus, for example, a scope “from A to B” or “from about A to about B” includes both A and B. Disclosing values ​​and ranges for a particular parameter (such as quantity, weight percentage, etc.) does not exclude other values ​​and ranges useful herein. This disclosure contemplates that two or more specific exemplary values ​​of a given parameter can define endpoints of a claimed range of values ​​for that parameter. For example, if parameter X is exemplified herein as having a value A and also as having a value Z, then parameter X is contemplated to have a range of values ​​from about A to about Z. Similarly, this disclosure contemplates that two or more ranges of values ​​for a parameter (whether nested, overlapping, or distinct) are grouped together using all possible combinations of ranges of values ​​that the endpoints of the scope of disclosure may claim. For example, if the parameter X is exemplified in this paper as having a value in the range of 1-10, 2-9, or 3-8, it is also conceivable that the parameter X may have other ranges of values, including 1-9, 1-8, 1-3, 1-2, 2-10, 2-8, 2-3, 3-10, 3-9, etc.

[0033] When an element or level is described as being "in," "attached," "connected," or "coupled" to another element or level, that element or level may be directly in, attached to, connected to, or coupled to the other element or level, or there may be an intervening element or level. Conversely, when an element is described as being "directly in," "directly attached," "directly connected," or "directly coupled" to another element, there is no intervening element. Other terms used to describe relationships between elements should be interpreted similarly (e.g., "between" and "directly," "adjacent" and "next to"), etc. As used herein, the term "and / or" includes any and all combinations of one or more of the related columns.

[0034] While the terms first, second, third, etc., may be used herein to describe various elements, components, regions, hierarchies, and / or segments, these elements, components, regions, hierarchies, and / or segments should not be limited to these terms. These terms may be used only to distinguish one element, component, region, hierarchy, or segment from another. Terms such as “first,” “second,” and other numerical terms used herein do not imply order or sequence unless the context clearly indicates otherwise. Therefore, without departing from the teachings of the exemplary embodiments, the first element, component, region, hierarchy, or segment discussed below may be referred to as a second element, component, region, hierarchy, or segment.

[0035] For ease of description, spatially related terms such as “inside,” “outside,” “below,” “below,” “under,” “above,” and “above” are used to describe the relationship between one element or feature and another, as shown in the figure. Spatially relative terms may be intended to cover different orientations in the use or operation of the device beyond those depicted in the figure. For example, if the device is flipped in the figure, some elements or features described as being “below” or “under” other elements or features should be oriented as being “above” other elements or features. Therefore, the example term “below” can encompass both above and below directions. The device may take other orientations (rotated 90 degrees or other orientations), and the spatially relative descriptors used herein should be interpreted accordingly.

[0036] All documents (including patents, patent applications, and scientific literature) cited in this Detailed Description section are incorporated herein by reference unless otherwise expressly stated. In the event of any conflict or ambiguity between the cited documents and this Detailed Description section, this Detailed Description section shall prevail.

[0037] As used herein, the term "head-mounted device" (or "headset" or "HMD") refers to a display device configured to be worn on the head, having one or more display optics (including lenses) in front of one or more eyes. This term can even be used more broadly by the term "augmented reality system." An example of a suitable head-mounted device is Microsoft...

[0038] Figure 1 A system for dynamically registering anatomical structures using augmented reality 100 is illustrated. System 100 may include an augmented reality system 102, an imaging system 104, a measurement system 106, and a computer system 108. Augmented reality system 102 can be configured to display, for example,... Figures 3 to 4 and Figures 6 to 9 The enhanced representation 110 is shown. The enhanced representation 110 can be a two-dimensional (2D) or three-dimensional (3D) depiction of information relevant to the current medical procedure. Non-limiting examples of relevant information may include preoperative and / or intraoperative data, such as a three-dimensional depiction of anatomical features 112 of patient 114. Anatomical features 112 may be organic structures and / or regions of patient 114 that are the focus of the current procedure. Other non-limiting examples of anatomical features 112 may include organs, organ parts, tissues, joints, bones, tumors, implants, etc. The enhanced representation 110 can have many applications and uses, such as preoperative planning, procedural guidance, and training. It should be understood that those skilled in the art may choose other information to depict the enhanced representation 110. It should also be understood that anatomical features 112 may include any anatomical structure portion of patient 114.

[0039] Although still referring to Figures 3 to 4 and Figures 6 to 9 However, the augmented reality system 102 can be configured to display the augmented representation 110 within the augmented reality environment 116. The augmented reality environment 116 may include virtual windows and / or different modes, such as... Figures 3 to 4 , Figure 6 and Figure 9 The "Heads-Up Display" or "HUD" mode shown and Figures 7 to 8 The “flash” mode shown, as illustrated in U.S. Patent Application US17 / 110,991 by Black et al., the entire disclosure of which (including definitions) is incorporated herein by reference. Ideally, this would allow a physician to view the enhanced representation 110 in the same field of vision as patient 114. See now... Figure 6 The augmented reality system 102 can be configured to display an augmented representation 110 on a portion 118 of the patient 114 in an augmented reality environment 116. In some examples, the portion 118 of the patient 114 can be an anatomical feature 112 of the patient 114. Advantageously, this allows the augmented representation 110 to be drawn directly on the anatomical feature 112 to provide relevant feedback within the context of the location of the anatomical feature 112. For example, the augmented representation 110 can be an intraoperative scan of the anatomical feature 112, which can be overlaid on the anatomical feature 112 for the physician. In other cases, the portion 118 of the patient 114 can be adjacent to the anatomical feature 112 of the patient 114. Ideally, this allows the physician to observe the augmented representation 110 while also being able to observe the anatomical feature 112 within the same field of view; for example, a mixed reality view.

[0040] In some examples, the augmented representation 110, utilizing augmented reality, may be displayed at an approximate location of the anatomical feature 112 of the patient 114. For example, the computer system 108 may employ algorithms, machine learning, artificial intelligence, and / or combinations thereof to estimate the location of the anatomical feature 112 of the patient 114 according to medically acceptable tolerances. However, it should be understood that the augmented representation 110 may also be displayed on other surfaces and / or augmented representations as needed.

[0041] Augmented reality system 102 may be a wearable head-mounted display. A non-limiting example of augmented reality system 102 may be Microsoft... Augmented reality system 102 and methods of operation (including methods of displaying augmented representation 110) can be found in: U.S. Patent 10,478,255 to West et al.; U.S. Patent 10,895,906 to West et al.; U.S. Patent 10,869,727 to Yanof et al.; U.S. Patent Publication 2021 / 0081035 to West et al.; U.S. Patent Application 17 / 117,841 to Martin et al.; U.S. Patent Application 17 / 213,636 to Black et al.; U.S. Patent Application 17 / 163,975 to Black et al.; and U.S. Patent Application 17 / 110,991 to Black et al., the entire disclosures of which (including definitions) are incorporated herein by reference. However, it should be understood that, within the scope of this disclosure, those skilled in the art may employ other AR devices and methods of operation for augmented reality system 102.

[0042] It should be understood that in some embodiments, the augmented reality system 102 may also include one or more position sensors 138. The position sensors 138 of the augmented reality system 102 may be configured to determine and generate position data 140 of the augmented reality system 102, such as the approximate three-dimensional (3D) spatial position, orientation, angular velocity, and acceleration of the augmented reality system 102. For example, it should be understood that this would allow for the accurate display of holographic images within a physician's field of vision during operation. Non-limiting examples of position sensors 138 include accelerometers, gyroscopes, electromagnetic sensors, and / or optical tracking sensors. It should also be understood that those skilled in the art may employ different types and numbers of position sensors 138 for the augmented reality system 102, for example, depending on the needs of the process or situation utilizing the augmented reality system 102.

[0043] Imaging system 104 is configured to image anatomical features 112 of patient 114 and generate anatomical imaging data 142. Anatomical imaging data 142 may include information and / or media associated with the structure, rotation, and / or position of anatomical features 112 relative to patient 114. It should be understood that those skilled in the art may select the data types to be included in the anatomical imaging data 142. Ideally, imaging system 104 may be used to image anatomical features 112 of patient 114 and generate anatomical imaging data 142 before, during, and / or during the combination phase of surgery.

[0044] As will be described in further detail below, the anatomical imaging data 142 can be used by the computer system 108 to generate the enhanced representation 110. In other words, the imaging system 104 can be used to perform scanning and / or other imaging procedures to generate the anatomical imaging data 142 for use in generating the enhanced representation 110. For example, the imaging system 104 may include an ultrasound system having at least one ultrasound probe 120. A physician may move the ultrasound probe 120 over the anatomical features 112 of the patient 114 to capture the anatomical imaging data 142, which may include a 2D image 124. Figure 2 An ultrasound system with an ultrasound probe 120 is shown as imaging system 104. The ultrasound probe 120 can move along a path 122 of the patient 114 to generate a 2D image 124. The 2D image 124 can then be converted into an enhanced representation 110 by a computer system 108. Other non-limiting examples of imaging system 104 may include a computed tomography (CT) system, an electromagnetic system, a cone-beam computed tomography (or CBCT) system, a gas exchange system, a mechanically controlled ventilation system, a vital capacity measurement system, an electrocardiogram (ECG) system, a magnetic resonance imaging (MRI) system, an electromechanical wave propagation system, a transesophageal echocardiography (TEE) system, and combinations thereof. However, it should be understood that other imaging procedures and systems may be used by those skilled in the art as imaging system 104 within the scope of this disclosure.

[0045] Measurement system 106 is configurable to measure anatomical movements of patient 114 and generate anatomical movement data 144. Anatomical movements may include one or more movements occurring in one or more parts of patient 114's body, possibly based on partial or complete movement cycles of patient 114. Non-limiting examples of movement cycles may include external or internal forces, such as movements based on respiratory cycles, cardiac cycles, joint movements including range of motion, migration of internal anatomical structures, and / or physiological processes. Within the scope of this disclosure, other movement cycles that affect patient anatomical features may also be measured. Furthermore, anatomical movements may include multiple movement cycles. Anatomical movements affect several aspects of anatomical features 112 that can be captured in anatomical movement data, such as structure, position, rotation, etc.

[0046] Anatomical motion data 144 may include information and / or media associated with anatomical motion and its effect on anatomical features 112. In some examples, anatomical motion data 144 may include displacement and strain measurements that can be used to delineate and visually visualize areas of diseased or sclerotic tissue to enable optimal implantation and implant repair procedures. It should be understood that those skilled in the art may choose to include other data and information in the anatomical motion data as needed.

[0047] As will be further detailed below, anatomical motion data 144 can be used by computer system 108 to update augmented representation 110 based on anatomical motion data, which allows for multifaceted selective updates of augmented representation 110 based on internal movement. Ideally, augmented representation 110 will remain available regardless of the internal movement of patient 114's body. Measurement system 106 can be used to measure the anatomical movement of patient 114 before, during, and / or combined stages of surgery and generate anatomical motion data 144. Measurement system 106 can be configured to allow gating of the anatomical movement of patient 114 and generate anatomical motion data. (Refer to...) Figure 5 The measurement system 106 may include a vital capacity measurement system for measuring respiratory-related anatomical movements and generating anatomical movement data. Other non-limiting examples of the measurement system 106 may include a computed tomography (CT) system, an electromagnetic system, a cone-beam computed tomography (CBCT) system, a gas exchange system, a mechanically controlled ventilation system, a vital capacity measurement system, an electrocardiogram (ECG) system, a magnetic resonance imaging (MRI) system, an electromechanical wave propagation system, a transesophageal echocardiography (TEE) system, and combinations thereof. It should be understood that those skilled in the art may employ other systems for the measurement system 106 as needed.

[0048] It should be understood that, within the scope of this disclosure, those skilled in the art can combine the imaging system 104 and the measurement system 106 into a single system and / or add one or more additional systems. Similarly, anatomical imaging data 142 and anatomical motion data 144 can be combined into a single data entry and / or additional data entries as needed. Furthermore, more examples and methods of operation of the imaging system 104 and the measurement system 106 will be detailed below.

[0049] Now refer to Figure 1Computer system 108 can communicate with augmented reality system 102, imaging system 104, and / or measurement system 106. This can be achieved via wireless connection, wired connection, or through network 152. It should be understood that network 152 of system 100 can include various wireless and wired communication networks, including, as non-limiting examples, radio access networks such as LTE or 5G, local area networks (LANs), wide area networks (WANs) such as the Internet, or wireless LANs (WLANs). It should be understood that such network examples are not intended to be limiting, and the scope of this disclosure includes implementations in which one or more computing platforms of system 100 can be operatively linked through some other communication coupling, including wireless and wired communication networks. One or more components and sub-components of system 100 can be configured to communicate with a networked environment via wireless or wired connection. In some embodiments, one or more computing platforms can be configured to communicate directly with each other via wireless or wired connection. Examples of various computing platforms and networked devices include, but are not limited to, smartphones, wearable devices, tablet computers, laptop computers, desktop computers, Internet of Things (IoT) devices, or other mobile or fixed devices such as standalone servers, networked servers, or server arrays.

[0050] Computer system 108 may have processor 146 and memory 148. Memory 148 may include non-transient processor-executable instructions 150 to perform several different operations. For example, computer system 108 may be configured to receive anatomical imaging data 142 from imaging system 104 and anatomical motion data 144 from measurement system 106. Furthermore, as previously described, imaging system 104 and measurement system 106 may be a single system and / or additional or discrete systems. Therefore, computer system 108 may receive anatomical imaging data 142 and anatomical motion data 144 from a single system and / or multiple systems.

[0051] Computer system 108 can also be configured to generate augmented representation 110 based on anatomical imaging data 142. This can be achieved through user input, algorithms, machine learning, artificial intelligence, and / or combinations thereof. In some examples, computer system 108 may generate augmented representation 110 based on the systems and methods described in U.S. Patent 10,478,255 to West et al. and / or U.S. Patent 10,895,906 to West et al.

[0052] Reference Figures 3 to 4The computer system 108 can be configured to apply post-processing effects to enhance the augmented representation 110. For example, the computer can apply anti-aliasing to the augmented representation 110 to smooth jagged edges 126 into smooth edges 128 (which can be formed during the generation process). This can be accomplished using a variety of different anti-aliasing techniques. Non-limiting examples may include Supersample Anti-Aliasing (SSAA), Multi-Sampling Anti-Aliasing (MSAA), Fast Approximate Anti-Aliasing (FXAA), Temporal Anti-Aliasing (TXAA), etc. Smoothing can also include simple operations such as averaging the data between measurement points; for example, smoothing noise between datasets and / or estimating gaps between data points. The computer system 108 can also apply anti-aliasing individually or in combination using algorithms, machine learning, artificial intelligence. Advantageously, this allows the enhanced representation 110 to be clearer to the physician, providing optimization for the position, size, orientation, and / or animation of the enhanced representation 110. It should be understood that those skilled in the art can use other processes and methods to smooth the jagged edges 126 into smooth edges 128 of the enhanced representation 110.

[0053] Computer system 108 can also be configured to associate augmented representation 110 with anatomical features 112 of patient 114. In some examples, this can be achieved by integrating augmented representation 110 into a holographic registration relative to patient 114, as described in U.S. Patent 10,478,255 to West et al. and / or U.S. Patent 10,895,906 to West et al. Ideally, associating augmented representation 110 with anatomical features 112 of patient 114 allows anatomical imaging and location data to be placed into a common holographic coordinate system used by augmented reality system 102 to provide augmented representation 110 to augmented reality environment 116. However, it should be understood that those skilled in the art may employ different methods as needed to associate augmented representation 110 with anatomical features 112 of patient 114.

[0054] The computer system 108 can be further configured to correlate the augmented representation 110 with the anatomical motion data. In other words, the computer system 108 can determine the relationship between the augmented representation 110 and the anatomical motion data, which determines whether the augmented representation 110 needs to be updated based on the anatomical motion of the patient 114. This can be accomplished using various algorithms, machine learning, and artificial intelligence. Furthermore, some applications will be discussed below to further explain how the correlation between the augmented representation 110 and the anatomical motion data 144 may occur.

[0055] Reference Figures 3 to 4 and Figures 6 to 9 The computer system 108 may also be configured to render an augmented representation 110 in the augmented reality environment 116 on the augmented reality system 102. In some examples, the computer system 108 may be configured to render an augmented representation 110 on a portion 118 of the patient 114 in the augmented reality environment 116 on the augmented reality system 102, such as... Figure 6 As shown. As previously described, this allows physicians to view the augmented representation 110 within the same field of vision as patient 114 in the augmented reality environment 116. In some examples, the augmented representation 110 may be associated with implants, devices, and / or tools for optimal deployment and / or utilization, such as Figures 7 to 8 As shown.

[0056] Now refer to Figure 4 , Figure 6 and Figure 7 The computer system 108 may be further configured to selectively update the augmented representation 110 based on the anatomical motion data 144. In some examples, updating the augmented representation 110 may include updating the position of the augmented representation 110 in the augmented reality environment 116 according to the anatomical motion data 144. As a non-limiting example, Figure 6 and Figure 7 The illustration shows the enhancement representation 110 being updated so that its position changes according to anatomical motion data 144 (original position 130 is shown as a solid line, updated position 132 as a dashed line). Ideally, the updated position 132 of the enhancement representation 110 could reflect how the position of the anatomical feature 112 changes due to the anatomical motion of the patient 114. For example, liver cancer in patient 114 might undergo positional changes based on the patient 114's respiratory cycle. Therefore, the computer system 108 can compensate for the positional changes of the anatomical feature 112 caused by the anatomical motion of patient 114. Advantageously, this allows the enhancement representation 110 to remain useful even if the position of the anatomical feature 112 changes due to anatomical motion.

[0057] Reference Figure 4 Updating the augmented representation 110 may include animate the augmented representation 110 based on anatomical motion data 144 to display structural changes in the augmented representation 110. For example, the structure of the augmented representation 110 may be animate based on anatomical motion data 144 to expand, contract, deform, or reorganize. As a non-limiting example, Figure 4 The augmented representation 110 is depicted expanding based on anatomical motion data 144 (the original state 134 is shown as a solid line, and the boundary 136 to which the feature will expand is shown as a dashed line). Ideally, this would allow the physician to visually see the augmented representation 110 expanding to boundary 136, reflecting how the anatomical feature 112 changes due to anatomical motion. This is believed to aid in relating the current state of the motion-based anatomical feature 112 to the physician's process.

[0058] In other examples, updating the augmented representation 110 may include updating the position and / or structure of the augmented representation 110 based on anatomical motion data 144 without displaying transition animations. Ideally, this would save processing power without needing to display transitions.

[0059] Computer system 108 can be configured to selectively update augmentation representation 110 based on anatomical motion data 144 at set update intervals. The update interval can be scaled according to the requirements of the current medical procedure. For example, if the procedure requires higher accuracy, the update interval can be shortened to allow for a higher update frequency. If the procedure does not require high accuracy, the update interval can be lengthened to reduce the update frequency. In some examples, augmentation representation 110 can be updated nearly continuously based on anatomical motion data 144. However, it should be understood that those skilled in the art can scale the update interval according to the requirements of the procedure.

[0060] It should be understood that the computer system 108 can automatically and selectively update the augmentation representation 110 based on the anatomical motion data 144. However, in some examples, the computer system 108 may be configured to allow a physician to decide when to update the augmentation representation 110. For example, input may be provided to the system by a physician or another person to enable manual updates of the augmentation representation 110 as needed.

[0061] Furthermore, computer system 108 can adapt to various computational capability levels by using rolling averages to predict motion within a specific confidence interval, thereby reducing the computational burden required for clinically acceptable performance levels. It should be understood that, in some cases, computer system 108 may be outside of augmented reality system 102. However, in other cases, computer system 108 may be included within augmented reality system 102.

[0062] System 100 may further include a tracking sensor for use in assisting imaging anatomical imaging data 142 and measuring anatomical motion data 144. For example, the tracking sensor may include an electromagnetic system for tracking instruments such as an ultrasound probe 120 and generating tracking data that can be incorporated into anatomical images and position data, as well as anatomical motion data 144. The tracking data may also include approximate position in 3D space, orientation of the tracked instrument, angular velocity, and acceleration. Non-limiting examples of tracking sensors may include accelerometers, gyroscopes, electromagnetic sensors, and optical tracking sensors. However, it should be understood that, within the scope of this disclosure, those skilled in the art may employ different devices and instruments as tracking sensors.

[0063] Applications of Respiration:

[0064] like Figures 5 to 6As shown, the measurement system 106 can be configured to measure anatomical movements that occur due to respiration and generate anatomical movement data 144. For example, the measurement system 106 may include a vital capacity measurement system, a gas exchange system, and / or a mechanically controlled ventilation system to measure respiratory-related anatomical movements and generate anatomical movement data 144. Other technologies such as CT systems, MRI systems, ultrasound systems, and ECG systems may also be used to capture anatomical movements and generate and / or supplement anatomical movement data 144.

[0065] Typical respiratory rate is approximately 5-7 mg / kg body weight, but inhalation and exhalation of the lungs are not linear. Inhalation and exhalation, as specific phases in the respiratory cycle, can cause significant changes in lung deformation and airflow. Imaging system 104 may include performing vital capacity measurement, which may include measuring the average air volume of seven (7) breaths.

[0066] Measurement system 106 can use wireless spirometry or mechanical ventilation to capture real-time spirometry data about the target area for surgical intervention. An example of a surgical intervention includes liver cancer. The cancer may change location according to the respiratory cycle. This location change can be captured by measurement system 106 and generated into anatomical motion data 144. Computer system 108 can then selectively update and enhance the representation 110 by displaying animation or visual feedback to a physician. The location of the cancer can be captured by measurement system 106 with respect to a known phase of the respiratory cycle at a given time point. Computer system 108 can correlate the location of the cancer with a known amount of air in the lungs of the patient 114 associated with that time point. Measurement system 106 can then capture and / or refine the anatomical motion data 144 by capturing the location of anatomical features 112 (such as cancer) using real-time 2D, 3D, or 4D ultrasound. Other systems and methods may be employed by those skilled in the art within the scope of this disclosure.

[0067] Computer system 108 can apply anatomical motion data 144 to the X, Y, and Z coordinates at different time points to demonstrate the gating and deformation of anatomical feature 112. Measurement system 106 may include an ultrasound system, MRI system, or CT system, which can generate anatomical motion data 144, such as the elastography features of anatomical feature 112. Computer system 108 can use anatomical motion data 144 to provide a dynamic deformation of enhanced representation 110, reflecting how motion affects anatomical feature 112.

[0068] In some examples, measurement system 106 may measure anatomical movements that may occur in patient 114 during inhalation and / or exhalation. Based on these measurements, measurement system 106 may include vital capacity data, which may be incorporated into anatomical movement data 144. Measurement system 106 may include a spirometer and / or a bellows-type device for performing the measurement and generating the vital capacity data. However, it should be understood that different systems and methods may be used by those skilled in the art to collect the vital capacity data for measurement system 106. Vital capacity data may include flow cycles to create reference points in the respiratory cycle of patient 114. In particular, Figure 5 A table of vital capacity data is shown, including first measurement (M1), second measurement (M2), third measurement (M3), fourth measurement (M4), fifth measurement (M5), and sixth measurement (M6). These measurements M1, M2, M3, M4, M5, and / or M6 can be used to correlate the position of the enhanced representation 110 with the known volume of air in the lungs of the patient 114 associated with a specific time point. The computer system 108 can then update the enhanced representation 110 based on the vital capacity data. For example, the computer system 108 can update the position of the enhanced representation 110 to reflect how the position of the anatomical feature 112 moves with inhalation and / or exhalation and with a complete respiratory cycle or multiple cycles. This may be particularly applicable to anatomical features 112, including tumors, cysts, blood vessels, cardiac structures, muscles, bones, and / or nerves.

[0069] Cardiac Cycle Applications:

[0070] During structural heart repair or replacement procedures and electrophysiological ablation procedures, diagnostic imaging of the cardiac cycle or cardiac pacing can be used to apply the methods of this disclosure. Similar solutions can be achieved using diagnostic imaging systems such as CT systems, MRI systems, TEE systems, ECG systems, and / or electromechanical wave propagation systems. However, it should be understood that those skilled in the art may employ other different types of diagnostic measurement systems as needed for the imaging system 104 and measurement system 106. During the cardiac cycle, the measurement system 106 may include known cardiac output (Q) or pacing characteristics, and gating techniques may be used to visualize cardiac motion.

[0071] In some examples, measurement system 106 may include an ECG system. The ECG system generates ECG data, including the ECG electrical cycle, and correlates the ECG data with the cardiac mechanical cycle, which may be incorporated into the anatomical motion data 144. The ECG electrical cycle may have three main components: the P wave, representing atrial depolarization; the QRS complex, representing ventricular depolarization; and the T wave, representing ventricular repolarization. (See reference...) Figure 9The computer system 108 can update the enhancement representation 110 based on ECG data incorporated into the anatomical motion data 144 by assigning time points to the ECG electrical cycle. For example, a first time point (T1) can be associated with the P wave and atrial depolarization in the ECG electrical cycle. A second time point (T2) can be associated with the Q wave in the QRS complex of the ECG electrical cycle and normal left-to-right depolarization of the interventricular septum. A third time point (T3) can be associated with the R wave in the QRS complex of the ECG electrical cycle and early ventricular depolarization. A fourth time point (T4) can be associated with the S wave in the QRS complex of the ECG electrical cycle and Purkinje fiber depolarization. Figure 9 As shown, the computer system updates 3D features by relating each time point to a segment of the enhanced representation 110. The enhanced representation 110 can then be animated based on this relationship, visually demonstrating the minimum and maximum ranges of motion during the cardiac cycle. Ideally, this would visually show the orientation of key structures (such as the apex, chambers, valves, vessels, nodules, leaflets, chordae tendineae, and commissures), the orientation of septal punctures, and the relationships and ranges of motion of other structures. It should be understood that other anatomical features may also benefit from this application. Visual representations of such structures with dynamic and / or deformable features can provide clinically important information to assist in the placement of replacement or repair products.

[0072] Applications of spinal surgery and orthopedic deformity:

[0073] Measurement system 106 can also be configured to measure and generate anatomical motion data 144 regarding soft tissue biomechanical stress during orthodontic and spinal surgery. Measurement system 106 can be configured to measure strain-measured forces and preoperatively generated forces, and how these forces affect joint orientation, long bone deformities, and spinal alignment. In some examples, this can be performed using measurement system 106 incorporating an ultrasound system. The known location of a joint or bone can be measured and referenced with respect to strain measurements and overall tendon or muscle length. Predictive modeling and intraoperative calculations can then be performed based on common orthodontic and spinal surgeries.

[0074] Applications of ultrasound:

[0075] As previously described, the imaging system 104 may include an ultrasound system having at least one ultrasound probe 120, such as Figure 2 As shown. The physician can move the ultrasound probe 120 over the anatomical features 112 of the patient 114 to capture anatomical imaging data 142, which may include 2D images 124. Figure 2An ultrasound system with an ultrasound probe 120 is shown as an imaging system 104. The ultrasound probe 120 can move along a path 122 of the patient 114 to generate a 2D image 124. The 2D image 124 can then be converted into an enhanced representation 110 by a computer system 108.

[0076] In some examples, the measurement system 106 may include an ultrasound system. While moving the ultrasound probe 120 over the anatomical features 112 of the patient 114, the physician may pause for predetermined times at set pause intervals to indicate the start and end of the patient 114's breathing. By pausing for predetermined times at set pause intervals, multiple phases of breathing can be captured at multiple locations and incorporated into the anatomical motion data 144. Therefore, the system 100 can indicate anatomical deformations and translations by updating the enhanced representation 110 based on the anatomical motion data 144. For example, referring to… Figure 4 The enhanced representation 110 can be updated to dynamically animate between two phases of the patient 114's breathing. However, it should be understood that those skilled in the art can update other aspects of the enhanced representation 110 based on the anatomical motion data 144.

[0077] The ultrasound system may also include multiple sensors. These sensors may be positioned around the patient 114's body to indicate anatomical motion, which may include movement and respiration. The multiple sensors may be selectively positioned to serve as reference points for the generated ultrasound images. Furthermore, anatomical motion data 144, which may include translation and rotation data, may be generated from the multiple sensors to enhance augmented reality registration, overlay, and orientation by updating the augmented representation 110 based on the anatomical motion data 144.

[0078] Applications of Ultrasound Color Doppler:

[0079] Reference Figures 7 to 8 The measurement system 106 may include an ultrasound color Doppler system. The ultrasound color Doppler system may be configured to estimate blood flow through anatomical features 112, such as blood vessels, by bouncing high-frequency sound waves (ultrasound waves) from circulating red blood cells, thereby generating anatomical motion data 144. Specifically, the ultrasound color Doppler may be further configured to estimate the direction and velocity of blood flow through anatomical features 112 and incorporate this into the anatomical motion data 144. This may include using color coding to indicate the direction of flow. For example, blue may be used to indicate that blood is flowing out of the ultrasound probe 120 (…). Figure 8 (Displayed as a square pattern). Red can be used to indicate that blood is flowing towards the ultrasound probe 120 (…). Figures 7 to 8 (Displayed as a striped pattern).

[0080] In some examples, computer system 108 may update the enhanced representation 110 based on blood flow direction and velocity incorporated as anatomical motion data 144 to visualize the direction and / or velocity of blood flow in anatomical feature 112. This process is particularly useful when anatomical feature 112 includes anechoic or difficult-to-visualize vessel walls. Ideally, an ultrasound color Doppler system could be used to dynamically register cardiac anatomy during the cardiac cycle for optimal implant or repair implant deployment.

[0081] During operation, the ultrasound color Doppler system, as measurement system 106, measures the direction and / or velocity of blood flow through anatomical feature 112. Measurement system 106 then determines and generates relevant data as anatomical motion data 144, including how movement of anatomical feature 112 or changes in cardiac cycle occur based on the direction and / or velocity of blood flow through anatomical feature 112. Computer system 108 correlates enhancement representation 110 with anatomical motion data 144. Computer system 108 then updates enhancement representation 110 by adjusting its position according to anatomical motion data 144 to reflect changes in the position of anatomical feature 112 over the cardiac cycle.

[0082] How to use System 100:

[0083] For example Figure 10a and Figure 10b As shown, method 200 includes step 202, providing system 100. Step 204, imaging system 104 images anatomical features 112 of patient 114. Step 206, imaging system 104 generates anatomical imaging data 142 based on the imaged anatomical features 112 of patient 114. Step 208, measurement system 106 measures anatomical movements of patient 114. Step 210, measurement system 106 generates anatomical motion data 144 based on the measured anatomical movements of patient 114. Step 212, computer system 108 receives anatomical imaging data 142 and anatomical motion data 144. Step 214, computer system 108 generates an enhanced representation 110 based on anatomical imaging data 142. Step 216, computer system 108 correlates the enhanced representation 110 with the anatomical features 112 of patient 114. Step 218, computer system 108 correlates the enhanced representation 110 with the anatomical motion data 144 of patient 114. Step 220: The computer system 108 may render the augmented representation 110 in the augmented reality environment 116 on the augmented reality system 102. As described above, the computer system 108 may render the augmented representation 110 on a portion 118 of the patient 114 in the augmented reality environment 116 on the augmented reality system 102. Step 222: The computer system 108 may selectively render the augmented representation 110 based on anatomical motion data 144.

[0084] Now refer to Figure 11a and Figure 11b Method 200' may include step 224', applying post-processing to the enhanced representation 110. For example, as Figures 3 to 4 As shown, the computer can apply anti-aliasing to the enhanced representation 110 to smooth jagged edges 126 that may form during the generation process into smooth edges 128.

[0085] Advantageously, the aforementioned system 100 and method enable dynamic registration of anatomical structures using augmented reality. This approach reduces overall procedural costs, provides real-time procedural data, and reduces and / or eliminates radiation exposure because CT scans are not required before or during the procedure to generate images. Ideally, this disclosure could be a solution that takes into account dynamic body movements such as the breathing of the patient 114 in the registration, correction, and dynamic motion of anatomical structures represented in the augmented reality environment 116.

[0086] This document provides exemplary embodiments to provide a thorough understanding of the disclosure and to fully communicate the scope of the disclosure to those skilled in the art. Numerous specific details, such as examples of specific components, apparatus, and methods, are set forth herein to provide a thorough understanding of embodiments of the disclosure. It will be apparent to those skilled in the art that these specific details are not required, and exemplary embodiments may be embodied in many different forms, none of which should be construed as limiting the scope of the disclosure. Some exemplary embodiments do not detail known processes, known equipment structures, and known technologies. Within the scope of protection of the technology disclosed, equivalent changes, modifications, and alterations can be made to certain embodiments, materials, compositions, and methods to obtain substantially similar results.

Claims

1. A system for dynamically registering patient anatomical structures using augmented reality, comprising: An augmented reality system configured to display an augmented representation of a patient's anatomical features in an augmented reality environment; An imaging system configured to image the patient’s anatomical features and generate anatomical imaging data; The measurement system is configured to measure the in vivo movement of soft tissues in the patient's anatomical structure and generate anatomical motion data prior to performing surgery on the patient, wherein the anatomical motion data of the in vivo movement of soft tissues in the patient's anatomical structure includes: anatomical motion data collected during the motion cycle of the patient's anatomical structure. as well as A computer system, communicating with the augmented reality system, the imaging system, and the measurement system, is configured as follows: Receives anatomical imaging data from the imaging system and anatomical motion data of soft tissue movement within the patient's anatomical structures from the measurement system. The enhanced representation is generated based on the anatomical imaging data. This enhancement is associated with the patient's anatomical features. The enhancement is associated with anatomical motion data of the in vivo movement of soft tissues within the patient's anatomy. The enhanced representation is selectively updated based on anatomical motion data of soft tissue movement within the patient's anatomy. Render augmented representations of the augmented reality environment on the augmented reality system.

2. The system according to claim 1, wherein, The augmented reality system is configured to display an augmented representation of a portion of the patient in the augmented reality environment.

3. The system according to claim 1, wherein, The imaging system includes members selected from the group consisting of: ultrasound systems, computed tomography (CT) systems, electromagnetic systems, cone-beam computed tomography (CBCT) systems, gas exchange systems, mechanically controlled ventilation systems, vital capacity measurement systems, electrocardiogram (ECG) systems, magnetic resonance imaging (MRI) systems, electromechanical wave propagation systems, transesophageal echocardiography (TEE) systems, and combinations thereof.

4. The system according to claim 1, wherein, The imaging system is configured to image the patient's anatomical features and generate the anatomical imaging data prior to performing surgery on the patient.

5. The system according to claim 1, wherein, The imaging system is configured to image the patient's anatomical features and generate the anatomical imaging data during surgical procedures.

6. The system according to claim 1, wherein, The measurement system includes members selected from the group consisting of: ultrasound systems, computed tomography (CT) systems, electromagnetic systems, cone-beam computed tomography (CBCT) systems, gas exchange systems, mechanical ventilation systems, vital capacity measurement systems, electrocardiogram (ECG) systems, magnetic resonance imaging (MRI) systems, electromechanical wave propagation systems, transesophageal echocardiography (TEE) systems, and combinations thereof.

7. The system according to claim 1, wherein, The measurement system is configured to measure anatomical movements and generate the anatomical movement data during surgical procedures on a patient.

8. The system according to claim 1, wherein, The computer system is also configured to smooth the enhanced representation, thereby smoothing the jagged edges of the enhanced representation.

9. The system according to claim 1, wherein, The computer system is configured to update the enhanced representation by updating the position of the patient's anatomical features, thereby compensating for the movement of the anatomical features based on the anatomical motion data.

10. The system according to claim 1, wherein, The computer system is configured to update the augmented representation by updating the structure of the augmented representation, thereby displaying structural changes in the augmented representation based on the anatomical motion data.

11. The system according to claim 1, wherein, The computer system is configured to update at least one of the anatomical features and the structure of the enhanced representation by animing the transition between the original state of the enhanced representation and the updated state of the enhanced representation, thereby updating the enhanced representation.

12. The system according to claim 1, wherein, The computer system is configured to selectively update the enhanced representation based on the anatomical motion data at predetermined intervals.

13. The system according to claim 1, wherein, The computer system is configured to selectively update the enhanced representation in response to input.

14. A method for dynamically registering patient anatomical structures using augmented reality, the method comprising the following steps: A system is provided for dynamic registration of anatomical structures using augmented reality, comprising an augmented reality system, an imaging system, a measurement system, and a computer system; The imaging system images the patient's anatomical features; Anatomical imaging data is generated by the imaging system based on imaging the anatomical features of the patient; The measurement system measures the in vivo movement of soft tissues in the patient's anatomy prior to performing surgery on the patient; The measurement system generates anatomical motion data of the soft tissues in the patient's anatomical structure based on the measurement of the patient's anatomical motion, wherein the anatomical motion data of the soft tissues in the patient's anatomical structure includes: anatomical motion data collected within the motion cycle of the patient's anatomical structure. The computer system receives the anatomical imaging data and the anatomical motion data of the soft tissues in the patient's anatomical structure. The computer system generates the enhanced representation based on anatomical imaging data; The computer system associates the enhanced representation with the patient's anatomical features; The computer system correlates the enhanced representation with anatomical motion data of the in vivo movement of soft tissues in the patient's anatomy. The enhanced representation is selectively updated by the computer system based on anatomical motion data of soft tissue movement within the patient's anatomy; and The computer system renders an augmented representation of the augmented reality environment on the augmented reality system.

15. The method according to claim 14, wherein, The measurement system includes an ultrasound system with an ultrasound probe, which captures 2D images by moving the ultrasound probe over the patient's anatomical features and pausing for a predetermined time at set pause intervals.

16. The method of claim 14, wherein, The measurement system includes a vital capacity measurement system, wherein the anatomical motion data includes a flow loop with multiple reference points that correlate the location of the enhanced representation with a known amount of air in the patient's lungs.

17. The method according to claim 14, wherein, The measurement system includes an electrocardiogram (ECG) system, wherein the anatomical motion data includes ECG electrical cycles with multiple components, each of which is associated with a segment of the enhanced representation.

18. The method of claim 14, further comprising the step of: applying post-processing to the enhanced representation to smooth the enhanced representation, thereby smoothing jagged edges of the enhanced representation.

Images