Conversational planner for repair or replacement surgery

Abstract

According to certain aspects of the present disclosure, a computer-implemented method includes receiving 3D imaging. The method includes generating a 3D model based on the 3D imaging. The method includes, in response to a predetermined set of surgical steps, generating a finite number of 4D time-transformation scenes for each predetermined surgical step associated with the 3D model. The method includes displaying a selective interactive simulation based on the finite number of 4D time-transformation scenes. A system and a machine-readable medium are further provided.

Description

[Technical Field]

[0001] This application claims the benefit of priority under 35 U.S.C. § 119 of U.S. Provisional Patent Application No. 63 / 371,444, entitled "Interactive Planner for Repair or Replacement Surgery," filed August 15, 2022, the disclosure of which is incorporated herein by reference in its entirety for all purposes.

[0002] TECHNICAL FIELD This disclosure relates generally to integrated surgical simulation, and more particularly to integrated surgical simulation using an interactive planner for repair or replacement surgery. [Background technology]

[0003] Structural heart disease (SHD) interventions are rapidly growing in cardiac surgery. New procedures and devices are developed and introduced every year. These interventions include, for example, aortic valve, mitral valve, atrial septum, ventricular septum, left atrial appendage closure, tricuspid valve, and pulmonary valve interventions.

[0004] Heart disease is the leading cause of death in the United States. As of 2019, it is estimated that approximately 48% of Americans have cardiovascular disease. Cardiac surgery has advanced over the last 15 years, particularly with the development of transcatheter heart valve replacement (THVR) and surgical heart valve repair procedures.

[0005] In some scenarios, hospital heart teams meet regularly to determine treatment paths for their heart valve patients. Patients may undergo a variety of treatments, including heart valve replacement or repair. While there are patient populations best served by each treatment, there is also a third patient population that is a candidate for either repair or replacement. To date, there have been no randomized, prospective clinical studies to definitively conclude the most appropriate treatment path for these patient subsets. Therefore, physicians are challenged to visualize the outcomes of repair versus replacement in a patient-specific manner when determining a patient's treatment path. While both replacement and repair procedures are legitimate treatment options for some patients, each procedure has its own unique combination of potential complications and benefits that physicians must objectively analyze.

[0006] There is growing evidence that valve-related complications, such as leaflet thrombosis, prosthesis malmatch, and coronary ostial obstruction, as well as leaks, are strongly correlated with the interaction of transcatheter heart valve (THV) devices with a patient's individual anatomy and the resulting fluid dynamics, resulting in costs to hospitals and, in some cases, reduced quality of patient care in certain circumstances when there is a strong concern about potential complications. With regard to surgical repairs (e.g., valve repair, congenital defect repair, etc.), there are very limited simulation methods available for preoperative planning.

[0007] The description provided in the Background Art section should not be considered to be prior art merely because the prior art is mentioned in or related to the Background Art section. The Background Art section may contain information that describes one or more aspects of the subject technology. Summary of the Invention [Means for solving the problem]

[0008] According to certain aspects of the disclosed technology, systems and methods are provided for preoperative planning of medical surgical repair or implantation procedures, such as, but not limited to, structural heart procedures. The disclosed technology enables physicians to have multiple integrated simulations for, for example, navigating between multiple structural heart procedures in order to visualize simulated predictions of the long-term functionality of cardiac structures as they may require surgical intervention. The disclosed technology surpasses other similar technologies by allowing physicians to visualize changes in simulated results in real time as a continuous response to changes in initial parameters by interpolating between simulated results generated off-time. This feature is important for users to experience real-time, independent surgical exploration with real-time feedback on the functional effectiveness of repair or implantation configurations.

[0009] According to certain aspects of the present disclosure, a computer-implemented method includes receiving 3D imaging data. The method includes generating a 3D model based on the 3D imaging. The method includes generating, in response to a predetermined set of surgical steps, a finite number of 4D time deformation scenes for each predetermined surgical step associated with the 3D model. The method includes displaying a selective interactive simulation based on the finite number of 4D time deformation scenes.

[0010] According to another aspect of the present disclosure, a system is provided. The system includes a memory containing instructions and a processor configured to execute the instructions, which, when executed, cause the processor to receive 3D imaging. The processor is configured to execute the instructions, which, when executed, cause the processor to generate a 3D model based on the 3D imaging. The processor is configured to execute the instructions, which, when executed in response to a predetermined set of surgical steps, cause the processor to generate a finite number of 4D time-transformation scenes for each predetermined surgical step associated with the 3D model. The processor is configured to execute the instructions, which, when executed, cause the processor to display a selective interactive simulation based on the finite number of 4D time-transformation scenes.

[0011] According to another aspect of the present disclosure, a non-transitory machine-readable storage medium having machine-readable instructions for causing a processor to execute a method is provided. The method includes generating a 3D model based on 3D imaging. The method includes generating, in response to a predetermined set of surgical steps, a finite number of 4D time-transformation scenes for each predetermined surgical step associated with the 3D model. The method includes displaying a selective interactive simulation based on the finite number of 4D time-transformation scenes.

[0012] According to another aspect of the present disclosure, a method for seamless real-time interactive navigation between various surgical repair scenarios or configurations or device implantation scenarios or configurations is provided.

[0013] According to another aspect of the present disclosure, a method is provided for visualization of altered physical properties of tissue deformation integrated into a simulation.

[0014] According to another aspect of the present disclosure, a method is provided for visualizing how surrounding structural heart tissue will heal over time after a structural heart procedure.

[0015] According to another aspect of the present disclosure, a method is provided for visualizing how healing of structural heart tissue after a structural heart procedure changes cardiac function over time.

[0016] According to another aspect of the present disclosure, a method is provided for measuring changes in valve deployment depth, angle, and eccentric deployment for further procedure optimization.

[0017] According to another aspect of the present disclosure, a method is provided for simulating structural cardiac tissue manipulation and resulting tissue function.

[0018] According to another aspect of the present disclosure, a method is provided for real-time interactive visualization and manipulation of various structural cardiac procedures (congenital heart defects, valve replacement, valve repair).

[0019] According to another aspect of the present disclosure, a method is provided for visualization of various rescue strategies during structural heart surgery, when indicated.

[0020] According to another aspect of the present disclosure, a method is provided for integrating physical properties of cardiac tissue into a deep learning artificial neural network algorithm.

[0021] According to another aspect of the present disclosure, a method is provided for training a structural cardiac procedure database aimed at identifying new tissue interactions and potential procedural complications.

[0022] According to another aspect of the present disclosure, a method is provided that enables digital measurement of structural cardiac components to assist in planning cardiac surgery and transcatheter procedures.

[0023] According to another aspect of the present disclosure, a method is provided for training a database regarding the healing / remodeling of a particular cardiac tissue that is the focus of a surgical or transcatheter intervention.

[0024] It will be understood that other configurations of the subject technology will be readily apparent to those skilled in the art from the following detailed description, in which various configurations of the subject technology are shown and described by way of example. As will be understood, the subject technology is capable of other and different configurations, and its several details can be modified in various other respects, all without departing from the scope of the subject technology. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature, and not as restrictive.

[0025] The accompanying drawings, which are included to provide a further understanding and are incorporated in and constitute a part of this specification, illustrate disclosed embodiments and, together with the description, serve to explain the principles of the disclosed embodiments. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 illustrates an exemplary architecture for generating an integrated interactive simulation of 3D images. [Figure 2] FIG. 1 is an exemplary block diagram illustrating a server and a user device according to certain aspects of the present disclosure. [Figure 3] FIG. 10 shows an example wireframe rendering looking down on the ascending aorta, without valves, demonstrating the triangular graph structure of the mesh. [Figure 4]FIG. 4 shows an example rendering from the same angle as FIG. 3, entering the aorta. All zones are assigned different colors for illustrative purposes. The aortic wall 400 is red, calcium 402 is green, and the aortic valve leaflets 404, 406, and 408 are yellow, purple, and blue, respectively. In this rendering, the transcatheter valve leaflets share zone 410, which is assigned brown, and the stent is composed of two zones 412 and 414, which are cyan and white. [Figure 5A] FIG. 5 illustrates an example mesh showing a simulated rendering of FIG. 4 at full resolution. [Figure 5B] FIG. 5B illustrates an example mesh showing the simulated rendering of FIG. 5A after significant simplification. [Figure 6] FIG. 6 illustrates regions of large vertex movement 600 (highlighted in red) that directly correlate with regions of high importance when interpreting the results of a simulation. [Figure 7] Figure 1 shows an example 2D bounding box drawn outside the aorta, indicating regions of high sensitivity; however, the actual algorithm uses an axis-aligned 3D bounding box, which avoids recursive subdivision of regions of low sensitivity. [Figure 8A] FIG. 10 shows a portion of an aorta rendering without normal data. [Figure 8B] A portion of a 3D rendering with normal data. Normal data provides the rendering engine with information that can be used to smooth lighting, thereby giving the impression of a higher quality model without increasing geometric complexity. [Figure 9]9A-9C illustrate example sub-scenarios. The top example in FIG. 9 shows interpolation between two sub-scenarios (varying valve deployment depth). Because these sub-scenarios share the same triangulation, vertex data can be smoothly interpolated. The bottom example in FIG. 9 shows interpolation between two scenarios (varying prosthetic valve brands). Because different scenarios involve meshes using different triangulations, crossfading is required between incompatible meshes. Note that because the meshes represent calcium and the aortic root shares triangulation across different scenarios, the vertex data for these components for visualization can likewise be smoothly interpolated when the scenarios share the same patient. [Figure 10] Figure 10 shows a rendering of a GLTF file encompassing all animation frames of a sub-scenario in which an Evolut® transcatheter valve is expanded at the aortic root to a depth of 1 mm. [Figure 11] FIG. 1 illustrates an example slider-based user interface used to control the exploration of parameter space. Vertical lines in the slider explicitly indicate the sub-scenario being simulated. Moving the slider thumb between these click stops drives the interpolation between the simulated outcomes. The slider thumb magnetically stops cleanly at these click stops, ensuring that the physician does not accidentally observe an interpolated outcome during the planning process. [Figure 12A] FIG. 10 illustrates an example extension force demonstrated by a color gradient, with opacity and redness corresponding to base rupture risk. [Figure 12B] FIG. 10 illustrates an example extension force demonstrated by a color gradient, with opacity and redness corresponding to base rupture risk. [Figure 12C] FIG. 10 illustrates an example extension force demonstrated by a color gradient, with opacity and redness corresponding to base rupture risk. [Figure 12D]FIG. 10 illustrates an example extension force demonstrated by a color gradient, with opacity and redness corresponding to base rupture risk. [Figure 12E] FIG. 10 illustrates an example extension force demonstrated by a color gradient, with opacity and redness corresponding to base rupture risk. [Figure 13] Figure 10 shows the cursor hovering over the part of the simulation that allows the physician to inspect the extension force by direct value. [Figure 14A] FIG. 10 shows exemplary linear measurements demonstrating the evolution of an object's length and interpolated deployment depth over time. [Figure 14B] FIG. 10 shows exemplary linear measurements demonstrating the evolution of an object's length and interpolated deployment depth over time. [Figure 14C] FIG. 10 shows exemplary linear measurements demonstrating the evolution of an object's length and interpolated deployment depth over time. [Figure 14D] FIG. 10 shows exemplary linear measurements demonstrating the evolution of an object's length and interpolated deployment depth over time. [Figure 15] FIG. 2 is a block diagram illustrating an exemplary computer system with which the server and user device of FIG. 1 may be implemented. [Figure 16] FIG. 1 illustrates an exemplary process for generating an integrated interactive simulation of a 3D image. DETAILED DESCRIPTION OF THE INVENTION

[0027] Not all of the components depicted in each figure may be required in one or more implementations, and one or more implementations may include additional components not shown in the figures. Variations in the arrangement and type of components may be made without departing from the scope of the present subject disclosure. Additional, different, or fewer components may also be utilized within the scope of the present subject disclosure.

[0028] The detailed description set forth below is intended to be a description of various implementations and is not intended to represent the only implementations in which the technology of the present subject matter can be practiced. Those skilled in the art will understand that the described implementations can be modified in various ways without departing from the scope of the present disclosure. Accordingly, the drawings and this description are to be regarded as illustrative in nature and not as restrictive.

[0029] According to certain aspects of the disclosed technology, systems and methods are provided for preoperative planning of medical-surgical repair or implantation procedures, such as, but not limited to, structural heart procedures. The disclosed technology enables physicians to have multiple integrated simulations for, for example, navigating between multiple structural heart procedures, with the goal of visualizing simulated predictions of the long-term functionality of cardiac structures as they may require surgical intervention. The disclosed technology surpasses other similar technologies by allowing physicians to visualize in real time changes in simulated results as a continuous response to changes in initial parameters by interpolating between simulated results generated off-time. This feature is important for users to experience real-time, independent surgical exploration with real-time feedback on the functional effectiveness of repair or implantation configurations.

[0030] In certain aspects, the disclosed technology enables (1) optimizing patient outcomes by providing every heart team with expert, biomechanically optimized strategies for valve selection and deployment guidance, including risk mitigation strategies, and (b) saving hospital stays and money while increasing impact in a transition from a "fee-for-service" to an "outcomes-based" reimbursement environment.

[0031] While the disclosed technology is applicable to any medical-surgical repair or implantation procedure, an exemplary description in the context of structural heart surgery follows below. The field of structural heart intervention is a broad and rapidly evolving medical specialty. Therefore, it is essential that surgeons and interventional cardiologists / imagers be able to visualize simulations of various procedures in a patient-specific manner.

[0032] The disclosed technology includes a computational model simulation component that allows physicians to visualize biophysical-based cardiac tissue interactions during various structural cardiac procedures. The deep learning algorithms of the disclosed technology take into account or model the pre- and post-procedural biomechanics of cardiac structures and quantitatively assess the function of diseased anatomy pre- and post-procedure in both valve replacement and repair. Tissue biomechanical properties are derived from pre-procedure CT (computed tomography) imaging by solving an inverse problem to fit reduced-order model parameters through deep learning, which is applied to pre- and post-procedure imaging datasets (including clinical patient databases and bench tests) and then incorporated into the deep learning algorithm. In addition, the disclosed technology takes into account how the manipulated tissue heals and changes over time, which can be considered a fifth dimension. This allows physicians to gain a better understanding of the long-term physiological function of the intervention, rather than just simulating the acute surgical procedure. This physiological function takes into account post-intervention tissue remodeling as well as hemodynamic variability associated with the procedure being performed. The material properties used to generate the simulation will be trained from multi-phase CT scans or other imaging modalities such as MRI (magnetic resonance imaging), ultrasound, PET (positron emission tomography), nuclear imaging modalities, or other functional imaging modalities, as these contain information about the structure-function relationships of specific structural heart disease states.

[0033] 1 illustrates an exemplary architecture 100 for generating an integrated interactive simulation of 3D images. For example, the architecture 100 includes a server 10 and a user device 12 connected via a network 14.

[0034] The server 10 may be any device having an appropriate processor, memory, and communications capabilities for communicating with the user devices 12. For load balancing purposes, the server 10 may include multiple servers. The user devices 12, with which the server 10 communicates via the network 14, may be, for example, tablet computers, mobile phones, mobile computers, laptop computers, portable media players, eBook readers, or any other devices having appropriate processors, memory, and communications capabilities. In certain aspects, the server 10 may be an infrastructure-as-a-service (IaaS) cloud computing server and may be capable of supporting platform-as-a-service (PaaS) and software-as-a-service (SaaS) services.

[0035] Network 14 may include, for example, any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, etc. Additionally, network 14 may include, but is not limited to, any one or more of the following network topologies: a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree network, or a hierarchical network.

[0036] Figure 2 is a block diagram illustrating an example of a server 10 and a user device 12 within the architecture of Figure 1, in accordance with certain aspects of the present disclosure. For purposes of illustration, a user device 12 is described, but it should be understood that any number of user devices 12 may be used.

[0037] The server 10 and the user device 12 are connected over a network 14 via respective communication modules 16, 18. The communication modules 16, 18 are configured to interface with the network 14 to send and receive information, such as data, requests, responses, and commands, to other devices on the network 14. The communication modules 16, 18 may be, for example, modems or Ethernet cards. The server 10 is connected over the network 14 to a database 30. In certain embodiments, the database 30 is a compiled serial image database of tissue healing. In certain embodiments, the database 30 is a trained database containing functional quantification performed postoperatively using invasive or non-invasive imaging.

[0038] Server 10 includes a processor 20, a communications module 16, and a memory 22. Processor 20 of server 10 is configured to execute instructions, such as instructions physically coded into processor 20, instructions received from software in memory 22, or a combination of both. Processor 20 of server 10 is configured to perform the functions described herein.

[0039] The user device 12 includes a processor 24, a communications module 18, and a memory 26. The processor 24 of the user device 12 is configured to execute instructions, such as instructions physically coded into the processor 24, instructions received from software in the memory 26, or a combination of both. The processor 24 of the user device 12 is configured to perform the functions described herein.

[0040] 1 and 2, any imaging (e.g., CT, MRI, ECHO) capable of producing a 3D image 28 serves as input for the computational algorithm of the server 10. In this simulation, patient imaging (e.g., CT, MRI, ECHO, 3D ECHO) is first used to generate a 3D model of the organ / tissue that is the subject of a surgical repair or implantation procedure. Then, an off-time computer simulation generates a finite number of time deformations of the organ / tissue and any implantation device or instrument in response to a predetermined set of possible surgical steps (e.g., shortening the leaflet edge length, repositioning the leaflet to an artificial base, moving the commissural posts in a given direction, or reducing the size of the annulus by a given amount). The simulation generates time deformations (4D data) of the organ / tissue for each possible surgical step.

[0041] The a priori generated scenes are unique because each individual scene is a 4D data set (or 5D or greater if healing is also modeled) capable of representing the time transformation of the engineered tissue. These scenes are used as input to a novel, devised planner, thereby enabling real-time simulation and functional quantification for planning and decision-making. The generation of each scene is achieved using finite element modeling, reduced-order modeling, and other artificial intelligence and machine learning alternatives for modeling. For example, a surgeon can visualize what is happening to the entire organ tissue as they constrict a specific patient's valve annulus. The surgeon can then view the time transformation of this tissue, as well as a patient-specific animation of the tissue in response to the constriction surgical technique (Scene 1). In addition, a second scene will be generated that will generate a patient-specific animation of the tissue in response to (for example) trimming of a valve leaflet. All possible steps involved in device repair, replacement, or implantation are generated to simulate the corresponding scene.

[0042] The scene simulation will further analyze the behavior of the organ / tissue system after surgery. Functional quantification can include computational fluid modeling, solid modeling using finite element analysis (FEA), computational fluid dynamics modeling (CFD), AI / ML, or reduced order models. The above-mentioned models can be trained on a database including functional quantification performed after surgery using invasive or non-invasive imaging or measurements (e.g., ECHO, cath (catheterization), CT, MRI, PET, fMRI (functional magnetic resonance imaging), etc.).

[0043] This will be achieved by allowing artificial intelligence algorithms to be trained on a compiled database of serial images of tissue healing. The serial images can be acquired using non-invasive or invasive techniques. For applications involving the skin (e.g., plastic surgery), the images can be optical (photographic).

[0044] The conversational planner takes all generated scenes and allows the surgeon to virtually explore the data, interacting between multiple scenes and their corresponding scenarios. All scenes are pre-computed using a novel interpolation method, allowing the surgeon to see how each surgical procedure will alter the entire patient-specific organ system. The planner is equipped with augmented reality (AR) and virtual reality (VR) capabilities for an immersive user experience.

[0045] Additionally, the ability to visualize multiple treatment scenarios and smoothly navigate through a multidimensional scenario space is a desirable feature that will enable data-driven surgical planning. The disclosed simulation generates animations of a patient's surgical outcome according to valve surgical parameters as well as the patient's physiology. A "scenario" in this section will refer to a unique combination of circumstances that distinguish a procedure, including but not limited to the patient, transcatheter valve type, and anatomical region. A collection of simulated animations that share these contextual variables will be referred to as a "sub-scenario" because they all represent the same procedure but differ in the continuous values ​​that govern the outcome.

[0046] 3-14D, exemplary methods according to certain aspects of the present disclosure that may be implemented by the server 10 and the user device 12 are described below.

[0047] "Mesh" in this section means a set of points in triangles (or other polygonal structures) that provide a 3D representation of the geometry being CT scanned.

[0048] Referring to Figure 3, this visualization capability is applicable for heart teams planning structural heart procedures, but visualization of surgical sub-scenario can be extended to other surgical specialties, such as orthopedic surgery, neurosurgery, neurovascular intervention, plastic and reconstructive surgery, and gynecological surgery. The generated scenes need not be specific to the cardiovascular system. The novel interpolation method smoothly generates progressions between different 3D files generated from the predictive model. The interpolation method can be linear, nonlinear, or any other method, including, but not limited to, reduced-order modeling involving a combination of predetermined primitives.

[0049] Referring to Figure 4, sub-scenario frames contain frames of mesh data that can be rendered in sequence to animate the procedure results according to the sub-scenario's individual parameters. Each frame of mesh data contains a set of meshes that represent a "zone" of the reconstructed 3D volume of the patient's body. For example, a zone within the 3D volume containing the patient's ascending aorta may include (1) a section of the aortic wall, (2) the left coronary leaflet, (3) the right coronary leaflet, (4) the non-coronary leaflet of the aortic valve, and (5) a transcatheter valve stent.

[0050] One example scenario is "Expansion of the Evolut transcatheter valve inside the aortic root to a depth of 3 mm," designed to reveal potential occlusions in the coronary arteries. Sub-scenario parameters for this scenario include deployment depth, valve rotation, etc.

[0051] A novel visualization device for reviewing simulated results in an accessible, intuitive, and informative manner requires the following functionality: 1. Accurate real-time rendering of simulated data on mobile devices 2. Smooth transitions between simulated scenarios 3.Physically-based inspection tool.

[0052] The visualization accuracy of simulated results is directly proportional to the complexity of the mesh displayed in the visualization device. Rendering performance is inversely proportional to the mesh complexity and therefore accuracy. This is especially evident on memory-constrained mobile devices.

[0053] FIG. 5A is an example mesh showing a simulated rendering of FIG. 4 at full resolution.

[0054] Figure 5B shows an example mesh that shows a simulated rendering of Figure 5A after significant simplification: while the shape is preserved, the number of vertices and triangles (and thus the total data size for storing and transmitting the mesh) is reduced by approximately 78%.

[0055] Mesh simplification is an adjustable algorithm that can help reduce the complexity of a mesh without significantly sacrificing accuracy by prioritizing the removal of vertices that already lie within the plane formed by their neighboring vertices in the mesh. Many algorithms exist that can perform this simplification with various tradeoffs.

[0056] However, the disclosed technology requires a more sophisticated approach using custom solutions that selectively maintain the original simulation level of accuracy in the more sensitive portions of the simulated results. This performance cost can be borne by simplifying other portions of the simulated results, because many simulations have large portions of the mesh that remain fixed over time and are less relevant to the physician's decision-making process.

[0057] These "sensitive regions" can be determined manually (the simulation engineer defines an axis-aligned bounding box that encompasses the sensitive region) or dynamically (an algorithm analyzes a group of simulated results to determine which parts are fixed and finds a bounding box around this static region).

[0058] Referring to Figures 6 and 7, we can see that in the dynamic case, all vertices in each scenario do not change position by more than an adjustable difference across all frames of all sub-scenarios, where the distance is calculated by a function proportional to the pythagorean length. Note that this algorithm can also use triangle deformation as a heuristic to locate sensitive regions. This means that instead of looking only at the movement of a vertex as it is, it also considers how much a vertex moves relative to its neighbors in the mesh. It then computes an axis-aligned bounding box around these relatively unchanging vertices and recursively divides the bounding box so that its children contain more unchanging vertices, until the box has no changing vertices. The end result is a set of bounding boxes, each of which encompasses an area of ​​the mesh that is a good candidate for simplification.

[0059] Once the bounding boxes marking high or low sensitivity are obtained, the original mesh is split into high and low sensitivity sub-meshes. The low sensitivity sub-mesh is simplified. To improve rendering performance, the low sensitivity sub-mesh can be kept separate and shared between frames and between sub-scenarios, so that data is not copied across multiple frames and can be maintained within the GPU.

[0060] For meshes with additional per-vertex data, such as fluid pressure, the change in these values ​​over time can be used as candidates for sensitive discrimination, or texture mapping can be used to visually preserve high-frequency features while reducing the number of vertices, provided that loss of geometric precision is acceptable as long as no surface-visible data is lost.

[0061] 8A and 8B, once the high sensitivity regions are determined, well-known mesh simplification algorithms reduce the complexity of the low sensitivity regions, thereby improving rendering performance. Other common post-processing algorithms are then used to add per-vertex data such as normals, thereby improving rendering quality by smoothing lighting at a negligible cost relative to performance.

[0062] For scenarios with sub-scenarios that move across large portions of the enclosing volume, a level of detail system is employed that further optimizes performance by switching between different sets of low-sensitivity regions. The algorithm computes only low-sensitivity bounding boxes across animation frames of individual sub-scenarios, and tracks the movement of bounding boxes across the sub-scenario space, only combining similar bounding boxes by an adjustable factor.

[0063] Data accessibility is crucial to a physician's ability to efficiently draw conclusions from the disclosed simulations. Leverage open-source web technologies to make data visualizations accessible on the widest possible range of devices. In certain aspects, GLTF is a commonly preferred mesh representation format because it is easily compressed to conserve bandwidth, is accepted by nearly all 3D rendering engines, and supports representation of rendering techniques that allow the visualization to convincingly depict the data.

[0064] Once the mesh data is finalized, we construct a GLTF representation of the data that can be rendered by any suitable rendering engine. We describe this construction strategy in the next section because its details are important to our interpolation method.

[0065] The disclosed technology empowers physicians to smoothly navigate diverse surgical scenarios by fostering novel mental models, where decisions already considered fixed and discrete are expressed as organic potentialities within a continuous space of possibilities.

[0066] A given scenario has a set of sub-scenarios that can be depicted as a diagram within this continuous space of possibilities. The axes of this space include time, as well as any linearly separable parameters that can be explored for the scenario. To demonstrate continuity across sub-scenarios, a data visualizer generates linear interpolations between the sub-scenario meshes.

[0067] Referring to Figure 9, in order for a mesh to be compatible with interpolation, it must have a compatible triangulation. Triangulation is when successive indices point to the vertices that make up the mesh. It is assumed that all sub-scenarios that share a scenario utilize the same triangulation, and that the mesh simplification described in the previous section is applied to all meshes in each sub-scenario as well. For interpolation between sub-scenarios that employ a level-of-detail strategy, a combination of interpolation and crossfading is used to provide an approximate experience.

[0068] Interpolation between meshes is achieved within the visualization device through a well-known technique called morph targets. Morph targets allow interpolation between any number of meshes by rendering a weighted average of the positions of the meshes' vertices. For interpolation between sub-scenario involving non-positional data, a similar technique is used to render a weighted average of color data or textures, with the average computed in a custom shader program running on the device's graphics processing unit.

[0069] The challenge of smooth interpolation between an unlimited number of sub-scenarios is maintaining a seamless experience while downloading and exchanging large amounts of visualized data. The disclosed technology solves this problem by making optimal use of data compression, data sharing, data streaming, and data caching.

[0070] Another problem solved by the disclosed technique is determining fit zones for smooth interpolation across variables.

[0071] GLTF files generated from simplified mesh data are compressed using a lossless compression algorithm designed specifically for the binary data in the GLTF format, which produces state-of-the-art compression results that are satisfactory for our use case.

[0072] 10, multiple batches of sub-scenarios that are expected to be examined by a physician soon, depending on usage metrics, are included in a single GLTF file. For example, sub-scenarios can be batched along a timeline, resulting in animation playback as soon as the file containing the sub-scenario is loaded into memory.

[0073] For example, when a physician examines a particular sub-scenario across the timeline, other nearby sub-scenarios in the parameter space are partially loaded, allowing for rapid interpolation to the adjacent sub-scenarios. When navigating to sub-scenario defined by a different set of parameters, the remaining frames along the timeline are optionally downloaded and then unpacked and loaded for visualization. The details of the unpacking and loading implementation are handled by the rendering engine that performs the visualization.

[0074] By leveraging the bounding boxes from the previous section, we have completely fixed options for the treatment portion of the simulation. These fixed components typically serve to contextually adapt the informational portion of the simulated results and can be treated separately from the sub-scenario. In this case, we can create a separate GLTF file containing only the fixed components. If we have dynamic data that does not affect the vertex positions in the fixed components, we can have a separate file containing only the data per sub-scenario, allowing us to perform interpolation without incurring the bandwidth cost of downloading a copy of the vertex data every time the user switches between sub-scenarios.

[0075] 11, with these optimizations, the disclosed technology provides a smooth experience that enables the physician to understand the continuous relationships between the parameters explored by the physician. The user interface provided to the physician for exploring the sub-scenarios is primarily driven by a set of sliders that describe the set of available parameters. These sliders indicate the range of simulated sub-scenarios and use click stops to move the physician to the simulated sub-scenarios rather than the interpolated sub-scenarios, because the interpolated mesh does not require medical accuracy.

[0076] Visualization devices for simulated results are useful not only in a conceptual sense, facilitating the synthesis of data into actionable conclusions, but also in grounding simulated data in the real world.

[0077] 12A-12E and 14D, the disclosed technology reveals a set of inspection tools that assist physicians in their ability to understand information and present that information to others. These inspection tools include, but are not limited to, zone-by-zone opacity control, zone-by-zone slicing, and measurement tools for extension force length, angle, and rate of change of each. These inspection tools enable physicians to glean specially designed real-world information from simulated data.

[0078] To inspect the data and place measurement points, real-time ray casting against the mesh is required. Given a large number of triangles in the mesh, it is necessary to employ sophisticated techniques to determine the locations to ray cast. Typical ray casting uses a two-step approach, where a simple collider around the mesh is inspected for intersections, and then all individual triangles are inspected for intersections.

[0079] A distinct advantage of mixed reality head-mounted displays (HMDs) is that they provide a highly intuitive sense of scale. Using a visualization device with an HMD allows for life-size rendering and even arbitrary scaling. When inspecting life-size simulated results in an augmented reality headset, such as the Microsoft HoloLens 2, physicians have physically accurate predictions readily available in the operating room. The rendering quality, shape, and scale realism of the disclosed technology allow physicians to perform procedures without having to incur the cognitive load of mapping the data visualization device's display to its real-world counterpart.

[0080] The surgical planner will provide the surgeon with a detailed virtual plan of the intended procedure. The surgical plan can be provided as input data for a surgical robot that performs the surgical procedure. By recreating a physically accurate simulation of the outcome, the disclosed technology's unique combination of physical measurement tools and interactive patterns of mixed reality provides the physician with a virtual laboratory where they can empirically plan the surgical approach with highly realistic insight and devise an optimal surgical plan specific to each patient and their anatomy / pathology. This plan can further include any "bailout" strategies that are simulated by the surgeon prior to surgery in case of planning an emergency life-saving intervention if necessary.

[0081] 15 is a block diagram illustrating an example computer system 1500 in which the server 10 and user device 12 of FIG. 2 may be implemented. In certain aspects, the computer system 1500 may be implemented using hardware or a combination of software and hardware, such as in a dedicated server, or integrated into another entity or distributed across multiple entities.

[0082] Computer system 1500 (e.g., server 10 and user system 12) includes a bus 1508 or other communication mechanism for communicating information and a processor 1502 (e.g., processors 20, 24) coupled to bus 1508 for processing information. According to one aspect, computer system 1500 can be an IaaS cloud computing server capable of supporting PaaS and SaaS services.

[0083] In addition to hardware, computer system 1500 can include code that creates an execution environment for a subject computer program, such as code comprising one or more of processor firmware, protocol stacks, database management systems, operating systems, or a combination of these, stored in included memory 1504 (e.g., memory 22, 26), such as random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable PROM (EPROM), registers, hard disk, removable disk, CD-ROM, DVD, or any other suitable storage device coupled to bus 1508 for storing information and instructions executed by processor 1502. Processor 1502 and memory 1504 can be supplemented by, or incorporated in, special purpose logic circuitry.

[0084] The instructions may be stored in memory 1504 and may be embodied in one or more computer program products, e.g., one or more modules of computer program instructions encoded on a computer-readable medium for execution by or to control the operation of computer system 1500.

[0085] Computer programs discussed herein do not necessarily correspond to files in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple collaborative files (e.g., files storing one or more modules, subprograms, or portions of code). A computer program may be deployed to run on one computer or on multiple computers located at a single location or distributed across multiple locations and interconnected by a communications network, such as in a cloud computing environment. The processes and logic flows described herein may be implemented by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating results.

[0086] The computer system 1500 further includes a data storage device 1506, such as a magnetic disk or optical disk, coupled to the bus 1508 for storing information and instructions. The computer system 1500 may be coupled to various devices via an input / output module 1510. The input / output module 1510 may be any input / output module. An exemplary input / output module 1510 includes a data port, such as a USB port. Additionally, the input / output module 1510 may be configured to communicate with the processor 1502 to enable short-range communication of the computer system 1500 with other devices. The input / output module 1510 may, for example, provide wired communication in some implementations or wireless communication in other implementations, and multiple interfaces may be used. The input / output module 1510 is configured to connect to a communication module 1512. An exemplary communication module 1512 (e.g., communication modules 16, 18) includes a network interface card, such as an Ethernet card and an Ethernet modem.

[0087] In certain embodiments, the input / output module 1510 is configured to connect to multiple devices, such as input devices 1514 and / or output devices 1516. Exemplary input devices 1514 include a keyboard and a pointing device, such as a mouse or trackball, that allow a user to provide input to the computer system 1500. Other types of input devices 1514, such as tactile input devices, visual input devices, audio input devices, or brain-computer interface devices, may also be used to provide user interaction.

[0088] According to one aspect of the present disclosure, the server 10 and the user device 12 may be implemented using a computer system 1500 in response to the processor 1502 executing one or more sequences of one or more instructions contained in the memory 1504. These instructions may be read into the memory 1504 from another machine-readable medium, such as the data storage device 1506. Execution of the sequences of instructions contained in the main memory 1504 causes the processor 1502 to perform the processes described herein. One or more processors in a multi-processing configuration may also be employed to execute the sequences of instructions contained in the memory 1504. The processor 1502 may process the executable instructions and / or data structures by remotely accessing the computer program product, for example, by downloading the executable instructions and / or data structures from a remote server via the communications module 1512 (e.g., in a cloud computing environment). In alternative aspects, hardwired circuitry may be used in place of or in combination with software instructions to implement various aspects of the present disclosure. Thus, aspects of the present disclosure are not limited to any specific combination of hardware circuitry and software.

[0089] FIG. 16 shows an example process 1600 for generating an integrated interactive simulation of a 3D image.

[0090] The process begins by proceeding to step 1610 when processor 20 of server 10 receives 3D imaging. As depicted in step 1612, processor 20 of server 10 generates a 3D model based on the 3D imaging. In response to a predetermined set of surgical steps, processor 20 of server 10 generates a finite number of 4D time-transformation scenes for each predetermined surgical step associated with the 3D model, as depicted in step 1614. Processor 20 of server 10 displays selective interactive simulations based on the finite number of 4D time-transformation scenes, as depicted in step 1616.

[0091] Various aspects of the subject matter described herein may be implemented in a computing system that includes back-end components, such as a data server; middleware components, such as an application server; front-end components, such as a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein; or any combination of one or more such back-end, middleware, or front-end components. For example, some aspects of the subject matter described herein may be implemented in a cloud computing environment. Thus, in certain aspects, a user of the systems and methods disclosed herein can perform at least some of the steps by accessing a cloud server through a network connection. Furthermore, data files, circuit diagrams, performance specifications, and the like obtained from the present disclosure may be stored in a database server within the cloud computing environment or downloaded from the cloud computing environment to a private storage device.

[0092] The terms "machine-readable storage medium" or "computer-readable medium" as used herein refer to any medium that participates in providing instructions or data to the processor 502 for execution. The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific format. Such media may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media.

[0093] The terms "computer-readable storage medium" and "computer-readable medium" as used herein are generally limited to tangible, physical objects that store information in a form that is readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral signals. Storage media are distinct from, but may be used in conjunction with, transmission media. Transmission media involves transmitting information between storage media. For example, transmission media include coaxial cable, copper wire, and fiber optics, including the wires that comprise bus 508. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave or infrared data communications. Furthermore, as used herein, the terms "computer," "server," "processor," and "memory" all refer to electronic or other technological devices. These terms exclude humans or groups of humans. In this application, the terms "display" or "displaying" refer to displaying on an electronic device.

[0094] In one aspect, a method can be an operation, an instruction, or a function, and vice versa. In one aspect, a section or claim can be modified to include some or all of the words (e.g., instructions, operations, functions, or components) recited in one or more sections, one or more words, one or more sentences, one or more phrases, one or more paragraphs, and / or one or more claims.

[0095] To illustrate the interchangeability of hardware and software, various illustrative blocks, modules, components, methods, operations, instructions, algorithms, etc. are generally described in terms of their functionality. Whether such functionality is implemented as hardware, software, or a combination of hardware and software is determined by the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in a variety of ways for each particular application.

[0096] When used herein with the word "and" or "or" separating any of the items, the phrase "at least one" preceding a list of items modifies the list as a whole, not each member (e.g., each item) of the list. The phrase "at least one" does not require the selection of at least one item; rather, the phrase can mean including at least one of any one of the items, at least one of any combination of the items, and / or at least one of each of the items. By way of example, "at least one of A, B, and C" or "at least one of A, B, or C" means, respectively, A only, B only, or C only; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0097] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Terms such as "one aspect," "another aspect," "some aspects," "one or more aspects," "one implementation," "an implementation," "another implementation," "some implementations," "one or more implementations," "one embodiment," "an embodiment," "another embodiment," "some embodiments," "one or more embodiments," "one configuration," "configuration," "another configuration," "some configurations," "one or more configurations," the subject technology, the disclosure, the present disclosure, and variations thereof, are used for convenience and do not imply that disclosure associated with such phrases is essential to the subject technology or that such disclosure applies to all configurations of the subject technology. Disclosure associated with such phrases may apply to all configurations or one or more configurations. Disclosure associated with such phrases may provide one or more examples. Phrases such as one aspect or some aspects can mean one or more aspects, and vice versa, and this is equally true for the other above phrases.

[0098] Reference to an element in the singular is not intended to mean "only one," but rather "one or more," unless otherwise specified. The term "some" means one or more. Underlined and / or italicized headings and subheadings are used merely for convenience and do not limit the subject technology, and are not to be relied upon in connection with interpreting the description of the subject technology. Relative terms such as "first" and "second" may be used to distinguish one entity or action from another, but do not necessarily require or imply such an actual relationship or order between such entities or actions. All structural and functional equivalents to the elements of the various configurations described throughout this disclosure, as known or later become known to those skilled in the art, are expressly incorporated herein by reference and are intended to be encompassed by the subject technology. Furthermore, nothing disclosed herein is intended to be made available to the public, regardless of whether that disclosure is expressly set forth in the above description. A claim element shall not be construed under the provisions of 35 U.S.C. 112, sixth paragraph, unless the element is expressly recited using the phrase "means for," or, in the case of a method claim, the element is recited using the phrase "step for."

[0099] While this specification contains many details, these details should not be construed as limitations on the scope that may be claimed, but rather as descriptions of particular implementations of the present subject matter. Certain features described herein in the context of separate embodiments may also be implemented in a single embodiment in combination. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable subcombination. Furthermore, while features may be described above as functioning in a particular combination and may even initially be claimed as such, one or more features from a claimed combination may in some instances be deleted from the combination, or the claimed combination may be associated with a subcombination or a variation of the subcombination.

[0100] Although the subject matter herein has been described with reference to particular aspects, other aspects may be implemented and are within the scope of the following claims. For example, while the figures depict actions in a particular order, this should not be understood as requiring that these actions be performed in the particular order or sequential order shown, or that all of the actions shown be performed to achieve desired results. Actions recited in the claims may be performed in a different order and still achieve desired results. As one example, processes depicted in the accompanying figures do not necessarily require the particular order or sequential order shown to achieve desired results. In certain environments, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above-described aspects should not be understood as requiring such separation in all aspects, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0101] The title, background art, brief description of the drawings, abstract, and drawings are provided in this disclosure as illustrative examples of the disclosure, not as a limiting description. They are presented with the understanding that they will not be used to limit the scope or meaning of the claims. In addition, it will be appreciated that the detailed description provides illustrative examples, and that various features are grouped together in various implementations for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed subject matter requires features other than those expressly recited in each claim. Rather, as reflected in the claims, inventive subject matter lies in less than all features of a single disclosed structure or operation. The claims are incorporated into the detailed description, with each claim standing on its own as separately claimed subject matter.

[0102] The claims are not intended to be limited to only the embodiments described herein, but are intended to be accorded the full scope consistent with the claims as set forth and to encompass all legal equivalents. However, no claim is intended to, nor should any claim be construed to, encompass subject matter that fails to satisfy applicable patent law requirements.

Claims

1. The steps of receiving a medical image, The steps include generating a 3D model based on the aforementioned medical imaging, The process includes the step of generating a finite number of time-deformed scenes for each predetermined surgical step associated with the 3D model in response to a predetermined set of surgical steps, The aforementioned time-deformed scene is two-dimensional (2D) or greater. Computerized implementation method.

2. The computer implementation method according to claim 1, wherein the time-deformed scene is 4D and / or 5D.

3. The computer implementation method according to claim 1, further comprising the step of displaying a selective conversational simulation based on a finite number of time-deformed scenes.

4. The computer implementation method according to claim 3, wherein the display step includes a display using conversational augmented reality (AR), virtual reality (VR), and / or mixed reality.

5. The computer implementation method according to claim 3, further comprising the step of adjusting the selective conversational simulation in response to user input and based on the 3D model.

6. The computer implementation method according to claim 5, further comprising the step of analyzing the performance of the adjusted selective conversational simulation.

7. The computer implementation method according to claim 6, wherein the step of analyzing the performance of the adjusted selective conversational simulation includes the step of analyzing using at least one of computational fluid modeling, solid modeling using finite element analysis, computational fluid dynamic modeling, artificial intelligence, machine learning, and low-dimensionality modeling.

8. The computer-aided method according to claim 7, wherein the step of analyzing performance is based on a trained database that includes functional quantification performed postoperatively using one of invasive imaging and non-invasive imaging.

9. The computer implementation method according to claim 7, wherein the step of analyzing using artificial intelligence includes the step of training the artificial intelligence on a compiled serial image database of tissue healing.

10. The computer implementation method according to claim 1, wherein the step of generating a finite number of time-deformed scenes is performed using artificial intelligence and machine learning modeling.

11. The computer-aided method according to claim 1, wherein the medical imaging includes computed tomography (CT) imaging, magnetic resonance imaging (MRI), ultrasound imaging, positron emission tomography (PET) imaging, echocardiography (ECHO) imaging, nuclear medicine imaging, functional imaging, or a combination thereof.

12. Memory containing instructions, When executed, the processor Receiving medical imaging, To generate a 3D model based on the aforementioned medical imaging, In response to a predetermined set of surgical steps, generate a finite number of time-deformed scenes for each predetermined surgical step associated with the 3D model. A processor configured to execute the instruction that causes the operation to be performed, The aforementioned time-deformed scene is two-dimensional (2D) or greater. system.

13. The system according to claim 12, wherein the time-deformed scene is 4D and / or 5D.

14. The processor, Displaying selective conversational simulations based on the aforementioned finite time-deformed scenarios. The system according to claim 12, further comprising an instruction to cause the system to perform the following.

15. The system according to claim 14, further comprising instructions for causing the processor to display via augmented reality (AR), virtual reality (VR), and / or mixed reality.

16. The aforementioned processor, Adjusting the selective conversational simulation in response to user input and based on the 3D model. The system according to claim 14, further comprising an instruction to cause the system to perform the following.

17. The aforementioned processor, To analyze the performance of the adjusted selective conversational simulation. The system according to claim 16, further comprising an instruction to cause the system to perform the following.

18. The system according to claim 17, wherein the performance of the tuned selective conversational simulation is analyzed using at least one of computational fluid modeling, solid modeling using finite element analysis, computational fluid dynamic modeling, artificial intelligence, machine learning, and dimensionality reduction modeling.

19. The system according to claim 18, wherein the performance of the adjusted selective conversational simulation is analyzed based on a trained database that includes functional quantification performed postoperatively using one of invasive and non-invasive imaging.

20. The system according to claim 18, wherein the analysis using artificial intelligence includes training the artificial intelligence on a compiled serial image database of tissue healing.

21. The system according to claim 12, wherein the finite number of time-deformed scenes are generated using artificial intelligence and machine learning modeling.

22. The system according to claim 12, wherein the medical imaging includes computed tomography (CT) imaging, magnetic resonance imaging (MRI), ultrasound imaging, positron emission tomography (PET) imaging, echocardiography (ECHO) imaging, nuclear medicine imaging, functional imaging, or a combination thereof.

23. A non-temporary machine-readable storage medium having machine-readable instructions for causing a processor to execute a method, wherein the method is Receiving medical imaging, To generate a 3D model based on the aforementioned medical imaging, This includes generating a finite number of time-deformed scenes for each predetermined surgical step associated with the 3D model in response to a predetermined set of surgical steps, The aforementioned time-deformed scene is two-dimensional (2D) or greater. Non-temporary machine-readable storage medium.

24. The non-temporary machine-readable storage medium according to claim 23, wherein the time-deformed scene is 4D and / or 5D.

25. The non-temporary machine-readable storage medium according to claim 23, further comprising instructions for causing the processor to perform the method, which includes displaying a selective conversational simulation based on a finite number of time-deformed scenes.

26. The non-temporary machine-readable storage medium according to claim 25, wherein the display includes a conversational augmented reality (AR), virtual reality (VR), and / or mixed reality display.

27. The aforementioned processor, Adjusting the selective conversational simulation in response to user input and based on the 3D model. A non-temporary machine-readable storage medium according to claim 25, further comprising instructions for performing the method including the above.

28. The aforementioned processor, To analyze the performance of the adjusted selective conversational simulation. A non-temporary machine-readable storage medium according to claim 27, further comprising instructions for performing the method including the above.

29. The non-temporary machine-readable storage medium according to claim 28, wherein the analysis of the performance of the adjusted selective conversational simulation includes analysis using at least one of computational fluid modeling, solid modeling using finite element analysis, computational fluid dynamic modeling, artificial intelligence, machine learning, and low-dimensionality modeling.

30. The non-temporary machine-readable storage medium according to claim 29, wherein performance analysis is based on a trained database including functional quantification performed postoperatively using one of invasive and non-invasive imaging.

31. The non-temporary machine-readable storage medium according to claim 29, wherein the analysis using artificial intelligence includes training the artificial intelligence on a compiled serial image database of tissue healing.

32. The non-temporary machine-readable storage medium according to claim 23, wherein the medical imaging includes computed tomography (CT) imaging, magnetic resonance imaging (MRI), ultrasound imaging, positron emission tomography (PET) imaging, echocardiography (ECHO) imaging, nuclear medicine imaging, functional imaging, or a combination thereof.

Images