Hybrid simulation model for simulating medical procedures

Abstract

The invention relates to a hybrid simulation model for simulating a medical procedure. A method and apparatus for simulating a medical procedure to be performed on a patient. Patient-specific data specific to a patient on which a medical procedure is to be performed is received. The generic model is modified using the patient-specific data to generate a hybrid simulation model customized for the patient. Simulation of the medical procedure is performed using the hybrid simulation model and dynamic simulation information corresponding to at least one of the medical procedure or the patient. A simulation is customized for both a patient and a medical procedure.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 2019800542449 (PCT / US2019 / 038911), titled "Hybrid Simulation Model for Simulating Medical Procedures", filed on June 25, 2019.

[0002] Cross - Reference to Related Applications

[0003] This application claims the benefit of U.S. Provisional Application No. 62 / 689,584, filed on June 25, 2018, which is incorporated herein by reference in its entirety. Technical Field

[0004] The present disclosure relates to medical procedures and methods for simulating minimally invasive medical procedures. More specifically, the present disclosure relates to systems and methods for providing a hybrid simulation model that has been customized for a specific patient and a specific medical procedure to be performed on the patient. Background Art

[0005] Minimally invasive medical techniques are designed to reduce the amount of extracorporeal tissue damaged during a diagnostic or surgical procedure, thereby reducing the patient's recovery time, discomfort, and harmful side effects. Such minimally invasive techniques can be performed through natural orifices in the patient's anatomy or through one or more surgical incisions. Through these natural orifices or incisions, a clinician can insert medical tools to reach the target tissue location. Minimally invasive medical tools include instruments such as therapeutic instruments, diagnostic instruments, and surgical instruments. Minimally invasive medical tools can also include imaging instruments, such as endoscopic instruments, which provide a field of view within the patient's anatomy to the user.

[0006] Some minimally invasive medical tools can be telesurgical, or remotely operated, or computer - assisted. A clinician near a telesurgical system may need to receive guidance in the form of instructions, warnings, confirmations, etc., before, during, or after performing a medical procedure using the telesurgical system. There is a need for systems and methods for improving the process of generating a simulation model for simulating a medical procedure. Summary of the Invention

[0007] The appended claims summarize embodiments of the present invention.

[0008] In an embodiment, a method for simulating a medical procedure to be performed on a patient is provided. Patient - specific data specific to the patient on whom the medical procedure is to be performed is received. A general model is modified using the patient - specific data to generate a patient - customized hybrid simulation model. The simulation of the medical procedure is performed using the hybrid simulation model and dynamic simulation information corresponding to at least one of the medical procedure or the patient. The simulation is customized for both the patient and the medical procedure.

[0009] In another embodiment, a simulation system includes a computer system configured to receive patient-specific data specific to a patient on whom a medical procedure is to be performed; modify a generic model using the patient-specific data to generate a patient-customized hybrid simulation model. The computer system is configured to perform a simulation of the medical procedure using the hybrid simulation model and dynamic simulation information corresponding to at least one of the medical procedure or the patient, wherein the simulation is customized for both the patient and the medical procedure.

[0010] In yet another embodiment, a method for simulating a medical procedure to be performed on a patient is provided. Anthropometric data and patient diagnostic data of the patient are received. A generic model of a representative patient is selected from a database system. The generic model is modified based on the anthropometric data and the patient diagnostic data to generate a patient-customized hybrid simulation model. A simulation of the medical procedure is performed using the hybrid simulation model and dynamic simulation information corresponding to at least one of the medical procedure or the patient. The simulation is customized for both the patient and the medical procedure. The simulation takes into account the effect of the simulated interaction between the patient's anatomy and the instruments used in the medical procedure. The simulation dynamically adapts to each step of the medical procedure such that a portion of the patient's anatomy interacting with the medical procedure in each step is rendered at a higher level of fidelity.

[0011] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, additional aspects, features, and advantages of the present disclosure will be apparent to those skilled in the art from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be emphasized that, in accordance with standard practice in the industry, the various features are not drawn to scale. In fact, for clarity of discussion, the dimensions of the various features may be arbitrarily increased or decreased. Additionally, the present disclosure may repeat reference numerals and / or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself indicate a relationship between the various embodiments and / or configurations discussed.

[0013] Figure 1A is a schematic diagram of a medical system according to an embodiment.

[0014] Figure 1B is a perspective view of an assembly according to an embodiment.

[0015] Figure 1C is a perspective view of a surgeon's console for a medical system according to an embodiment.

[0016] Figure 2is a block diagram of a simulation system according to an embodiment, the simulation system being for simulating a medical procedure to be performed on a patient.

[0017] Figure 3 is a block diagram illustrating different types of general data that can be found in a database system according to an embodiment.

[0018] Figure 4 is a block diagram illustrating different types of patient-specific data that can be found in a database system according to an embodiment.

[0019] Figure 5 is a block diagram illustrating different types of dynamic simulation information according to an embodiment.

[0020] Figure 6 is a flowchart illustration of a method for simulating a medical procedure to be performed on a patient according to an embodiment.

[0021] Figure 7 is a flowchart illustration of a method for generating a hybrid simulation model for simulating a medical procedure to be performed on a patient according to an embodiment.

[0022] Figure 8 is a flowchart illustration of a method for running a simulation of a medical procedure using a hybrid simulation model according to an embodiment. Detailed Description

[0023] To facilitate understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the accompanying drawings, and specific language will be used to describe these embodiments. However, it will be understood that no limitation of the scope of the present disclosure is intended. In the following detailed description of various aspects of the invention, numerous specific details are set forth to provide a thorough understanding of the disclosed embodiments. However, as will be understood by those skilled in the art, the embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the invention.

[0024] As would be typically contemplated by those skilled in the art in the field to which the present disclosure pertains, any changes and further modifications to the described devices, instruments, methods of the principles of the present disclosure, and any further applications of the principles of the present disclosure can be fully considered. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one embodiment can be combined with the features, components, and / or steps described with respect to other embodiments of the present disclosure. Additionally, the dimensions provided herein are for specific examples, and it is anticipated that different dimensions, sizes, and / or ratios can be utilized to implement the concepts of the present disclosure. To avoid unnecessary descriptive repetition, one or more components or actions described according to one illustrative embodiment can be used or omitted according to the applicability of other illustrative embodiments. For the sake of brevity, many iterations of these combinations will not be described separately. For simplicity, in some cases, the same reference numerals are used in all the figures to refer to the same or similar parts.

[0025] The following embodiments will describe various instruments and parts of instruments based on their states in three-dimensional space. As used herein, the term "position" refers to the location of an object or a part of an object in three-dimensional space (e.g., three translational degrees of freedom along the Cartesian X, Y, Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or a part of an object (three rotational degrees of freedom - e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the positioning of an object or a part of an object in at least one translational degree of freedom and the orientation of the object or a part of the object in at least one rotational degree of freedom (up to a total of six degrees of freedom).

[0026] Now referring to the drawings, Drawing 1 includes Figure 1A , Figure 1B and Figure 1C , which together provide a schematic overview of a medical system 10 that can be used in a medical procedure such as including a diagnostic, therapeutic, or surgical procedure. The medical system 10 is located in a surgical environment 11. In one or more embodiments, the medical system 10 can be a teleoperated medical system under the remote operation control of a surgeon. In alternative embodiments, the medical system 10 can be under the partial control of a computer programmed to perform a medical procedure or subroutine. In other alternative embodiments, the medical system 10 can be a fully automated medical system under the full control of a computer programmed to utilize the medical system 10 to perform a medical procedure or subroutine. An example of a medical system 10 that can be used to implement the systems and techniques described in the present disclosure is the da Surgical System manufactured by Intuitive Surgical, Inc of Sunnyvale, California.

[0027] As Figure 1A shown, the medical system 10 generally includes an assembly 12 that can be mounted on or positioned near an operating table O where a patient P is located. The assembly 12 can be referred to as a patient-side cart, an operating cart, or a surgical robot. In one or more embodiments, the assembly 12 can be a teleoperated assembly. A teleoperated assembly can be referred to as, for example, a teleoperated arm cart. A medical device system 14 and an endoscopic imaging system 15 are operatively coupled to the assembly 12. An operator input system 16 allows a surgeon or other type of clinician S to view images of or representations of a surgical site and to control the operation of the medical device system 14 and / or the endoscopic imaging system 15.

[0028] The medical device system 14 can include one or more medical devices. In embodiments where the medical device system 14 includes multiple medical devices, the multiple medical devices can include multiple identical medical devices and / or multiple different medical devices. Similarly, the endoscopic imaging system 15 can include one or more endoscopes. In the case of multiple endoscopes, the multiple endoscopes can include multiple identical endoscopes and / or multiple different endoscopes.

[0029] The operator input system 16 can include a surgeon's console and can be in the same room as the operating table O. In some embodiments, the surgeon S and the operator input system 16 can be in a different room from the patient P or in a completely different building. The operator input system 16 generally includes one or more control devices for controlling the medical device system 14. The (one or more) control devices can include any number and combination of a variety of input devices, such as a handle, a joystick, a trackball, a data glove, a trigger gun, a foot pedal, a manual controller, a voice recognition device, a touch screen, a human motion or presence sensor, and other types of input devices.

[0030] In some embodiments, the (one or more) control devices will be provided with the same degrees of freedom as the (one or more) medical devices of the medical device system 14 to provide the surgeon with a sense of telepresence that the (one or more) control devices and the instruments are integral, such that the surgeon has a strong sense of directly controlling the instruments as if at the surgical site. In other embodiments, the (one or more) control devices can have more or fewer degrees of freedom than the associated medical devices and still provide the surgeon with telepresence. In some embodiments, the (one or more) control devices are manual input devices that move in six degrees of freedom and can also include an actuatable handle for actuating the instruments (e.g., for closing grasping jaws of an end effector, applying an electrical potential to an electrode, delivering a drug treatment, and actuating other types of instruments).

[0031] When surgeon S observes the surgical site through the operator input system 16, the assembly 12 supports and manipulates the medical device system 14. Images of the surgical site can be obtained through the endoscopic imaging system 15, which can be manipulated by the assembly 12. The assembly 12 can include the endoscopic imaging system 15 and can also include multiple medical device systems 14. The number of single-use medical device systems 14 will typically depend on factors such as the diagnostic or surgical procedure to be performed and the space limitations within the operating room. The assembly 12 can include a kinematic structure of one or more non-servo-controlled links (e.g., one or more links that can be manually positioned and locked in place, often referred to as mounting structures) and a manipulator. When the manipulator takes the form of a teleoperator, the assembly 12 is a teleoperation assembly. The assembly 12 includes multiple motors that drive inputs on the medical device system 14. In an embodiment, these motors move in response to commands from a control system (e.g., control system 20). The motors include a drive system that can advance the medical device into natural or surgically created anatomical orifices when coupled to the medical device system 14. Other electric drive systems can move the distal end of the medical device in multiple degrees of freedom, which can include three linear degrees of motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three rotational degrees of motion (e.g., rotation about the X, Y, Z Cartesian axes). Additionally, the motors can be used to actuate an articulable end effector of the medical device to grasp tissue in the jaws of a biopsy device or the like. The medical device of the medical device system 14 can include an end effector having a single working member, such as a scalpel, a blunt knife, an optical fiber, or an electrode. Other end effectors can include, for example, forceps, grippers, scissors, or clip appliers.

[0032] The medical system 10 further includes a control system 20. The control system 20 includes: at least one memory 24 and at least one processor 22 for implementing control between the medical device system 14, the operator input system 16, and other auxiliary systems 26; and other auxiliary systems 26, which include, for example, an imaging system, an audio system, a fluid delivery system, a display system, a lighting system, a steering control system, a flushing system, and / or a suction system. Clinician C can circulate within the surgical environment 11 and can access, for example, the assembly 12 during a setup procedure or view the display of the auxiliary system 26 from the patient's bedside.

[0033] Although in Figure 1A the control system 20 is depicted as being external to the assembly 12, in some embodiments, the control system 20 can be fully contained within the assembly 12. The control system 20 also includes programmed instructions (e.g., stored on a non-transitory computer-readable medium) to implement some or all of the methods described in accordance with the aspects disclosed herein. Although inFigure 1A In the simplified schematic diagram, the control system 20 is shown as a single box, but the control system 20 may include two or more data processing circuits, where a portion of the processing is optionally performed on or near the assembly 12, another portion of the processing is performed on the operator input system 16, and so on.

[0034] Any of a variety of centralized or distributed data processing architectures may be employed. Similarly, the programmed instructions may be implemented as multiple separate programs or subroutines, or they may be integrated into many other aspects of the system described herein, including the teleoperation system. In one embodiment, the control system 20 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry.

[0035] The control system 20 communicates with a database 27, which may store one or more clinician profiles, lists of patients and patient profiles, lists of procedures to be performed on patients, lists of clinicians who are scheduled to perform the procedures, other information, or combinations thereof. Clinician profiles may include information about the clinician, including how long the clinician has worked in the medical field, the level of education the clinician has obtained, the clinician's level of experience with the medical system 10 (or similar systems), or any combination thereof.

[0036] The database 27 may be stored in the memory 24 and may be dynamically updated. Additionally or alternatively, the database 27 may be stored on a device such as a server or a portable storage device, which may be accessed by the control system 20 via an internal network (e.g., the secure network of a medical facility or a teleoperation system provider) or an external network (e.g., the Internet). The database 27 may be distributed across two or more locations. For example, the database 27 may exist on multiple devices that may include different entities and / or cloud servers. Additionally or alternatively, the database 27 may be stored on a portable user-assigned device such as a computer, a mobile device, a smartphone, a laptop computer, an electronic badge, a tablet, a pager, and other similar user devices.

[0037] In some embodiments, the control system 20 may include one or more servo controllers that receive force and / or torque feedback from the medical device system 14. In response to the feedback, the servo controllers transmit signals to the operator input system 16. The (one or more) servo controllers may also transmit signals to direct the assembly 12 to move the (one or more) medical device systems 14 and / or the endoscopic imaging system 15, which extend through an opening in the body into an internal surgical site within the patient's body. Any suitable conventional or dedicated servo controller may be used. The servo controller may be separate from or integrated with the assembly 12. In some embodiments, the servo controller and the assembly 12 are provided as part of a teleoperated arm cart positioned near the patient's body.

[0038] The control system 20 may be coupled to the endoscopic imaging system 15 and may include a processor to process the captured images for subsequent display, such as to a surgeon on a console of the surgeon, or on other suitable displays located locally and / or remotely. For example, in the case of using a stereoscopic endoscope, the control system 20 may process the captured images to present a coordinated stereoscopic image of the surgical site to the surgeon. Such coordination may include alignment between the relative images and may include adjusting the stereoscopic working distance of the stereoscopic endoscope.

[0039] In alternative embodiments, the medical system 10 may include more than one assembly 12 and / or more than one operator input system 16. The exact number of assemblies 12 will depend on the surgical procedure and space limitations within the operating room and other factors. The operator input systems 16 may be juxtaposed, or they may be located in separate positions. The multiple operator input systems 16 allow more than one operator to control the one or more assemblies 12 in various combinations.

[0040] The medical system 10 may also be used for training and practicing medical procedures. For example, the medical system 10 may be used in conjunction with the simulation system 200 described below in Figure 2 which simulates one or more medical procedures to be performed on a patient.

[0041] Figure 1Bis a perspective view of an embodiment of the assembly 12, which may be referred to as a patient-side cart, a surgical cart, a teleoperation arm cart, or a surgical robot. The illustrated assembly 12 provides manipulation of three surgical tools 30a, 30b, 30c (e.g., a medical device system 14) and an imaging device 28 (e.g., an endoscopic imaging system 15), such as a stereoscopic endoscope for capturing images of a surgical site. The imaging device may transmit signals to the control system 20 via a cable 56. Manipulation is provided by a teleoperation mechanism having multiple joints. The imaging device 28 and the surgical tools 30a - 30c may be positioned and manipulated through incisions in a patient so as to maintain a kinematic remote center at the incision to minimize the size of the incision. When the distal ends of the surgical tools 30a - 30c are positioned within the field of view of the imaging device 28, the images of the surgical site may include images of the distal ends of the surgical tools 30a - 30c.

[0042] The assembly 12 includes a drivable base 58. The drivable base 58 is connected to a telescoping column 57 that allows adjustment of the height of the arm 54. The arm 54 may include a rotational joint 55 that both rotates and moves up and down. Each arm 54 may be connected to an orientation platform 53. The arms 54 may be marked to facilitate troubleshooting. For example, each arm 54 may be decorated with a different number, letter, symbol, other identifier, or a combination thereof. In Figure 1B the example, the arms 54 are numbered from 1 to 4. The orientation platform 53 may be capable of rotating 360 degrees. The assembly 12 may also include a telescoping horizontal cantilever 52 for moving the orientation platform 53 in a horizontal direction.

[0043] In this example, each arm 54 is connected to a robotic arm 51. The robotic arm 51 may be directly connected to a medical device, such as one of the surgical tools 30a - 30c. The robotic arm 51 may be telescopically operable. In some examples, the arms 54 connected to the orientation platform 53 may not be telescopically operable. Instead, these arms 54 may be positioned as needed before the surgeon S begins operating with the telescopically operable components. Throughout the surgical procedure, the medical device may be removed and replaced with other instruments, such that the association of the instrument with the arm may change during the procedure.

[0044] Endoscopic imaging systems (e.g., endoscopic imaging system 15 and imaging device 28) can be provided in a variety of configurations including rigid or flexible endoscopes. A rigid endoscope includes a rigid tube that houses a relay lens system for transmitting an image from the distal end of the endoscope to the proximal end. A flexible endoscope uses one or more flexible optical fibers to transmit an image. A digital image-based endoscope has a "tip chip" design in which a distal digital sensor (such as one or more charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) devices) stores image data. An endoscopic imaging system can provide a two-dimensional or three-dimensional image to a viewer. A two-dimensional image can provide limited depth perception. A three-dimensional stereoscopic endoscopic image can provide a viewer with a more accurate depth perception. A stereoscopic endoscopic instrument uses a stereoscopic camera to capture a stereoscopic image of a patient's anatomy. The endoscopic instrument can be a fully sterilizable assembly in which the endoscope cable, handle, and shaft are all firmly coupled and hermetically sealed.

[0045] Figure 1C is a perspective view of an embodiment of an operator input system 16, which can be referred to as a surgeon's console. The operator input system 16 includes a left-eye display 32 and a right-eye display 34 for presenting a coordinated stereogram of a surgical environment capable of depth perception to a surgeon S. The operator input system 16 also includes one or more input control devices 36, which in turn cause the assembly 12 to manipulate one or more instruments of the endoscopic imaging system 15 and / or the medical device system 14. The input control devices 36 can provide the same degrees of freedom as the instruments associated with them to provide a telepresence for the surgeon S, or the input control devices 36 are integrated with the perception of the instruments, giving the surgeon a strong sense of directly controlling the instruments. For this purpose, position sensors, force sensors, and tactile feedback sensors (not shown) can be employed to transmit the position, force, and touch from a medical device (e.g., surgical tools 30a - 30c or imaging device 28) back to the surgeon's hand through the input control devices 36. The input control device 37 is a foot pedal that receives input from the user's foot. Aspects of the operator input system 16, the assembly 12, and the auxiliary system 26 can be adjustable and customizable to meet the physical needs, skill level, or preferences of the surgeon S.

[0046] Figure 2 is a block diagram of a simulation system 200 for simulating a medical procedure 202 performed on a patient 203 (e.g., patient P) in a medical training environment 205. In one or more embodiments, the medical procedure 202 will be performed in a surgical environment (e.g., Figure 1A the surgical environment 11 in

[0047] The simulation system 200 includes a computer system 204. The computer system 204 may include a single computer or multiple computers communicating with each other. The computer system 204 may include at least one processor 206. The processor 206 may be used to run executable code stored in a memory to simulate a medical procedure 202. In one embodiment, the simulation system 200 is separate from the control system 20 described in FIG. 1. In other embodiments, the simulation system 200 may be implemented as part of the control system 20. In other embodiments, the processor 206 of the simulation system 200 may be implemented using the processor 22 of the control system 20 in FIG. 1.

[0048] In one or more embodiments, the simulation system 200 further includes a display system 208 coupled to the processor 206. In other embodiments, the display system 208 may be considered separate from the simulation system 200. The display system 208 may include one or more display devices communicatively coupled to the processor 206. In one or more embodiments, the display system 208 may be coupled to the surgeon's console of the operator input system 16.

[0049] The simulation system 200 creates a hybrid simulation model 210 that integrates general medical data 212 (e.g., medical data of a representative patient) with patient-specific data 214 of a patient 203, and then dynamically modifies the hybrid simulation model 210 based on information or parameters that change during the simulation (i.e., virtual reality simulation) of the medical procedure 202. This process of generating the hybrid simulation model 210 reduces the total time, effort, and processing resources required to generate the simulation model.

[0050] The simulation system 200 may retrieve general medical data 212 from a database system 218. The database system 218 may include one or more databases. The general medical data 212 includes information that is not specific to a particular patient 203. For example, the general medical data 212 may include general information about the medical procedure 202 and / or information about a representative patient or group of representative patients. In one or more embodiments, the general medical data 212 includes a general model 219 of a representative patient or group of representative patients. Examples of different types of general medical data 212 that may be retrieved from the database system 218 are described below. In one or more embodiments, the simulation system 200 receives patient-specific data 214 that includes information specific to a particular patient 203. Examples of different types of patient-specific data 214 are described below. Figure 3 In the following Figure 4 Examples of different types of patient-specific data 214 are described.

[0051] The simulation system 200 integrates the general medical data 212 and the patient-specific data 214 to generate a hybrid simulation model 210 customized for the patient 203. In one or more embodiments, the simulation system 200 uses the patient-specific data 214 to modify, enhance, or otherwise refine the general medical data 212 for the region of interest in the patient anatomy 222 of the patient 203. As an example, the patient-specific data 214 can be used to scale and / or morph the general model 219. Additionally, the patient-specific data 214 can be used to modify the portion of the general model 219 that represents, is adjacent to, and / or is involved with the set of target regions 224 that the medical procedure 202 is directed to.

[0052] The set of target regions 224 can include, for example, one or more diseased anatomical structures of the patient anatomy 222 of the patient 203. The anatomical structures can include, for example, one or more or a combination of organs, organ systems, muscles, nerves, arteries, veins, bones, bone structures, ligaments, and certain other types of anatomical structures or tissues. By modifying only the portion of the general model 219 that represents, is adjacent to, and / or is involved with the set of target regions 224, the simulation system 200 reduces the total time, effort, and processing resources required to generate the hybrid simulation model 210 customized for the patient 203.

[0053] In one or more embodiments, due to the integration of the general medical data 212 and the patient-specific data 214, the hybrid simulation model 210 represents the patient anatomy 222 of the patient 203 with varying levels of fidelity. For example, each anatomical structure of the patient anatomy 222 is represented in the hybrid simulation model 210 with a specific level of fidelity. The level of fidelity indicates the accuracy with which the modeled anatomical structure in the hybrid simulation model 210 matches the actual geometry and properties of the corresponding actual anatomical structure of the patient 203.

[0054] In some embodiments, the hybrid simulation model 210 represents the patient anatomy 222 of the patient 203 with a gradient 226 of levels of fidelity. For example, the level of fidelity of the modeled anatomical structure can decrease as the distance from the operative path of the medical procedure 202 increases. The operative path can be, for example, the surgical path, the procedural path, or some other path (external path and / or internal path) with respect to the patient anatomy 222 for performing the medical procedure 202. When the medical system 10 described in FIG. 1 is to be used to perform the medical procedure 202, the surgical path can include, for example, a path with respect to the patient anatomy 222 of the patient 203, and during the medical procedure 202, each arm 54 and each surgical instrument and tool of the medical device system 14 that makes up the assembly 12 described in Figure 1A and Figure 1B will follow this path.

[0055] For example, the representation of the set of target regions 224 in the hybrid simulation model 210 has a higher fidelity level than the (one or more) fidelity levels of the portion of the hybrid simulation model 210 that represents anatomical structures that will not be affected or will not interact with them during the medical procedure 202. If the medical procedure 202 is a cardiac surgical procedure, the set of target regions 224 may include the patient 203's heart and the regions surrounding the heart. However, a cardiac surgical procedure may never involve the patient 203's limbs. Thus, in the hybrid simulation model 210, the limbs may be represented at a lower fidelity level than the heart and even other anatomical structures within the patient 203's thoracic cavity. In some cases, the anatomical structures in the patient 203's abdomen will be represented at a higher fidelity level than the limbs but at a lower fidelity level compared to the heart.

[0056] The resulting hybrid simulation model 210 generated by the simulation system 200 can be used to run highly customized and high-quality simulations of the medical procedure 202 performed on the patient 203. For example, the simulation system 200 can perform a simulation 228 of the medical procedure 202 using the hybrid simulation model 210 and dynamic simulation information 230 corresponding to at least one of the medical procedure 202 or the patient 203. The simulation 228 can be a dynamic simulation. The dynamic simulation information 230 can include information about how the patient 203 and / or the patient's anatomy 222 will be affected by the medical procedure 202, as well as other information about the medical procedure 202 that does not remain static during the simulation of the medical procedure 202. Examples of the dynamic simulation information 230 are described below in Figure 5 .

[0057] The simulation 228 can be rendered on the display system 208 such that a surgeon, a member of the surgical team, or other medical personnel can use the simulation 228 to rehearse the medical procedure 202. Because the simulation 228 of the medical procedure 202 is performed using the hybrid simulation model 210 and the dynamic simulation information 230, the simulation 228 provides a higher overall fidelity level in representing how the medical procedure 202 will be performed for a particular patient 203 compared to simulations run using the generic model 219 for the patient 203.

[0058] The simulation 228 can be performed, for example, Figure 1A in the environment 11 in to rehearse or otherwise train for the medical procedure 202. In one or more embodiments, the simulation 228 can be performed using, for example, the medical system 10 in FIG. 1. For example, the simulation 228 can be performed using the operator input system 16, the assembly 12, or both. The operator input system 16 can include a single surgeon console or a dual surgeon console.

[0059] When running simulation 228, simulation 228 uses dynamic simulation information 230 to account for the effects of the simulated interaction between the anatomy of patient anatomy 222 and the instrument used to perform medical procedure 202. Accordingly, simulation system 200 uses dynamic simulation information 230 to dynamically scale or deform hybrid simulation model 210 such that the experience provided by simulation 228 is more similar to actual medical procedure 202. This type of simulation 228 allows a surgeon or other medical personnel to rehearse medical procedure 202 with a higher level of realism. However, because hybrid simulation model 210 is generated starting from general medical data 212 customized using patient-specific data 214, the total time, effort, and processing resources spent in generating hybrid simulation model 210 are reduced compared to generating a model that captures the entire patient anatomy 222 of patient 203 at the highest fidelity level.

[0060] Figure 3 FIG. is a block diagram illustrating different types of general medical data 212 that can be found in database system 218. Database system 218 can include, for example, a procedure library 300, a general model library 302, and a disease state library 304. Each of these three libraries can be implemented using one or more databases or other types of data repositories.

[0061] Procedure library 300 can store information related to one or more medical procedures, including the medical procedure 202 to be performed on patient 203 as described in Figure 2 . In one example embodiment, procedure library 300 stores procedure information 306 for medical procedure 202. Procedure information 306 identifies the various phases and / or steps of medical procedure 202. Procedure information 306 can identify, for example but not limited to: the anatomy known to be adjacent to one or more diseased anatomies to be surgically treated by medical procedure 202, the surgical path that a medical instrument should follow during the performance of medical procedure 202, the key anatomical landmarks at which intraoperative decisions are to be made, the length or time range expected to be spent on various anatomies along the surgical path during the medical procedure, and other types of information related to medical procedure 202.

[0062] The general model library 302 stores one or more general models representing the anatomy of representative patients. In one or more embodiments, the general model library 302 stores a first general model for a first group of representative patients and a second general model for a second group of representative patients. The first group of representative patients can be healthy females, while the second group of representative patients can be healthy males. In other embodiments, the general model library 302 stores a first plurality of general models for a first group of representative patients and a second plurality of general models for a second group of representative patients. The first plurality of general models can include general models scaled to average female anthropometry, while the second plurality of general models can include general models scaled to average male anthropometry. For example, the first plurality of general models can include one general model for a certain height and weight category of a healthy female, another general model for another height and weight category of a healthy female, and yet another general model for yet another height and weight category of a healthy female.

[0063] Thus, the general model library 302 stores general, simulation-ready models that can be retrieved by the simulation system for generating the hybrid simulation model 210 in the figures. Any one of the general models stored in the general model library 302 can be retrieved by the simulation system 200 and used as the general model 219, and subsequently modified using patient-specific data 214 as Figure 2 described.

[0064] In one or more embodiments, the disease state library 304 stores information about the properties of any number of anatomical structures when afflicted with various types of disease states. For example, the disease state library 304 can store information about the material properties of a heart afflicted with one disease and information about the material properties of a heart afflicted with another disease.

[0065] Figure 4 is a block diagram illustrating different types of patient-specific data 214. Patient-specific data 214 can include, for example, anthropometric data 400 and patient diagnostic data 402 specific to the Figure 2 patient 203 described. The anthropometric data 400 for the patient 203 includes data or measurements for evaluating the size, shape, and composition of the patient 203. The anthropometric data 400 can include, but is not limited to, for example: height measurements 404, weight measurements 406, anatomical landmark measurements 408, body mass index (BMI) measurements 410, waist-to-hip ratio 412, skinfold test data 414, bioelectrical impedance data 416, some other types of anthropometric measurements, or combinations thereof. The anatomical landmark measurements 408 can include, for example, measurements of the distances between key bones or other anatomical landmarks of the patient's anatomy 222 of the patient 203. Figure 2The simulation system 200 therein can use anthropometric data 400 to generate a scaling factor, which can be applied to the general model of a representative patient retrieved from the general model library 302 in Figure 3 therein to customize the general model for patient 203.

[0066] Patient diagnostic data 402 includes information about the specific anatomy of patient 203. In one or more embodiments, the patient diagnostic information includes information about the Figure 2 geometry of the target region set 224 in Figure 2 the medical procedure 202 is directed to, and information about the geometry of one or more anatomies adjacent to or functionally involved with each of the target region sets 224. For example, patient diagnostic data 402 can include preoperative imaging data 418, physical examination data 420, and / or other types of diagnostic information that provide information about the target region set 224 (and, in some cases, anatomies adjacent to or functionally involved with each of the target region sets 224). The preoperative imaging data 418 can include, but is not limited to, for example, ultrasound images, x-ray images, medical resonance imaging (MRI) images, computed tomography (CT) images, or other types of imaging data. The physical examination data 420 can include information about the patient's anatomy 222 that can be collected from, for example, observation, palpation, percussion, and auscultation of patient 203. Figure 2 The simulation system 200 therein can use patient diagnostic data 402 to modify the geometry and / or other types of attributes of the anatomies in the general model of a representative patient that can be retrieved from the general model library 302 in Figure 3 therein to customize the general model for patient 203.

[0067] Figure 5 is a block diagram illustrating different types of dynamic simulation information 230. The dynamic simulation information 230 includes information about one or more parameters expected to change during the simulation 220 of the medical procedure 202 run using the hybrid simulation model 210. For example, the dynamic simulation information 230 can include physiological parameter data 500, instrument positioning information 502, field of view information 504, patient pose information 506, and other types of information. In one or more embodiments, Figure 2 the simulation system 200 of

[0068] The physiological parameter data 500 includes data for a plurality of physiological parameters that may change or are expected to change during a medical procedure 202. In particular, the physiological parameter data 500 includes information on how the anatomical structures within the patient 203 should be viewed and how they respond to simulated interactions between the anatomical structures and the instruments and / or agents used to perform the medical procedure 202. For example, during the simulation 228 of the medical procedure 202, the effects of simulations of aspiration, irrigation, impedance cardiography, and other steps or procedures performed as part of the medical procedure 202 can cause simulated changes in a plurality of physiological parameters. The plurality of physiological parameters can include, for example but not limited to, arterial and venous pressures, respiratory rate, insufflation pressure, tissue response to energy application, tissue response to physical contact with one or more instruments, and / or other types of physiological parameters. Figure 2 The simulation system 200 can use the physiological parameter data 500 to ensure that the simulation 228 of the medical procedure 202 takes into account the simulated changes in the physiological parameters. For example, the simulation system 200 can use the physiological parameter data 500 to dynamically adjust the rendering of the hybrid simulation model 210 on the display system 208 during the simulation 228 in a manner that takes into account the simulated changes in the physiological parameters. More specifically, the simulation system 200 can adjust the visualization of the hybrid simulation model 210 to account for these simulated changes.

[0069] The instrument positioning information 502 includes information on how the positioning of an instrument (e.g., the instruments of the medical device system 14 in FIG. 1, retractors, endoscopes, scalpels, suction tubes, irrigation tubes, laser devices, grasping tools, etc.) can change or is expected to change during the course of the medical procedure 202. As an example, the instrument positioning information 502 can include information on how the positioning of an endoscope (e.g., the endoscope imaging system 15 in FIG. 1) is expected to move or change during the medical procedure 202. The simulation system 200 can use the instrument positioning information 502 to dynamically adjust the rendering of the hybrid simulation model 210 on the display system 208 during the simulation 228 in a manner that takes into account these changes in the positioning of the endoscope.

[0070] Similarly, the simulation system 200 can use the field of view information 504 to dynamically adjust the rendering of the hybrid simulation model 210 on the display system 208 during the simulation 228 in a manner that takes into account changes in one or more fields of view or perspectives. The field of view information 504 includes information on one or more different fields of view. For example, the field of view information 504 can include field of view information based on the instrument, vision-based field of view information, some other type of field of view or perspective information, or a combination thereof.

[0071] The instrument-based field-of-view information can include, for example, information about what anatomical structures a surgeon or other medical personnel can actually see in the field of view of one or more imaging instruments (such as endoscopes, cameras, etc.). For example, the field-of-view information 504 can include information about the positioning of the imaging instrument and the calculation of the field of view of the imaging instrument during movement of the imaging instrument during the simulation of the medical procedure 202. The vision-based field-of-view information includes information from the perspective of the surgeon or other clinician. For example, the field-of-view information can be the perspective that a surgeon or other clinician will be able to view during the medical procedure 202.

[0072] Patient pose information 506 includes information about how the positioning and / or orientation of the patient 203 can change during the medical procedure 202. In some embodiments, the positioning and / or orientation of the patient 203 is determined by the positioning and / or orientation of the bed, operating table, or other type of structure on which the patient 203 is positioned during the medical procedure 202. The patient pose information 506 can include information about how gravity affects the patient 203 based on the positioning and / or orientation of the patient 203. For example, the patient pose information 506 can include information about how the positioning and / or orientation of the patient 203 during the medical procedure 202 may cause one or more anatomical structures to deform. The simulation system 200 can use the patient pose information 506 to dynamically adjust the rendering of the hybrid simulation model 210 on the display system 208 during the simulation 228 in a manner that takes into account these changes in the pose of the patient 203.

[0073] Figure 6 is a flowchart illustration of a method 600 for simulating a medical procedure to be performed on a patient. In one or more embodiments, the medical procedure will be performed in a surgical environment (such as the surgical environment 11 of FIG. 1). The method 600 is illustrated in Figure 6 as a set of operations or processes 602 to 608, and will continue to be described with reference to Figures 2 - 5 herein. Not all of the illustrated processes 602 to 608 may be performed in all embodiments of the method 600. Additionally, one or more processes not explicitly illustrated in Figure 6 may be included before, after, between, or as part of the processes 602 to 608. In some embodiments, one or more of the processes 602 to 608 can be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which when run by one or more processors (such as the processor of the control system) can cause the one or more processors to perform one or more processes.

[0074] At process 602, patient-specific data 214 specific to patient 203 on whom medical procedure 202 is to be performed is received. Patient-specific data 214 can include, for example, anthropometric data 400, patient diagnostic data 402, or a combination of both.

[0075] At process 604, the general model 219 of a representative patient is modified using patient-specific data 214 to generate a hybrid simulation model 210 customized for patient 203. The general model 219 can be retrieved from the database system 218. In particular, in one embodiment, the general model 219 can be retrieved from the general model library 302. Depending on the gender of patient 203, the general model 219 can be a model of the anatomy of a healthy female or male. In one or more embodiments, the general model 219 selected for use can represent the anatomy of a healthy female or male having anthropometric results common to patient 203 or within a selected range of patient 203's anthropometric results.

[0076] In one or more embodiments, modifying the general model 219 includes modifying the representation of each of the plurality of anatomical structures in the general model 219 along the operation path along which medical procedure 202 is to be performed based on the received patient-specific data 214, thereby generating the hybrid simulation model 210. In some embodiments, modifying the general model 219 using patient-specific data 214 in process 604 includes scaling and / or deforming the general model 219 based on anthropometric data 400. Additionally, the modification can include adjusting one or more parts of the general model 219 based on patient diagnostic data 402. For example, if preoperative imaging data 418 indicates that patient 203 has an enlarged liver and medical procedure 202 to be performed involves taking a biopsy of the liver, the general model 219 can be modified such that the representation of the liver in the general model 219 is enlarged accordingly. In this way, this part of the general model 219 is customized for patient 203. Thus, the general model 219 can be customized based on the set of target regions 224 that medical procedure 202 is directed to.

[0077] At process 606, a simulation 228 of medical procedure 202 is performed using the hybrid simulation model 210 and dynamic simulation information 230 corresponding to at least one of medical procedure 202 or patient 203. In one or more embodiments, performing the simulation 228 includes running the simulation 228 using the hybrid simulation model 210 and a simulated (or computerized) version of the instrument that will interact with patient 203 during medical procedure 202. In other embodiments, performing the simulation 228 can include a surgeon, a member of the surgical team, or other medical personnel interacting with the hybrid simulation model 210 using the medical system 10 described in FIG. 1, thereby training or rehearsing medical procedure 202.

[0078] During simulation 228 in process 606, dynamic simulation information 230 is used to render a first portion of hybrid simulation model 210 at a fidelity level that is higher than the fidelity level of a second portion of hybrid simulation model 210. The first portion of hybrid simulation model 210 represents patient anatomy 222 of patient 203 that interacts with medical procedure 202, and the second portion of hybrid simulation model 210 represents patient anatomy 222 that does not interact with medical procedure 202 at any step of medical procedure 202. In one or more embodiments, performing simulation 228 includes dynamically simulating medical procedure 202 using hybrid simulation model 210 and dynamic simulation information 230 such that patient anatomy 222 of patient 203 is represented by a gradient 226 of fidelity levels. For example, the fidelity level of anatomical structures in hybrid simulation model 210 can decrease during simulation as the distance from the surgical path of medical procedure 202 increases.

[0079] Performing simulation 228 can include using patient pose information 506 to adjust how one or more anatomical structures of patient anatomy 222 in hybrid simulation model 210 deform in response to an expected change in the pose of patient 203 during simulation 228. Performing simulation 228 can also include adjusting the visualization of hybrid simulation model 210 during simulation 228 based on dynamic simulation information 230 to account for the effects of simulated interactions between anatomical structures and instruments. These instruments can include any number of manually controlled instruments, robotically controlled instruments, or both.

[0080] For example, simulation system 200 can consider how a change in the positioning of one or more instruments during medical procedure 202 can affect a portion of patient anatomy 222 along the surgical path in hybrid simulation model 210. Simulation system 200 can use this instrument positioning information 502 to adjust the rendering of hybrid simulation model 210 on display system 208 to assist a surgeon or clinician who is rehearsing medical procedure 202. Simulation system 200 can also use field of view information 504 to adjust the rendering of hybrid simulation model 210. In some embodiments, simulation system 200 can actually modify or recreate one or more portions of hybrid simulation model 210 based on the positioning of the instruments.

[0081] Accordingly, a simulation 228 using the hybrid simulation model 210 is customized for patient 203 and the medical procedure 202 to be performed on patient 203. The simulation 228 causes the hybrid simulation model 210 to dynamically adapt to each step or stage of the medical procedure 202 such that the portion of the patient anatomy 222 that interacts with the medical procedure 202 at each step is rendered at a higher level of fidelity than other portions of the patient anatomy 222. This type of simulation 228 provides a representation of the medical procedure 202 to be performed on patient 203 with a desired level of accuracy.

[0082] Figure 7 is a flowchart illustration of a method 700 for generating a hybrid simulation model for simulating a medical procedure to be performed on a patient. In one or more embodiments, the medical procedure is to be performed in a surgical environment (such as the surgical environment 11 of FIG. 1). The method 700 is illustrated in Figure 7 as a set of operations or processes 702 to 716 and is described with continued reference to Figures 2 to 5 herein.

[0083] In addition, compared to the method 600 described above Figure 6 herein, Figure 7 the method 700 illustrated in Figure 7 herein can be a more detailed method for generating a hybrid simulation model. Not all of the illustrated processes 702 to 716 may be performed in all embodiments of the method 700. Additionally, one or more processes not explicitly illustrated in Figure 7 herein may be included before, after, between, or as part of the processes 702 to 716. In some embodiments, one or more of the processes 702 to 716 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium that, when run by one or more processors (e.g., a processor of a control system), can cause the one or more processors to perform one or more processes.

[0084] At process 702, anthropometric data 400 and patient diagnostic data 402 are received at the simulation system 200. At process 704, a generic model 219 of a representative patient is selected based on the received anthropometric data 400. For example, a generic model 219 of a representative patient of the same gender and in the same height and weight category as patient 203 may be selected. At process 706, the generic model 219 is scaled and / or deformed based on the anthropometric data 400. The scaling and / or deformation may be a further refinement of the generic model 219 based on the exact height and weight of patient 203, the body mass index of patient 203, and / or other measurements of patient 203.

[0085] At process 708, based on the procedure information 306 regarding the medical procedure 202 to be performed on patient 203, a surgical path and a plurality of anatomical structures along the surgical path are identified. At process 710, based on the received patient diagnostic data 402, the representation of each of the plurality of anatomical structures along the surgical path is modified to complete the hybrid simulation model 210. Thus, the hybrid simulation model 210 is generated by combining general information with relevant patient-specific information.

[0086] At process 712, a simulation 228 of the medical procedure 202 to be performed on patient 203 is run using the hybrid simulation model 210 and the dynamic simulation information 230. For example, a surgeon may use the medical system 10 described in FIG. 1 to simulate performing the medical procedure 202 using the hybrid simulation model 210. During the simulation 228, the simulation system 200 uses the dynamic simulation information 230 to adjust the visualization or rendering of the hybrid simulation model 210 on the display system 208 to account for the effects of the simulated interaction between the anatomy of patient 203 and the instruments and / or agents used in the medical procedure 202. The simulation 228 dynamically adapts to each step of the medical procedure 202 such that the portion of the patient anatomy 222 of patient 203 that interacts with the medical procedure 202 at each step of the medical procedure 202 is rendered at a higher fidelity level.

[0087] In one or more example embodiments, the hybrid simulation model 210 may be “intelligent” as to what is rendered at a higher fidelity during the simulation 228 of the medical procedure 202. For example, during the simulation of the medical procedure 202, a higher fidelity rendering may be applied on the fly as needed to anatomical structures that interact dynamically with instruments, endoscopes, other agents (e.g., suction, irrigation, etc.), and / or gravity. By this method, when a surgeon performs an initial dissection step (e.g., dropping the bladder), it may not be necessary to render the diseased anatomy (e.g., the prostate) at a high fidelity.

[0088] In other embodiments, the hybrid simulation model 210 may allow the simulation 228 to start with a lower fidelity rendering. As the simulation 228 progresses, the rendering of the corresponding portions of the hybrid simulation model 210 will “snap” to an appropriate fidelity level based on the step or phase of the medical procedure 202 being simulated. In some embodiments, during the simulation 228 of the medical procedure, the hybrid simulation model 210 will be able to provide the desired fidelity level for any relevant anatomical structure whenever an instrument, agent, or gravity interacts with the anatomy.

[0089] At process 714, simulation data is calculated based on simulation 228 of medical procedure 202. The simulation data that can be feedback data can be calculated during simulation 228, after simulation 228 is completed, or both. The simulation data can include information about simulation 228 itself and patient-specific intraoperative data. For example, the simulation data can include data on how simulation 228 of medical procedure 202 compares to the expected performance and outcomes of medical procedure 202. For example, the simulation data can include data on the efficiency of simulation 228, a measure of the success of simulation 228, data on the time taken for each step of medical procedure 202, data on the simulated interaction between medical system 10 and patient anatomy 222, data on the impact of changes in patient posture during simulation 228 on the simulation of patient anatomy 222 compared to patient posture information 506, other types of data, or a combination thereof.

[0090] In some embodiments, the simulation data includes information that can be used for preoperative planning. As an example, the simulation data can be used to improve medical device selection, improve medical device calibration, improve patient preparation prior to medical procedure 202, or a combination thereof, thereby increasing the procedural efficiency or operational efficiency of medical procedure 202.

[0091] At process 716, the calculated simulation data is used to modify the hybrid simulation model 210 to form an updated hybrid simulation model 210. The updated hybrid simulation model 210 can allow the second simulation 228 to be run with a higher fidelity than the first simulation 228. In some embodiments, at process 716, the hybrid simulation model 210 is modified based on user input. At process 718, the dynamic simulation information 230 is modified based on the calculated simulation data to form updated dynamic simulation information 230. The updated dynamic simulation information 230 takes into account the results of simulation 228 to provide updated information that can be used to dynamically adjust another simulation of medical procedure 202. In some embodiments, at process 718, the dynamic simulation information 230 is modified based on user input.

[0092] At process 720, a new simulation of medical procedure 202 is performed using the updated hybrid simulation model 210 and the updated dynamic simulation information 230. Performing the new simulation using the updated hybrid simulation model 210 and the updated dynamic simulation information 230 can provide a higher level of customization for both patient 203 and medical procedure 202.

[0093] In some embodiments, processes 714 through 720 may be repeated multiple times to further refine and customize the hybrid simulation model 210. Processes 714 through 720 may be repeated any number of times until a desired version of the hybrid simulation model 210 with a desired level of refinement and overall fidelity and customization has been generated. The simulation data generated during multiple runs of the simulation 228 may be used as patient-specific intraoperative guidance data. For example, the simulation data may be used to generate instructions to control the medical system 10 to actually perform the medical procedure 202 on the patient 203.

[0094] In some embodiments, the simulation data may be fed into a learning algorithm or learning program for providing information on how to improve the medical procedure 202. The simulation data may provide useful information about actions that, if performed during the medical procedure 202, would result in less than ideal outcomes. In one or more embodiments, the simulation data may be used to identify the best set and / or sequence of actions to perform during the medical procedure 202 to allow for the best possible outcome. In some embodiments, the simulation data may be used to help train other surgeons or clinicians.

[0095] Figure 8 is a flowchart illustration of a method for running a simulation of a medical procedure using a hybrid simulation model. In one or more embodiments, the medical procedure will be performed in a surgical environment (such as the surgical environment 11 of FIG. 1). Method 800 is illustrated in Figure 8 as a set of operations or processes 802 through 806, and continues to be described with reference to Figures 2 - 5 for description.

[0096] In addition, Figure 8 the method 800 illustrated in Figure 7 may be a more detailed method for running a simulation of a medical procedure as described in process 712 above in Figure 8 Not all of the illustrated processes 802 through 806 may be performed in all embodiments of method 800. Additionally, one or more processes not explicitly illustrated in

[0097] At process 802, the hybrid simulation model 210 is rendered to provide a visualization of the hybrid simulation model 210 on the display system 208. This visualization of the hybrid simulation model 210 may include one or more different views of the hybrid simulation model 210. For example, the visualization may include one or more external views of the patient anatomy 222, one or more internal views of the patient anatomy 222, or a combination thereof.

[0098] At process 804, an input is detected during the simulation 228 of the medical procedure 202. The input may include any number of inputs. For example, the input may include at least one of the following: user input, system input received from the medical system 10 used to perform the simulation 228, Figure 1A input received from the simulation system 200 described in Figure 2 or some other type of input. User input may be received from any number of input devices or systems. For example, user input may be received via the display system 208, a touch screen, a gesture recognition system, a gaze-based recognition system, a voice-based recognition system, an augmented or virtual reality system, a keyboard, a joystick, one or more touch controls, a mouse, some other type of input device, or any combination thereof. In some cases, user input may be received from the operator input system 16.

[0099] In one or more embodiments, the system input received from the medical system 10 may include any number of inputs or combinations of inputs from the assembly 12. For example, system input may be received from the medical device system 14, the endoscopic imaging system 15, or a combination thereof. In one or more embodiments, the input from the simulation system 200 may include an input selected from the dynamic simulation information 230 (e.g., the value of a parameter included in the dynamic simulation information 230), an input from the hybrid simulation model 210, or both.

[0100] At process 806, the input is used to update the hybrid simulation model 210 and the dynamic simulation information 230. In one or more embodiments, updating the hybrid simulation model 210 may include modifying a portion of the hybrid simulation model 210, recreating a portion of the hybrid simulation model 210, or both. In some embodiments, updating the dynamic simulation information 230 may include updating the value of one or more parameters included in the dynamic simulation information 230. For example, the input detected at process 804 may represent Figure 1B the movement of the simulation of the surgical tool 30b as shown in Figure 5 . This movement may be, for example, a deeper insertion of the surgical tool 30b within the patient anatomy 222. The input may be used, for example, to update the

[0101] At process 808, the visualization of the hybrid simulation model 210 during the simulation 228 of the medical procedure 202 is controlled based on the updated hybrid simulation model 210 and the updated dynamic simulation information 230. In some embodiments, controlling the visualization of the hybrid simulation model 210 means that the visualization does not change. For example, the updated hybrid simulation model 210 and the updated dynamic simulation information 230 may not require a change in the visualization of the hybrid simulation model 210.

[0102] In one or more embodiments, the visualization of the hybrid simulation model 210 is controlled so as to render the updated hybrid simulation model 210 in a manner that provides the highest level of fidelity along the operative path (e.g., surgical path) of the medical procedure 202. As an example, the updated instrument positioning information 502 may cause a change in the portion of the hybrid simulation model 210 that is displayed. For instance, a portion of the hybrid simulation model 210 corresponding to a deeper location within the patient anatomy 222 may be displayed and rendered at a higher level of fidelity compared to the level of fidelity prior to receiving the input at process 804.

[0103] In one or more embodiments, the simulation system 200 allows the rendering of the hybrid simulation model 210 to be adapted based on the updated dynamic simulation information 230 associated with and / or corresponding to the current step and / or one or more next steps of the medical procedure 202. For example, when the simulation 228 is at the current step in the medical procedure 202, the simulation system 200 may use the dynamic simulation information 230 to make predictions regarding how to render or visualize the hybrid simulation model 210 in one or more subsequent steps. Thus, once those steps are reached, the simulation system 200 is able to more quickly and efficiently render the portion of the hybrid simulation model 210 corresponding to one or more next steps.

[0104] At optional process 810, feedback may optionally be provided using an output system. The output system may include, for example, the display system 208, an audio device, a haptic feedback system, a lighting system, or a combination thereof. The audio device may include, but is not limited to, for example, a speaker system, headphones, one or more earbuds, some other type of audio device, or a combination thereof. Feedback may be provided to assist the user (e.g., a surgeon or other clinician) during the simulation 228. For example, audio feedback may be used to indicate when an impassable boundary within the patient anatomy 222 is reached or when the hybrid simulation model 210 and / or the dynamic simulation information 230 indicates that an adverse effect or reaction is being experienced or will be experienced. In some cases, haptic feedback may be used to provide a more realistic experience during the simulation 228. For example, haptic feedback may be used to simulate contact with a hardened mass within the patient anatomy 222.

[0105] In some cases, at process 810, the visual feedback can include text detailed instructions for performing the current step and / or the next step of the medical procedure 202. In some embodiments, the visual feedback includes labels, distance information, positioning information, simulated physiological parameter information, or other types of information. The distance information can include, for example but not limited to, the distance between the tip of the probe and the tissue of the patient 203. The positioning information can include, for example but not limited to, the patient's posture, the positioning of the instrument or tool, or some other type of positioning information. The simulated physiological parameter information can include, for example, simulated body temperature, simulated blood pressure, simulated heart rate, simulated brain waves, or some other type of simulated physiological parameter value.

[0106] Accordingly, embodiments provide a simulation system 200 and process for generating a hybrid simulation model 210 customized for a patient 203 and a medical procedure 202 to be performed on the patient 203 for use in a simulated medical procedure 202. The hybrid simulation model 210 is created in part from patient-specific data 214 and in part from a generic model 219 of a representative patient. In one or more embodiments, the patient-specific data 214 can be used to scale the generic model 219 and deform only the regions of interest in the generic model 219 that will interact with or be affected by the instruments and / or agents and gravity used to perform the medical procedure 202. Additionally, the hybrid simulation model 210 can be dynamically adjusted based on dynamic simulation information 230 obtained from running a simulation of the medical procedure 202. The resulting hybrid simulation model 210 enables a high-fidelity virtual reality simulation of the medical procedure 202. This type of hybrid simulation model 210, which focuses only on the anatomy of the patient's anatomy 222 of the patient 203 that will interact with or be affected by the instruments and / or agents of the medical procedure 202, can have a reduced size compared to a model that captures the entire patient anatomy 222 in high fidelity. Additionally, this type of hybrid simulation model 210 can increase the efficiency with which training simulations for performing the medical procedure 202 can be run and stored.

[0107] One or more elements in embodiments of the present invention may be implemented in software to be executed on a processor of a computer system such as a control processing system. When implemented in software, the elements of embodiments of the present invention are essentially code segments for performing necessary tasks. The program or code segments may be stored in a processor-readable storage medium or device, and the program or code segments may be downloaded via a computer data signal embodied in a carrier wave on a transmission medium or communication link. The processor-readable storage device may include any medium that can store information, including optical media, semiconductor media, and magnetic media. Examples of the processor-readable storage device include electronic circuits; semiconductor devices, semiconductor storage devices, read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM); floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. The code segments may be downloaded via a computer network such as the Internet, an intranet, etc.

[0108] Note that the processes and displays presented may not be inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the described operations. The structure required for various such systems will appear as elements in the claims. Additionally, embodiments of the present invention have not been described with reference to any particular programming language. It should be understood that various programming languages may be used to implement the teachings of the present invention as described herein.

[0109] Although certain exemplary embodiments of the present invention have been described and shown in the drawings, it should be understood that such embodiments are merely exemplary and do not limit the present invention, and since various other modifications can be made by those of ordinary skill in the art, embodiments of the present invention are not limited to the specific constructions and arrangements shown and described.

Claims

1. A method for simulating a medical procedure to be performed on a patient, the method comprising: Receiving patient-specific data specific to the patient on whom the medical procedure is to be performed; Modifying a generic model using the patient-specific data to generate a hybrid simulation model customized for the patient; And Performing a simulation of the medical procedure using the hybrid simulation model and dynamic simulation information corresponding to at least one of the medical procedure or the patient, wherein the simulation is customized for both the patient and the medical procedure.

2. The method according to claim 1, wherein performing the simulation comprises: Rendering a first portion of the hybrid simulation model at a fidelity level higher than a fidelity level of a second portion of the hybrid simulation model, the first portion representing the patient's anatomy that interacts with the medical procedure, the second portion representing the patient's anatomy that does not interact with the medical procedure at any step of the medical procedure.

3. The method according to claim 1, wherein performing the simulation comprises: Dynamically simulating the medical procedure using the hybrid simulation model and the dynamic simulation information such that the patient's anatomy is represented as a gradient of fidelity levels, wherein the fidelity level of the anatomical structures in the hybrid simulation model decreases during the simulation as the distance from the surgical path of the medical procedure increases.

4. The method according to claim 1, wherein performing the simulation comprises: Adjusting the visualization of the hybrid simulation model during the simulation based on the dynamic simulation information to account for the effects of the simulated interaction between the anatomical structures and the instrument.

5. The method according to claim 1, wherein performing the simulation comprises: Rendering the hybrid simulation model to provide a visualization of the hybrid simulation model on a display system; Receiving an input during the simulation; And Updating the hybrid simulation model and the dynamic simulation information based on the input.

6. The method according to claim 5, wherein performing the simulation further comprises: Controlling the visualization of the hybrid simulation model during the simulation of the medical procedure based on the updated hybrid simulation model and the updated dynamic simulation information.

7. The method according to claim 1, wherein the dynamic simulation information includes physiological parameter data, instrument positioning information, field of view information, and patient pose information.

8. The method according to claim 1, wherein the patient-specific data includes patient diagnostic data, which includes at least one of preoperative imaging data, physical examination data, the geometry of a target region set, or the geometry of anatomical structures adjacent to the target region set.

9. The method according to claim 1, wherein the patient-specific data includes anthropometric data, which includes height measurements, weight measurements, and anatomical landmark measurements.

10. The method according to claim 1, further comprising: Selecting the generic model from a database system, wherein the generic model is a generic model of a representative patient or a group of representative patients.

Images