Hybrid Simulation Model for Simulating Medical Procedures

Abstract

A method and apparatus for simulating a medical procedure to be performed on a patient. Patient-specific data specific to the patient on whom the medical procedure is to be performed is received. A generic model is modified using the patient-specific data to generate a hybrid simulation model customized for the patient. A simulation of the medical procedure is performed using the hybrid simulation model and dynamic simulation information corresponding to at least one of the medical procedure or the patient. The simulation is customized for both the patient and the medical procedure.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 689,584, filed June 25, 2018, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to medical procedures and methods for simulating minimally invasive medical procedures. More specifically, the present disclosure relates to systems and methods for providing a hybrid simulation model that has been customized for a specific patient and a specific medical procedure to be performed on the patient. Background Art

[0004] Minimally invasive medical techniques are intended to reduce the amount of extracorporeal tissue that is damaged during a diagnostic or surgical procedure, thereby reducing the patient's recovery time, discomfort, and harmful side effects. Such minimally invasive techniques can be performed through natural orifices in the patient's anatomy or through one or more surgical incisions. Through these natural orifices or incisions, clinicians can insert medical tools to reach the target tissue location. Minimally invasive medical tools include instruments such as therapeutic instruments, diagnostic instruments, and surgical instruments. Minimally invasive medical tools may also include imaging instruments, such as endoscopic instruments, which provide the user with a field of view within the patient's anatomy.

[0005] Some minimally invasive medical tools may be teleoperated, either remotely or computer-assisted. A clinician near a teleoperating system may need to receive guidance in the form of instructions, warnings, confirmations, etc. before, during, or after performing a medical procedure using the teleoperating system. Systems and methods are needed for improving the process of generating simulation models for simulating medical procedures. Summary of the invention

[0006] The accompanying claims outline embodiments of the invention.

[0007] In an embodiment, a method for simulating a medical procedure to be performed on a patient is provided. Patient-specific data specific to the patient on whom the medical procedure is to be performed is received. A generic model is modified using the patient-specific data to generate a hybrid simulation model customized for the patient. A simulation of the medical procedure is performed using the hybrid simulation model and dynamic simulation information corresponding to at least one of the medical procedure or the patient. The simulation is customized for both the patient and the medical procedure.

[0008] In another embodiment, a simulation system includes a computer system configured to receive patient-specific data specific to a patient for whom a medical procedure is to be performed; modify a generic model using the patient-specific data to generate a hybrid simulation model customized for the patient. The computer system is configured to perform a simulation of the medical procedure using the hybrid simulation model and dynamic simulation information corresponding to at least one of the medical procedure or the patient, wherein the simulation is customized for both the patient and the medical procedure.

[0009] In yet another embodiment, a method for simulating a medical procedure to be performed on a patient is provided. Anthropometric data and patient diagnostic data of the patient are received. A generic model of a representative patient is selected from a database system. The generic model is modified based on the anthropometric data and the patient diagnostic data to generate a hybrid simulation model customized for the patient. A simulation of the medical procedure is performed using the hybrid simulation model and dynamic simulation information corresponding to at least one of the medical procedure or the patient. The simulation is customized for both the patient and the medical procedure. The simulation takes into account the effects of simulated interactions between the patient's anatomical structure and instruments used in the medical procedure. The simulation dynamically adapts to each step of the medical procedure so that a portion of the patient's anatomy that interacts with the medical procedure at each step is rendered at a higher level of fidelity.

[0010] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the present disclosure, but not to limit the scope of the present disclosure. In this regard, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] When read in conjunction with the accompanying drawings, various aspects of the present disclosure can be best understood from the following detailed description. It is emphasized that, in accordance with standard practice in the industry, various features are not drawn to scale. In fact, for clarity of discussion, the sizes of various features can be increased or reduced at will. In addition, the present disclosure may repeat reference numerals and / or letters in various examples. This repetition is for simplicity and clarity purposes and does not in itself indicate the relationship between the various embodiments and / or configurations discussed.

[0012] Figure 1A is a schematic diagram of a medical system according to an embodiment.

[0013] Figure 1B is a perspective view of an assembly according to an embodiment.

[0014] Figure 1C is a perspective view of a surgeon's console for a medical system according to an embodiment.

[0015] Figure 2is a block diagram of a simulation system for simulating a medical procedure to be performed on a patient according to an embodiment.

[0016] Figure 3 is a block diagram illustrating different types of common data that may be found in a database system according to an embodiment.

[0017] Figure 4 is a block diagram illustrating different types of patient-specific data that may be found in a database system according to an embodiment.

[0018] Figure 5 is a block diagram illustrating different types of dynamic simulation information according to an embodiment.

[0019] Figure 6 is a flowchart illustration of a method for simulating a medical procedure to be performed on a patient, according to an embodiment.

[0020] Figure 7 is a flowchart illustration of a method for generating a hybrid simulation model for simulating a medical procedure to be performed on a patient, according to an embodiment.

[0021] Figure 8 is a flowchart illustration of a method for running a simulation of a medical procedure using a hybrid simulation model, according to an embodiment. DETAILED DESCRIPTION

[0022] In order to promote the understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the accompanying drawings, and specific language will be used to describe these embodiments. However, it will be understood that it is not intended to limit the scope of the present disclosure. In the following detailed description of various aspects of the present invention, many specific details are set forth to provide a thorough understanding of the disclosed embodiments. However, as will be appreciated by those skilled in the art, embodiments of the present disclosure may be practiced without these specific details. In other cases, well-known methods, procedures, components and circuits are not described in detail to avoid unnecessarily covering up various aspects of embodiments of the present invention.

[0023] As would be expected by a person skilled in the art to which the present disclosure relates, any changes and further modifications to the described equipment, apparatus, methods of the principles of the present disclosure and any further applications of the principles of the present disclosure are fully contemplated. In particular, it is fully contemplated that the features, components and / or steps described with respect to one embodiment may be combined with the features, components and / or steps described with respect to other embodiments of the present disclosure. In addition, the dimensions provided herein are for specific examples, and it is contemplated that different dimensions, sizes and / or ratios may be utilized to implement the concepts of the present disclosure. In order to avoid unnecessary descriptive repetitions, one or more components or actions described with respect to an illustrative embodiment may be used or omitted according to the applicability of other illustrative embodiments. For the sake of brevity, many iterations of these combinations will not be described separately. For the sake of simplicity, in some cases, the same reference numerals are used in all drawings to refer to the same or similar parts.

[0024] The following embodiments will describe various instruments and parts of instruments according to their states in three-dimensional space. As used herein, the term "positioning" refers to the position of an object or a portion of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, and Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or a portion of an object (three rotational degrees of freedom - e.g., roll, pitch, and yaw). As used herein, the term "posture" refers to the positioning of an object or a portion of an object in at least one translational degree of freedom and the orientation of the object or a portion of an object in at least one rotational degree of freedom (up to six total degrees of freedom).

[0025] Now referring to the drawings, FIG. 1 includes Figure 1A , Figure 1B and Figure 1C , which together provide a schematic overview of a medical system 10 that can be used in a medical procedure including, for example, a diagnostic, therapeutic, or surgical procedure. The medical system 10 is located in a surgical environment 11. In one or more embodiments, the medical system 10 can be a remotely operated medical system under the remote operating control of a surgeon. In alternative embodiments, the medical system 10 can be under partial control of a computer programmed to perform a medical procedure or subroutine. In other alternative embodiments, the medical system 10 can be a fully automated medical system that is under the full control of a computer programmed to perform a medical procedure or subroutine using the medical system 10. An example of a medical system 10 that can be used to implement the systems and techniques described in the present disclosure is a da Surgical system.

[0026] like Figure 1A As shown, the medical system 10 generally includes an assembly 12, which can be mounted on an operating table O where a patient P is located or positioned near the operating table O. The assembly 12 can be referred to as a patient side cart, a surgical cart, or a surgical robot. In one or more embodiments, the assembly 12 can be a teleoperation assembly. The teleoperation assembly can be referred to as, for example, a teleoperation arm cart. A medical device system 14 and an endoscopic imaging system 15 are operably coupled to the assembly 12. An operator input system 16 allows a surgeon or other type of clinician S to view an image of the surgical site or an image representing the surgical site and to control the operation of the medical device system 14 and / or the endoscopic imaging system 15.

[0027] The medical device system 14 may include one or more medical devices. In embodiments where the medical device system 14 includes multiple medical devices, the multiple medical devices may include multiple identical medical devices and / or multiple different medical devices. Similarly, the endoscopic imaging system 15 may include one or more endoscopes. In the case of multiple endoscopes, the multiple endoscopes may include multiple identical endoscopes and / or multiple different endoscopes.

[0028] The operator input system 16 may include a surgeon's console and may be located in the same room as the operating table O. In some embodiments, the surgeon S and the operator input system 16 may be located in a different room than the patient P or in an entirely different building. The operator input system 16 typically includes one or more control devices for controlling the medical instrument system 14. The control device(s) may include any number or more of a variety of input devices, such as a handle, a joystick, a trackball, a data glove, a trigger gun, a foot pedal, a hand controller, a voice recognition device, a touch screen, a human motion or presence sensor, and other types of input devices.

[0029] In some embodiments, the control device(s) are provided with the same degrees of freedom as the medical device(s) of the medical device system 14 to provide the surgeon with a sense of telepresence that the control device(s) are integral with the instrument, so that the surgeon has a strong sense of direct control of the instrument, just as if at the surgical site. In other embodiments, the control device(s) may have more or fewer degrees of freedom than the associated medical instrument and still provide telepresence to the surgeon. In some embodiments, the control device(s) are manual input devices that move with six degrees of freedom and may also include an actuatable handle for actuating the instrument (e.g., for closing a grasping jaw end effector, applying an electrical potential to an electrode, providing medication, and actuating other types of instruments).

[0030] When the surgeon S observes the surgical site through the operator input system 16, the assembly 12 supports and manipulates the medical device system 14. Images of the surgical site can be obtained by the endoscopic imaging system 15, which can be manipulated by the assembly 12. The assembly 12 may include the endoscopic imaging system 15, and may also include multiple medical device systems 14. The number of medical device systems 14 used at one time will generally depend on factors such as the diagnostic or surgical procedures to be performed and the space limitations in the operating room. The assembly 12 may include a kinematic structure of one or more non-servo-controlled links (e.g., one or more links that can be manually positioned and locked in place, generally referred to as mounting structures) and a manipulator. When the manipulator takes the form of a teleoperated manipulator, the assembly 12 is a teleoperated assembly. The assembly 12 includes multiple motors that drive inputs on the medical device system 14. In an embodiment, these motors move in response to commands from a control system (e.g., control system 20). The motor includes a drive system that, when coupled to the medical device system 14, can advance the medical device into an anatomical orifice created naturally or surgically. Other electric drive systems can move the distal end of the medical device in multiple degrees of freedom, which can include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). In addition, the motor can be used to actuate the articulable end effector of the medical device to grasp tissue in the jaws of a biopsy device, etc. The medical device of the medical device system 14 may include an end effector having a single working member, such as a scalpel, a blunt knife, an optical fiber, or an electrode. Other end effectors may include, for example, forceps, graspers, scissors, or clip applicators.

[0031] The medical system 10 also includes a control system 20. The control system 20 includes: at least one memory 24 and at least one processor 22 for implementing control between the medical device system 14, the operator input system 16, and other auxiliary systems 26; and other auxiliary systems 26, which include, for example, imaging systems, audio systems, fluid delivery systems, display systems, lighting systems, steering control systems, irrigation systems, and / or suction systems. The clinician C can circulate within the surgical environment 11 and can access, for example, the assembly 12 during the setup procedure or view the display of the auxiliary system 26 from the patient's bedside.

[0032] Despite Figure 1A The control system 20 is depicted as being external to the assembly 12, but in some embodiments, the control system 20 may be completely contained within the assembly 12. The control system 20 also includes programmed instructions (e.g., stored on a non-transitory computer readable medium) to implement some or all of the methods described in accordance with various aspects disclosed herein. Figure 1A The control system 20 is shown as a single box in the simplified schematic of , but the control system 20 may include two or more data processing circuits, where part of the processing is optionally performed on or near the assembly 12, another part of the processing is performed on the operator input system 16, and so on.

[0033] Any of a variety of centralized or distributed data processing architectures may be employed. Similarly, the programmed instructions may be implemented as a plurality of separate programs or subroutines, or they may be integrated into many other aspects of the systems described herein, including remote operating systems. In one embodiment, the control system 20 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry.

[0034] The control system 20 is in communication with a database 27, which may store one or more clinician profiles, a list of patients and patient profiles, a list of procedures to be performed on the patients, a list of clinicians scheduled to perform the procedures, other information, or a combination thereof. The clinician profile may include information about the clinician, including how long the clinician has worked in the medical field, the level of education the clinician has obtained, the clinician's level of experience with the medical system 10 (or similar systems), or any combination thereof.

[0035] The database 27 may be stored in the memory 24 and may be dynamically updated. Additionally or alternatively, the database 27 may be stored on a device such as a server or a portable storage device that may be accessed by the control system 20 via an internal network (e.g., a secure network of a medical facility or a remote operating system provider) or an external network (e.g., the Internet). The database 27 may be distributed across two or more locations. For example, the database 27 may be present on multiple devices that may include devices of different entities and / or cloud servers. Additionally or alternatively, the database 27 may be stored on a portable user-assigned device such as a computer, mobile device, smart phone, laptop, electronic badge, tablet computer, pager, and other similar user devices.

[0036] In some embodiments, the control system 20 may include one or more servo controllers that receive force and / or torque feedback from the medical device system 14. In response to the feedback, the servo controller transmits a signal to the operator input system 16. The servo controller (one or more) may also transmit a signal to instruct the assembly 12 to move the medical device system (one or more) 14 and / or the endoscopic imaging system 15, which extends to the internal surgical site in the patient's body via an opening in the body. Any suitable conventional or dedicated servo controller can be used. The servo controller can be separated from the assembly 12 or integrated with the assembly 12. In some embodiments, the servo controller and the assembly 12 are provided as a part of a teleoperated arm cart positioned near the patient's body.

[0037] The control system 20 can be coupled to the endoscopic imaging system 15 and can include a processor to process the captured images for subsequent display, such as to a surgeon at a surgeon's console, or on other suitable displays located locally and / or remotely. For example, where a stereoscopic endoscope is used, the control system 20 can process the captured images to present a coordinated stereoscopic image of the surgical site to the surgeon. Such coordination can include alignment between relative images and can include adjusting the stereo working distance of the stereoscopic endoscope.

[0038] In alternative embodiments, the medical system 10 may include more than one assembly 12 and / or more than one operator input system 16. The exact number of assemblies 12 will depend on the surgical procedure and the space limitations in the operating room, as well as other factors. The operator input systems 16 may be collocated, or they may be located in separate locations. Multiple operator input systems 16 allow more than one operator to control one or more assemblies 12 in various combinations.

[0039] The medical system 10 can also be used to train and practice medical procedures. For example, the medical system 10 can be used with the following Figure 2 The simulation system 200 described in the accompanying drawings is used in conjunction with the simulation system 200, which simulates one or more medical procedures to be performed on a patient.

[0040] Figure 1B1 is a perspective view of an embodiment of an assembly 12, which can be referred to as a patient side cart, a surgical cart, a teleoperation arm cart or a surgical robot. The assembly 12 shown provides manipulation of three surgical tools 30a, 30b, 30c (e.g., medical instrument system 14) and an imaging device 28 (e.g., an endoscopic imaging system 15), which is, for example, a stereo endoscope for capturing images of a surgical site. The imaging device can transmit a signal to a control system 20 via a cable 56. Manipulation is provided by a teleoperation mechanism with multiple joints. The imaging device 28 and surgical tools 30a-30c can be positioned and manipulated by an incision in the patient, thereby keeping the kinematic remote center at the incision to minimize the size of the incision. When the distal end of the surgical tool 30a-30c is positioned in the field of view of the imaging device 28, the image of the surgical site can include the image of the distal end of the surgical tool 30a-30c.

[0041] The assembly 12 includes a drivable base 58. The drivable base 58 is connected to a telescopic column 57, which allows the height of the arm 54 to be adjusted. The arm 54 may include a swivel joint 55 that both rotates and moves up and down. Each arm 54 may be connected to an orientation platform 53. The arms 54 may be marked to facilitate troubleshooting. For example, each arm 54 may be decorated with a different number, letter, symbol, other identifier, or combination thereof. Figure 1B , arms 54 are numbered from 1 to 4. Orientation platform 53 may be capable of rotating 360 degrees. Assembly 12 may also include a telescopic horizontal cantilever 52 for moving orientation platform 53 in a horizontal direction.

[0042] In this example, each arm 54 is connected to a robotic arm 51. The robotic arm 51 can be directly connected to a medical instrument, such as one of the surgical tools 30a-30c. The robotic arm 51 can be remotely operable. In some examples, the arms 54 connected to the orientation platform 53 may not be remotely operable. Instead, these arms 54 can be positioned as needed before the surgeon S begins to operate with the remotely operated components. Throughout the surgical procedure, medical instruments may be removed and replaced with other instruments, so that the association of instruments with arms may change during the procedure.

[0043] Endoscopic imaging systems (e.g., endoscopic imaging system 15 and imaging device 28) can be provided in various configurations including rigid or flexible endoscopes. Rigid endoscopes include a rigid tube that houses a relay lens system for transmitting images from the distal end of the endoscope to the proximal end. Flexible endoscopes use one or more flexible optical fibers to transmit images. Endoscopes based on digital images have a "chip-on-the-tip" design in which a distal digital sensor (such as one or more charge-coupled devices (CCDs) or complementary metal oxide semiconductor (CMOS) devices) stores image data. Endoscopic imaging systems can provide two-dimensional or three-dimensional images to a viewer. Two-dimensional images can provide limited depth perception. Three-dimensional stereoscopic endoscopic images can provide a more accurate depth perception for the viewer. Stereoscopic endoscopic instruments use stereo cameras to capture stereoscopic images of the patient's anatomy. Endoscopic instruments can be fully sterilizable assemblies with endoscope cables, handles, and shafts that are securely coupled and hermetically sealed.

[0044] Figure 1C 1 is a perspective view of an embodiment of an operator input system 16, which may be referred to as a surgeon's console. The operator input system 16 includes a left eye display 32 and a right eye display 34 for presenting a coordinated stereoscopic view of the surgical environment capable of depth perception to the surgeon S. The operator input system 16 also includes one or more input control devices 36, which in turn enable the assembly 12 to manipulate one or more instruments of the endoscopic imaging system 15 and / or the medical instrument system 14. The input control devices 36 may provide the same degrees of freedom as the instruments they are associated with, to provide the surgeon S with a remote presentation, or the input control devices 36 may be integrated with the instruments to give the surgeon a strong sense of direct control of the instruments. To this end, position sensors, force sensors, and tactile feedback sensors (not shown) may be employed to transmit the position, force, and tactile sensation from the medical instruments (e.g., surgical tools 30a-30c or imaging devices 28) back to the surgeon's hands through the input control devices 36. The input control device 37 is a foot pedal that receives input from the user's foot. Aspects of the operator input system 16 , assembly 12 , and auxiliary system 26 may be adjustable and customizable to meet the surgeon's S physical needs, skill level, or preferences.

[0045] Figure 2 2 is a block diagram of a simulation system 200 for simulating a medical procedure 202 performed on a patient 203 (e.g., patient P) in a medical training environment 205. In one or more embodiments, the medical procedure 202 will be performed in a surgical environment (e.g., Figure 1A The procedure is performed in a surgical environment 11).

[0046] The simulation system 200 includes a computer system 204. The computer system 204 may include a single computer or multiple computers in communication with each other. The computer system 204 may include at least one processor 206. The processor 206 may be used to run executable code stored in a memory to simulate the medical procedure 202. In one embodiment, the simulation system 200 is separate from the control system 20 described in FIG. 1. In other embodiments, the simulation system 200 may be implemented as part of the control system 20. In other embodiments, the processor 22 of the control system 20 in FIG. 1 may be used to implement the processor 206 of the simulation system 200.

[0047] In one or more embodiments, the simulation system 200 also includes a display system 208 coupled to the processor 206. In other embodiments, the display system 208 may be considered separate from the simulation system 200. The display system 208 may include one or more display devices communicatively coupled to the processor 206. In one or more embodiments, the display system 208 may be coupled to a surgeon's console of the operator input system 16.

[0048] The simulation system 200 creates a hybrid simulation model 210 that integrates generic medical data 212 (e.g., medical data of a representative patient) with patient-specific data 214 of the patient 203, and then dynamically modifies the hybrid simulation model 210 based on information or parameters that change during the simulation (i.e., virtual reality simulation) of the medical procedure 202. This process of generating the hybrid simulation model 210 reduces the overall time, effort, and processing resources required to generate the simulation model.

[0049] The simulation system 200 can retrieve generic medical data 212 from a database system 218. The database system 218 can include one or more databases. The generic medical data 212 includes information that is not specific to a particular patient 203. For example, the generic medical data 212 can include generic information about the medical procedure 202 and / or information about a representative patient or a representative group of patients. In one or more embodiments, the generic medical data 212 includes a generic model 219 of the representative patient or representative group of patients. In the following Figure 3 Examples of different types of generic medical data 212 that can be retrieved from the database system 218 are described in . In one or more embodiments, the simulation system 200 receives patient-specific data 214 that includes information specific to a particular patient 203. In the following Figure 4 Examples of different types of patient-specific data 214 are described in .

[0050] The simulation system 200 integrates the generic medical data 212 and the patient-specific data 214 to generate a hybrid simulation model 210 customized for the patient 203. In one or more embodiments, the simulation system 200 uses the patient-specific data 214 to modify, enhance, or otherwise refine the generic medical data 212 for regions of interest in the patient anatomy 222 of the patient 203. As an example, the patient-specific data 214 can be used to scale and / or morph the generic model 219. In addition, the patient-specific data 214 can be used to modify portions of the generic model 219 that represent, are adjacent to, and / or relate to a set of target regions 224 to which the medical procedure 202 is directed.

[0051] The target region set 224 may include, for example, one or more diseased anatomical structures of the patient anatomy 222 of the patient 203. The anatomical structures may include, for example, one or more of an organ, an organ system, a muscle, a nerve, an artery, a vein, a bone, a skeletal structure, a ligament, some other type of anatomical structure or tissue, or a combination thereof. By modifying only the portion of the generic model 219 that represents, is adjacent to, and / or relates to the target region set 224, the simulation system 200 reduces the overall time, effort, and processing resources required to generate a hybrid simulation model 210 customized for the patient 203.

[0052] In one or more embodiments, the hybrid simulation model 210 represents the patient anatomy 222 of the patient 203 with varying fidelity levels due to the integration of the generic medical data 212 with the patient-specific data 214. For example, each anatomical structure of the patient anatomy 222 is represented with a particular fidelity level in the hybrid simulation model 210. The fidelity level indicates how accurately the modeled anatomical structure in the hybrid simulation model 210 matches the actual geometry and properties of the corresponding actual anatomical structure of the patient 203.

[0053] In some embodiments, the hybrid simulation model 210 represents the patient anatomy 222 of the patient 203 with a gradient 226 of fidelity levels. For example, the fidelity level of the modeled anatomical structure can decrease as the distance from the operating path of the medical procedure 202 increases. The operating path can be, for example, a surgical path, a procedural path, or some other path (external path and / or internal path) relative to the patient anatomy 222 for performing the medical procedure 202. When the medical system 10 described in FIG. 1 is to be used to perform the medical procedure 202, the surgical path can include, for example, a path relative to the patient anatomy 222 of the patient 203 during which each arm 54 and the components in the medical procedure 202 are moved. Figure 1A and Figure 1B Each surgical instrument and tool of the medical device system 14 of the assembly 12 described in will follow this path.

[0054] For example, the representation of the target region set 224 in the hybrid simulation model 210 has a higher fidelity level than the fidelity level(s) of the portions of the hybrid simulation model 210 representing anatomical structures that will not be affected or interacted with during the medical procedure 202. If the medical procedure 202 is a cardiac surgery procedure, the target region set 224 may include the heart of the patient 203 and the areas surrounding the heart. However, the cardiac surgery procedure may never involve the limbs of the patient 203. Therefore, in the hybrid simulation model 210, the limbs may be represented at a lower fidelity level than the heart or even other anatomical structures within the chest cavity of the patient 203. In some cases, the anatomical structures of the abdomen of the patient 203 will be represented at a higher fidelity level than the limbs but at a lower fidelity level than the heart.

[0055] The resulting hybrid simulation model 210 generated by the simulation system 200 can be used to run a highly customized, high-quality simulation of the medical procedure 202 performed on the patient 203. For example, the simulation system 200 can perform a simulation 228 of the medical procedure 202 using the hybrid simulation model 210 and dynamic simulation information 230 corresponding to at least one of the medical procedure 202 or the patient 203. The simulation 228 can be a dynamic simulation. The dynamic simulation information 230 can include information about how the patient 203 and / or the patient anatomy 222 will be affected by the medical procedure 202, as well as other information about the medical procedure 202 that does not remain static during the simulation of the medical procedure 202. Figure 5 An example of dynamic simulation information 230 is described in .

[0056] The simulation 228 can be rendered on the display system 208 so that the surgeon, members of the surgical team, or other medical personnel can use the simulation 228 to rehearse the medical procedure 202. Because the simulation 228 of the medical procedure 202 is performed using the hybrid simulation model 210 and the dynamic simulation information 230, the simulation 228 provides a higher overall level of fidelity in representing how the medical procedure 202 will be performed for the specific patient 203 than a simulation run using the generic model 219 for the patient 203.

[0057] In e.g. Figure 1A The simulation 228 is performed in the environment 11 in the embodiment to practice or otherwise train for the medical procedure 202. In one or more embodiments, the simulation 228 can be performed using, for example, the medical system 10 in FIG. 1. For example, the simulation 228 can be performed using the operator input system 16, the assembly 12, or both. The operator input system 16 can include a single surgeon console or dual surgeon consoles.

[0058] When the simulation 228 is run, the simulation 228 uses the dynamic simulation information 230 to account for the effects of the simulated interaction between the anatomical structures of the patient anatomy 222 and the instruments used to perform the medical procedure 202. Thus, the simulation system 200 uses the dynamic simulation information 230 to dynamically scale or deform the hybrid simulation model 210 so that the experience provided by the simulation 228 is more similar to the actual medical procedure 202. This type of simulation 228 allows a surgeon or other medical personnel to rehearse the medical procedure 202 with a higher sense of realism. However, because the hybrid simulation model 210 is generated starting from generic medical data 212 that is customized using patient-specific data 214, the overall time, effort, and processing resources spent generating the hybrid simulation model 210 are reduced compared to generating a model that captures the entire patient anatomy 222 of the patient 203 at the highest level of fidelity.

[0059] Figure 3 is a block diagram illustrating different types of generic medical data 212 that may be found in database system 218. Database system 218 may include, for example, program library 300, generic model library 302, and disease state library 304. Each of these three libraries may be implemented using one or more databases or other types of data repositories.

[0060] The procedure library 300 may store information related to one or more medical procedures, including Figure 2 202 to be performed on a patient 203. In an example embodiment, the procedure library 300 stores procedure information 306 for the medical procedure 202. The procedure information 306 identifies various stages and / or steps of the medical procedure 202. The procedure information 306 may, for example, but not limited to, identify: anatomical structures known to be adjacent to one or more diseased anatomical structures to be surgically treated by the medical procedure 202, a surgical path that a medical device should follow when performing the medical procedure 202, key anatomical landmarks where intraoperative decisions are to be made, the length of time or time range expected to be spent on various anatomical structures along the surgical path during the medical procedure, and other types of information about the medical procedure 202.

[0061] The generic model library 302 stores one or more generic models representing the anatomical bodies of representative patients. In one or more embodiments, the generic model library 302 stores a first generic model for a first representative patient group and a second generic model for a second representative patient group. The first representative patient group may be healthy women, and the second representative patient group may be healthy men. In other embodiments, the generic model library 302 stores a first plurality of generic models for the first representative patient group and a second plurality of generic models for the second representative patient group. The first plurality of generic models may include generic models scaled to common female anthropometric measurements, and the second plurality of generic models may include generic models scaled to common male anthropometric measurements. For example, the first plurality of generic models may include a generic model for a certain height and weight category of healthy women, another generic model for another height and weight category of healthy women, and another generic model for another height and weight category of healthy women.

[0062] Thus, the generic model library 302 stores generic, simulation-ready models that can be retrieved by the simulation system 200 of the figure for generating the hybrid simulation model 210. Any of the generic models stored in the generic model library 302 can be retrieved by the simulation system 200 and used as the generic model 219, and then as Figure 2 The generic model 219 is modified using the patient-specific data 214 .

[0063] In one or more embodiments, the disease state library 304 stores information about the properties of any number of anatomical structures when afflicted with various types of disease states. For example, the disease state library 304 may store information about the material properties of a heart afflicted with one disease and information about the material properties of a heart afflicted with another disease.

[0064] Figure 4 2 is a block diagram illustrating different types of patient-specific data 214. The patient-specific data 214 may include, for example, Figure 2 4 and 5. The anthropometric data 400 and patient diagnostic data 402 for the patient 203 described in the accompanying drawings may include anthropometric data 400 and patient diagnostic data 402 for the patient 203 described in the accompanying drawings. The anthropometric data 400 for the patient 203 includes data or measurements used to assess the size, shape, and composition of the patient 203. The anthropometric data 400 may include, but is not limited to, for example: height measurement 404, weight measurement 406, anatomical landmark measurement 408, body mass index (BMI) measurement 410, waist-to-hip ratio 412, skin fold test data 414, bioelectrical impedance data 416, some other type of anthropometric measurement, or a combination thereof. The anatomical landmark measurement 408 may include, for example, measurements of distances between key bones or other anatomical landmarks of the patient anatomy 222 of the patient 203. Figure 2The simulation system 200 in the embodiment of the present invention can use the anthropometric data 400 to generate a scaling factor that can be applied to the image obtained from Figure 3 The generic model of a representative patient is retrieved from the generic model library 302 in order to customize the generic model for the patient 203 .

[0065] Patient diagnostic data 402 includes information about specific anatomical structures of patient 203. In one or more embodiments, patient diagnostic information includes information about the specific anatomical structures of patient 203. Figure 2 The patient diagnostic data 402 may include information about the geometry of the target region set 224 in the patient anatomy 222, and information about the geometry of one or more anatomical structures adjacent to or functionally related to each of the target region set 224. For example, the patient diagnostic data 402 may include preoperative imaging data 418, physical examination data 420, and / or other types of diagnostic information that provide information about the target region set 224 (and, in some cases, anatomical structures adjacent to or functionally related to each of the target region set 224). The preoperative imaging data 418 may include, but is not limited to, for example, ultrasound images, x-ray images, medical resonance imaging (MRI) images, computed tomography (CT) images, or other types of imaging data. The physical examination data 420 may include information about the patient anatomy 222 that may be collected from, for example, observation, palpation, percussion, and auscultation of the patient 203. Figure 2 The simulation system 200 in the embodiment may use the patient diagnostic data 402 to modify the Figure 3 The geometry and / or other types of properties of the anatomical structures in the generic model of the representative patient retrieved from the generic model library 302 in the embodiment of the present invention are used to customize the generic model for the patient 203 .

[0066] Figure 5 2 is a block diagram illustrating different types of dynamic simulation information 230. The dynamic simulation information 230 includes information about one or more parameters that are expected to change during a simulation 220 of a medical procedure 202 run using a hybrid simulation model 210. For example, the dynamic simulation information 230 may include physiological parameter data 500, instrument positioning information 502, field of view information 504, patient posture information 506, and other types of information. In one or more embodiments, Figure 2 The simulation system 200 uses the instrument positioning information 502 , the field of view information 504 , and the patient posture information 506 during the simulation 228 to dynamically adjust the rendering or presentation of the hybrid simulation model 210 during the simulation 228 to improve the accuracy and realism of the simulation 228 .

[0067] The physiological parameter data 500 includes data for a plurality of physiological parameters that may change or be expected to change during the medical procedure 202. In particular, the physiological parameter data 500 includes information about how anatomical structures within the patient 203 should appear and respond to simulated interactions between the anatomical structures and instruments and / or agents used to perform the medical procedure 202. For example, during the simulation 228 of the medical procedure 202, the simulated effects of aspiration, irrigation, impedance cardiography, and other steps or procedures performed as part of the medical procedure 202 may result in simulated changes in the plurality of physiological parameters. The plurality of physiological parameters may include, for example, but not limited to, arterial and venous pressures, respiratory rate, insufflation pressure, tissue response to energy application, tissue response to physical contact with one or more instruments, and / or other types of physiological parameters. Figure 2 The simulation system 200 can use the physiological parameter data 500 to ensure that the simulation 228 of the medical procedure 202 takes into account the simulated changes in the physiological parameters. For example, the simulation system 200 can use the physiological parameter data 500 to dynamically adjust the rendering of the hybrid simulation model 210 on the display system 208 during the simulation 228 in a manner that takes into account the simulated changes in the physiological parameters. More specifically, the simulation system 200 can adjust the visualization of the hybrid simulation model 210 to take into account these simulated changes.

[0068] The instrument positioning information 502 includes information about how the positioning of an instrument (e.g., an instrument of the medical instrument system 14 in FIG. 1 , a retractor, an endoscope, a scalpel, a suction tube, an irrigation tube, a laser device, a grasping tool, etc.) may change or is expected to change during the course of the medical procedure 202. As an example, the instrument positioning information 502 may include information about how the positioning of an endoscope (e.g., the endoscopic imaging system 15 in FIG. 1 ) is expected to move or change during the medical procedure 202. The simulation system 200 may use the instrument positioning information 502 to dynamically adjust the rendering of the hybrid simulation model 210 on the display system 208 during the simulation 228 in a manner that accounts for these changes in the positioning of the endoscope.

[0069] Similarly, the simulation system 200 can use the field of view information 504 to dynamically adjust the rendering of the hybrid simulation model 210 on the display system 208 during the simulation 228 in a manner that takes into account changes in one or more fields of view or perspectives. The field of view information 504 includes information about one or more different fields of view. For example, the field of view information 504 can include instrument-based field of view information, vision-based field of view information, some other type of field of view or perspective information, or a combination thereof.

[0070] Instrument-based field of view information may include, for example, information about what anatomical structures a surgeon or other medical personnel can actually see in the field of view of one or more imaging instruments (such as an endoscope, a camera, etc.). For example, field of view information 504 may include information about the positioning of an imaging instrument and the calculation of the field of view of the imaging instrument as the imaging instrument moves during the simulation of the medical procedure 202. Vision-based field of view information includes information from the perspective of a surgeon or other clinician. For example, the field of view information may be from the perspective that a surgeon or other clinician will see during the medical procedure 202.

[0071] The patient posture information 506 includes information about how the positioning and / or orientation of the patient 203 may change during the medical procedure 202. In some embodiments, the positioning and / or orientation of the patient 203 is determined by the positioning and / or orientation of the bed, operating table, or other type of structure on which the patient 203 is positioned during the medical procedure 202. The patient posture information 506 may include information about how gravity affects the patient 203 based on the positioning and / or orientation of the patient 203. For example, the patient posture information 506 may include information about how the positioning and / or orientation of the patient 203 during the medical procedure 202 may cause one or more anatomical structures to deform. The simulation system 200 may use the patient posture information 506 to dynamically adjust the rendering of the hybrid simulation model 210 on the display system 208 during the simulation 228 in a manner that takes into account these changes in the posture of the patient 203.

[0072] Figure 6 1 is a flowchart illustration of a method 600 for simulating a medical procedure to be performed on a patient. In one or more embodiments, the medical procedure is performed in a surgical environment (such as surgical environment 11 of FIG. 1 ). Figure 6 602 to 608, and continue to refer to Figure 2-Figure 5 Not all of the illustrated processes 602 to 608 may be performed in all embodiments of the method 600. Figure 6 One or more processes not explicitly illustrated in the drawings may be included before, after, between, or as part of processes 602 to 608. In some embodiments, one or more of processes 602 to 608 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when executed by one or more processors (e.g., processors of a control system), may cause the one or more processors to perform the one or more processes.

[0073] At process 602, patient specific data 214 specific to a patient 203 on whom a medical procedure 202 is to be performed is received. The patient specific data 214 may include, for example, anthropometric data 400, patient diagnostic data 402, or a combination of both.

[0074] At process 604, the generic model 219 of the representative patient is modified using the patient-specific data 214 to generate a hybrid simulation model 210 customized for the patient 203. The generic model 219 can be retrieved from the database system 218. In particular, in one embodiment, the generic model 219 can be retrieved from the generic model library 302. Depending on the gender of the patient 203, the generic model 219 can be a model of a healthy female or male anatomy. In one or more embodiments, the generic model 219 selected for use can represent the anatomy of a healthy female or male having anthropometric measurements in common with the patient 203 or anthropometric measurements within a selected range of the patient 203.

[0075] In one or more embodiments, modifying the generic model 219 includes modifying the representation of each of the plurality of anatomical structures in the generic model 219 along the operation path along which the medical procedure 202 is to be performed based on the received patient-specific data 214, thereby generating the hybrid simulation model 210. In some embodiments, the modification of the generic model 219 using the patient-specific data 214 in the process 604 includes scaling and / or deforming the generic model 219 based on the anthropometric data 400. In addition, the modification may include adjusting one or more portions of the generic model 219 based on the patient diagnostic data 402. For example, if the preoperative imaging data 418 indicates that the patient 203 has an enlarged liver and the medical procedure 202 to be performed involves performing a biopsy on the liver, the generic model 219 may be modified so that the representation of the liver in the generic model 219 is enlarged accordingly. In this manner, the portion of the generic model 219 is customized for the patient 203. Thus, the generic model 219 may be customized based on the set of target regions 224 to which the medical procedure 202 is directed.

[0076] At process 606, a simulation 228 of the medical procedure 202 is performed using the hybrid simulation model 210 and the dynamic simulation information 230 corresponding to at least one of the medical procedure 202 or the patient 203. In one or more embodiments, performing the simulation 228 includes running the simulation 228 using the hybrid simulation model 210 and simulated (or computerized) versions of instruments that will interact with the patient 203 during the medical procedure 202. In other embodiments, performing the simulation 228 may include a surgeon, a member of a surgical team, or other medical personnel interacting with the hybrid simulation model 210 using the medical system 10 described in FIG. 1 to train or rehearse the medical procedure 202.

[0077] During the simulation 228 in process 606, the dynamic simulation information 230 is used to render a first portion of the hybrid simulation model 210 at a higher fidelity level than a second portion of the hybrid simulation model 210, the first portion of the hybrid simulation model 210 representing the patient anatomy 222 of the patient 203 interacting with the medical procedure 202, and the second portion of the hybrid simulation model 210 representing the patient anatomy 222 not interacting with the medical procedure 202 at any step of the medical procedure 202. In one or more embodiments, performing the simulation 228 includes dynamically simulating the medical procedure 202 using the hybrid simulation model 210 and the dynamic simulation information 230 such that the patient anatomy 222 of the patient 203 is represented at a gradient 226 of fidelity levels. For example, the fidelity level of the anatomical structures in the hybrid simulation model 210 may decrease during the simulation as the distance from the surgical path of the medical procedure 202 increases.

[0078] Executing the simulation 228 may include using the patient posture information 506 to adjust how one or more anatomical structures of the patient anatomy 222 in the hybrid simulation model 210 deform in response to anticipated changes in the posture of the patient 203 during the simulation 228. Executing the simulation 228 may also include adjusting the visualization of the hybrid simulation model 210 during the simulation 228 to account for the effects of simulated interactions between the anatomical structures and the instruments based on the dynamic simulation information 230. The instruments may include any number of manually controlled instruments, robotically controlled instruments, or both.

[0079] For example, the simulation system 200 may consider how changes in the positioning of one or more instruments during the medical procedure 202 may affect portions of the patient anatomy 222 along the surgical path in the hybrid simulation model 210. The simulation system 200 may use the instrument positioning information 502 to adjust the rendering of the hybrid simulation model 210 on the display system 208 to assist a surgeon or clinician who is rehearsing the medical procedure 202. The simulation system 200 may also use the field of view information 504 to adjust the rendering of the hybrid simulation model 210. In some embodiments, the simulation system 200 may actually modify or recreate one or more portions of the hybrid simulation model 210 based on the positioning of the instruments.

[0080] Thus, a simulation 228 using the hybrid simulation model 210 is customized to the patient 203 and the medical procedure 202 to be performed on the patient 203. The simulation 228 dynamically adapts the hybrid simulation model 210 to each step or phase of the medical procedure 202 so that at each step the portion of the patient anatomy 222 that interacts with the medical procedure 202 is rendered at a higher level of fidelity than other portions of the patient anatomy 222. This type of simulation 228 provides a representation of the medical procedure 202 to be performed on the patient 203 with a desired level of accuracy.

[0081] Figure 7 1 is a flowchart illustration of a method 700 for generating a hybrid simulation model for simulating a medical procedure to be performed on a patient. In one or more embodiments, the medical procedure will be performed in a surgical environment (such as surgical environment 11 of FIG. 1 ). The method 700 is Figure 7 702 to 716 and continue to refer to Figures 2 to 5 Give a description.

[0082] In addition, compared with the above Figure 6 The method 600 described in Figure 7 The method 700 illustrated in FIG. 700 may be a more detailed method for generating a hybrid simulation model. Not all of the illustrated processes 702 to 716 may be performed in all embodiments of the method 700. Figure 7 One or more processes not explicitly illustrated in the drawings may be included before, after, between, or as part of processes 702 to 716. In some embodiments, one or more of processes 702 to 716 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when executed by one or more processors (e.g., processors of a control system), may cause the one or more processors to perform the one or more processes.

[0083] At process 702, anthropometric data 400 and patient diagnostic data 402 are received at the simulation system 200. At process 704, a generic model 219 of a representative patient is selected based on the received anthropometric data 400. For example, a generic model 219 of a representative patient of the same gender and the same height and weight category as the patient 203 may be selected. At process 706, the generic model 219 is scaled and / or deformed based on the anthropometric data 400. The scaling and / or deformation may be a further refinement of the generic model 219 based on the precise height and weight of the patient 203, the body mass index of the patient 203, and / or other measurements of the patient 203.

[0084] At process 708, a surgical path and a plurality of anatomical structures along the surgical path are identified based on the procedure information 306 regarding the medical procedure 202 to be performed on the patient 203. At process 710, a representation of each of the plurality of anatomical structures along the surgical path is modified based on the received patient diagnostic data 402, thereby completing the hybrid simulation model 210. Thus, the hybrid simulation model 210 is generated by combining generic information with relevant patient-specific information.

[0085] At process 712, a simulation 228 of a medical procedure 202 to be performed on a patient 203 is run using the hybrid simulation model 210 and the dynamic simulation information 230. For example, a surgeon may use the medical system 10 described in FIG1 to simulate performing the medical procedure 202 using the hybrid simulation model 210. During the simulation 228, the simulation system 200 uses the dynamic simulation information 230 to adjust the visualization or rendering of the hybrid simulation model 210 on the display system 208 to account for the effects of the simulated interactions between the anatomy of the patient 203 and the instruments and / or agents used in the medical procedure 202. The simulation 228 dynamically adapts to each step of the medical procedure 202 such that the portion of the patient anatomy 222 of the patient 203 that interacts with the medical procedure 202 at each step of the medical procedure 202 is rendered at a higher level of fidelity.

[0086] In one or more example embodiments, the hybrid simulation model 210 can be "intelligent" about what to render at a higher fidelity during the simulation 228 of the medical procedure 202. For example, higher fidelity rendering can be applied on the fly as needed to anatomical structures that dynamically interact with instruments, endoscopes, other agents (e.g., suction, irrigation, etc.), and / or gravity during the simulation of the medical procedure 202. In this way, when the surgeon is performing an initial dissection step (e.g., lowering the bladder), it may not be necessary to render the diseased anatomical structure (e.g., the prostate) at a high fidelity.

[0087] In other embodiments, the hybrid simulation model 210 may allow the simulation 228 to begin with a lower fidelity rendering. As the simulation 228 proceeds, the rendering of the corresponding portion of the hybrid simulation model 210 will "snap" to the appropriate fidelity level based on the step or phase of the medical procedure 202 being simulated. In some embodiments, during the simulation 228 of a medical procedure, whenever an instrument, agent, or gravity interacts with the anatomical structure, the hybrid simulation model 210 will be able to provide the desired fidelity level for any relevant anatomical structure.

[0088] At process 714, simulation data is calculated based on the simulation 228 of the medical procedure 202. The simulation data, which may be feedback data, may be calculated during the simulation 228, after the simulation 228 is completed, or both. The simulation data may include information about the simulation 228 itself as well as patient-specific intraoperative data. For example, the simulation data may include data about how the simulation 228 of the medical procedure 202 compares to the expected performance and results of the medical procedure 202. For example, the simulation data may include data about the efficiency of the simulation 228, measures of success of the simulation 228, data about the time taken for each step of the medical procedure 202, data about the simulated interaction between the medical system 10 and the patient anatomy 222, data about the effect of changes in the patient's posture during the simulation 228 on the simulation of the patient's anatomy 222 as compared to the patient posture information 506, other types of data, or combinations thereof.

[0089] In some embodiments, the simulation data includes information that can be used for preoperative planning. As an example, the simulation data can be used to improve medical device selection, improve medical device calibration, improve patient preparation before the medical procedure 202, or a combination thereof, thereby increasing the procedural efficiency or operational efficiency of the medical procedure 202.

[0090] At process 716, the hybrid simulation model 210 is modified using the calculated simulation data to form an updated hybrid simulation model 210. The updated hybrid simulation model 210 may allow the second simulation 228 to be run at a higher fidelity than the first simulation 228. In some embodiments, at process 716, the hybrid simulation model 210 is modified based on user input. At process 718, the dynamic simulation information 230 is modified based on the calculated simulation data to form updated dynamic simulation information 230. The updated dynamic simulation information 230 takes into account the results of the simulation 228 to provide updated information that can be used to dynamically adjust another simulation of the medical procedure 202. In some embodiments, at process 718, the dynamic simulation information 230 is modified based on user input.

[0091] At process 720, a new simulation of the medical procedure 202 is performed using the updated hybrid simulation model 210 and the updated dynamic simulation information 230. Performing a new simulation using the updated hybrid simulation model 210 and the updated dynamic simulation information 230 may provide both the patient 203 and the medical procedure 202 with a finer level of customization.

[0092] In some embodiments, processes 714 to 720 can be repeated multiple times to further refine and customize the hybrid simulation model 210. Processes 714 to 720 can be repeated any number of times until a desired version of the hybrid simulation model 210 with the desired level of refinement and overall fidelity and customization has been generated. The simulation data generated in multiple runs of the simulation 228 can be used as patient-specific intraoperative guidance data. For example, the simulation data can be used to generate instructions to control the medical system 10 to actually perform the medical procedure 202 on the patient 203.

[0093] In some embodiments, the simulation data may be fed into a learning algorithm or learning program for providing information about how to improve the medical procedure 202. The simulation data may provide useful information about actions that would lead to less than ideal results if performed during the medical procedure 202. In one or more embodiments, the simulation data may be used to identify an optimal set and / or sequence of actions to be performed during the medical procedure 202 to allow for the best possible results. In some embodiments, the simulation data may be used to help train other surgeons or clinicians.

[0094] Figure 8 is a flowchart illustration of a method for running a simulation of a medical procedure using a hybrid simulation model. In one or more embodiments, the medical procedure will be performed in a surgical environment (such as surgical environment 11 of FIG. 1 ). The method 800 Figure 8 802 to 806, and continue to refer to Figures 2 to 5 Give a description.

[0095] also, Figure 8 The method 800 illustrated in FIG. 8 may be used as described above. Figure 7 A more detailed method of running a simulation of a medical procedure is described in process 712 of the method 800. Not all of the illustrated processes 802 to 806 may be performed in all embodiments of the method 800. Figure 8 One or more processes not explicitly illustrated in the drawings may be included before, after, between, or as part of processes 802 to 806. In some embodiments, one or more of processes 802 to 806 may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when executed by one or more processors (e.g., processors of a control system), may cause the one or more processors to perform the one or more processes.

[0096] At process 802, the hybrid simulation model 210 is rendered to provide a visualization of the hybrid simulation model 210 on the display system 208. The visualization of the hybrid simulation model 210 may include one or more different views of the hybrid simulation model 210. For example, the visualization may include one or more external views of the patient anatomy 222, one or more internal views of the patient anatomy 222, or a combination thereof.

[0097] At process 804, input is detected during the simulation 228 of the medical procedure 202. The input may include any number of inputs. For example, the input may include at least one of the following: user input, Figure 1A The medical system 10 receives system inputs from Figure 2 , or some other type of input received by the simulation system 200 described in . The user input may be received from any number of input devices or systems. For example, the user input may be received via the display system 208, a touch screen, a gesture recognition system, a gaze-based recognition system, a speech-based recognition system, an augmented or virtual reality system, a keyboard, a joystick, one or more touch controls, a mouse, some other type of input device, or any combination thereof. In some cases, the user input may be received from the operator input system 16.

[0098] In one or more embodiments, the system input received from the medical system 10 may include any number or combination of inputs from the assembly 12. For example, the system input may be received from the medical instrument system 14, the endoscopic imaging system 15, or a combination thereof. In one or more embodiments, the input from the simulation system 200 may include input selected from the dynamic simulation information 230 (e.g., the value of a parameter included in the dynamic simulation information 230), input from the hybrid simulation model 210, or both.

[0099] At process 806, the input is used to update the hybrid simulation model 210 and the dynamic simulation information 230. In one or more embodiments, updating the hybrid simulation model 210 may include modifying a portion of the hybrid simulation model 210, recreating a portion of the hybrid simulation model 210, or both. In some embodiments, updating the dynamic simulation information 230 may include updating the values ​​of one or more parameters included in the dynamic simulation information 230. For example, the input detected at process 804 may represent Figure 1B The movement may be, for example, a deeper insertion of the surgical tool 30b into the patient's anatomy 222. The input may be used, for example, to update Figure 5 The instrument positioning information 502 described in . Any number or type of algorithms may be used to perform the updating of the hybrid simulation model 210 and the dynamic simulation information 230 .

[0100] At process 808, visualization of the hybrid simulation model 210 during the simulation 228 of the medical procedure 202 is controlled based on the updated hybrid simulation model 210 and the updated dynamic simulation information 230. In some embodiments, controlling the visualization of the hybrid simulation model 210 means that the visualization is not changed. For example, the updated hybrid simulation model 210 and the updated dynamic simulation information 230 may not require a change in the visualization of the hybrid simulation model 210.

[0101] In one or more embodiments, visualization of the hybrid simulation model 210 is controlled so that the updated hybrid simulation model 210 is rendered in a manner that provides the highest level of fidelity along the operational path (e.g., surgical path) of the medical procedure 202. As an example, the updated instrument positioning information 502 may cause a change in the displayed portion of the hybrid simulation model 210. For example, portions of the hybrid simulation model 210 corresponding to deeper locations within the patient anatomy 222 may be displayed and rendered at a higher level of fidelity than the level of fidelity prior to receiving the input at process 804.

[0102] In one or more embodiments, the simulation system 200 allows the rendering of the hybrid simulation model 210 to adapt based on updated dynamic simulation information 230 related to or corresponding to the current step and / or the next step(s) of the medical procedure 202. For example, when the simulation 228 is at the current step in the medical procedure 202, the simulation system 200 can use the dynamic simulation information 230 to make predictions about how to render or visualize the hybrid simulation model 210 in one or more subsequent steps. As a result, the simulation system 200 can render the portions of the hybrid simulation model 210 corresponding to the one or more next steps more quickly and efficiently once those steps are reached.

[0103] At optional process 810, an output system may be optionally used to provide feedback. The output system may include, for example, a display system 208, an audio device, a tactile feedback system, a lighting system, or a combination thereof. The audio device may include, but is not limited to, for example, a speaker system, earphones, (one or more) earplugs, some other types of audio devices, or a combination thereof. Feedback may be provided to help a user (e.g., a surgeon or other clinician) during simulation 228. For example, audio feedback may be used to indicate when an impassable boundary in the patient's anatomy 222 is reached or when a hybrid simulation model 210 and / or dynamic simulation information 230 indicates that an adverse effect or reaction is being experienced or will be experienced. In some cases, tactile feedback may be used to provide a more realistic experience during simulation 228. For example, tactile feedback may be used to simulate contact with a hardened block in the patient's anatomy 222.

[0104] In some cases, at process 810, visual feedback may include textual detailed instructions for executing the current step and / or the next step of medical procedure 202. In some embodiments, visual feedback includes labels, distance information, positioning information, simulated physiological parameter information, or other types of information. Distance information may include, for example, but not limited to, the distance between the tip of the probe and the tissue of the patient 203. Positioning information may include, for example, but not limited to, the patient's posture, the positioning of an instrument or tool, or some other type of positioning information. Simulated physiological parameter information may include, for example, simulated body temperature, simulated blood pressure, simulated heart rate, simulated brain waves, or some other type of simulated physiological parameter values.

[0105] Thus, embodiments provide a simulation system 200 and process for generating a hybrid simulation model 210 customized for a patient 203 and a medical procedure 202 to be performed on the patient 203 for use in a simulated medical procedure 202. The hybrid simulation model 210 is created in part from patient-specific data 214 and in part from a generic model 219 of a representative patient. In one or more embodiments, the patient-specific data 214 can be used to scale the generic model 219 and deform only the regions of interest in the generic model 219 that will interact with or be affected by the instruments and / or agents and gravity used to perform the medical procedure 202. In addition, the hybrid simulation model 210 can be dynamically adjusted based on dynamic simulation information 230 obtained from a simulation of running the medical procedure 202. The resulting hybrid simulation model 210 enables a high-fidelity virtual reality simulation of the medical procedure 202 to be performed. This type of hybrid simulation model 210, which focuses only on the anatomical structures of the patient anatomy 222 of the patient 203 that will interact with or be affected by the instruments and / or agents of the medical procedure 202, can be of reduced size compared to a model that captures the entirety of the patient anatomy 222 with high fidelity. Additionally, this type of hybrid simulation model 210 can increase the efficiency with which training simulations for performing the medical procedure 202 can be run and stored.

[0106] One or more elements in the embodiments of the present invention may be implemented in software to be executed on a processor of a computer system such as a control processing system. When implemented in software, the elements of the embodiments of the present invention are essentially code segments that perform necessary tasks. The program or code segment may be stored in a processor-readable storage medium or device, which may be downloaded via a computer data signal embodied in the form of a carrier wave on a transmission medium or communication link. The processor-readable storage device may include any medium that can store information, including optical media, semiconductor media, and magnetic media. Examples of processor-readable storage devices include electronic circuits; semiconductor devices, semiconductor storage devices, read-only memories (ROMs), flash memories, erasable programmable read-only memories (EPROMs); floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. The code segments may be downloaded via a computer network (e.g., the Internet, an intranet, etc.).

[0107] Note that the processes and displays presented may not be inherently related to any particular computer or other device. Various general purpose systems may be used with programs according to the teachings herein, or it may prove convenient to construct more specialized devices to perform the described operations. The structures required for various of these systems will appear as elements in the claims. In addition, embodiments of the present invention are described without reference to any particular programming language. It should be understood that various programming languages ​​may be used to implement the teachings of the present invention as described herein.

[0108] Although certain exemplary embodiments of the present invention have been described and shown in the accompanying drawings, it should be understood that such embodiments are merely exemplary and not limiting of the present invention, and that since various other modifications may be made by a person skilled in the art, the embodiments of the present invention are not limited to the specific constructions and arrangements shown and described.

Claims

1. A method for simulating a medical procedure to be performed on a patient, the method comprising: receiving patient-specific data specific to the patient on whom the medical procedure is to be performed; modifying a generic model using the patient-specific data to generate a hybrid simulation model customized for the patient; as well as Prior to the medical procedure, a simulation of the medical procedure is performed using the hybrid simulation model and dynamic simulation information corresponding to at least one of the medical procedure or the patient, wherein the simulation is customized for both the patient and the medical procedure.

2. The method of claim 1 , wherein performing the simulation comprises: A first portion of the hybrid simulation model is rendered at a higher fidelity level than a second portion of the hybrid simulation model, the first portion representing patient anatomy of the patient that interacts with the medical procedure, and the second portion representing patient anatomy that does not interact with the medical procedure at any step of the medical procedure.

3. The method of claim 1 , wherein performing the simulation comprises: The medical procedure is dynamically simulated using the hybrid simulation model and the dynamic simulation information such that the patient anatomy of the patient is represented with a gradient of fidelity levels, wherein as the distance from a surgical path of the medical procedure increases, the fidelity level of the anatomical structure in the hybrid simulation model decreases during the simulation.

4. The method of claim 1 , wherein performing the simulation comprises: Based on the dynamic simulation information, a visualization of the hybrid simulation model is adjusted during the simulation to account for the effects of the simulated interaction between the anatomy and the instrument.

5. The method of claim 1 , wherein performing the simulation comprises: rendering the hybrid simulation model to provide a visualization of the hybrid simulation model on a display system; receiving input during said simulation; as well as The hybrid simulation model and the dynamic simulation information are updated based on the input.

6. The method of claim 5, wherein performing the simulation further comprises: Based on the updated hybrid simulation model and the updated dynamic simulation information, the visualization of the hybrid simulation model is controlled during the simulation of the medical procedure.

7. The method according to claim 1, wherein the dynamic simulation information includes physiological parameter data, instrument positioning information, field of view information and patient posture information.

8. The method of claim 1 , wherein the patient-specific data comprises patient diagnostic data, the patient diagnostic data comprising at least one of preoperative imaging data, physical examination data, the geometry of a target region set, or the geometry of anatomical structures adjacent to the target region set.

9. The method of claim 1, wherein the patient-specific data comprises anthropometric data including height measurements, weight measurements, and anatomical landmark measurements.

10. The method according to claim 1, further comprising: The general model is selected from a database system, wherein the general model is a general model for a representative patient or a representative group of patients.

11. The method of claim 1 , wherein the generic model is a generic model of a representative patient having anthropometric measurements within a selected range for the patient.

12. The method of claim 1, wherein performing the simulation comprises: During the simulation of the medical procedure, how one or more anatomical structures of the patient's anatomy are deformed is adjusted in response to changes to the patient's posture.

13. The method of claim 1, wherein performing the simulation comprises: A visualization of the hybrid simulation model is adjusted during the simulation to represent changes in positioning of an imaging modality relative to one or more anatomical structures of a patient anatomy of the patient.

14. The method according to claim 1, further comprising: Simulation data is calculated based on the simulation of the medical procedure.

15. The method according to claim 14, further comprising: modifying the hybrid simulation model based on the calculated simulation data to form an updated hybrid simulation model; as well as The dynamic simulation information is modified based on the calculated simulation data to form updated dynamic simulation information.

16. The method according to claim 15, further comprising: A new simulation of the medical procedure is performed using the updated hybrid simulation model and the updated dynamic simulation information.

17. The method of claim 1, wherein using the patient-specific data to modify the generic model comprises: The hybrid simulation model is generated by modifying a representation of each of a plurality of anatomical structures in the generic model along a surgical path along which the medical procedure is to be performed based on the received patient-specific data.

18. A simulation system comprising: A processor configured to: receiving patient-specific data specific to the patient on whom the medical procedure is to be performed; modifying the generic model using the patient-specific data to generate a hybrid simulation model customized for the patient; and Prior to the medical procedure, a simulation of the medical procedure is performed using the hybrid simulation model and dynamic simulation information corresponding to at least one of the medical procedure or the patient, wherein the simulation is customized for both the patient and the medical procedure.

19. A simulation system according to claim 18, wherein the processor is further configured to dynamically simulate the medical procedure using the hybrid simulation model and the dynamic simulation information to represent the patient anatomy of the patient with a gradient of fidelity levels, and wherein as the distance from the surgical path of the medical procedure increases, the fidelity level of the anatomical structure in the hybrid simulation model decreases during the simulation.

20. The simulation system of claim 18, wherein the processor is further configured to adjust visualization of the hybrid simulation model during the simulation based on the dynamic simulation information to account for effects of simulated interactions between anatomical structures and instruments used in the medical procedure.

21. The simulation system according to claim 18, wherein the dynamic simulation information includes physiological parameter data, instrument positioning information, field of view information and patient posture information.

22. A simulation system according to claim 18, wherein the patient-specific data includes patient diagnostic data, and the patient diagnostic data includes at least one of preoperative imaging data, physical examination data, the geometry of the target region set, or the geometry of anatomical structures adjacent to the target region set.

23. The simulation system of claim 18, wherein the patient-specific data comprises anthropometric data including height measurements, weight measurements, and anatomical landmark measurements.

24. The simulation system of claim 18, wherein the processor is further configured to retrieve the generic model from a database system, and wherein the generic model is a generic model of a representative patient or a representative group of patients.

25. The simulation system of claim 18, wherein the generic model is a generic model of a representative patient having anthropometric measurements within a selected range of the patient.

26. The simulation system of claim 18, wherein the processor is further configured to adjust how one or more anatomical structures of the patient anatomy of the patient deform in response to a change in a posture of a patient couch during the simulation of the medical procedure.

27. The simulation system of claim 18, wherein the processor is further configured to adjust a visualization of the hybrid simulation model during the simulation of the medical procedure to represent changes in the position of an endoscope relative to one or more anatomical structures of the patient's anatomy.

28. A method for simulating a medical procedure to be performed on a patient, the method comprising: receiving anthropometric data and patient diagnostic data of the patient; A generic model for selecting representative patients from a database system; modifying the generic model based on the anthropometric data and the patient diagnostic data to generate a hybrid simulation model customized for the patient; as well as prior to the medical procedure, performing a simulation of the medical procedure using the hybrid simulation model and dynamic simulation information corresponding to at least one of the medical procedure or the patient, wherein the simulation is customized to both the patient and the medical procedure; wherein the simulation takes into account the effects of simulated interactions between the patient's anatomy and instruments used in the medical procedure; and Wherein the simulation dynamically adapts to each step in the medical procedure such that at each step in the medical procedure a portion of the patient's anatomy of the patient interacting with the medical procedure is rendered with a higher level of fidelity.

Images