Surgical Simulation Object Modification System

Abstract

A computing device for simulating a surgical procedure may be configured to receive first two-dimensional (2D) image data of a first human organ of a first patient and second 2D image data of a second human organ of a second patient. The computing device may be configured to generate 3D image data based on 3D reconstruction reference data from an aggregation of the first 2D image data and the second 2D image data. The computing device may be configured to map a shape of the deformable 3D model of the simulated human organ to correspond to spatial information of the 3D image data. The computing device may be configured to map a surface of the deformable 3D model of the simulated human organ to correspond to visual information of the 3D image data. The computing device may be configured to output the deformable 3D model.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 191,681, filed on May 21, 2021, the entire disclosure of which is incorporated herein by reference.

[0002] This application is related to the following applications filed simultaneously, the content of each of which is incorporated herein by reference. · U.S. Patent Application No. 17 / 332,594, titled "METHODS FOR SURGICAL SIMULATION", filed on May 27, 2021 (Attorney Docket No. END9338USNP1) · U.S. Patent Application No. 17 / 332,399, titled "SURGICAL SIMULATION NAVIGATION SYSTEM", filed on May 27, 2021 (Attorney Docket No. END9338USNP3) · U.S. Patent Application No. 17 / 332,441, titled "SURGICAL SIMULATION SYSTEM WITH COORDINATED IMAGINING", filed on May 27, 2021 (Attorney Docket No. END9338USNP4) · U.S. Patent Application No. 17 / 332,462, titled "SURGICAL SIMULATION SYSTEM WITH SIMULATED SURGICAL EQUIPMENT COORDINATION", filed on May 27, 2021 (Attorney Docket No. END9338USNP5) · U.S. Patent Application No. 17 / 332,197, titled "SIMULATION - BASED SURGICAL PROCEDURE PLANNING SYSTEM", filed on May 27, 2021 (Attorney Docket No. END9338USNP6) · U.S. Patent Application No. 17 / 332,407, titled "SIMULATION - BASED DIRECTED SURGICAL DEVELOPMENT SYSTEM", filed on May 27, 2021 (Attorney Docket No. END9338USNP7) • U.S. Patent Application No. 17 / 332,449, filed on May 27, 2021, entitled "SURGICAL ADVERSE EVENT SIMULATION SYSTEM" (Agent Reference Number END9338USNP8) • U.S. Patent Application No. 17 / 332,496, filed on May 27, 2021, entitled "SIMULATION-BASED SURGICAL ANALYSIS SYSTEM" (Agent Reference Number END9338USNP9) • U.S. Patent Application No. 17 / 332,480, filed on May 27, 2021, entitled "DYNAMIC ADAPTATION SYSTEM FOR SURGICAL SIMULATION" (Agent Reference Number END9338USNP10) [Background technology]

[0003] For example, surgical simulations, such as computer-based three-dimensional simulations of the surgical environment and / or surgical procedures, present opportunities to advance surgical techniques. Surgical simulations have potential to be useful for surgical training, planning, and development. For instance, surgical simulations can be used to train surgeons in new procedures and / or to improve surgeons' performance of known procedures. Surgical simulations can be used as a virtual "final rehearsal" to help surgeons prepare for the next procedure. Furthermore, surgical simulations can be used for experimentation with unproven procedures and techniques.

[0004] However, surgical simulation platforms are complex systems that face many limitations in terms of capability, scope, and applicability. For example, many platforms are technology "silos" that are specifically programmed and tuned to address specific learning objectives or to simulate the operation of a single instrument, such as the operation of a surgical robot. Such limitations can reduce the effectiveness of the platform as a tool for advancing surgical techniques. Furthermore, such limitations can represent a significant technical obstacle to integrating simulation-based applications into other aspects of the surgical process, such as preoperative planning, intraoperative support, and postoperative analysis. [Overview of the project] [Problems that the invention aims to solve]

[0005] Therefore, innovations in surgical simulation technology, such as technological advancements addressing surgical simulation capabilities, scope, and applicability, can accelerate further progress in the field of surgery. [Means for solving the problem]

[0006] This specification describes systems, methods, and means for surgical simulation object modification. A computing device for simulating a surgical procedure may include a processor. The processor may be configured to receive first two-dimensional (2D) image data of a first human organ of a first patient. The first 2D image data may include first 2D visible image data captured within the body of the first patient and first three-dimensional (3D) reconstruction reference data. The first 2D image data may be captured using structured light imaging. The processor may be configured to receive second 2D image data of a second human organ of a second patient. The second 2D image data may include second 2D visible image data captured within the body of the second patient and second 3D reconstruction reference data. The processor may be configured to generate 3D image data from an aggregation of the first 2D visible image data and the second 2D visible image data, based on first 3D reconstruction reference data and second 3D reconstruction reference data. The 3D image data may include spatial information and visual information. The processor may be configured to map the shape of a deformable 3D model of a simulated human organ to the spatial information of the 3D image data. The processor may be configured to map the surface of a deformable 3D model of a simulated human organ to the visual information of the 3D image data. The processor may be configured to output a deformable 3D model.

[0007] The present invention relates to a computing device, and the computer device is A processor is provided, the processor is configured to: receive first two-dimensional (2D) image data of a first human organ, the first 2D image data including first 2D visible image data and first three-dimensional (3D) reconstruction reference data captured in the body of a first patient; receive second 2D image data of a second human organ, the second 2D image data including second 2D visible image data and second 3D reconstruction reference data captured in the body of a second patient; generate 3D image data from an aggregation of the first 2D visible image data and second 2D visible image data, based on first 3D reconstruction reference data and second 3D reconstruction reference data, the 3D image data including spatial information and visual information; map the shape of a deformable 3D model of the human organ to correspond to the spatial information of the 3D image data; map the surface of the deformable 3D model of the human organ to correspond to the visual information of the 3D image data; and output the deformable 3D model.

[0008] It should be understood that a "deformable 3D model" can refer to any volumetric model containing multiple spatial data points (such as a model of a human organ rendered in 3D). This allows the model to be easily "deformed" (or interacted with to change the shape / size of the model) by modifying the spatial data points within the model.

[0009] "Mapping" the shape / surface of a first model can also be understood as modifying the shape / surface of the first model based on a separate 3D model (i.e., 3D image data). This may involve using point-to-point mapping, where each point on the model can be adjusted to minimize the difference with the separate 3D model (in this case, the generated 3D image data).

[0010] One technical effect of this embodiment is that the accuracy of both the shape and surface details of the 3D model can be improved through the capture and processing of 2D images.

[0011] In some embodiments, the processor is configured to simulate surgical procedures.

[0012] In some embodiments, the 3D model of a human organ is a 3D model of a simulated human organ.

[0013] In some embodiments, the first human organ and the second human organ are human organs of the same type.

[0014] In some embodiments, the first patient is identical to the second patient.

[0015] In some embodiments, 2D image data is captured using structured light imaging.

[0016] In some embodiments, the processor is further configured to receive tissue property data of one or more human organs, the tissue property data being determined by tracking the displacement of at least one or more physical reference markers attached to the human organs, and to deform the deformable 3D model based on the physical property data when a simulated force is applied to the deformable 3D model.

[0017] In some embodiments, the first 2D image data further includes first invisible image data captured inside the body of a first patient, the second 2D image data further includes second invisible image data captured inside the body of a second patient, the processor is further configured to generate 3D image data from an aggregate of the first visible image data and the second visible image data based on the first and second criteria and the first invisible image data and the second invisible image data, the 3D image data further includes non-visual information, and the processor is further configured to generate subsurface data of a deformable 3D model of a human organ based on the non-visual information of the 3D image data.

[0018] In some embodiments, the processor is further configured to receive patient-specific data and to map the surface of a deformable 3D model of a human organ to correspond to the visual information of the 3D image data and the patient-specific data.

[0019] In some embodiments, the processor is further configured to define expected modes of a simulated surgical procedure, wherein the expected modes are one of a surgical step, a job which is part of a surgical step, a location in a surgical scene, or a surgical instrument; to monitor user input data for the simulated surgical procedure; and, when it is determined that the input data corresponds to an expected mode, to present the relevant instructions to the user.

[0020] In some embodiments, the processor is further configured to define expected aspects of a simulated surgical procedure, wherein the expected aspects are one of a surgical step, a job which is part of a surgical step, a location within a surgical site, or a surgical instrument; to monitor user input data for the simulated surgical procedure; and, when it is determined that the input data corresponds to an expected aspect, to present the instructor with the option to initiate an instructional interaction with the user.

[0021] In some embodiments, the processor is further configured to define the expected surgical steps of a simulated surgical procedure, monitor user input data for the simulated surgical procedure, and, when it is determined that the input data corresponds to the expected surgical steps, present the user with the option to view corresponding segments of one or more actual surgical procedure videos, wherein the one or more actual surgical procedure videos include segments indexed to the surgical steps.

[0022] In some embodiments, the option to view corresponding segments of one or more actual surgical procedure videos may be further customized depending on surgical complications or outcomes.

[0023] According to the present invention, a computer-implemented method is also provided, the computer-implemented method comprising receiving first two-dimensional (2D) image data of a first human organ, the first 2D image data including first 2D visible image data and first three-dimensional (3D) reconstruction reference data captured within a first patient; receiving second 2D image data of a second human organ, the second 2D image data including second 2D visible image data and second 3D reconstruction reference data captured within a second patient; generating 3D image data based on the first 3D reconstruction reference data and the second 3D reconstruction reference data from an aggregation of the first 2D visible image data and the second 2D visible image data, the 3D image data including spatial information and visual information; mapping the shape of a deformable 3D model of the human organ to correspond to the spatial information of the 3D image data; mapping the surface of the deformable 3D model of the human organ to correspond to the visual information of the 3D image data; and outputting the deformable 3D model.

[0024] In some embodiments, the 3D model of the human organ is a 3D model of a virtual human organ.

[0025] In some embodiments, the first human organ and the second human organ are of the same type of human organ.

[0026] In some embodiments, the first patient is the same as the second patient.

[0027] In some embodiments, the 2D image data is captured using structured light imaging.

[0028] In some embodiments, the method further includes receiving tissue property data of one or more human organs, the tissue property data being determined by tracking the displacement of physical reference markers attached to at least one or more human organs, and deforming the deformable 3D model based on the physical property data when a simulated force is applied to the deformable 3D model.

[0029] In some embodiments, the first 2D image data further includes first invisible image data captured inside the body of a first patient, the second 2D image data further includes second invisible image data captured inside the body of a second patient, the processor is further configured to generate 3D image data from an aggregate of the first visible image data and the second visible image data based on the first and second criteria and the first invisible image data and the second invisible image data, the 3D image data further includes non-visual information, and the processor is further configured to generate subsurface data of a deformable 3D model of a human organ based on the non-visual information of the 3D image data.

[0030] In some embodiments, the processor is further configured to receive patient-specific data and to map the surface of a deformable 3D model of a human organ to correspond to the visual information of the 3D image data and the patient-specific data.

[0031] In some embodiments, the method further includes defining an expected mode of a simulated surgical procedure, wherein the expected mode is one of a surgical step, a job which is part of a surgical step, a location in a surgical scene, or a surgical instrument; monitoring user input data for the simulated surgical procedure; and, when it is determined that the input data corresponds to an expected mode, presenting the user with the relevant instructions.

[0032] In some embodiments, the method further includes defining an expected mode of a simulated surgical procedure, wherein the expected mode is one of a surgical step, a job which is part of a surgical step, a location within a surgical site, or a surgical instrument; monitoring user input data for the simulated surgical procedure; and, when it is determined that the input data corresponds to an expected mode, presenting the instructor with the option to initiate an instructional interaction with the user.

[0033] In some embodiments, the processor is further configured to define the expected surgical steps of a simulated surgical procedure, monitor input data from the user for the simulated surgical procedure, and, when it is determined that the input data corresponds to the expected surgical steps, present the user with the option to view corresponding segments of one or more actual surgical procedure videos, wherein the one or more actual surgical procedure videos include segments indexed to the surgical steps.

[0034] In some embodiments, the option to view corresponding segments of one or more actual surgical procedure videos may be further customized depending on surgical complications or outcomes. A computer-readable medium is also provided, which, when executed by a computer, contains instructions that cause the computer to perform the method of the present invention. [Brief explanation of the drawing]

[0035] [Figure 1] This is a block diagram of a computer-implemented interactive surgical system. [Figure 2] This shows an exemplary surgical system used to perform surgical procedures in an operating room. [Figure 3] This disclosure illustrates, in at least one aspect, a visualization system, a robotic system, and an exemplary surgical hub paired with an intelligent instrument. [Figure 4]The present disclosure, in at least one aspect, describes a surgical data network having a communication hub configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to the cloud. [Figure 5] This figure shows an exemplary computer-implemented interactive surgical system. [Figure 6] An exemplary surgical hub is shown, including multiple modules connected to a modular control tower. [Figure 7] This is a block diagram of an exemplary surgical simulator system. [Figure 8] This is a block diagram of an exemplary surgical simulator system. [Figure 9] This is a block diagram depicting the user interface device of an exemplary surgical simulator. [Figure 10] This is a flowchart illustrating the operation of an illustrative surgical simulator. [Figure 11A] This document describes the data structure of an exemplary surgical procedure plan for use in a computer-implemented interactive surgical system and / or surgical simulator. [Figure 11B] This document describes the data structure of an exemplary surgical procedure plan for use in a computer-implemented interactive surgical system and / or surgical simulator. [Figure 12] This shows an exemplary data flow of the operation of an exemplary surgical simulator. [Figure 13] This shows an exemplary data flow of the operation of an exemplary surgical simulator. [Figure 14] This is a flowchart illustrating the exemplary actions of an exemplary surgical simulator. [Modes for carrying out the invention]

[0036] Surgical simulation systems, devices, and methods may include forms of integration with other medical devices, data sources, processes, and institutions. For example, surgical simulation systems, devices, and methods may include forms of integration with computer-implemented interactive surgical systems and / or one or more elements of computer-implemented interactive surgical systems.

[0037] Referring to Figure 1, the computer-implemented interactive surgical system 100 may comprise one or more surgical systems 102 and a cloud-based system (for example, a cloud 104 which may include a remote server 113 connected to a storage device 105). Each surgical system 102 may comprise at least one surgical hub 106 that communicates with the cloud 104 which may include a remote server 113.

[0038] One or more simulation devices 103, 111 may communicate with and / or be integrated as part of the computer-implemented interactive surgical system 100. For example, simulation device 103 may be an element of one or more surgical systems 102. For example, simulation device 103 may communicate with one or more surgical hubs 106. For example, simulation device 111 may communicate with the computer-implemented interactive surgical system 100 via the cloud 104.

[0039] In one example, as shown in Figure 1, the surgical system 102 includes a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112, which are configured to communicate with each other and / or with a hub 106. In some embodiments, the surgical system 102 may comprise M hubs 106, N visualization systems 108, O robotic systems 110, and P handheld intelligent surgical instruments 112, where M, N, O, and P are integers of 1 or more.

[0040] In various embodiments, the visualization system 108 may comprise one or more imaging sensors strategically positioned relative to a sterile field, one or more image processing units, one or more storage arrays, and one or more displays, as shown in Figure 2. In one embodiment, the visualization system 108 may include interfaces for HL7, PACS, and EMR. Various components of the visualization system 108 are described under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety.

[0041] As shown in Figure 2, the primary display 119 is positioned in the sterile field so that it is visible to the operator on the operating table 114. In addition, a visualization tower 111 is positioned outside the sterile field. The visualization tower 111 may comprise a first non-sterile display 107 and a second non-sterile display 109, facing opposite directions from each other. The visualization system 108, guided by the hub 106, is configured to utilize displays 107, 109, and 119 to coordinate the flow of information to operators inside and outside the sterile field. For example, the hub 106 can cause the visualization system 108 to display snapshots of the surgical site recorded by the imaging device 124 on the non-sterile displays 107 or 109 while maintaining live video of the surgical site on the primary display 119. The snapshots on the non-sterile displays 107 or 109 allow, for example, a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0042] In one embodiment, the hub 106 can also be configured to send diagnostic input or feedback entered by a non-sterile operator in the visualization tower 111 to a primary display 119 in the sterile field, which can then be viewed by a sterile operator at the operating table. In one example, the input may take the form of modifications to a snapshot displayed on the non-sterile display 107 or 109, which can then be sent to the primary display 119 by the hub 106.

[0043] Referring to Figure 2, the surgical instrument 112 is used as part of the surgical system 102 in a surgical procedure. The hub 106 can also be configured to coordinate the flow of information to the display of the surgical instrument 112. For example, in U.S. Patent Application Publication 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety. Diagnostic input or feedback entered by a non-sterile operator in the visualization tower 111 can be sent by the hub 106 to the surgical instrument display 115 in the sterile field, which can then be viewed by the operator of the surgical instrument 112. Examples of surgical instruments suitable for use in the surgical system 102 are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385) filed December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety.

[0044] Figure 2 shows an example of a surgical system 102 used to perform a surgical procedure on a patient lying on an operating table 114 in an operating room 116. A robotic system 110 may be used as part of the surgical system 102 in a surgical procedure. The robotic system 110 may include a surgeon's console 118, a patient-side cart 120 (surgical robot), and a surgical robot hub 122. While the surgeon views the surgical site through the surgeon's console 118, the patient-side cart 120 may operate at least one detachably connected surgical tool 117 through a minimally invasive incision in the patient's body. Images of the surgical site may be acquired by a medical imaging device 124, which can be operated by the patient-side cart 120 to change the orientation of the imaging device 124. Images of the surgical site may be processed using the robotic hub 122 and then displayed to the surgeon through the surgeon's console 118.

[0045] Other types of robotic systems can be readily adapted for use with surgical system 102. Various examples of robotic systems and surgical tools suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0201137(A1) (U.S. Patent Application No. 16 / 209,407), filed 4 December 2018, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," the disclosure of which is incorporated herein by reference in its entirety.

[0046] Various examples of cloud-based analytical methods implemented by Cloud104 and suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1) (U.S. Patent Application No. 16 / 209,403), filed 4 December 2018, entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," the disclosure of which is incorporated herein by reference in its entirety.

[0047] In various embodiments, the imaging device 124 may include at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0048] The optical components of the imaging device 124 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors can receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0049] One or more illumination sources may be configured to emit electromagnetic energy in the visible and invisible spectra. The visible spectrum, sometimes also called the light spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye), and is sometimes called visible light or simply light. The typical human eye responds to wavelengths in air from approximately 380 nm to approximately 750 nm.

[0050] The invisible spectrum (e.g., the non-emission spectrum) is a portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths below approximately 380 nm and above approximately 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths above approximately 750 nm are longer than the red visible spectrum and consist of invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths below approximately 380 nm are shorter than the violet spectrum and consist of invisible ultraviolet, X-rays, and gamma-ray electromagnetic radiation.

[0051] In various embodiments, the imaging device 124 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, duodenoscopes, intestinaloscopes, esophagogastroduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes.

[0052] The imaging device may employ multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within a specific wavelength range from the entire electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, such as IR and ultraviolet light. Spectral imaging makes it possible to extract additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in more detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after the surgical task is completed to perform one or more of the tests described above on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical instruments is required in any surgical procedure. The strict sanitary and sterilization conditions required in the “operating room,” i.e., the operating room or treatment room, require the highest possible sterility of all medical devices and instruments. Part of the sterilization process is the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device 124 and its accessories and components. It will be understood that the sterile field may be considered a specific area that is deemed to be free of microorganisms, such as inside a tray or on a sterile towel, or the sterile field may be considered the area immediately surrounding the patient who is ready for surgical treatment. The sterile field may include cleaned team members wearing appropriate clothing, as well as all equipment and fixation devices within that area.

[0053] Referring here to Figure 3, a hub 106 is shown that communicates with a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112. The hub 106 includes a hub display 135, an imaging module 138, a generator module 140, a communication module 130, a processor module 132, a storage array 134, and an operating room mapping module 133. In certain embodiments, as shown in Figure 3, the hub 106 further includes a fume extraction module 126 and / or aspiration / irrigation module 128. During surgical procedures, applying energy to tissue for sealing and / or cutting is generally associated with fume extraction, aspiration of excess fluid, and / or tissue irrigation. Fluid lines, power lines, and / or data lines from different sources often become entangled during surgical procedures. Dealing with this problem during surgical procedures can result in the loss of valuable time. Untangling lines may require disconnecting them from their corresponding modules, which may require resetting the modules. The hub modular enclosure 136 provides an integrated environment for managing power lines, data lines, and fluid lines, reducing the frequency of entanglement between such lines. An aspect of the present disclosure presents a surgical hub for use in surgical procedures involving the application of energy to tissue at a surgical site. The surgical hub includes a hub enclosure and a combination generator module slidably receivable within a docking station of the hub enclosure. The docking station includes data contacts and power contacts. The combination generator module includes two or more ultrasonic energy generator components, bipolar RF energy generator components, and unipolar RF energy generator components housed in a single unit. In one aspect, the combination generator module also includes a fume exhaust component, at least one energy supply cable for connecting the combination generator module to a surgical instrument, at least one fume exhaust component configured to exhaust smoke, fluid, and / or particulate matter generated by the application of therapeutic energy to tissue, and a fluid line extending from a remote surgical site to the fume exhaust component.In one embodiment, the above-mentioned fluid line is a first fluid line, and a second fluid line extends from the remote surgical site to a suction and irrigation module slidably received within the hub enclosure. In one embodiment, the hub enclosure comprises a fluid interface. Certain surgical procedures may require the application of two or more energy types to tissue. One energy type may be more beneficial for cutting tissue, while another different energy type may be more beneficial for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Embodiments of the present disclosure present a solution in which a modular enclosure 136 of the hub is configured to house various generators and facilitate interactive communication between them. One advantage of the modular enclosure 136 of the hub is that it allows for the rapid removal and / or replacement of various modules. Embodiments of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking station having a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and the first energy generator module is slidably movable to disengage from the first power and data contacts. In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy for application to tissue, different from the first energy, and a second docking station having a second docking port including second data contacts and second power contacts, wherein the second energy generator module is slidably movable to electrically engage with the power and data contacts, and the second energy generator module is slidably movable to disengage from the second power and second data contacts.In addition, the modular surgical enclosure also includes a communication bus between a first docking port and a second docking port, configured to facilitate communication between a first energy generator module and a second energy generator module. Referring to Figure 3, an aspect of the present disclosure is presented relating to a hub modular enclosure 136 that enables modular integration of a generator module 140, a smoke evacuation module 126, and a suction / irrigation module 128. The hub modular enclosure 136 further facilitates interactive communication between modules 140, 126, and 128. The generator module 140 may be a generator module comprising integrated unipolar, bipolar, and ultrasonic components, supported within a single housing unit that is slidably inserted into the modular enclosure 136 of the hub. The generator module 140 may be configured to connect to a unipolar device 142, a bipolar device 144, and an ultrasonic device 146. Alternatively, the generator module 140 may comprise a series of unipolar generator modules, bipolar generator modules, and / or ultrasonic generator modules interacting via a hub modular enclosure 136. The hub modular enclosure 136 can be configured to facilitate the insertion of multiple generators and bidirectional communication between generators docked to the hub modular enclosure 136, so that multiple generators function as a single generator.

[0054] Figure 4 shows a surgical data network 201 comprising a modular communication hub 203 configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to a cloud-based system (a cloud 204 which may include, for example, a remote server 213 connected to a storage device 205). In one embodiment, the modular communication hub 203 comprises a network hub 207 and / or a network switch 209 that communicate with a network router. The modular communication hub 203 can also be coupled to a local computer system 210 to provide local computer processing and data manipulation. The surgical data network 201 may be configured as passive, intelligent, or switching. A passive surgical data network acts as a data conduit, enabling data to go from one device (or segment) to another device (or segment) and to cloud computing resources. An intelligent surgical data network enables traffic to pass through a monitored surgical data network and includes additional feature units that constitute each port in the network hub 207 or network switch 209. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0055] Modular devices 1a-1n, located in the operating room, can be connected to a modular communication hub 203. A network hub 207 and / or a network switch 209 can be connected to a network router 211 to connect devices 1a-1n to a cloud 204 or a local computer system 210. Data associated with devices 1a-1n can be transferred to a cloud-based computer via the router for remote data processing and manipulation. Data associated with devices 1a-1n can also be transferred to a local computer system 210 for local data processing and manipulation. Modular devices 2a-2m, located in the same operating room, can also be connected to a network switch 209. The network switch 209 can be connected to a network hub 207 and / or a network router 211 to connect devices 2a-2m to the cloud 204. Data associated with devices 2a-2n may be transferred to the cloud 204 via the network router 211 for data processing and manipulation. Data associated with devices 2a-2m can also be transferred to a local computer system 210 for local data processing and manipulation.

[0056] It will be understood that the surgical data network 201 can be expanded by interconnecting multiple network hubs 207 and / or multiple network switches 209 with multiple network routers 211. A modular communication hub 203 may be housed in a modular control tower configured to accept multiple devices 1a-1n / 2a-2m. A local computer system 210 may also be housed in the modular control tower. The modular communication hub 203 is connected to a display 212 to display images acquired by some of the devices 1a-1n / 2a-2m, for example, during a surgical procedure. In various embodiments, the devices 1a-1n / 2a-2m may include a variety of modules, particularly among modular devices that can be connected to the modular communication hub 203 of the surgical data network 201, such as an imaging module 138 connected to an endoscope, a generator module 140 connected to an energy-based surgical device, a smoke extraction module 126, a suction / irrigation module 128, a communication module 130, a processor module 132, a storage array 134, a surgical device connected to a display, and / or a non-contact sensor module.

[0057] In one embodiment, the surgical data network 201 may include a combination of a network hub, network switches, and network routers connecting devices 1a-1n / 2a-2m to the cloud. One or all of the devices 1a-1n / 2a-2m connected to the network hub or network switch can collect data in real time and transfer the data to a cloud computer for data processing and manipulation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to handle software applications. The term “cloud” may be used as a metaphor for “Internet,” but the term is not limited in that way. Thus, the term “cloud computing” may be used herein to refer to “one type of Internet-based computing,” in which various services such as servers, storage, and applications are delivered to a modular communication hub 203 and / or computer system 210 located in an operating room (e.g., a fixed, mobile, temporary, or on-site operating room or space), and to devices connected to the modular communication hub 203 and / or computer system 210 via the Internet. Cloud infrastructure may be maintained by a cloud service provider. In this context, the cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located in one or more operating rooms. Cloud computing services can perform numerous calculations based on data collected by smart surgical instruments, robots, and other computerized devices located in the operating room. Hub hardware enables multiple devices or connections to connect to a computer that communicates with cloud computing resources and storage.

[0058] By applying cloud computing data processing technology to data collected by devices 1a-1n / 2a-2m, the surgical data network can provide improved surgical outcomes, reduced costs, and improved patient satisfaction. At least some of devices 1a-1n / 2a-2m can be used to observe the condition of tissue after tissue sealing and cutting procedures to assess leakage or perfusion of the sealed tissue. At least some of devices 1a-1n / 2a-2m can be used to examine data, including images of body tissue samples, for diagnostic purposes using cloud-based computing to identify pathologies such as the effects of disease. Such data may include tissue localization and margin confirmation, as well as phenotype. At least some of devices 1a-1n / 2a-2m can be used to identify anatomical structures of the body using various sensors integrated with imaging devices and techniques such as overlaying images captured by multiple imaging devices. Data collected by devices 1a-1n / 2a-2m, including image data, may be transferred to the cloud 204 or the local computer system 210, or both, for data processing and manipulation, including image processing and manipulation. The data may be analyzed to improve the outcome of surgical procedures by determining whether further treatments, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robots, can be performed on tissue-specific sites and conditions. Such data analysis may further involve prognostic analysis processing, and the use of standardized methods can provide useful feedback for either confirming surgical treatment and surgeon behavior, or suggesting modifications to surgical treatment and surgeon behavior.

[0059] The operating room devices 1a-1n may be connected to the modular communication hub 203 via a wired or wireless channel, depending on the configuration of the devices 1a-1n with respect to the network hub. In one embodiment, the network hub 207 may be implemented as a local network broadcast device operating on the physical layer of the Open System Interconnection (OSI) model. The network hub can provide connectivity to the devices 1a-1n located within the same operating room network. The network hub 207 may collect data in packet form and transmit them to the router in half-duplex mode. The network hub 207 cannot store any media access control / Internet protocol (MAC / Internet Protocol, IP) for transferring device data. Only one of the devices 1a-1n can transmit data through the network hub 207 at a time. The network hub 207 cannot have a routing table or intelligence regarding the destination of information and broadcasts all network data to each connection and to the remote server 213 (Figure 4) on the cloud 204. While the Network Hub 207 can detect basic network errors such as collisions, broadcasting all information to multiple ports poses a security risk and could cause bottlenecks.

[0060] Operating room devices 2a-2m can be connected to network switch 209 via a wired or wireless channel. Network switch 209 operates within the data link layer of the OSI model. Network switch 209 may be a multicast device for connecting devices 2a-2m located within the same operating room to a network. Network switch 209 can transmit data in the form of frames to network router 211 and operates in full-duplex mode. Multiple devices 2a-2m can transmit data simultaneously through network switch 209. Network switch 209 stores and uses the MAC addresses of devices 2a-2m to transfer data.

[0061] The network hub 207 and / or network switch 209 may be connected to the network router 211 to connect to the cloud 204. The network router 211 operates within the network layer of the OSI model. The network router 211 creates a route for sending data packets received from the network hub 207 and / or network switch 211 to cloud-based computing resources for further processing and manipulation of data collected by one or all of the devices 1a-1n / 2a-2m. The network router 211 may be used to connect two or more different networks located in different locations, for example, different networks located in different operating rooms of the same medical facility or different networks located in different operating rooms of different medical facilities. The network router 211 can send data in packet form to the cloud 204 and operates in full-duplex mode. Multiple devices can send data simultaneously. The network router 211 uses IP addresses to transfer data.

[0062] In one embodiment, the network hub 207 may be implemented as a USB hub that enables the connection of multiple USB devices to a host computer. The USB hub can extend a single USB port into several layers so that there are more ports available for connecting devices to the host system computer. The network hub 207 may include wired or wireless functionality for receiving information via a wired or wireless channel. In one embodiment, a wireless USB short-range high-bandwidth wireless communication protocol may be used for communication between devices 1a-1n and devices 2a-2m located in the operating room.

[0063] In this example, the operating room devices 1a-1n / 2a-2m can exchange data over short distances from fixed and mobile devices (using short-wavelength UHF radio waves in the 2.4-2.485 GHz ISM band) and communicate with the modular communication hub 203 via the Bluetooth wireless technology standard to establish a personal area network (PAN). The surgical site devices 1a-1n / 2a-2m may communicate with the modular communication hub 203 via several wireless or wired communication standards or protocols, including, but not limited to, Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, new radio (NR), long-term evolution (LTE), and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, and their Ethernet derivatives, as well as any other wireless and wired protocols designated as 3G, 4G, 5G, and later. The computing module may include multiple communication modules. For example, the first communication module may be dedicated to shorter-range wireless communication such as Wi-Fi and Bluetooth, while the second communication module may be dedicated to longer-range wireless communication such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, and Ev-DO.

[0064] The modular communication hub 203 can function as a central connection point for one or all of the operating room devices 1a-1n / 2a-2m and can handle a data type known as a frame. Frames can carry data generated by devices 1a-1n / 2a-2m. When a frame is received by the modular communication hub 203, it is amplified and transmitted to the network router 211, which then transfers this data to cloud computing resources using a number of wireless or wired communication standards or protocols as described herein.

[0065] The modular communication hub 203 can be used as a standalone device or connected to compatible network hubs and network switches to form a larger network. Because the modular communication hub 203 is generally easy to install, configure, and maintain, it can be a good choice for networking operating room devices 1a-1n / 2a-2m.

[0066] Figure 5 shows a computer-implemented interactive surgical system 200. The computer-implemented interactive surgical system 200 is similar in many respects to the computer-implemented interactive surgical system 100. For example, the computer-implemented interactive surgical system 200 includes one or more surgical systems 202 that are similar in many respects to the surgical system 102. Each surgical system 202 includes at least one surgical hub 206 that communicates with a cloud 204 which may include a remote server 213. In one embodiment, the computer-implemented interactive surgical system 200 includes a modular control tower 236 connected to a plurality of operating room devices, such as intelligent surgical instruments, robots, and other computerized devices located in the operating room. As shown in Figure 6, the modular control tower 236 includes a modular communication hub 203 connected to the computer system 210.

[0067] As shown in the embodiment of Figure 5, the modular control tower 236 may be connected to an imaging module 238 which can be connected to an endoscope 239, a generator module 240 which can be connected to an energy device 241, a fume exhaust module 226, a suction / irrigation module 228, a communication module 230, a processor module 232, a storage array 234, a smart device / instrument 235 optionally connected to a display 237, and a non-contact sensor module 242. Operating room devices may be connected to cloud computing resources and data storage via the modular control tower 236. The robot hub 222 may also be connected to the modular control tower 236 and cloud computing resources. In particular, the device / instrument 235 and the visualization system 208 may be connected to the modular control tower 236 via wired or wireless communication standards or protocols as described herein. The modular control tower 236 may be connected to a hub display 215 (e.g., a monitor, screen) to display and overlay images received from the imaging module, device / instrument display and / or other visualization systems 208. The hub display may also display data received from devices connected to the modular control tower, along with images and superimposed images.

[0068] Figure 6 shows a surgical hub 206 comprising multiple modules connected to a modular control tower 236. The modular control tower 236 may comprise a modular communication hub 203, such as a network connectivity device, and a computer system 210, such as for local processing, visualization, and imaging. As shown in Figure 6, the modular communication hub 203 may be connected in a hierarchical configuration to expand the number of modules (e.g., devices) that may be connected to the modular communication hub 203, and data associated with the modules may be transferred to the computer system 210, cloud computing resources, or both. As shown in Figure 6, each network hub / switch within the modular communication hub 203 may include three downstream ports and one upstream port. The upstream network hub / switch may be connected to a processor to provide communication connectivity to cloud computing resources and local displays 217. Communication to the cloud 204 can be done via either a wired communication channel or a wireless communication channel.

[0069] The surgical hub 206 may utilize a non-contact sensor module 242 to measure the dimensions of the operating room and generate a map of the operating room using either an ultrasonic non-contact measuring device or a laser-type non-contact measuring device. The ultrasonic-based non-contact sensor module can scan the operating room by transmitting bursts of ultrasound and receiving echoes as they bounce off the surrounding walls of the operating room, as described under the heading "Surgical Hub Spatial Awareness Within an Operating Room" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on December 4, 2018, which is incorporated herein by reference in its entirety, and the sensor module is configured to determine the size of the operating room and adjust the Bluetooth pairing distance limits. A laser-based non-contact sensor module can, for example, scan an operating room by transmitting laser light pulses, receive the laser light pulses reflected from the outer walls of the operating room, compare the phase of the transmitted pulses with the received pulses to determine the size of the operating room, and adjust the Bluetooth pairing distance limit.

[0070] The computer system 210 may include a processor 244 and a network interface 245. The processor 244 may be connected via a system bus to a communication module 247, a storage device 248, a memory 249, a non-volatile memory 250, and an input / output interface 251. The system bus may be any of several types of bus structures, including memory buses or memory controllers, peripheral buses or external buses, and / or local buses, using any variety of available bus architectures. Examples of such architectures include, but are not limited to, 9-bit buses, Industrial Standard Architecture (ISA), Micro-Charmel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association (PCMCIA), Small Computer Systems Interface (SCSI), or any other proprietary bus.

[0071] The processor 244 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer to improve performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse-width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels. Further details are available in the product datasheet.

[0072] In one embodiment, the processor 244 may include a safety controller, including two controller-based families such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured, in particular, specifically for IEC61508 and ISO26262 safety limit applications, to provide a highly integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0073] System memory can be categorized into volatile and non-volatile memory. The Basic Input / Output System (BIOS), which includes basic routines for transferring information between elements within the computer system during startup, is stored in non-volatile memory. Examples of non-volatile memory include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Examples of volatile memory include random access memory (RAM), which functions as external cache memory. Furthermore, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), sync-link DRAM (SLDRAM), and direct rhombus RAM (DRRAM).

[0074] The computer system 210 may also include removable / non-removable volatile / non-volatile computer storage media, such as disk storage. Examples of disk storage devices include, but are not limited to, magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, the disk storage device may include the above-mentioned storage media independently or in combination with other storage media. Examples of other storage media include, but are not limited to, optical disk drives such as compact disk ROM devices (CD-ROMs), compact disk recordable drives (CD-R drives), compact disk rewritable drives (CD-RW drives), or digital multi-purpose disk ROM drives (DVD-ROMs). Removable or non-removable interfaces may be used to facilitate connection of the disk storage device to the system bus.

[0075] It should be understood that the computer system 210 may include software that acts as an intermediary between the described user and basic computer resources in a suitable operating environment. Such software may include an operating system. An operating system, which may be stored on disk storage, may function to control and allocate the resources of the computer system. System applications may leverage resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein can be implemented using various operating systems or combinations of operating systems.

[0076] The user can input commands or information to the computer system 210 via input devices connected to the I / O interface 251. Examples of input devices include, but are not limited to, pointing devices such as mice, trackballs, styluses, and touchpads; keyboards; microphones; joysticks; gamepads; satellite receivers; scanners; TV tuner cards; digital cameras; digital video cameras; and webcams. These and other input devices connect to the processor via interface ports and the system bus. Examples of interface ports include serial ports, parallel ports, game ports, and USB ports. Output devices use some of the same types of ports as the input devices. Therefore, for example, a USB port may be used to provide input to the computer system and output information from the computer system to the output device. Output adapters may be provided to indicate that some output devices, such as monitors, displays, speakers, and printers, may be present, among others, which may require special adapters. Examples of output adapters include, but are not limited to, video and sound cards that provide means of connection between the output device and the system bus. Note that other devices and / or systems of devices, such as remote computers, can provide both input and output functions.

[0077] Computer system 210 may operate in a networked environment using logical connections to one or more remote computers, such as cloud computers, or to local computers. Remote cloud computers may be personal computers, servers, routers, network PCs, workstations, microprocessor-based devices, peer devices, or other common network nodes, but typically include many or all of the elements described in relation to computer systems. For brevity, only memory storage devices are shown along with remote computers. Remote computers may be logically connected to the computer system via a network interface, and subsequently physically connected via a communication interface. Network interfaces may encompass communication networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies include fiber optic distributed data interfaces (FDDI), copper distributed data interfaces (CDDI), Ethernet / IEEE 802.3, and Token Ring / IEEE 802.5. WAN technologies include, but are not limited to, point-to-point links, integrated service digital networks (ISDN) and their variations, packet-switched networks, and digital subscriber lines (DSL).

[0078] In various embodiments, the computer system 210 in Figure 6, the imaging module 238 in Figures 5 and 6, and / or the visualization system 208, and / or the processor module 232 may include an image processor, an image processing engine, a media processor, or any dedicated digital signal processor (DSP) used for processing digital images. The image processor can be made faster and more efficient by using parallel computing with single-instruction multiple data (SIMD) or multiple-instruction multiple data (MIMD) techniques. The digital image processing engine can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0079] A communication connection(s) may refer to hardware / software used to connect a network interface to a bus. For illustrative purposes only, the communication connection(s) are shown as being inside the computer system, but they may be outside the computer system 210. For illustrative purposes only, hardware / software required for connecting to a network interface may include internal and external technologies such as standard telephone-grade modems, cable modems and DSL modems, ISDN adapters and Ethernet cards.

[0080] Figure 7 is a block diagram of an exemplary surgical simulator system. The surgical simulator system may include a simulation device 30000. The surgical simulator system may also include an application creation device 30002, a human interface device 30004, a surgeon agent device 30006, and / or a surgical data system 30008.

[0081] The simulation device 30000 may provide core simulation functions. For example, loading / running one or more simulations, receiving and processing user control information inputs, generating and transmitting visual, auditory, and / or tactile information outputs, collecting simulation operation and activity information, and primary simulation cycle processing may be performed by the simulation device 30000.

[0082] The application creation device 30002 may provide simulation creation functionality. Individual simulation applications may be stored in the simulation device 30000 as application modules 30010. Application modules 30010 may be created, modified, and / or deleted by the application creation device 30002. Application modules 30010 may contain computer-readable and / or executable instructions that direct the operation of the simulation device 30000. For example, application modules 30010 may contain any file type suitable for storing information and running surgical simulations, such as simulation scripts, programming code, structural data files such as Extensible Markup Language (XML) files, and database files.

[0083] The application creation device 30002 may include a graphical user interface with a control unit for creating application modules 30010. The application creation device 3002 can communicate with the simulation device 30000 to search for, modify, and / or load application modules 30010 for simulation operation. For example, the graphical user interface may include an interface structure that allows the user to select simulation activities, input various simulation parameters, set simulation objectives, and verify the execution of the simulation. The application creation device 30002 may be provided as a standalone device and / or integrated with one or more other devices of a surgical simulation system, such as being integrated with the simulation device 30000.

[0084] The human interface device 30004 may include any hardware, software, and / or combination thereof that enables a human user to interact with a simulation provided by the simulation device 30000. The human interface device 30004 may enable the user to provide control inputs to the simulation device 300000 and / or receive output information (such as visual, auditory, and / or tactile information) from the simulation device 30000. In one embodiment, the human interface device 30004 may include a conventional desktop computer.

[0085] The human interface device 30004 may include suitable physical equipment. For example, the human interface device 30004 may include physical equipment that physically and / or virtually mimics aspects of surgical procedures. For example, such equipment could include a desktop unit, a partial task virtual reality unit, a high-fidelity virtual reality unit, a high-fidelity full-size patient unit, a suite unit, a high-fidelity full operating room unit, a full physical virtual reality unit, or a surgical robot console unit. For example, the human interface device 30004 may include devices such as a computer-based simulator interface disclosed by Gallager et al, "Simulations for Procedural Training," Fundamentals of Surgical Simulation, Principles and Practice, Springer (2012).

[0086] Human interface devices 30004 may include physical devices that physically and / or virtually mimic surgical instruments. For example, human interface devices 30004 may include physical devices that mimic surgical instruments, appliances, and consumables such as access devices such as trocars, hand access ports, pneumoperitoneum needles, and guide sheaths; auxiliary hemostatic agents such as patches, gelatin, and powders; craniofacial instruments such as stretchers and plates; balloons and inflators; catheters such as diagnostic catheters, access catheters, vascular catheters, and therapeutic catheters; energy sealing and dissection devices such as tissue sealers, shears, blades, and forceps; and pull-down wires, compression screws, plates, etc. This may include orthopedic instruments such as plants, drills, burs, rods, and connectors; ligators such as incision and endoscopic clip applicators; microwave ablation equipment; auxiliary endoscopic instruments such as drains, sutures, ligatures, needle holders, retrievers, and suture clips; surgical stapling equipment such as open staplers, endoscopic staplers, cutter staplers, electric staplers, circular staplers, vascular staplers, linear staplers, staple cartridges, and staple line reinforcement applicators; wound closure materials such as sutures, adhesives, needles, and knotless tissue control devices; and imaging devices such as minimally invasive imaging devices. For example, human interface device 30004 may include a virtual reality handheld controller that, when operated with a virtual reality headset, mimics surgical instruments, appliances, and consumables such as those disclosed above.

[0087] The human interface device 30004 may include a display that conveys a visual representation of the simulation to the user. The human interface device 30004 may include a computer display. The human interface device 30004 may include a virtual reality headset display. For example, a virtual reality headset display may be used to display a surgical environment, such as the one disclosed in Figure 2 of this specification. A user having such a virtual reality headset display may view and / or interact with any of the elements in the surgical room 116, such as the patient, the robotic system 110, the surgeon's console 118, the surgical robot hub 122, one or more surgical tools 117, the imaging device 124, the patient-side cart 120, and one or more displays 119, 107, 109.

[0088] The human interface device 30006 can present visual information representing the surgeon's perspective. The human interface device 30006 can present visual information from simulated imaging devices such as arthroscopes, angioscopes, bronchoscopes, coredocoscopes, conoscopes, cytoscopes, duodenoscopy, enteroscopes, upper gastrointestinal endoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngo-neproscopes, sigmoidoscopy, thoracoscopy, ureteroscopes, and their related instruments and control units. The human interface device 30006 can also present visual information from simulated auxiliary intraoperative imaging equipment such as computed tomography (CT) units, magnetic resonance imaging (MRI) units, image-guided surgery units, and intraoperative ultrasound units; and fluoroscopy units. Such perspective visual information, surgical imaging information, and supplemental intraoperative imaging information can be displayed to the user in any combination suitable for the operation of the simulation. For example, such information may be presented to the user as a single full-screen view, a tiled window view, a picture-in-picture view, or it may be registered in a simulated display unit of a virtual reality view.

[0089] The human interface device 30004 may include a physical and / or virtual reality surgical robotic surgeon console. For example, an exemplary surgeon console-like human interface device 30004 may include a display, such as a stereoscopic display, and control inputs, such as a handheld manipulator and foot pedals. For example, the surgeon console-like human interface device 30004 may include the interface of the surgeon's console 118 disclosed herein. The human interface device 30004 may enable voice control, for example, by a microphone and voice recognition function. The human interface device 30004 may provide audible feedback, for example, by a speaker. The human interface device 30004 may provide tactile feedback, for example, by vibration, force feedback, vortex rings, and ultrasonic techniques.

[0090] When implemented, the human interface device 30004 may be provided as a standalone device and / or may be integrated with one or more other devices of the surgical simulation system, such as being integrated with the simulation device 30000. The simulation device 30000 may include an interface module 30012 for communicating with the human interface device 30004. In one embodiment, the human interface device 30004 may be integrated with one or more elements of the computer-implemented interactive surgical system 100. For example, the human interface device 30004 may be integrated with the computer system 210. For example, the human interface device 30004 may be integrated with the hub 106. For example, the human interface device 30004 may be integrated with the visualization system 108. The interface module 30012 may communicate with one or more elements of the computer-implemented interactive surgical system 100, for example, via the surgical data system interface module 30014.

[0091] In one embodiment, two or more human interface devices 30004 can operate simultaneously with the simulation device 30000. For example, in a multi-person simulation application.

[0092] The surgical agent device 30006 may include any hardware and / or software suitable for providing computer-based control and response to the inputs and outputs of the simulation device 30000. The surgical agent device 30006 may include computer processing that mimics human input to the simulation device 30000. For example, the surgical agent device 30006 may be capable of recording and registering control inputs, such as basic instrument operations. The surgical agent device 30006 may include computer processing that can access the input / output application programming interface (API) of the simulation device 30000. For example, the API may reveal one or more input / output functions that can be directed according to the surgical agent device 3006. Functions may include fine-grained operation and physical-based input / output functions, such as functions that directly control the position and movement of instruments. Functions may include coarser-grained activity-based input / output functions, such as ligation activities, suturing activities, and stapling activities. The functions may include coarser-grained extrasurgical tasks such as surgical access functions and organ mobilization functions, and / or stage-based input / output functions. Each function may include parameters corresponding to the level of granularity. The parameters may provide specific details for directing the operation of the function within the simulation. The surgical agent 30006 may include functions for generating and manipulating multiple simulation runs. For example, a user may want to estimate the duration of various suturing techniques. The surgical agent device 30006 may be used to plan simulations of any number of different techniques, each of which may be run via the simulation device, and metrics collected by the simulation device may be used to estimate the differences in duration.

[0093] The surgical agent device 30006 may be provided as a standalone device and / or may be integrated with one or more other devices of a surgical simulation system, such as being integrated with the simulation device 30000. The simulation device 30000 may include an interface module 30012 for communicating with the surgical agent device 30006. For example, the surgical agent device 30006 may be integrated as a module of the simulation device 30000. For example, the surgical agent device 30006 may be integrated into the application module 30010 of the simulation device.

[0094] The surgical data system 30008 may include any hardware and / or software suitable for providing external structured surgical information and functions to the simulation device 30000. The surgical data system 30008 may include structures and / or functions described in relation to Figures 1 to 6 of this specification. For example, the surgical data system 30008 may include one or more elements of the computer-implemented interactive surgical system 100. The surgical data system 30008 may include, for example, a surgical hub 106. For example, the simulation device 30000 includes a surgical data system interface module 30014 that enables communication with the surgical hub 106 via the surgical hub's communication module 130. The surgical data system 30008 may include, for example, one or more surgical data repositories. For example, the surgical data system 30008 may include a computer system 210 located in the operating room. For example, the surgical data system 30008 may include a remote server 213 in the cloud 204.

[0095] For example, a surgical data system 30008, such as a surgical hub 106, may provide data to a simulation device 30000 and / or an application creation device 30002. For example, the data may include any surgical data collected and / or generated by the surgical hub 106. Alternatively, for example, the simulation device 30000 may directly receive similar data from any of the networked devices disclosed in Figures 1 to 6. Such data may include, for example, information about live surgical procedures. Such data may include information about past surgical procedures. Such data may include information about future scheduled surgical procedures.

[0096] Information regarding surgical procedures may include information about the patient, staff, planned procedures, procedures performed, and postoperative activities such as patient outcomes. For example, information received and used by a simulation device may include patient records, patient images, models of the patient's anatomical structure, patient test results, and patient history. For example, information received and used by a simulation device may include the staff roster for the procedure, details of past procedures for specific staff members, staff metrics, experience, recent schedules and workloads, and past surgical activities (such as instrument usage statistics and procedure duration). For example, information received and used by a simulation device may include procedure plans, equipment and inventory information, pull lists, checklists, procedure plan analyses, and recommendations. For example, information received and used by a simulation device may include any data collected or generated during a live procedure, such as procedure progress, milestones, patient information, vital signs, operating room setup, staff movements, images, instrument use, surgical techniques, such as those captured by video, manually recorded, and / or inferred from reports by smart devices, such as duration and abnormal event reports. Any data captured during a live procedure can also be stored and made available as past procedures. For example, information received and used by a simulation device may include postoperative records, patient recovery information, and postoperative diagnostic information such as patient outcomes, tests, and images.

[0097] The simulation device 30000 may include any computer or processing platform suitable for running one or more simulations. The simulation may include a computer-modeled surgical procedure environment. For example, the simulation may include a model of the patient's anatomical and / or physiological structure. For example, the simulation may include a model of the actions of one or more healthcare professionals, such as a surgeon, nurse, other physician, or technician, and / or a model of instruments.

[0098] The simulation device 30000 may include one or more additional functional modules. Each module may include hardware, software, or a combination thereof that enables the module's functionality. One or more modules working together may represent a computer framework on which simulations of medical procedures can be performed. Modules may include hardware elements such as computer processing units, graphics processing units, field-programmable gate arrays (FPGAs), communication hardware, and memory. Modules may include software elements that, when executed by the processor, cause the module to perform specific functions.

[0099] The simulation device may include, for example, a core simulation module 30016, a simulation application module directory 30018, an interface module 30012, an object properties module 30020, a physics module 30022, a physiological model 30024, a texture model 30026, a 3D graphics pipeline 30028, a surgical data system interface module 30014, a metric extraction module 30030, and a session storage and management module 30032. The simulation device may also include an operating system module 30034.

[0100] The core simulation model 30016 may provide the primary simulation functions of the simulation device 30000. For example, the core simulation module 30016 may include code for initializing the simulation, communicating and interacting with other modules of the simulation device 30000, and / or managing architecture-level simulation parameters. For example, the core simulation module 30016 may include a master event clock to provide time synchronization and / or coordination of the operation of the modules of the simulation device 30000. For example, the core simulation module 30016 may establish the overall simulation frame rate.

[0101] The core simulation module 30016 may include a core for providing a master simulation cycle. The core simulation module 30016 may execute one or more iterations of the master simulation cycle. Each iteration of the master simulation cycle may represent an individual time slice of the simulation. In one embodiment, the core simulation module 30016 may execute the master simulation cycle according to the flow disclosed in Figure 10.

[0102] The simulation application module directory 30018 may manage the storage, retrieval, and / or linking of one or more application modules 30010. Each application module 30010 may contain code that indicates an application-level aspect of the simulation. For example, an application module 30010 may include functionality that provides simulation of a specific anatomical structure, a specific teaching scope, or a specific instrument. In the exemplary simulation device 30000, the application-specific simulation device 30000 may operate with a single application module 30010, with or without the simulation application module directory 30010. The simulation application module directory 30018 may operate based on its interaction with the core simulation module 30016 and / or the application creation device 30002.

[0103] Interface module 30012 may provide functions for interacting with human interface device 30004 and / or surgical agent device 30006. For example, interface module 30012 may include one or more drivers for converting information received from human interface device 30004 into software commands, interrupts, etc. For example, interface module 30012 may include a software application programming interface (API) for interacting with surgical agent 30006. Interface module 30012 may provide information received from human interface module 30004 and / or surgical agent device 30006 to other modules of simulation device 30000. For example, interface module 30012 may receive control inputs representing the movement of simulated instruments from human interface module 30004 and / or surgical agent device 30006 and provide this information to one or more other modules of simulation device 30000 so that the movement can be represented in the simulation.

[0104] Interface module 30012 may provide an API to enable finer-grained interaction with the surgical agent device 30006. For example, the API may provide an interface for receiving simulation parameters and simulation settings from the surgical agent device 30006. Such simulation parameters and / or simulation settings may be similar to those input by the user via, for example, the application creation device 30002. For example, the surgical agent device 30006 may be capable of running one or more computer-controlled simulation trials by the simulation device 30000. For example, the surgical agent device 30006 may be capable of running multiple simulations, each with alternative interactions.

[0105] The interface module 30012 can transmit the output from the simulation device 30000 to the human interface device 30004 and / or the surgical agent device 30006. For example, the output may include visual output, tactile output, audio output, and / or structured data output.

[0106] The Object Properties module 30020 may provide functions for managing the simulated appearance and / or behavior of objects in a simulation. Examples of simulated objects include anatomical structures, instruments, equipment, consumables, and fluids. The appearance of an object can be managed by object properties such as position, dimensions, scale, material, parent / child relationships, vertices, faces, interactivity, transparency, trajectory, rendering properties, texture, surface reflectivity, motion blur, and layering. The behavior of an object can be managed by object properties such as physical properties, mass, motion, collision behavior, elasticity, viscosity, surface tension, rigging constraints, hardness, shear strength, tearing behavior, and grain size.

[0107] The physics module 30022 may provide the functionality to calculate the physical response and / or interaction of objects in a simulation. The physics module may determine such responses and / or interactions according to classical mechanics, fluid dynamics, soft body mechanics, Brownian motion, collision detection, cloth behavior, finite element analysis, etc. The physics module 30022 may include commercial and / or open-source modules such as PhysX®, Simulation Open Framework Architecture (SOFA)®, and VisSim®.

[0108] The physiological module 30024 may provide the functionality to calculate anatomical structures and / or the patient's overall physiological responses and / or interactions in a simulation. The physiological module 30024 may provide physiological models of important organs and / or systems. These physiological models may include mathematical models, statistical models, etc. For example, the physiological module 30024 may modularize the patient's vital signs to calculate the patient's responses and / or interactions to activities performed during the simulation. For example, a circulatory model may calculate blood pressure in response to severed blood vessels in the simulation. The physiological module 30024 and the physics module 30022 may cooperate with each other during the calculation of each state of the simulation. For example, blood pressure calculated by the circulatory model may be used to determine hydrodynamic properties calculated by the physics module 30022 and managed by the object properties module 30020.

[0109] The texture module 30026 may provide the functionality to determine, search for, and / or generate appropriate surfaces for objects in the simulation. The texture module 30026 may include one or more surface modalities that can be controlled according to the simulation parameters. Surface modalities may include artificially generated surfaces, surfaces based on real-world images, and combinations thereof. The texture module 30026 may coordinate its operation with the physics module 30022 to provide accurate haptic feedback to the user via the user interface module 30012.

[0110] The 3D graphics pipeline 30028 can provide functionality for the visual rendering of the simulation environment. The 3D graphics pipeline 30028 can receive object properties and depth. The 3D graphics pipeline 30028 can determine the visualization to be presented to the user, representing objects in 3D space as seen from the camera's viewpoint. The 3D graphics pipeline 30028 can determine the geometric aspects of rendering, such as lighting, projection, clipping, and view transformation. The 3D graphics pipeline 30028 can determine the rasterization aspects of rendering, such as fragmentation, pixel shading, vertex shading, geometry sharing, and texture filtering. The 3D graphics pipeline 30028, in cooperation with the texture module 30026, can provide accurate visual feedback to the user via the interface module 30012.

[0111] The surgical data system interface module 30014 can provide interactive connectivity to one or more elements of the computer-implemented interactive surgical system 100. Information from one or more elements of the computer-implemented interactive surgical system 100 can be communicated via the surgical data system interface module 30014 to one or more modules of the simulation device 30000 and may affect the operation of the simulation. For example, the surgical data system interface module 30014 may receive information about surgical procedures and communicate it to the corresponding application module 30010. For example, the surgical data system interface module 30014 may receive information about instruments and communicate it to the object properties module 30020. For example, the surgical data system interface module 30014 may receive information about patients and communicate it to the physiological module. For example, the surgical data system interface module 30014 may receive information about tissue images and communicate it to the texture module 30026.

[0112] Information from the modules of the simulation device 30000 can be provided to one or more elements of the computer-implemented interactive surgical system 100 via the surgical data system interface 30014. For example, one or more elements of the computer-implemented interactive surgical system 100 may receive statistics related to the simulated treatment plan from the metric extraction module 30030. For example, one or more elements of the computer-implemented interactive surgical system 100 may receive a reproduced simulation visualization treatment plan from the session memory and management module 30032. For example, the surgical data system interface module 30014 may provide a communication path between the interface module 30012 and one or more elements of the computer-implemented interactive surgical system 100. For example, a surgeon during a live surgical procedure may access simulation information from the operating room and / or manipulate the simulation. For example, a surgeon may use the surgeon console 118 to access and / or interact with a simulation corresponding to a live surgical procedure.

[0113] The metric extraction module 30014 may provide a function to record various parameters related to the operation of the simulation. For example, the metric extraction module 30014 may record metrics related to the simulation as a whole, such as duration, number of activities, number of movements, complexity of movements, participating staff, staff movements, and changes in equipment and / or instruments. For example, the metric extraction module 30014 may record metrics related to specific aspects of the simulation, such as simulated patient vital signs, complications, collisions, and bleeding. The metric extraction module 30014 may maintain a master log of metric-related events during the simulation. The metric extraction module 30014 may record metric-related events according to the configuration from the application module 30010 used in the simulation.

[0114] The session memory and management module 30032 may provide management functions for the main simulation execution record. For example, the session memory and management module 30032 may store information that enables the simulation to be re-executed, displayed, and / or analyzed as a whole. The session memory and management module 30032 may store information about each input, simulation state, and output, such as the inputs, simulation states, and outputs disclosed with respect to Figure 10. The session memory and management module 30032 may enable previous simulations to be recalled, copied, and initialized by new user inputs. For example, a trainee surgeon can recall a simulation performed by an experienced surgeon, pause the simulation at a critical step, and attempt that step themselves. The session memory and management module 30032 may provide an overlay function between different executions of a particular simulation. Such overlays may highlight similarities and differences and enhance training.

[0115] The operating system module 30034 can manage the hardware and / or software resources of the simulation device 30000. The operating system module 30034 can provide common computing system-level services to other modules of the simulation device 30000. For example, the operating system module 30034 may provide hardware I / O processing, memory allocation, hardware interrupt processing, software interrupt processing, thread processing, single-task processing, multi-task processing, etc. The simulation device 30000 may be a real-time computing device. The operating system module 30034 may include a real-time operating system. For example, the operating system module 30034 may be driven by events and frame rates established by the core simulation module 30016.

[0116] Figure 8 is a block diagram of an exemplary surgical simulator system. The simulation device 30000 is shown along with an exemplary hardware architecture. For example, the simulation device 30000 may include a processor 30034, memory 30036, storage 30038, display adapter 30040, operation interface adapter 30042, surgical data system adapter 30044, and / or network adapter 30046. One or more of the processor 30034, memory 30036, storage 30038, display adapter 30040, operation interface adapter 30042, surgical data system adapter 30044, and / or network adapter 30046 may be used to enable the operation of the modules of the simulation device 30000 disclosed herein.

[0117] The processor 30046 may include computer processing units, graphics processing units, any suitable microcontroller, microprocessor, field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), and / or any combination thereof that is suitable for processing and providing a 3D simulated environment for interaction with a computer agent and / or a human user. In one embodiment, the processor 30046 may include one or more processing units. The processor 30046 may be a processor of any suitable depth to carry out the digital processing requirements disclosed herein. For example, the processor 30046 may be a 32-bit processor, a 64-bit processor, a 128-bit processor, and so on.

[0118] Such a processor may include, or communicate with, a medium, such as a computer-readable medium, that can store instructions that, when executed by the processor, cause the processor to perform the steps described herein as being executed or assisted by the processor. Some embodiments of the computer-readable medium may include, but are not limited to, electronic, optical, magnetic, or other storage devices that can provide computer-readable instructions to the processor, such as a processor in a web server. Other examples of media include, but are not limited to, floppy disks, CD-ROMs, magnetic disks, memory chips, ROMs, RAMs, ASICs, configured processors, all optical media, all magnetic tapes or other magnetic media, or any other media that a computer processor can read. The processors and processes described may be one or more structures and may be distributed through one or more structures. The processor may include code for performing one or more (or part of) the methods described herein.

[0119] Memory 30036 may include any component or set of components suitable for storing data. For example, memory 30036 may include volatile memory and / or non-volatile memory. Memory 30036 may include random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, and the like.

[0120] Storage 30038 may include any component or set of components suitable for storing large amounts of data. For example, storage 30038 may include hard disk drives (HDDs), solid state drives (SSDs), network-attached storage (NAS), etc. Storage 30038 may include database structures and / or database management systems (DBMS).

[0121] The display adapter 30040 may include any components or sets of components suitable for outputting a visual representation of a 3D simulation environment. For example, the display adapter 30040 may include a graphics card, display card, graphics adapter, etc. The display adapter 30040 may be used to generate a feed of output images to a display device, such as the display of the human interface device 30004. The display adapter 30040 may include a graphics processing unit (GPU). The display adapter 30040 may include, for example, hardware for rendering a graphics pipeline. The operation interface adapter 30042 may include any components or sets of components suitable for receiving operation information from a human interface device and / or outputting feedback information to a human interface device. For example, the operation interface adapter 30042 may receive motion tracking information from a virtual reality headset and then manipulate the view displayed to the user. For example, the operation interface adapter 30042 may receive a control input indicating a user operating a surgical instrument and then output haptic feedback to the user's handheld device. For example, the operation interface adapter 30042 can receive control information from a conventional desktop keyboard and mouse.The operating interface adapter may include input / output hardware such as serial input / output ports, parallel input / output ports, universal asynchronous receiver / transmitter (UART), discrete logic input / output pins, analog-to-digital converters, digital-to-analog converters, universal serial bus (USB) ports, USB-C ports, FireWire ports, High Performance Parallel Interface (HIPPI), Thunderbolt ports, Yapbus, Ethernet, Gigabit Ethernet, and / or any other suitable peripheral interface technologies.

[0122] The surgical data system adapter 30044 may include any components or sets of components suitable for communicating with the surgical data system 30008. The surgical data system adapter 30044 may include communication hardware for establishing a physical channel between the simulation device 30000 and the surgical data system 30008. For example, the surgical data system adapter 30044 may include a USB port, a USB-C port, a FireWire port, a HIPPI port, a Thunderbolt port, a Yapbus port, an Ethernet port, a Gigabit Ethernet port, and / or any other suitable peripheral interface. The surgical data system adapter 30044 may include hardware, software, and / or a combination thereof for establishing a logical channel between the simulation device 30000 and the surgical data system 30008 via the network adapter 30046 and the network 30048.

[0123] The network adapter 30046 may include any components or sets of components suitable for communication over a network, such as network 30048. The network adapter 30046 can enable communication over local area networks (LANs), wide area networks (WANs), and / or mobile networks. Examples of LAN technologies include fiber optic distributed data interfaces (FDDI), copper distributed data interfaces (CDDI), Ethernet / IEEE 802.3, Token Ring / IEEE 802.5, and Wi-Fi / IEEE 802.11. Examples of WAN technologies include, but are not limited to, point-to-point links, integrated service digital networks (ISDN) and their variations, packet-switched networks, and digital subscriber lines (DSL). A mobile network may include communication links based on one or more mobile communication protocols such as GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long-Term Evolution (LTE) or 4G, LTE Advanced (LTE-A), New Radio (NR), or 5G.

[0124] In one embodiment, the network adapter 30046 may include a wireless network adapter such as a 5G network adapter. Such a 5G network adapter 30046 may use 5G New Radio (NR) transceivers to provide ultra-reliable and low latency communication (URLLC) to enhanced mobile broadband (eMBB). Such a 5G network adapter 30046 may use radio bands such as 3.5GHz to 7GHz and / or higher radio bands such as 24GHz to 48GHz. The network 30048 serving such a 5G network adapter 30046 may include a public wireless network, a semi-private (e.g., network slicing-based) network, and / or a fully private wireless network.

[0125] Figure 9 is a block diagram depicting a user interface device 30004 of an exemplary surgical simulator. The human user interface device 30004 is shown together with an exemplary hardware architecture. For example, the human user interface device 30004 may include a processor 30050, memory 30052, a display subsystem 30054, and / or an operation subsystem 30056.

[0126] The processor 30050 may include a computer processing unit, a graphics processing unit, any suitable microcontroller, microprocessor, field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), and / or any combination thereof suitable for handling processing related to the display of visual information received from the simulation device 30000, processing of operation information for transmission to the simulation device, and processing of feedback information received from the simulation device 30000. The processor 30050 may include a microcontroller for interface with one or more local sensors to detect control operations from the user and / or for interface with one or more local actuators to provide feedback from the user.

[0127] Such a processor may include, or communicate with, a medium, such as a computer-readable medium, that can store instructions that, when executed by the processor, cause the processor to perform the steps described herein as being executed or assisted by the processor. Some embodiments of the computer-readable medium may include, but are not limited to, electronic, optical, magnetic, or other storage devices that can provide computer-readable instructions to the processor, such as a processor in a web server. Other examples of media include, but are not limited to, floppy disks, CD-ROMs, magnetic disks, memory chips, ROMs, RAMs, ASICs, configured processors, all optical media, all magnetic tapes or other magnetic media, or any other media that a computer processor can read. The processors and processes described may be one or more structures and may be distributed through one or more structures. The processor may include code for performing one or more (or part of) the methods described herein.

[0128] Memory 30036 may include any component or set of components suitable for storing data. For example, memory 30036 may include volatile memory and / or non-volatile memory. Memory 30036 may include random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, and the like.

[0129] The display subsystem 30054 may include any components or sets of components suitable for displaying a visual representation of a 3D simulation from the simulation device 30000 to the user. The display subsystem may include display hardware such as monitors, digital projectors, smartphones, digital headsets, virtual reality headsets, stereoscopic displays, console displays for robotic surgeons, surgical display units, and surgical microscopes.

[0130] The operation subsystem 30056 may include any components or sets of components suitable for collecting operation controls from the user and transmitting them to the simulation device 30000, and / or providing the user with feedback information received from the simulation device 30000. User operations may include any interfaces with sensors that involve the user, for example, to indicate the user's intentions in the simulation. Examples of interfaces include keyboards, mice, joysticks, physical devices that mimic the size, shape, and operation of actual surgical instruments, virtual reality handheld controllers, smart gloves, motion detection systems (e.g., hand tracking systems), console manipulators and / or control units of a robotic surgeon, and physical units that mimic the size, shape, and operation of a console manipulator and / or control unit of an actual robotic surgeon. For example, the interface may include viewpoint sensors, such as accelerometers, in a headset to indicate the user's viewpoint in the simulation.

[0131] Feedback from the simulation device 30000 may include any interface with actuators that provide sensory input to the user. For example, feedback may include tactile feedback, force feedback, temperature feedback, moisture feedback, sound feedback, and olfactory feedback. For example, force feedback and / or tactile actuators in a manipulator on a robotic surgical console can be used to simulate the feedback a user would feel when operating such a manipulator in a live procedure. For example, force feedback and / or tactile actuators in a user device that mimics the size, shape, and operation of an actual surgical stapler can be used to simulate the feedback a user would feel when operating such a device on biological tissue, such as force feedback when engaging tissue and firing the stapler.

[0132] Figure 10 is a flowchart of the operation of an exemplary surgical simulator. In 30058, a simulation application may be loaded. For example, the core simulation module 30016 may load data associated with a particular application module 30010 into memory 30036. The loaded data may include instructions for the processor 30034 to operate a particular simulation. The loaded data may include a procedure plan for the simulation. For example, the procedure plan may be structured as disclosed herein, for example with respect to Figures 11A-11B. The loaded data may include the initial state of the simulation.

[0133] In 30060, the simulation output may be determined and / or transmitted. For example, the simulation output may be determined and / or transmitted by the simulation device 30000. Here, the core simulation module 30016 may refer to the current state of the simulation (e.g., the initial state and / or subsequent states). The core simulation module 30016 may involve one or more other modules to process and output the current state. For example, the core simulation module may involve one of the following: the object properties module 30020, the texture module 30026, the application module 30010, the 3D graphics pipeline 30028, the interface module 30012, and / or the surgical data system interface module 30014 to process and output the current simulation state as information. Information related to the output may be processed and / or stored by, for example, the metric extraction module 30030 and / or the session storage and management module 30032.

[0134] For example, in a human-operated simulation session, output information may be transmitted to the display subsystem 30054 and / or operation subsystem 30056 of the human interface device 30004 via the display adapter 30040 and / or operation interface adapter 30042. For example, in a computer-controlled simulation session, output information may be transmitted to the surgical agent 30006 via the interface module 30012. Alternatively, in a computer-controlled simulation session, output information may be transmitted in the application module 30010 (for example, processed locally). For example, in a session accessed via the surgical data system 30008, output information may be transmitted by the surgical data system interface module 30014 via the surgical data system adapter 30044 and / or network adapter 30046.

[0135] In 30062, simulation inputs may be received and / or processed. For example, simulation inputs may be received and / or processed by the simulation device 30000. Here, the core simulation module may receive control inputs in conjunction with the interface device, the surgical data system interface module, and / or the application module 30010. Information related to the inputs may be processed and / or stored, for example, by the metric extraction module 30030 and / or the session storage and management module 30032.

[0136] For example, in a human-operated simulation session, input information may be transmitted from the operation subsystem 30056 of the human interface device 30004 and received via the operation interface adapter 30042. For example, in a computer-controlled simulation session, input information may be transmitted from the surgical agent 30006 and received via the interface module 30012. Also, for example, in a computer-controlled simulation session, input information may be received in the application module 30010 (for example, processed locally). For example, in a session accessed via the surgical data system 30008, input information may be received via the surgical data system adapter 30044 and / or the network adapter 30046 and first processed by the surgical data system interface module 30014.

[0137] In 30064, a subsequent simulation state can be determined. For example, a subsequent simulation state can be determined from the current simulation state and / or any received inputs. The core simulation module 30016 can determine a subsequent simulation state by engaging with one or more other modules of the simulation device 30000. For example, the code simulation module 30016 may engage with the application module, object properties module, physics module, physiological module, etc. The subsequent simulation state can be determined by the operation of the processor 30034. Information related to the input can be processed and / or stored, for example, by the metric extraction module 30030 and / or the session storage and management module 30032.

[0138] In this stage, the process may loop to receiving input at 30060. Each iteration of this flow may represent a corresponding time cycle in the simulation. The simulation frame rate may be set to a level suitable for the simulation's goals and the processing power of the surgical simulation device 30000. Lower frame rates may allow processing to achieve live human interaction simulation. Higher frame rates may allow for higher simulation fidelity. For example, when running a computer-controlled simulation, for example with a surgical agent 30006, a higher frame rate may be used even if it causes the simulation processing time to exceed the real-world time being simulated.

[0139] Figures 11A and 11B illustrate the data structure of an exemplary surgical procedure plan for use in a computer-implemented interactive surgical system and / or surgical simulator. A surgical procedure plan may include information outlining the staff, equipment, techniques, and steps that may be used to perform the surgical procedure. For example, a procedure plan may include a staff roster indicating the roles and / or specific medical professionals to be involved in the procedure. A procedure plan may include a list of equipment that may be used during the procedure, such as durable surgical instruments, imaging equipment, instruments, and consumables. For example, a procedure plan may include a list of candidates for surgical technicians to be used when preparing the operating room to assemble suitable tools and materials for the surgery and for the surgeon. A procedure plan may include information about the techniques to be performed in the procedure. For example, procedure plans for the same surgical objective may include different access methods, mobilization methods, examination methods, tissue joining methods, wound closure methods, etc.

[0140] A procedure plan may reflect the surgeon's expert judgment regarding individual cases. A procedure plan may reflect the surgeon's preferences and / or experience with specific techniques. A procedure plan may map specific surgical tasks to roles and equipment. A procedure plan may provide a timeline of the planned procedure.

[0141] A treatment plan may include one or more decision points and / or branches. Such decision points and / or branches may provide surgical options available for a particular aspect of the treatment, and the selection of one of these options may be based on information from the surgery itself. For example, the selection of one or more options may be based on a specific plane of the anatomical structure of a particular patient, and the surgeon may select an option based on an assessment of the patient's tissue during live surgery.

[0142] The procedure plan may include one or more contingencies. These may include information about possible but unlikely situations that may occur during live surgery. Contingencies may include one or more surgical tasks that may be used if such situations occur. Contingencies may be used to ensure that sufficient equipment, staff, and / or supplies are available at all times during the procedure.

[0143] A treatment plan may be recorded in one or more data structures. The treatment plan data structure may be used to record data about future live surgeries, completed live surgeries, future simulated surgeries, completed simulated surgeries, etc. A treatment plan data structure for live surgeries may be used by a computer-implemented interactive surgical system 100. For example, a treatment plan data structure for live surgeries may be used by a surgical hub 106 to enhance the situational awareness and / or operational modes of the computer-implemented interactive surgical system 100. The treatment plan data structure for live surgeries may be used by the surgical hub 106 to record distinct elements of the live surgery for structured analysis.

[0144] The treatment plan data structure can be used by the simulation device 30000. For example, the treatment plan data structure can be used by the simulation device 30000 to establish the settings for a simulation session and / or one or more objectives. For example, the treatment plan data structure can be used by the simulation device 30000 to record individual elements of a simulated surgery for structured analysis.

[0145] The treatment plan data structure may include any structure suitable for capturing data elements related to the treatment. For example, the treatment plan may be recorded in a tree-like data structure, such as shown in Figure 11A, where the root of the tree structure represents the core treatment data 30066. The core treatment data 30066 may include general information about the treatment, such as the treatment name, treatment code, patient name, date, and time. In the case of a simulation, the core treatment data 30066 may include information about the simulation device, such as the device ID, software version, user, and simulation execution settings (frame rate, resolution, connected user interface device, etc.).

[0146] The treatment data may include leaves in a tree structure. The first level of leaves may include data relating to key aspects of the treatment plan, such as data relating to the treatment setup 30068, one or more treatment stages 30070, one or more contingencies 30072, and the outcome of the treatment 30074.

[0147] Setup data 30068 may include information regarding the preparation and / or initial state of the procedure. For example, setup data 30068 may include elements such as staff roster, staff roles and / or staff IDs, operating room ID, equipment list, room layout, initial operating table position, list of instruments and / or consumables prepared in the surgical field, all initial settings related to the equipment, preoperative images, and patient records. In the case of a simulation, setup data 30068 may include information regarding the simulated environment, such as records of simulated anatomical structures, records of simulated physiological functions, and preoperative images.

[0148] Stage data 30070 may include data elements related to key milestones of the procedure. For example, the stages of the procedure may include milestones such as establishing access. Stage data 30070 may include information related to the staff, equipment, techniques, and steps that may be used to perform a particular stage of the procedure. Stage data 30070 may include a stage ID.

[0149] A stage may be further detailed by one or more sub-leaves, such as one or more surgical tasks 30076. A surgical task may represent a distinct surgical step within a given stage. For example, in the access stage, positioning a trocar may be a surgical task. Surgical task data 30076 may include a task ID. Surgical task data 30076 may include information relevant to a particular task, such as the staff and / or surgeon performing the task, the equipment used, the specific technique applied, patient vital signs during task execution, other environmental information, and lists. Each task may be further detailed using objective data 30078, data related to anatomical structure-instrument interaction 30080, and outcome data 30082. Objective data 30078 may include information indicating the relative success of task execution. Objective data 30078 may include information such as planned task duration, acceptable execution specificity, efficiency modality, and avoidance of complications. Outcome data 30082 may include information related to one or more objectives. Result data 30082 may record the surgical execution (e.g., live and / or simulated) against the target.

[0150] Task data 30076 may contain one or more elements of anatomical structure-instrument interaction data 30080. Anatomical structure-instrument interaction data 30080 may represent fine-grained instructions for surgical execution. Anatomical structure-instrument interaction data 30080 may represent one or more specific activities used in the execution of a surgical task. Anatomical structure-instrument interaction data 30080 may represent observable behaviors of the surgeon.

[0151] In one embodiment, anatomical structure-instrument interaction data 30080 may include specific positions, forces, angles, etc., applied to anatomical structures by the surgeon. For example, in live surgery, data recorded from smart instruments by the surgical hub 106 may be captured as anatomical structure-instrument interaction data 30080. For example, a surgical smart stapler working in conjunction with other elements of a computer-implemented interactive surgical system 100 may record the stapler's position, angle, tip force, jaw force, staple cartridge type, closure pressure, firing speed, etc. In simulated surgery, similar data elements may be captured.

[0152] Contingency data 30072 may indicate any complications that may be associated with the procedure. Each contingency data 30072 may include one or more task data elements 30084 that address an appropriate response to a specific complication. Contingency data 30072 may indicate a deviation from the original procedure plan. Also, for example, contingency data may be cross-referenced with one or more tasks 30078 and / or anatomical structure-instrument interactions 30080. For example, if a particular execution in an anatomical structure-instrument interaction 30080 may lead to a complication, the nature of that execution and the cross-reference of the contingency may be included in the outcome data 30082 related to that anatomical structure-instrument interaction 30080.

[0153] Outcome data 30074 may show the results of the procedure. Here, overall metrics of the surgical execution may be stored, and actual and / or simulated patient recovery information, as well as / or patient outcomes, may be recorded. For example, outcome data 30074 may include efficiency information, cost information, surgical duration, workload metrics, and the percentage of planned consumable usage.

[0154] Figure 11B shows a treatment plan data structure having the elements disclosed above, which further establishes the structure of alternative steps for completing a particular treatment, task, or activity. As shown, the treatment represented by treatment data 30086 may include two alternative setups, each represented by its respective setup data (first setup data 30088, 30090, and second setup data 30092). The first setup data 30088, 30090 may include two alternative tasks 30094, 30096. The second setup data 30092 may include one task 30098. In this figure, the treatment represented by treatment data 30086 can be achieved in three different ways. The first method uses the first setup 30088 and the first task 30094. The second method uses the first setup 30090 and the second task 30096. The third method uses the second setup 30092 and its corresponding task 30098.

[0155] Each path in the tree structure may represent a specific set of alternative methods for performing a procedure. Such a structure may be useful for assisting in the creation of specific procedure plans for particular live and / or simulated surgeries. Such a structure may be useful for simulating many possible alternatives to a procedure and evaluating the differences in outcomes.

[0156] Figure 12 shows an exemplary data flow 31000 of the operation of an exemplary surgical simulator 31016. The exemplary surgical simulator 31016 may be a simulation device 30000 (for example, shown in Figures 7 and 8). The simulator 31016 may include a processor 30034 of the simulation device 30000. Images of a patient's organs and / or tissues may be captured during a surgical procedure. For example, an image of a patient's stomach may be captured during a laparoscopic sleeve gastrectomy procedure. As shown in Figure 12, in an operating room 31002, a surgeon and surgical assistant are performing a laparoscopic sleeve gastrectomy procedure 31003. In procedure 31003, a laparoscope with structured light imaging capability may be used. The laparoscope may include an incident light channel for a light pattern from a structured light projector and an outgoing light channel for light reflected from the surgical site (e.g., stomach). The light pattern from the structured light projector may be narrowband light (e.g., blue light). Therefore, the light reflected from the stomach may include a distorted pattern of light reflected from the stomach and white light reflected from the stomach.

[0157] Images of the stomach reflected with white light can be filtered and captured by a white light camera, and these images can be presented to the surgeon and / or surgical assistant in a visualization user interface. Images of the stomach with a distorted light pattern superimposed can be filtered and captured intraoperatively, for example, in video format, by a corresponding narrowband light camera. The white light camera and narrowband light camera can be calibrated so that the red / green / blue (RGB) color values ​​of the image captured by the white light camera can be looked up for the image captured by the narrowband light camera. Thus, images of the stomach with a distorted light pattern and associated RGB color values ​​superimposed can be captured intraoperatively.

[0158] Captured images of a patient's organs and / or tissues may be transmitted to the surgical data system 31004. For example, a captured image of the stomach 31012 with a distorted light pattern superimposed may be transmitted to the surgical data system 31004 in the operating room 31002 (e.g., surgical data system 30006 as shown in Figure 7). Light patterns projected from a structured light projector may also be transmitted to the surgical data system 31004.

[0159] The surgical data system 31004 may transmit captured images of the patient's organs and / or tissues to the remote system 31008. For example, the surgical data system 31004 may transmit images of the stomach captured from multiple laparoscopic sleeve gastrectomy procedures 31013 to the remote system 31008 (e.g., the cloud 14 shown in Figure 1). The surgical data system 31004 may also transmit light patterns projected from a structured light projector to the remote system 31008. The remote system 31008 may include a remote server 31010 coupled to a storage device 31006. Multiple laparoscopic sleeve gastrectomy procedures may include procedures performed on the same patient and / or procedures performed on different patients.

[0160] The surgical simulator 31016 can simulate surgical procedures using two-dimensional (2D) image data of a patient's organs and / or tissues from a remote system 31008. For example, the surgical simulator 31016 can simulate a laparoscopic sleeve gastrectomy procedure using 2D image data 31014 of the stomach from a remote system 31008. The 2D image data 31014 of the stomach may include, for example, 2D image data of the stomach from multiple surgical data systems at the same facility, different facilities within the same geographical area, or different facilities within different geographical areas. The 2D image data 31014 of the stomach may be image frames from a video captured intraoperatively using structured optical imaging as described herein.

[0161] The surgical simulator 31016 can generate a deformable reference three-dimensional (3D) model of a human organ and combine this model with 2D image data of a human organ from a remote system 31016 to generate a simulated human organ for simulated surgical procedures. For example, the surgical simulator 31016 can generate a deformable reference 3D model 31018 of a stomach and combine this model 31018 with 2D image data 31014 of a stomach from a remote system 31008 to generate a simulated stomach. The surgical simulator 31016 can reconstruct 3D data of a stomach from the 2D image data 31014 of a stomach from the remote system 31008. The surgical simulator 31016 can map the shape of the deformable reference 3D model 31018 of a stomach to correspond to the reconstructed 3D data 31014 of a stomach from the remote system 31008. The surgical simulator 31016 can map the shape of a deformable reference 3D model 31018 of the stomach to correspond to the reconstructed 3D data 31014 of the stomach.

[0162] The surgical simulator 31016 can output a mapped, deformable reference 3D model of a simulated human organ 31020 after mapping the shape and surface of the simulated human organ to correspond to the reconstructed 3D data of the human organ. The simulation user 31024 can interact with the mapped, deformable reference 3D model of the simulated human organ 31020 in the simulated environment.

[0163] Figure 13 shows an exemplary data flow of the operation of the exemplary surgical simulator 31050. The exemplary surgical simulator 31050 may be a simulation device 30000 (for example, as shown in Figures 7 and 8). The exemplary data flow may be executed by the simulation device 30000. The exemplary data flow may be executed by the processor 30034 of the simulation device 30000. The surgical simulator 31050 may include a 3D reconstruction module 31056. The 3D reconstruction module 31056 may receive a 2D human organ image 31054 and associated 3D reconstruction reference data 31052. For example, the 2D human organ image 31054 may be a 2D image of the stomach 31014 as shown in Figure 12. For example, the associated 3D reconstruction reference data 31052 may be a light pattern projected from a structured light projector as shown in Figure 12.

[0164] The 3D reconstruction module 31056 can reconstruct 3D image data from a received 2D human organ image 31054 and associated 3D reconstruction reference data 31052. For example, the 3D reconstruction module 31056 can reconstruct a 3D point cloud 31060 from a 2D stomach image 31014 and associated 3D reconstruction reference data 31052. In one example, the 3D reconstruction module 31056 can reconstruct a 3D point cloud from a 2D image of the stomach 31014 and a light pattern projected from a structured light projector as shown in Figure 12. The 3D points can be reconstructed in pixels of the 2D image of the stomach 31014 using any suitable 3D reconstruction algorithm related to the light pattern. The 3D points reconstructed from the 2D image of the stomach 31014 may include RGB color values ​​from the 2D image of the stomach 31014. The 3D points reconstructed from the 2D image of the stomach 31014 may include 3D spatial coordinates. 3D points reconstructed from 2D images 31014 of different stomachs can be spatially aligned into a uniform 3D coordinate system using image alignment algorithms, such as non-rigid alignment algorithms and / or feature-based alignment algorithms.

[0165] The surgical instrument 31050 may include a 3D model generator 31055. The 3D model generator 31055 may generate a deformable 3D human organ model 31058. For example, the 3D model generator 31055 may generate a deformable 3D stomach model 31058 based on an anatomical template of the stomach (e.g., an anatomical atlas of the stomach). The deformable 3D stomach model 31058 may be a deformable volumetric tetrahedron model. The surface vertices of the deformable 3D stomach model 31058 may embody the surface shape of the stomach anatomical template. The surface vertices of the deformable 3D stomach model 31058 may be generated with default colors and default textures.

[0166] The surgical simulator 31050 may include an image mapping module 31062. For example, the image mapping module 31062 may receive a reconstructed 3D point cloud 31060 and a deformable 3D stomach model 31058 restored from a 2D image of the stomach 31014.

[0167] The mapping module 31062 can generate a mapped deformable 3D human organ model 31058 based on the reconstructed 3D point cloud 31060 and the deformable 3D human organ model 31064. For example, the mapping module 31062 can extract a 3D mesh surface stomach model from the deformable 3D stomach model 31058. The mapping module 31062 can convert the extracted 3D mesh surface stomach model into a corresponding 3D point cloud. Such a 3D point cloud can be mapped to a reconstructed 3D point cloud 31060, recovered from a 2D image of the stomach 31014, and the difference with the reconstructed 3D point cloud 31060 can be minimized, for example, using an iterative closest point (ICP) algorithm. The mapped 3D point cloud can be converted back to a 3D mesh surface stomach model, which can then be converted back to a deformable volumetric tetrahedron model. Therefore, the mapped deformable 3D stomach model 31064 best matches the shape, color, and texture of the reconstructed 3D point cloud 31060.

[0168] A mapped, deformable 3D human organ model 31064 can be sent to a visualization module for user interaction. For example, a mapped, deformable 3D stomach model can be sent to a visualization module for user interaction in a simulation environment.

[0169] Figure 14 is a flowchart of an exemplary operation of an exemplary surgical simulator. For example, the exemplary operation may include a process 31100 for generating 3D image data from 2D image data. Process 31100 may map a deformable 3D model to 3D image data. The exemplary surgical simulator may be a simulation device 30000 (for example, shown in Figures 7 and 8). Process 31100 may be performed by the simulation device 30000. Method 31100 may be performed by the processor 30034 of the simulation device 30000.

[0170] In 31102, a first two-dimensional (2D) image data of a first human organ of a first patient may be received. The first 2D image data may include a first 2D visible image data captured within the body of the first patient and a first three-dimensional (3-D) reconstruction reference data. The 2D image data of the first human organ of the first patient may be captured using structured light imaging.

[0171] In 31104, a second 2D image data of a second human organ of a second patient may be received. The second 2D image data may include a second 2D visible image data captured within the body of the second patient and a second 3D reconstruction reference data. For example, the first human organ of the first patient and the second human organ of the second patient may be the same type of human organ. For example, the first 2D image data may further include a first invisible image data captured within the body of the first patient. The second 2D image data may further include a second invisible image data captured within the body of the second patient. Based on the first and second criteria, and based on the first and second invisible image data, 3D image data may be generated from an aggregation of the first and second visible image data. The 3D image data may further include non-visual information. Subsurface data of a deformable 3D model of a simulated human organ may be generated based on the non-visual information of the 3D image data.

[0172] In 31106, 3D image data can be generated from the aggregation of first visible image data and second visible image data, based on first and second criteria. The 3D image data may include spatial information and visual information.

[0173] In 31108, the shape of a deformable 3D model of a simulated human organ can be mapped to correspond to the spatial information of 3D image data. For example, the method can receive tissue property data of one or more human organs. The tissue property data can be determined by tracking the displacement of physical reference markers attached to at least one or more human organs. A simulated force applied to the deformable 3D model can be determined. The shape of the deformable 3D model can be deformed based on the physical property data.

[0174] In 31110, the surface of a deformable 3D model of a simulated human organ can be mapped to correspond to the visual information of the 3D image data. For example, patient-specific data can be received. The surface of a deformable 3D model of a simulated human organ can be mapped to correspond to the visual information of the 3D image data and patient-specific data.

[0175] In 31112, a deformable 3D model can be output.

[0176] For example, the expected configuration of a simulated surgical procedure can be defined. The expected configuration may be one of the following: a surgical step, a job which is part of a surgical step, a location within a surgical scene, or a surgical instrument. User input data for the simulated surgical procedure can be monitored, and input data corresponding to the expected configuration can be determined. In 1202, relevant instructions can be presented to the user.

[0177] For example, the expected configuration of a simulated surgical procedure can be defined. The expected configuration could be one of the following: a surgical step, a job that is part of a surgical step, a location within the surgical site, or a surgical instrument. User input data for the simulated surgical procedure can be monitored. It can be determined that the input data corresponds to the expected configuration. The instructor may be presented with the option to initiate an instructional dialogue with the user.

[0178] For example, the expected surgical steps of a simulated surgical procedure can be defined. User input data for the simulated surgical procedure can be monitored. It can be determined that the input data corresponds to the expected surgical steps. The user may be presented with the option to view corresponding segments of one or more actual surgical procedure videos. One or more actual surgical procedure videos may include segments indexed to surgical steps. In one embodiment, the option to view corresponding segments of one or more actual surgical procedure videos may be further customized based on surgical complications or surgical outcomes.

[0179] The simulator may be configured to provide a realistic appearance through the integration of real-world information. The simulator may include a surgical simulator model having real-world images, textures, the shape of the organ being manipulated, and the characteristics of the organ (e.g., via computer adaptation). The simulator model (e.g., a 3D model) of the simulation may have real-world images overlaid on the model, providing accuracy and interactive characteristics of the surgery and organ. Images of the deformed organ may be generated by a combination of structured optical imaging of the actual surgery. Surface textures, features, and appearance may be generated from the aggregation of actual surgical images. The overlay may be customized by the user, facility, surgical data system (e.g., a hub), or remote system (e.g., a cloud-based system) based on the specific needs of the simulation and / or the specific characteristics of the simulated procedure and / or patient.

[0180] The simulator may be configured to provide a digital simulation using a video overlay of actual tissue imaging. The video overlay of actual tissue imaging may be adjusted to a 3D model of the deformed state by structured light imaging of the original dataset of actual tissue imaging.

[0181] The overlay of actual tissue imaging may be adaptive. The overlay of actual tissue imaging may be derived from aggregated real-world imaging of similar tissues and may include multispectral imaging of tissues and other imaging techniques. The aggregated real-world imaging may be adjusted by the user, based on predetermined characteristics, to provide a sense of reality for a specific patient or tissue. The aggregated real-world imaging may be updated from additional data aggregated in a remote system (e.g., a cloud-based system) as the number of procedures being imaged increases. Adaptations may be customized by region, population age, one or more comorbidities, and / or therapeutic effect.

[0182] The tissue model can be adjusted to the actual deformation and / or repositioning of other anatomical structures by intraoperative scanning (e.g., by structured light or time-of-flight distance sensor arrays). The tissue model can also be adjusted to data input into the simulation from the Crowd-Sourced Assessment of Technical Skills (CSATS) database or other live-view databases.

[0183] The simulator may be configured to simulate microtensiles and / or surface strains that can induce subsurface shear, deformation, and / or stress. Such a simulator may be performed using structured light in combination with physically reference markers on organs and / or tissues. Physically attached markers may be added to, for example, structured light projection tracking of organs and / or tissues, either in place of or in combination with it. Tracking using physically attached markers can monitor strain and stress within the organ surface and may allow extrapolation of deformation and movement implied by subsurface forces. Using such tracking, the generation of deformation models can more realistically reflect the movement and contraction and / or cutting of organs and / or tissues. Surface and subsurface strains and stresses can be calculated. Using such calculations, organ sections may be suitable for observation. Using such calculations, tissue tension, occlusion of blood flow due to tissue shear, and microtissue tension and maximum strain before tearing can be shown.

[0184] The simulator may be configured to include indexable recordings of real-world videos of surgeries that can be accessed and reviewed within the simulator (for example, to learn from actual surgical activities). The indexable recordings may include CSATS surgical stages extended into the simulator (for example, for learning). The indexable recordings may include ethnography.

[0185] Indexable records may include step-by-step displays of specific procedures. Step-by-step displays may allow users to see one or more examples of how the system suggests an approach to a step. Sources for exemplary videos may be thought leaders, crowdsourced resources, or best practices from a given network of hospitals. Network examples may be economically relevant. Such network examples may promote consistent behavior within the network. For example, for a particular network, payment and reimbursement for procedures may exemplify how surgeons within that network can handle a given approach, treatment, or product use. Different steps may be searched to draw exemplary videos. Examples of such steps may include vascular isolation for mesenteric resection, insertion of an anvil into the colon, or approach to the anvil.

[0186] Indexable records can provide a search for specific differences and draw upon examples of how specific physicians address those differences through their approaches.

[0187] Indexable records may be searchable by outcome / complication.

[0188] The simulator may be configured to provide live instructions to the user during the simulation via simulation control. Simulation control may include toggleable labeling, landmark IDs, complication callouts, tissue orientation indications, and notifications of deviations from best practice activities.

[0189] The simulator may be configured to operate in an automated tutorial mode. The simulation may have a predetermined start or steps that the user takes to reach a specific predetermined location, step, equipment, or job. The predetermined start and / or steps may include instructions relating to one or more of the following: step-by-step operation, interaction with other systems or instruments such as augmented imaging, or use of auxiliary or accessory tools or materials.

[0190] The simulator may be configured to operate in live command mode. The simulation may include interaction with an instructor, such as a faculty member or training physician. The simulation may allow the user to communicate with the instructor. The simulation may allow prompts to be provided at predetermined times. The simulation may be controlled by an external instructor, for example, via external control or indicators. The control may direct reorientation or guide the user through the steps. The control may include guiding the user through orientation, pointing, marking, labeling, or other means. The simulation may include a language overlay.

[0191] The simulator may be configured to include the coordinated interaction of alternative visualization means in the simulation. For example, the simulation may include multispectral imaging overlays or corrections, or other imaging interactions, to provide a occluded view of important structures. Examples of imaging interactions include scanning hospital take-ups for post-exposure treatment (PET) and magnified views of lymph nodes. Examples of important structures include blood vessels. The simulation may include clinical involvement.

[0192] For example, the simulation may include imaging from indocyanine green (ICG) to show blood flow. For example, the simulation may include simulating the optimal time for systemic ICG delivery. For example, the simulation may include simulating ICG fading (for example, over a period of time). For example, the simulation may include simulating the timing of direct ICG delivery.

[0193] The simulator may be configured to adapt simulated aggregate-based anatomical structures to the patient's irregularities, including imaging correction of the patient's specific anatomical structures. The simulation may affect organ size, the impact on disease state, and the location or positioning of organs with vascular morphologies specific to the patient. The simulation may include atypical patient organs and abnormal anatomical structures. The simulation may include input from preoperative imaging, such as multiple branching and / or trifoliation of vessels to lung lobes and lung segments. The simulation may include intraoperative and / or preoperative fluoroscopy providing information on collateral circulation to organs.

[0194] The following non-exclusive enumeration of embodiments also forms part of this disclosure.

[0195] Embodiment 1. A computing device, wherein the computing device is Equipped with a processor, the processor is The method involves receiving first two-dimensional (2D) image data of a first human organ, wherein the first 2D image data includes first 2D visible image data and first three-dimensional (3D) reconstruction reference data captured within the body of a first patient. The method involves receiving a second 2D image data of a second human organ, wherein the second 2D image data includes a second 2D visible image data captured within the body of a second patient and a second 3D reconstruction reference data. The method involves generating 3D image data from an aggregate of first 2D visible image data and second 2D visible image data, based on first 3D reconstruction reference data and second 3D reconstruction reference data, wherein the 3D image data includes spatial information and visual information. Mapping the shape of a deformable 3D model of a human organ to correspond to the spatial information of 3D image data, Mapping the surface of a deformable 3D model of a human organ to correspond to the visual information of 3D image data, A computing device configured to output deformable 3D models and perform the following actions.

[0196] It should be understood that a "deformable 3D model" can refer to any volumetric model containing multiple spatial data points (such as a model of a human organ rendered in 3D). This allows the model to be easily "deformed" (or interacted with to change the shape / size of the model) by modifying the spatial data points within the model.

[0197] "Mapping" the shape / surface of a first model can also be understood as modifying the shape / surface of the first model based on a separate 3D model (i.e., 3D image data). This may involve using point-to-point mapping, where each point on the model can be adjusted to minimize the difference with the separate 3D model (in this case, the generated 3D image data).

[0198] One technical effect of this embodiment is that the accuracy of both the shape and surface details of the 3D model can be improved through the capture and processing of 2D images.

[0199] Embodiment 2. The computing device according to Embodiment 1, wherein the processor is configured to simulate a surgical procedure.

[0200] Embodiment 3. The computing device according to any one of Embodiments 1 to 2, wherein the 3D model of the human organ is a 3D model of a simulated human organ.

[0201] Embodiment 4. A computing device according to any one of Embodiments 1 to 3, wherein the first human organ and the second human organ are human organs of the same type.

[0202] Embodiment 5. The computing device according to any one of Embodiments 1 to 4, wherein the first patient is the same as the second patient.

[0203] Embodiment 6.2D image data is captured using structured light imaging in a computing device according to any one of Embodiments 1 to 5.

[0204] Embodiment 7. The processor is Receiving tissue characteristic data from one or more human organs, wherein the tissue characteristic data is determined by tracking the displacement of at least one or more physical reference markers attached to the human organs. A computing device according to any one of embodiments 1 to 6, further configured to perform the following: when a simulated force is applied to a deformable 3D model, deform the deformable 3D model based on physical property data.

[0205] Embodiment 8. A computing device according to any one of Embodiments 1 to 7, wherein the first 2D image data further comprises first invisible image data captured inside the body of a first patient, the second 2D image data further comprises second invisible image data captured inside the body of a second patient, the processor is further configured to generate 3D image data from an aggregation of the first visible image data and the second visible image data based on a first criterion and a second criterion, and based on the first invisible image data and the second invisible image data, the 3D image data further comprises non-visual information, and the processor is further configured to generate subsurface data of a deformable 3D model of a human organ based on the non-visual information of the 3D image data.

[0206] Embodiment 9. The processor is Receive patient-specific data, A computing device according to any one of embodiments 1 to 8, further configured to map the surface of a deformable 3D model of a human organ to correspond to visual information and patient-specific data of 3D image data.

[0207] Embodiment 10. The processor is Defining the expected mode of a simulated surgical procedure, wherein the expected mode is one of the following: a surgical step, a job which is part of a surgical step, a location within a surgical scene, or a surgical instrument. Monitoring user input data for simulated surgical procedures, A computing device according to any one of Embodiments 2 to 9, which is further configured to present relevant instructions to the user when it is determined that the input data corresponds to an expected manner, as described in Embodiment 2.

[0208] Embodiment 11. The processor is Defining the expected form of a simulated surgical procedure, wherein the expected form is one of the following: a surgical step, a job that is part of a surgical step, a location within the surgical site, or a surgical instrument. Monitoring user input data for simulated surgical procedures, A computing device according to any one of Embodiments 2 to 9, as dependent on Embodiment 2, further configured to present the instructor with the option to initiate an instructional interaction with the user when it is determined that the input data corresponds to an expected form.

[0209] Embodiment 12. The processor is This involves defining the expected surgical steps of a simulated surgical procedure and monitoring user input data for the simulated surgical procedure. A computing device according to any one of Embodiments 2 to 9, as dependent on Embodiment 2, further configured to perform the following: when it is determined that the input data corresponds to an expected surgical step, it presents the user with the option to view corresponding segments of one or more actual surgical procedure videos, the one or more actual surgical procedure videos include segments indexed to the surgical steps.

[0210] Embodiment 13. The computing device according to Embodiment 12, wherein the option to view corresponding segments of one or more actual surgical procedure videos may be further customized depending on surgical complications or surgical outcomes.

[0211] Embodiment 14. Computer implementation method, wherein the computer implementation method is The method involves receiving first two-dimensional (2D) image data of a first human organ, wherein the first 2D image data includes first 2D visible image data and first three-dimensional (3D) reconstruction reference data captured within the body of a first patient. The method involves receiving a second 2D image data of a second human organ, wherein the second 2D image data includes a second 2D visible image data captured within the body of a second patient and a second 3D reconstruction reference data. The method involves generating 3D image data from an aggregate of first 2D visible image data and second 2D visible image data, based on first 3D reconstruction reference data and second 3D reconstruction reference data, wherein the 3D image data includes spatial information and visual information. Mapping the shape of a deformable 3D model of a human organ to correspond to the spatial information of 3D image data, Mapping the surface of a deformable 3D model of a human organ to correspond to the visual information of 3D image data, A computer implementation method, including outputting a deformable 3D model.

[0212] Embodiment 15. The computer implementation method according to Embodiment 14, wherein the 3D model of the human organ is a 3D model of a simulated human organ.

[0213] Embodiment 16. The computer implementation method according to any one of Embodiments 14 to 15, wherein the first human organ and the second human organ are human organs of the same type.

[0214] Embodiment 17. The computer-assisted method according to any one of Embodiments 14 to 16, wherein the first patient is the same as the second patient.

[0215] Embodiment 18.2D image data is captured using structured light imaging, as described in any one of Embodiments 14 to 17, by a computer-aided method.

[0216] Embodiment 19. Receiving tissue characteristic data from one or more human organs, wherein the tissue characteristic data is determined by tracking the displacement of at least one or more physical reference markers attached to the human organs. A computer-aided method according to any one of embodiments 14 to 18, further comprising: applying a simulated force to a deformable 3D model; and deforming the deformable 3D model based on physical property data.

[0217] Embodiment 20. A computer-aided method according to any one of Embodiments 14 to 19, wherein the first 2D image data further comprises first invisible image data captured inside the body of a first patient, the second 2D image data further comprises second invisible image data captured inside the body of a second patient, the processor is further configured to generate 3D image data from an aggregation of the first visible image data and the second visible image data based on the first and second criteria and the first invisible image data and the second invisible image data, the 3D image data further comprises non-visual information, and the processor is further configured to generate subsurface data of a deformable 3D model of a human organ based on the non-visual information of the 3D image data.

[0218] Embodiment 21. The processor is Receiving patient-specific data, and A computer-aided method according to any one of embodiments 14 to 20, further configured to perform mapping the surface of a deformable 3D model of a human organ to correspond to visual information and patient-specific data of 3D image data.

[0219] Embodiment 22. Defining the expected mode of a simulated surgical procedure, wherein the expected mode is one of the following: a surgical step, a job which is part of a surgical step, a location within a surgical scene, or a surgical instrument. Monitoring user input data for simulated surgical procedures, A computer implementation method according to any one of embodiments 14 to 21, further comprising: when it is determined that the input data corresponds to an expected pattern, presenting the relevant instructions to the user.

[0220] Embodiment 23. Defining the expected form of a simulated surgical procedure, wherein the expected form is one of the following: a surgical step, a job that is part of a surgical step, a location within the surgical site, or a surgical instrument. Monitoring user input data for simulated surgical procedures, A computer implementation method according to any one of embodiments 14 to 22, further comprising presenting the instructor with the option to initiate an instructional dialogue with the user when it is determined that the input data corresponds to an expected form.

[0221] Embodiment 24. The processor is To define the expected surgical steps of a simulated surgical procedure, Monitoring input data from users regarding simulated surgical procedures, and A computer implementation method according to any one of embodiments 14 to 23, further configured to perform the following: when it is determined that the input data corresponds to an expected surgical step, the user is presented with the option to view a corresponding segment of one or more actual surgical procedure videos, the one or more actual surgical procedure videos include segments indexed to the surgical step.

[0222] Embodiment 25. The computer-assisted method according to Embodiment 24, wherein the option to view corresponding segments of one or more actual surgical procedure videos may be further customized depending on surgical complications or outcomes.

[0223] Embodiment 26. A computer-readable medium containing instructions that, when executed by a computer, cause the computer to perform the method described in any one of Embodiments 14 to 25.

[0224] The following non-exclusive enumeration of embodiments also forms part of this disclosure.

[0225] Apparatus 1. A computing device for simulating a surgical procedure, wherein the computing device is Equipped with a processor, the processor is The objective is to receive first two-dimensional (2D) image data of a first human organ of a first patient, wherein the first 2D image data includes first 2D visible image data and first three-dimensional (3D) reconstruction reference data captured within the body of the first patient. The method involves receiving a second 2D image data of a second human organ of a second patient, wherein the second 2D image data includes a second 2D visible image data captured within the second patient's body and a second 3D reconstruction reference data. The method involves generating 3D image data from an aggregate of first 2D visible image data and second 2D visible image data, based on first 3D reconstruction reference data and second 3D reconstruction reference data, wherein the 3D image data includes spatial information and visual information. Mapping the shape of a deformable 3D model of a simulated human organ to correspond to the spatial information of 3D image data, Mapping the surface of a deformable 3D model of a simulated human organ to correspond to the visual information of 3D image data, A computing device configured to output deformable 3D models and perform the following actions.

[0226] Embodiment 2. A computing device for simulating the surgical procedure described in Embodiment 1, wherein the first human organ of the first patient and the second human organ of the second patient are human organs of the same type.

[0227] Embodiment 3. A computing device for simulating the surgical procedure described in Embodiment 1, wherein first 2D image data of a first human organ of a first patient is captured using structured light imaging.

[0228] Appearance 4. The processor is Receiving tissue characteristic data from one or more human organs, wherein the tissue characteristic data is determined by tracking the displacement of at least one or more physical reference markers attached to the human organs. A computing device for simulating a surgical procedure according to embodiment 1, further configured to determine that a simulated force has been applied to a deformable 3D model, and to deform the shape of the deformable 3D model based on tissue characteristic data.

[0229] Embodiment 5. A computing device for simulating a surgical procedure according to Embodiment 1, wherein the first 2D image data further comprises first invisible image data captured inside the body of a first patient, the second 2D image data further comprises second invisible image data captured inside the body of a second patient, the processor is further configured to generate 3D image data from an aggregation of the first 2D visible image data and the second 2D visible image data based on the first 3D reconstruction reference data and the second 3D reconstruction reference data, and based on the first invisible image data and the second invisible image data, the 3D image data further comprises non-visual information, and the processor is further configured to generate subsurface data of a deformable 3D model of a simulated human organ based on the non-visual information of the 3D image data.

[0230] Embodiment 6. The processor is Receive patient-specific data, A computing device for simulating a surgical procedure according to embodiment 1, further configured to map the surface of a deformable 3D model of a simulated human organ to correspond to visual information of 3D image data and patient-specific data.

[0231] Embodiment 7. The processor is Defining the expected mode of a simulated surgical procedure, wherein the expected mode is one of the following: a surgical step, a job which is part of a surgical step, a location within a surgical scene, or a surgical instrument. A computing device for simulating a surgical procedure as described in Embodiment 1, further configured to monitor user input data for a simulated surgical procedure, determine that the input data corresponds to an expected pattern, and present relevant instructions to the user.

[0232] Embodiment 8. The processor is Defining the expected form of a simulated surgical procedure, wherein the expected form is one of the following: a surgical step, a job that is part of a surgical step, a location within the surgical site, or a surgical instrument. This involves monitoring user input data for simulated surgical procedures and determining whether the input data corresponds to the expected behavior. A computing device for simulating a surgical procedure according to Embodiment 1, further configured to present an instructor with the option to initiate an instructional interaction with a user.

[0233] Embodiment 9. The processor is This involves defining the expected surgical steps of a simulated surgical procedure and monitoring user input data for the simulated surgical procedure. The system determines that the input data corresponds to the expected surgical steps of the simulated surgical procedure, A computing device for simulating a surgical procedure according to Embodiment 1, further configured to perform the following: presenting the user with the option to view corresponding segments of one or more actual surgical procedure videos, wherein the one or more actual surgical procedure videos include segments indexed to surgical steps.

[0234] Embodiment 10. A computing device for simulating a surgical procedure as described in Embodiment 9, wherein the option to view corresponding segments of one or more actual surgical procedure videos is further customized depending on surgical complications or surgical outcomes.

[0235] Apparatus 11. A computer-aided method for simulating a surgical procedure, wherein the method is: The objective is to receive first two-dimensional (2D) image data of a first human organ of a first patient, wherein the first 2D image data includes first 2D visible image data and first three-dimensional (3D) reconstruction reference data captured within the body of the first patient. The method involves receiving a second 2D image data of a second human organ of a second patient, wherein the second 2D image data includes a second 2D visible image data captured within the second patient's body and a second 3D reconstruction reference data. The method involves generating 3D image data from an aggregate of first 2D visible image data and second 2D visible image data, based on first 3D reconstruction reference data and second 3D reconstruction reference data, wherein the 3D image data includes spatial information and visual information. Mapping the shape of a deformable 3D model of a simulated human organ to correspond to the spatial information of 3D image data, Mapping the surface of a deformable 3D model of a simulated human organ to correspond to the visual information of 3D image data, A computer implementation method, including outputting a deformable 3D model.

[0236] Embodiment 12. A computer-aided method for simulating the surgical procedure described in Embodiment 11, wherein the first human organ of the first patient and the second human organ of the second patient are human organs of the same type.

[0237] Embodiment 13. A computer-aided method for simulating a surgical procedure as described in Embodiment 11, wherein a first 2D image data of a first human organ of a first patient is captured using structured light imaging.

[0238] Appearance 14. Receiving tissue characteristic data from one or more human organs, wherein the tissue characteristic data is determined by tracking the displacement of at least one or more physical reference markers attached to the human organs. A computer-aided method for simulating a surgical procedure according to embodiment 11, further comprising determining that a simulated force has been applied to a deformable 3D model, and deforming the shape of the deformable 3D model based on tissue characteristic data.

[0239] Embodiment 15. The first 2D image data further includes first invisible image data captured inside the body of a first patient, and the second 2D image data further includes second invisible image data captured inside the body of a second patient, and the method is The method involves generating 3D image data from an aggregation of first 2D visible image data and second 2D visible image data, based on first 3D reconstruction reference data and second 3D reconstruction reference data, and based on first invisible image data and second invisible image data, wherein the 3D image data further includes non-visible information. A computer-aided method for simulating a surgical procedure according to embodiment 11, further comprising generating subsurface data of a deformable 3D model of a simulated human organ based on non-visual information of 3D image data.

[0240] Appearance 16. Receiving patient-specific data, A computer-aided method for simulating a surgical procedure according to embodiment 11, further comprising mapping the surface of a deformable 3D model of a simulated human organ to correspond to visual information of 3D image data and patient-specific data.

[0241] Appearance 17. Defining the expected mode of a simulated surgical procedure, wherein the expected mode is one of the following: a surgical step, a job which is part of a surgical step, a location within a surgical scene, or a surgical instrument. A computer method for simulating a surgical procedure according to aspect 11, further comprising monitoring user input data for a simulated surgical procedure, determining that the input data corresponds to an expected pattern, and presenting relevant instructions to the user.

[0242] Appearance 18. Defining the expected form of a simulated surgical procedure, wherein the expected form is one of the following: a surgical step, a job that is part of a surgical step, a location within the surgical site, or a surgical instrument. This involves monitoring user input data for simulated surgical procedures and determining whether the input data corresponds to the expected behavior. A computer-aided method for simulating a surgical procedure according to embodiment 11, further comprising presenting an instructor with the option to initiate an instructional dialogue with a user.

[0243] Appearance 19. This involves defining the expected surgical steps of a simulated surgical procedure and monitoring user input data for the simulated surgical procedure. The system determines that the input data corresponds to the expected surgical steps of the simulated surgical procedure, A computer-aided method for simulating a surgical procedure according to embodiment 11, further comprising presenting the user with the option to view corresponding segments of one or more actual surgical procedure videos, wherein the one or more actual surgical procedure videos include segments indexed to surgical steps.

[0244] Embodiment 20. A computer-assisted method for simulating a surgical procedure as described in Embodiment 19, wherein the option to view corresponding segments of one or more actual surgical procedure videos is further customized depending on surgical complications or outcomes.

[0245] [Implementation Method] (1) A computing device, wherein the computing device is The processor comprises, Receiving first two-dimensional (2D) image data of a first human organ, wherein the first 2D image data includes first 2D visible image data and first three-dimensional (3D) reconstruction reference data captured within the body of a first patient. Receiving second 2D image data of a second human organ, wherein the second 2D image data includes second 2D visible image data captured within the body of a second patient and second 3D reconstruction reference data. The method involves generating 3D image data from the aggregation of the first 2D visible image data and the second 2D visible image data, based on the first 3D reconstruction reference data and the second 3D reconstruction reference data, wherein the 3D image data includes spatial information and visual information. Mapping the shape of a deformable 3D model of a human organ to correspond to the spatial information of the 3D image data, Mapping the surface of the deformable 3D model of the human organ to correspond to the visual information of the 3D image data, A computing device configured to output the aforementioned deformable 3D model. (2) The computing device according to Embodiment 1, wherein the processor is configured to simulate a surgical procedure. (3) The computing device according to Embodiment 1, wherein the 3D model of the human organ is a 3D model of a simulated human organ. (4) The computing device according to Embodiment 1, wherein the first human organ and the second human organ are human organs of the same type. (5) The computing device according to Embodiment 1, wherein the first patient is the same as the second patient.

[0246] (6) The computing device according to Embodiment 1, wherein the 2D image data is captured using structured light imaging. (7) The processor is Receiving tissue characteristic data of one or more human organs, wherein the tissue characteristic data is determined by tracking the displacement of at least one physical reference marker attached to the one or more human organs. The computing device according to Embodiment 1, further configured to perform the following: when a simulated force is applied to the deformable 3D model, deform the deformable 3D model based on the physical property data. (8) The computing device according to Embodiment 1, wherein the first 2D image data further comprises first invisible image data captured in the body of a first patient, the second 2D image data further comprises second invisible image data captured in the body of a second patient, the processor is further configured to generate the 3D image data from the aggregate of the first visible image data and the second visible image data based on the first and second criteria and the first invisible image data and the second invisible image data, the 3D image data further comprises non-visual information, and the processor is further configured to generate subsurface data of the deformable 3D model of the human organ based on the non-visual information of the 3D image data. (9) The processor is Receive patient-specific data, The computing device according to Embodiment 1, further configured to map the surface of the deformable 3D model of the human organ to correspond to the visual information and patient-specific data of the 3D image data. (10) The processor Defining the expected mode of the simulated surgical procedure, wherein the expected mode is one of a surgical step, a job which is part of the surgical step, a location in a surgical scene, or a surgical instrument, The monitoring of user input data for the aforementioned simulated surgical procedure, The computing device according to Embodiment 2, further configured to perform the following actions: when it is determined that the input data corresponds to the expected configuration, it presents the relevant instructions to the user.

[0247] (11) The processor is Defining the expected mode of the simulated surgical procedure, wherein the expected mode is one of a surgical step, a job which is part of the surgical step, a location within the surgical site, or a surgical instrument, monitoring input data from a user for the simulated surgical procedure; when it is determined that the input data corresponds to the expected manner, further configured to present an option to start an instructional dialogue with the user to an instructor, the computing device according to Embodiment 2. (12) The processor defines an expected surgical step of the simulated surgical procedure; monitors input data from the user for the simulated surgical procedure; when it is determined that the input data corresponds to the expected surgical step, presenting an option to view a corresponding segment of one or more actual surgical procedure videos to the user, wherein the one or more actual surgical procedure videos include segments indexed by surgical steps, further configured to perform, the computing device according to Embodiment 2. (13) The option to view the corresponding segment of the one or more actual surgical procedure videos can be further customized by surgical complications or surgical results, the computing device according to Embodiment 12. (14) A computer-implemented method, the computer-implemented method comprising: receiving first two-dimensional (2D) image data of a first human organ, the first 2D image data including first 2D visible image data captured within a first patient's body and first three-dimensional (3D) reconstruction reference data; receiving second 2D image data of a second human organ, the second 2D image data including second 2D visible image data captured within a second patient's body and second 3D reconstruction reference data; generating 3D image data based on the first 3D reconstruction reference data and the second 3D reconstruction reference data from an aggregation of the first 2D visible image data and the second 2D visible image data, the 3D image data including spatial information and visual information; Mapping the shape of a deformable 3D model of a human organ to correspond to the spatial information of the 3D image data, Mapping the surface of the deformable 3D model of the human organ to correspond to the visual information of the 3D image data, A computer implementation method comprising outputting the deformable 3D model. (15) The computer implementation method according to Embodiment 14, wherein the 3D model of the human organ is a 3D model of a simulated human organ.

[0248] (16) The computer implementation method according to Embodiment 14, wherein the first human organ and the second human organ are human organs of the same type. (17) The computer-assisted method according to Embodiment 14, wherein the first patient is the same as the second patient. (18) The computer implementation method according to Embodiment 14, wherein the 2D image data is captured using structured light imaging. (19) Receiving tissue characteristic data of one or more human organs, wherein the tissue characteristic data is determined by tracking the displacement of at least one physical reference marker attached to the one or more human organs, A computer-aided method according to Embodiment 14, further comprising: applying a simulated force to the deformable 3D model; and deforming the deformable 3D model based on the physical property data. (20) The computer implementation method according to Embodiment 14, wherein the first 2D image data further comprises first invisible image data captured in the body of a first patient, the second 2D image data further comprises second invisible image data captured in the body of a second patient, the processor is further configured to generate the 3D image data from the aggregate of the first visible image data and the second visible image data based on the first and second criteria and the first invisible image data and the second invisible image data, the 3D image data further comprises non-visual information, and the processor is further configured to generate subsurface data of the deformable 3D model of the human organ based on the non-visual information of the 3D image data.

[0249] (21) The processor is Receiving patient-specific data, and A computer-aided method according to Embodiment 14, further configured to perform mapping the surface of the deformable 3D model of the human organ to correspond to the visual information and patient-specific data of the 3D image data. (22) Defining the expected mode of the simulated surgical procedure, wherein the expected mode is one of a surgical step, a job which is part of the surgical step, a location in a surgical scene, or a surgical instrument, The monitoring of user input data for the aforementioned simulated surgical procedure, A computer implementation method according to embodiment 14, further comprising: when it is determined that the input data corresponds to the expected configuration, presenting the relevant instructions to the user. (23) Defining the expected mode of the simulated surgical procedure, wherein the expected mode is one of a surgical step, a job which is part of the surgical step, a location within the surgical site, or a surgical instrument, The monitoring of user input data for the aforementioned simulated surgical procedure, The computer implementation method according to Embodiment 14, further comprising presenting the instructor with the option to initiate an instructional dialogue with the user when it is determined that the input data corresponds to the expected configuration. (24) The processor is To define the expected surgical steps of the simulated surgical procedure, Monitoring input data from users regarding the simulated surgical procedure, and The computer implementation method according to Embodiment 14, further configured to perform the following: when it is determined that the input data corresponds to the expected surgical step, present the user with the option to view a corresponding segment of one or more actual surgical procedure videos, wherein the one or more actual surgical procedure videos include segments indexed to the surgical step. (25) The computer-assisted method according to Embodiment 24, wherein the option of viewing the corresponding segments of one or more actual surgical procedure videos may be further customized depending on surgical complications or surgical outcomes.

[0250] (26) A computer-readable medium that, when executed by a computer, includes instructions causing the computer to perform the method described in Embodiment 14.

Claims

1. A computing device, wherein the computing device is The processor comprises, The method involves receiving first two-dimensional (2D) image data of a first human organ, wherein the first 2D image data includes first 2D visible image data and first three-dimensional (3D) reconstruction reference data captured within the body of a first patient. The method involves receiving a second 2D image data of a second human organ, wherein the second 2D image data includes a second 2D visible image data captured within the body of a second patient and a second 3D reconstruction reference data. The method involves generating 3D image data from the aggregation of the first 2D visible image data and the second 2D visible image data, based on the first 3D reconstruction reference data and the second 3D reconstruction reference data, wherein the 3D image data includes spatial information and visual information. Mapping the shape of a deformable 3D model of a human organ to correspond to the spatial information of the 3D image data, Mapping the surface of the deformable 3D model of the human organ to correspond to the visual information of the 3D image data, Outputting the aforementioned deformable 3D model, The method involves receiving tissue characteristic data of the human organ, wherein the tissue characteristic data is determined by tracking the displacement of a physical reference marker attached to the human organ, A computing device configured to perform the following actions: when a simulated force is applied to the deformable 3D model, deform the deformable 3D model based on the tissue characteristics data.

2. The computing device according to claim 1, wherein the processor is configured to simulate a surgical procedure.

3. The computing device according to claim 1, wherein the 3D model of the human organ is a 3D model of a simulated human organ.

4. The computing device according to claim 1, wherein the first human organ and the second human organ are human organs of the same type.

5. The computing device according to claim 1, wherein the first patient is the same as the second patient.

6. The computing device according to claim 1, wherein the 2D image data is captured using structured light imaging.

7. The computing device according to claim 1, wherein the first 2D image data further comprises first invisible image data captured in the body of a first patient, the second 2D image data further comprises second invisible image data captured in the body of a second patient, the processor is further configured to generate the 3D image data from the aggregation of the first 2D visible image data and the second 2D visible image data based on the first 3D reconstruction reference data and the second 3D reconstruction reference data, and based on the first invisible image data and the second invisible image data, the 3D image data further comprises non-visual information, and the processor is further configured to generate subsurface data of the deformable 3D model of the human organ based on the non-visual information of the 3D image data.

8. The aforementioned processor, Receive patient-specific data, The computing device according to claim 1, further configured to map the surface of the deformable 3D model of the human organ to correspond to the visual information and patient-specific data of the 3D image data.

9. The aforementioned processor, Defining the expected mode of a simulated surgical procedure, wherein the expected mode is one of a surgical step, a job which is part of the surgical step, a location in a surgical scene, or a surgical instrument. The monitoring of user input data for the aforementioned simulated surgical procedure, The computing device according to claim 2, further configured to perform the following actions: when it is determined that the input data corresponds to the expected mode, it presents the relevant command to the user.

10. The aforementioned processor, Defining the expected mode of a simulated surgical procedure, wherein the expected mode is one of a surgical step, a job which is part of the surgical step, a location within the surgical site, or a surgical instrument. The monitoring of user input data for the aforementioned simulated surgical procedure, The computing device according to claim 2, further configured to perform the following: when it is determined that the input data corresponds to the expected configuration, it presents the instructor with the option to initiate an instructional dialogue with the user.

11. The aforementioned processor, To define the expected surgical steps of a simulated surgical procedure, Monitoring input data from users regarding the simulated surgical procedure, The computing device according to claim 2, further configured to perform the following: when it is determined that the input data corresponds to the expected surgical step, present the user with the option to view a corresponding segment of one or more actual surgical procedure videos, wherein the one or more actual surgical procedure videos include segments indexed to the surgical step.

12. The computing device according to claim 11, wherein the option to view the corresponding segments of one or more actual surgical procedure videos may be further customized depending on surgical complications or surgical outcomes.

13. A computer implementation method, wherein the computer implementation method is The processor, The method involves receiving first two-dimensional (2D) image data of a first human organ, wherein the first 2D image data includes first 2D visible image data and first three-dimensional (3D) reconstruction reference data captured within the body of a first patient. The method involves receiving a second 2D image data of a second human organ, wherein the second 2D image data includes a second 2D visible image data captured within the body of a second patient and a second 3D reconstruction reference data. The method involves generating 3D image data from the aggregation of the first 2D visible image data and the second 2D visible image data, based on the first 3D reconstruction reference data and the second 3D reconstruction reference data, wherein the 3D image data includes spatial information and visual information. Mapping the shape of a deformable 3D model of a human organ to correspond to the spatial information of the 3D image data, Mapping the surface of the deformable 3D model of the human organ to correspond to the visual information of the 3D image data, Outputting the aforementioned deformable 3D model, The method involves receiving tissue characteristic data of the human organ, wherein the tissue characteristic data is determined by tracking the displacement of a physical reference marker attached to the human organ, A computer implementation method comprising: applying a simulated force to the deformable 3D model, and then deforming the deformable 3D model based on the tissue characteristics data.

14. The computer implementation method according to claim 13, wherein the 3D model of the human organ is a 3D model of a simulated human organ.

15. The computer implementation method according to claim 13, wherein the first human organ and the second human organ are human organs of the same type.

16. The computer-assisted method according to claim 13, wherein the first patient is the same as the second patient.

17. The computer-aided method according to claim 13, wherein the 2D image data is captured using structured optical imaging.

18. Computerized method according to claim 13, wherein the first 2D image data further comprises first invisible image data captured in the body of a first patient, the second 2D image data further comprises second invisible image data captured in the body of a second patient, the processor is further configured to generate the 3D image data from the aggregation of the first 2D visible image data and the second 2D visible image data based on the first 3D reconstruction reference data and the second 3D reconstruction reference data, and based on the first invisible image data and the second invisible image data, the 3D image data further comprises non-visual information, and the processor is further configured to generate subsurface data of the deformable 3D model of the human organ based on the non-visual information of the 3D image data.

19. The aforementioned processor, Receiving patient-specific data, and The computer implementation method according to claim 13, further configured to perform mapping the surface of the deformable 3D model of the human organ to correspond to the visual information and patient-specific data of the 3D image data.

20. The processor is Defining the expected mode of a simulated surgical procedure, wherein the expected mode is one of a surgical step, a job which is part of the surgical step, a location in a surgical scene, or a surgical instrument. The monitoring of user input data for the aforementioned simulated surgical procedure, The computer implementation method according to claim 13, further configured to perform the following: when it is determined that the input data corresponds to the expected configuration, the computer presents the relevant command to the user.

21. The processor is Defining the expected mode of a simulated surgical procedure, wherein the expected mode is one of a surgical step, a job which is part of the surgical step, a location within the surgical site, or a surgical instrument. The monitoring of user input data for the aforementioned simulated surgical procedure, The computer implementation method according to claim 13, further configured to perform the following: when it is determined that the input data corresponds to the expected configuration, the instructor is presented with the option to initiate an instructional dialogue with the user.

22. The aforementioned processor, To define the expected surgical steps of a simulated surgical procedure, Monitoring input data from users regarding the simulated surgical procedure, and The computer implementation method according to claim 13, further configured to, when it is determined that the input data corresponds to the expected surgical step, present the user with the option to view a corresponding segment of one or more actual surgical procedure videos, wherein the one or more actual surgical procedure videos include segments indexed to the surgical step.

23. The computer-aided method according to claim 22, wherein the option of viewing the corresponding segments of one or more actual surgical procedure videos may be further customized depending on surgical complications or surgical outcomes.

24. A computer-readable medium that, when executed by a computer, includes instructions causing the computer to perform the method described in claim 13.

Images