Instruction-based, simulation-based surgical development system

Abstract

The interactive and dynamic surgical simulation system may be used in conjunction with a computer-implemented interactive surgical system. The surgical simulation system may enable instruction-based, simulation-based surgical development. The processor may be configured to receive a plurality of training task-environment parameter sets from a surgical data system. The processor may generate a plurality of candidate action sets for each training task-environment parameter set. Each candidate action set, when executed within a simulated environment represented by the corresponding training task-environment set, may cause the simulated surgical task to be performed with a corresponding effectiveness. This information may be used to train a model and then determine a target action set. The target action set may represent a way to perform the simulated surgical task optimized according to the defined effectiveness.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 191,681, filed on May 21, 2021, the entire disclosure of which is incorporated herein by reference.

[0002] This application is related to the following applications filed simultaneously, the content of each of which is incorporated herein by reference. ·U.S. Patent Application No. 17 / 332,594, entitled "METHODS FOR SURGICAL SIMULATION", filed on May 27, 2021 (Attorney Docket No. END9338USNP1) ·U.S. Patent Application No. 17 / 332,524, entitled "SURGICAL SIMULATION OBJECT RECTIFICATION SYSTEM", filed on May 27, 2021 (Attorney Docket No. END9338USNP2) ·U.S. Patent Application No. 17 / 332,399, entitled "SURGICAL SIMULATION NAVIGATION SYSTEM", filed on May 27, 2021 (Attorney Docket No. END9338USNP3) ·U.S. Patent Application No. 17 / 332,441, entitled "SURGICAL SIMULATION SYSTEM WITH COORDINATED IMAGINING", filed on May 27, 2,021 (Attorney Docket No. END9338USNP4) ·U.S. Patent Application No. 17 / 332,462, entitled "SURGICAL SIMULATION SYSTEM WITH SIMULATED SURGICAL EQUIPMENT COORDINATION", filed on May 27, 2021 (Attorney Docket No. END9338USNP5) ·U.S. Patent Application No. 17 / 332,197, entitled "SIMULATION - BASED SURGICAL PROCEDURE PLANNING SYSTEM", filed on May 27, 2021 (Attorney Docket No. END9338USNP6) • U.S. Patent Application No. 17 / 332,449, filed on May 27, 2021, entitled "SURGICAL ADVERSE EVENT SIMULATION SYSTEM" (Agent Reference Number END9338USNP8) • U.S. Patent Application No. 17 / 332,496, filed on May 27, 2021, entitled "SIMULATION-BASED SURGICAL ANALYSIS SYSTEM" (Agent Reference Number END9338USNP9) • U.S. Patent Application No. 17 / 332,480, filed on May 27, 2021, entitled "DYNAMIC ADAPTATION SYSTEM FOR SURGICAL SIMULATION" (Agent Reference Number END9338USNP10) [Background technology]

[0003] For example, surgical simulations, such as computer-based three-dimensional simulations of the surgical environment and / or surgical procedures, present opportunities to advance surgical techniques. Surgical simulations have potential to be useful for surgical training, planning, and development. For instance, surgical simulations can be used to train surgeons in new procedures and / or to improve surgeons' performance of known procedures. Surgical simulations can be used as a virtual "final rehearsal" to help surgeons prepare for the next procedure. Furthermore, surgical simulations can be used for experimentation with unproven procedures and techniques.

[0004] However, surgical simulation platforms are complex systems that face many limitations in terms of capability, scope, and applicability. For example, many platforms are technology "silos" that are specifically programmed and tuned to address particular learning objectives or to simulate the operation of a single instrument, such as simulating the movement of a surgical robot. Such limitations can reduce the effectiveness of the platform as a tool for advancing surgical techniques. Furthermore, such limitations can represent a significant technical obstacle to integrating simulation-based applications into other aspects of the surgical process, such as preoperative planning, intraoperative support, and postoperative analysis. [Overview of the project] [Problems that the invention aims to solve]

[0005] Therefore, innovations in surgical simulation technology, such as technological advancements addressing surgical simulation capabilities, scope, and applicability, can accelerate further progress in the field of surgery. [Means for solving the problem]

[0006] An interactive and dynamic surgical simulation system is disclosed. The surgical simulation system can be used in conjunction with a computer-implemented interactive surgical system. The surgical simulation system can enable instruction-based, simulation-based surgical development. The device can be used to determine a target action set for performing a simulated surgical task in a simulated environment. For example, the target action set may represent a method for performing a simulated surgical task optimized according to a defined effectiveness metric.

[0007] The processor may be configured to receive multiple training task-environment parameter sets from a surgical data system. Each training task-environment parameter set may include anatomical information, physiological information, surgical setup information, etc. Each training task-environment parameter set may correspond to a simulated environment in which a simulated surgical task is to be completed.

[0008] The processor may generate multiple candidate action sets for each training task-environment parameter set. Each candidate action set may include parameterized surgical activities within its respective simulated environment. For example, a simulated dissection activity may be parameterized by relative anatomical position, instrument selection, and direction of instrument application. When each candidate action set is executed within the simulated environment specified by the corresponding training task-environment set, it may perform a simulated surgical task. This execution may result in the respective effectiveness indicated by the corresponding effectiveness metric.

[0009] This information can be used to train a model. The processor may train the model using multiple task-environment parameter sets, each with multiple candidate action sets, and each with its corresponding effectiveness metric. The processor may also use the model to determine a target action set, then based on the target task-environment parameter set.

[0010] The present invention provides a computer implementation method for determining a target action set, the method comprising: receiving a plurality of environmental parameter sets, each environmental parameter set corresponding to a simulated environment in which a simulated surgical task is to be completed; generating a plurality of candidate action sets for each environmental parameter set, each candidate action set, when executed in its respective simulated environment, to perform its respective simulated surgical task with effectiveness indicated by a corresponding effectiveness metric; training a machine learning model using the plurality of environmental parameter sets, each plurality of candidate action sets, and each corresponding effectiveness metric, such that the trained machine learning model may include a functional relationship between model inputs and model outputs, wherein the model inputs include an input environmental parameter set and an input action set, and the outputs include an output effectiveness metric; and determining a target action set based on a target environmental parameter set and a corresponding effectiveness metric output by the trained machine learning model.

[0011] It will be understood that an action set (e.g., a candidate action set, a target action set, or an input action set) can refer to a set of parameters that define one or more actions performed during surgery. In some embodiments, an action set can refer to several surgical choices made by the operator. In some embodiments, an action set can include decisions regarding a particular layout (both surgical staff and surgical instruments). An action set can also refer to specific instruments used during surgery. In other words, an action set can refer to a set of surgical procedures that affect the surgery in terms of both duration, surgical outcome, and effectiveness of the procedure. It should be understood that candidate action sets, target action sets, and input action sets are all datasets having the same format. That is, a machine learning model can be trained on a candidate action set to help define relationships between any "input" action sets. This allows the machine learning model to be used to find a target action set with certain effectiveness within a given simulation.

[0012] The environmental parameter set (e.g., environmental parameter set, target environmental parameter set, or input environmental parameter set) can also be understood as a set of parameters that define the overall environment of a surgical simulation. In other words, the environmental parameter set refers to the parameters that initialize and define the constraints and physical properties of the surgical simulation. It should be understood that both the target parameter set and the environmental parameter set are datasets with the same format as the environmental parameter set. That is, a machine learning model can be trained on the environmental parameter set to help define the relationships between any "input" environmental parameter sets. This allows the machine learning model to be used to find a target action set that has some effectiveness within a given simulation.

[0013] In some embodiments, “training a machine learning model” may refer to training a model in a supervised manner using a labeled dataset. In this case, a combination of environment parameter sets and action sets can be considered input data (as it refers to a specific set of actions performed within a particular simulation environment). This data for training purposes may be “labeled” with its associated effectiveness metric (i.e., the simulation output regarding how effective a particular action was within that simulation). By feeding this labeled data to a machine learning model, the model can generate relationships between effectiveness associated with different actions / environments.

[0014] To avoid misunderstanding, the “environmental parameter set” may be the same parameter set that the application refers to as the “training task-environmental parameter set” (i.e., the parameter set that may be used to train a machine learning model).

[0015] Both the initialization of the surgical simulation (defined by the environment parameter set) and the specific choices / actions performed within the simulation (defined by the candidate action set) affect the overall effectiveness of the surgery (defined by the effectiveness metric).

[0016] In some embodiments, "determining the target action set" may refer to selecting a set of actions from a given set of action sets within a simulation (as defined by a set of environmental parameters). The target action set may be selected based on the target action set that produces the best output efficiency metric from the machine learning model (as input to the machine learning model).

[0017] In other words, the present invention improves the training of a machine learning model using the output data of a simulation, and then uses the improved trained artificial intelligence to draw improved conclusions regarding surgical actions / selections that can be performed within a particular surgical setup.

[0018] One advantage of this is that a machine learning model (such as a neural network) can be trained using additional data that would not normally be available during a live surgical procedure. In other words, the simulation can provide the machine learning model with data regarding "riskier" surgical options and their outcomes. Intelligence based on information from the machine learning model can inform future surgical decisions by highlighting decisions that have a truly beneficial effect without the need for an actual patient to bear the risk in order to collect the appropriate data.

[0019] The result of improving the training of the machine learning model by the method of the present embodiment is that improved surgical results can be achieved by the determined target action set.

[0020] In some embodiments, receiving a plurality of environmental parameter sets includes receiving a plurality of environmental parameter sets from a surgical data system that records real-world environmental data from the execution of a surgical task in the real world and generates a plurality of environmental parameter sets from the real-world environmental data.

[0021] In some embodiments, each environmental parameter set includes anatomical information, physiological information, and surgical setup information.

[0022] In some embodiments, each candidate action set includes parameterized surgical activities within their respective simulated environments.

[0023] In some embodiments, the parameterized activity includes a simulated dissection parameterized by any of relative anatomical position, instrument selection, and instrument application direction.

[0024] In some embodiments, the parameterized activity includes simulated suturing parameterized by any one of suture position, suture pattern, number of sutures, suture material, and suture tension.

[0025] In some embodiments, the parameterized activity includes a surgical robot guidance path.

[0026] In some embodiments, the simulated surgical task includes a lower anterior resection anastomosis, and each candidate action set, when executed in a simulated environment indicated by a corresponding environment set, executes the simulated surgical task with an effectiveness indicated by a corresponding effectiveness metric, and the corresponding effectiveness metric is an indicator of perfusion resulting from the execution of the simulated surgical task.

[0027] In some embodiments, the simulated surgical task includes either a segmentectomy or a lobectomy, and each candidate action set, when executed in a simulated environment indicated by a corresponding environment set, executes the simulated surgical task with an effectiveness indicated by a corresponding effectiveness metric, and the corresponding effectiveness metric is an indicator of the output lung volume resulting from the execution of the simulated surgical task.

[0028] In some embodiments, the method further includes generating an output that is a human-readable aggregation of a plurality of task environment parameter sets, each of the plurality of candidate action sets, and each corresponding effectiveness metric.

[0029] In some embodiments, training the machine learning model includes training the machine learning model using any one of a supervised learning algorithm, an unsupervised learning algorithm, and a reinforcement learning algorithm.

[0030] The present invention also provides a device for determining a target action set, the device comprising a processor, the processor receiving a plurality of environmental parameter sets from a surgical data system, each environmental parameter set corresponding to a simulated environment in which a simulated surgical task is to be completed; generating a plurality of candidate action sets for each environmental parameter set, each candidate action set, when executed in the respective simulated environment corresponding to the corresponding environmental set, causing the respective simulated surgical task to be performed with effectiveness indicated by the corresponding effectiveness metric; training a machine learning model using the plurality of environmental parameter sets, each plurality of candidate action sets, and each corresponding effectiveness metric, such that the trained machine learning model may include a functional relationship between the model input and the model output, the model input including an input environmental parameter set and an input action set, and the output including an output effectiveness metric; and determining a target action set based on a target environmental parameter set and the corresponding effectiveness metric output by the trained machine learning model.

[0031] In some embodiments, the surgical data system is configured to record real-world environmental data from the real-world execution of a surgical task and to further configure to generate multiple environmental parameter sets from the real-world environmental data.

[0032] In some embodiments, each environmental parameter set includes anatomical information, physiological information, and surgical setup information.

[0033] In some embodiments, each candidate action set includes parameterized surgical activities within their respective simulated environments.

[0034] In some embodiments, the parameterized activity includes simulated dissection, which is parameterized by either the relative anatomical position, the selection of the instrument, or the direction of application of the instrument.

[0035] In some embodiments, the parameterized activity includes simulated suturing, which is parameterized by one of the following: suture location, suture pattern, number of sutures, suture material, and suture tension.

[0036] In some embodiments, the parameterized activity includes a surgical robot guidance pathway.

[0037] In some embodiments, the simulated surgical task includes a lower anterior resection anastomosis, and each candidate action set is performed within a simulated environment indicated by the corresponding environment set, and the simulated surgical task is performed with the effectiveness indicated by the corresponding effectiveness metric, which is an indicator of perfusion resulting from the execution of the simulated surgical task.

[0038] In some embodiments, the simulated surgical task includes either a segmentectomy or a lobectomy, and each candidate action set is executed within a simulated environment indicated by the corresponding environment set, and the simulated surgical task is executed with the effectiveness indicated by the corresponding effectiveness metric, which is an indicator of the output lung volume resulting from the execution of the simulated surgical task.

[0039] In some embodiments, the processor is further configured to generate an output which is a human-readable aggregate of multiple task environment parameter sets, multiple candidate action sets for each, and their respective corresponding effectiveness metrics.

[0040] In some embodiments, the processor is configured to train a machine learning model using one of the following: a supervised learning algorithm, an unsupervised learning algorithm, or a reinforcement learning algorithm.

[0041] The present invention also provides a device for determining a target action set, the device comprising a processor, the processor receiving a plurality of environmental parameter sets from a surgical data system, each environmental parameter set corresponding to a simulated environment in which a simulated surgical task is to be completed; generating a plurality of candidate action sets for each environmental parameter set, each candidate action set, when executed in its respective simulated environment, causing the simulated surgical task to be performed with effectiveness indicated by a corresponding effectiveness metric; training a machine learning model using the plurality of environmental parameter sets, each plurality of candidate action sets, and each corresponding effectiveness metric, such that the trained machine learning model includes a functional relationship between the model input and the model output, the model input includes an input environmental parameter set and an input action set, and the output includes an output effectiveness metric; and determining a target action set based on a target environmental parameter set and a corresponding effectiveness metric output by the trained machine learning model.

[0042] In some embodiments, the surgical data system is configured to record real-world environmental data from the real-world execution of a surgical task and to further configure to generate multiple environmental parameter sets from the real-world environmental data.

[0043] In some embodiments, each environmental parameter set includes anatomical information, physiological information, and surgical setup information.

[0044] In some embodiments, each candidate action set includes parameterized surgical activities within their respective simulated environments.

[0045] The present invention also provides a device for determining a target action set, the device comprising a processor, the processor receiving a machine learning model trained with a plurality of environmental parameter sets, a plurality of candidate action sets, and a corresponding effectiveness metric, such that the trained machine learning model includes a functional relationship between the model input and the model output, wherein the model input includes an input environmental parameter set and an input action set, and the output includes an output effectiveness metric, and the device determines a target action set based on the environmental parameter sets and the machine learning model, where each environmental parameter set corresponds to a simulated environment in which a simulated surgical task is to be completed, the plurality of environmental parameter sets are from a surgical data system, and for each environmental parameter set, the plurality of candidate action sets, when executed in the simulated environment corresponding to the corresponding environmental set, execute the simulated surgical task with effectiveness indicated by the corresponding effectiveness metric.

[0046] In some embodiments, each environmental parameter set includes anatomical information, physiological information, and surgical setup information, and each candidate action set includes parameterized surgical activities within their respective simulated environments.

[0047] According to the present invention, a computer-readable medium is also provided which, when executed by a computer, contains instructions causing the computer to perform the method according to the present invention. [Brief explanation of the drawing]

[0048] [Figure 1] This is a block diagram of a computer-implemented interactive surgical system. [Figure 2] This shows an exemplary surgical system used to perform surgical procedures in an operating room. [Figure 3] This disclosure illustrates, in at least one aspect, a visualization system, a robotic system, and an exemplary surgical hub paired with an intelligent instrument. [Figure 4] The present disclosure, in at least one aspect, describes a surgical data network having a communication hub configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to the cloud. [Figure 5] This figure shows an exemplary computer-implemented interactive surgical system. [Figure 6] An exemplary surgical hub is shown, including multiple modules connected to a modular control tower. [Figure 7] This is a block diagram of an exemplary surgical simulator system. [Figure 8] This is a block diagram of an exemplary surgical simulator system. [Figure 9] This is a block diagram depicting the user interface device of an exemplary surgical simulator. [Figure 10] This is a flowchart illustrating the operation of an illustrative surgical simulator. [Figure 11A] This document describes the data structure of an exemplary surgical procedure plan for use in a computer-implemented interactive surgical system and / or surgical simulator. [Figure 11B] This document describes the data structure of an exemplary surgical procedure plan for use in a computer-implemented interactive surgical system and / or surgical simulator. [Figure 12] This is a block diagram of an exemplary machine learning framework. [Figure 13] This is a block diagram of an exemplary surgeon agent. [Figure 14] An exemplary simulation activity parameterized dataset is shown. [Figure 15] This document presents an exemplary surgical simulation flow for machine learning training. [Figure 16] This is a flowchart illustrating the operation of an exemplary surgical simulation. [Modes for carrying out the invention]

[0049] Surgical simulation systems, devices, and methods may include forms of integration with other medical devices, data sources, processes, and institutions. For example, surgical simulation systems, devices, and methods may include forms of integration with computer-implemented interactive surgical systems and / or one or more elements of computer-implemented interactive surgical systems.

[0050] Referring to Figure 1, the computer-implemented interactive surgical system 100 may comprise one or more surgical systems 102 and a cloud-based system (for example, a cloud 104 which may include a remote server 113 connected to a storage device 105). Each surgical system 102 may comprise at least one surgical hub 106 that communicates with the cloud 104 which may include a remote server 113.

[0051] One or more simulation devices 103, 111 may communicate with and / or be integrated as part of the computer-implemented interactive surgical system 100. For example, simulation device 103 may be an element of one or more surgical systems 102. For example, simulation device 103 may communicate with one or more surgical hubs 106. For example, simulation device 111 may communicate with the computer-implemented interactive surgical system 100 via the cloud 104.

[0052] In one example, as shown in Figure 1, the surgical system 102 includes a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112, which are configured to communicate with each other and / or with a hub 106. In some embodiments, the surgical system 102 may comprise M hubs 106, N visualization systems 108, O robotic systems 110, and P handheld intelligent surgical instruments 112, where M, N, O, and P are integers of 1 or more.

[0053] In various embodiments, the visualization system 108 may comprise one or more imaging sensors strategically positioned relative to a sterile field, one or more image processing units, one or more storage arrays, and one or more displays, as shown in Figure 2. In one embodiment, the visualization system 108 may include interfaces for HL7, PACS, and EMR. Various components of the visualization system 108 are described under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety.

[0054] As shown in Figure 2, the primary display 119 is positioned in the sterile field so that it is visible to the operator on the operating table 114. In addition, a visualization tower 111 is positioned outside the sterile field. The visualization tower 111 may comprise a first non-sterile display 107 and a second non-sterile display 109, facing opposite directions from each other. The visualization system 108, guided by the hub 106, is configured to utilize displays 107, 109, and 119 to coordinate the flow of information to operators inside and outside the sterile field. For example, the hub 106 can cause the visualization system 108 to display snapshots of the surgical site recorded by the imaging device 124 on the non-sterile displays 107 or 109 while maintaining live video of the surgical site on the primary display 119. The snapshots on the non-sterile displays 107 or 109 allow, for example, a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0055] In one embodiment, the hub 106 can also be configured to send diagnostic input or feedback entered by a non-sterile operator in the visualization tower 111 to a primary display 119 in the sterile field, which can then be viewed by a sterile operator on the operating table. In one example, the input may take the form of modifications to a snapshot displayed on the non-sterile display 107 or 109, which can then be sent to the primary display 119 by the hub 106.

[0056] Referring to Figure 2, the surgical instrument 112 is used as part of the surgical system 102 in a surgical procedure. The hub 106 can also be configured to coordinate the flow of information to the display of the surgical instrument 112. For example, in U.S. Patent Application Publication 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety. Diagnostic input or feedback entered by a non-sterile operator in the visualization tower 111 can be sent by the hub 106 to the surgical instrument display 115 in the sterile field, which can then be viewed by the operator of the surgical instrument 112. Examples of surgical instruments suitable for use in the surgical system 102 are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385) filed December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety.

[0057] Figure 2 shows an example of a surgical system 102 used to perform a surgical procedure on a patient lying on an operating table 114 in an operating room 116. A robotic system 110 may be used as part of the surgical system 102 in a surgical procedure. The robotic system 110 may include a surgeon's console 118, a patient-side cart 120 (surgical robot), and a surgical robot hub 122. While the surgeon views the surgical site through the surgeon's console 118, the patient-side cart 120 may operate at least one detachably connected surgical tool 117 through a minimally invasive incision in the patient's body. Images of the surgical site may be acquired by a medical imaging device 124, which can be operated by the patient-side cart 120 to change the orientation of the imaging device 124. Images of the surgical site may be processed using the robotic hub 122 and then displayed to the surgeon through the surgeon's console 118.

[0058] Other types of robotic systems can be readily adapted for use with surgical system 102. Various examples of robotic systems and surgical tools suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0201137(A1) (U.S. Patent Application No. 16 / 209,407), filed 4 December 2018, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," the disclosure of which is incorporated herein by reference in its entirety.

[0059] Various examples of cloud-based analytical methods implemented by Cloud104 and suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1) (U.S. Patent Application No. 16 / 209,403), filed 4 December 2018, entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," the disclosure of which is incorporated herein by reference in its entirety.

[0060] In various embodiments, the imaging device 124 may include at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0061] The optical components of the imaging device 124 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors can receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0062] One or more illumination sources may be configured to emit electromagnetic energy in the visible and invisible spectra. The visible spectrum, sometimes also called the light spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye), and is sometimes called visible light or simply light. The typical human eye responds to wavelengths in air from approximately 380 nm to approximately 750 nm.

[0063] The invisible spectrum (e.g., the non-emission spectrum) is a portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths below approximately 380 nm and above approximately 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths above approximately 750 nm are longer than the red visible spectrum and consist of invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths below approximately 380 nm are shorter than the violet spectrum and consist of invisible ultraviolet, X-rays, and gamma-ray electromagnetic radiation.

[0064] In various embodiments, the imaging device 124 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cystoscopes, duodenoscopes, intestinaloscopes, esophagogastroduodenoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngolaryngoscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes.

[0065] The imaging device may employ a multispectral monitor to distinguish between topography and underlying structures. A multispectral image captures image data within a specific wavelength range from the entire electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, such as IR and ultraviolet light. Spectral imaging makes it possible to extract additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in more detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after the surgical task is completed to perform one or more of the tests described above on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical instruments is required in any surgical procedure. The strict sanitary and sterilization conditions required in the “operating room,” i.e., the operating room or treatment room, require the highest possible sterility of all medical devices and instruments. Part of the sterilization process is the need to sterilize everything that comes into contact with the patient or enters the sterile field, including the imaging device 124 and its accessories and components. It will be understood that the sterile field may be considered a specific area that is deemed to be free of microorganisms, such as inside a tray or on a sterile towel, or the sterile field may be considered the area immediately surrounding the patient who is ready for surgical treatment. The sterile field may include cleaned team members wearing appropriate clothing, as well as all equipment and fixation devices within that area.

[0066] Referring here to Figure 3, a hub 106 is shown that communicates with a visualization system 108, a robotic system 110, and a handheld intelligent surgical instrument 112. The hub 106 includes a hub display 135, an imaging module 138, a generator module 140, a communication module 130, a processor module 132, a storage array 134, and an operating room mapping module 133. In certain embodiments, as shown in Figure 3, the hub 106 further includes a fume extraction module 126 and / or aspiration / irrigation module 128. During surgical procedures, applying energy to tissue for sealing and / or cutting is generally associated with fume extraction, aspiration of excess fluid, and / or tissue irrigation. Fluid lines, power lines, and / or data lines from different sources often become entangled during surgical procedures. Dealing with this problem during surgical procedures can result in the loss of valuable time. Untangling lines may require disconnecting them from their corresponding modules, which may require resetting the modules. The hub modular enclosure 136 provides an integrated environment for managing power lines, data lines, and fluid lines, reducing the frequency of entanglement between such lines. An aspect of the present disclosure presents a surgical hub for use in surgical procedures involving the application of energy to tissue at a surgical site. The surgical hub includes a hub enclosure and a combination generator module slidably receivable within a docking station of the hub enclosure. The docking station includes data contacts and power contacts. The combination generator module includes two or more ultrasonic energy generator components, bipolar RF energy generator components, and unipolar RF energy generator components housed in a single unit. In one aspect, the combination generator module also includes a fume exhaust component, at least one energy supply cable for connecting the combination generator module to a surgical instrument, at least one fume exhaust component configured to exhaust smoke, fluid, and / or particulate matter generated by the application of therapeutic energy to tissue, and a fluid line extending from a remote surgical site to the fume exhaust component.In one embodiment, the above-mentioned fluid line is a first fluid line, and a second fluid line extends from the remote surgical site to a suction and irrigation module slidably received within the hub enclosure. In one embodiment, the hub enclosure comprises a fluid interface. Certain surgical procedures may require the application of two or more energy types to tissue. One energy type may be more beneficial for cutting tissue, while another different energy type may be more beneficial for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Embodiments of the present disclosure present a solution in which a modular enclosure 136 of the hub is configured to house various generators and facilitate interactive communication between them. One advantage of the modular enclosure 136 of the hub is that it allows for the rapid removal and / or replacement of various modules. Embodiments of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking station having a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and the first energy generator module is slidably movable to disengage from the first power and data contacts. In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy for application to tissue, different from the first energy, and a second docking station having a second docking port including second data contacts and second power contacts, wherein the second energy generator module is slidably movable to electrically engage with the power and data contacts, and the second energy generator module is slidably movable to disengage from the second power and second data contacts.In addition, the modular surgical enclosure also includes a communication bus between a first docking port and a second docking port, configured to facilitate communication between a first energy generator module and a second energy generator module. Referring to Figure 3, an aspect of the present disclosure is presented relating to a hub modular enclosure 136 that enables modular integration of a generator module 140, a smoke evacuation module 126, and a suction / irrigation module 128. The hub modular enclosure 136 further facilitates interactive communication between modules 140, 126, and 128. The generator module 140 may be a generator module comprising integrated unipolar, bipolar, and ultrasonic components, supported within a single housing unit that is slidably inserted into the modular enclosure 136 of the hub. The generator module 140 may be configured to connect to a unipolar device 142, a bipolar device 144, and an ultrasonic device 146. Alternatively, the generator module 140 may comprise a series of unipolar generator modules, bipolar generator modules, and / or ultrasonic generator modules interacting via a hub modular enclosure 136. The hub modular enclosure 136 can be configured to facilitate the insertion of multiple generators and bidirectional communication between generators docked to the hub modular enclosure 136, so that multiple generators function as a single generator.

[0067] Figure 4 shows a surgical data network 201 comprising a modular communication hub 203 configured to connect modular devices located in one or more operating rooms of a medical facility, or any room within a medical facility equipped with specialized equipment for surgical procedures, to a cloud-based system (a cloud 204 which may include, for example, a remote server 213 connected to a storage device 205). In one embodiment, the modular communication hub 203 comprises a network hub 207 and / or a network switch 209 that communicate with a network router. The modular communication hub 203 can also be coupled to a local computer system 210 to provide local computer processing and data manipulation. The surgical data network 201 may be configured as passive, intelligent, or switching. A passive surgical data network acts as a data conduit, enabling data to go from one device (or segment) to another device (or segment) and to cloud computing resources. An intelligent surgical data network enables traffic to pass through a monitored surgical data network and includes additional feature units that constitute each port in the network hub 207 or network switch 209. An intelligent surgical data network may be referred to as a manageable hub or switch. A switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0068] Modular devices 1a-1n, located in the operating room, can be connected to a modular communication hub 203. A network hub 207 and / or a network switch 209 can be connected to a network router 211 to connect devices 1a-1n to a cloud 204 or a local computer system 210. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and manipulation. Data associated with devices 1a-1n may also be transferred to a local computer system 210 for local data processing and manipulation. Modular devices 2a-2m, located in the same operating room, can also be connected to a network switch 209. The network switch 209 can be connected to a network hub 207 and / or a network router 211 to connect devices 2a-2m to the cloud 204. Data associated with devices 2a-2n may be transferred to the cloud 204 via the network router 211 for data processing and manipulation. Data associated with devices 2a-2m may also be transferred to a local computer system 210 for local data processing and manipulation.

[0069] It will be understood that the surgical data network 201 can be expanded by interconnecting multiple network hubs 207 and / or multiple network switches 209 with multiple network routers 211. A modular communication hub 203 may be housed in a modular control tower configured to accept multiple devices 1a-1n / 2a-2m. A local computer system 210 may also be housed in the modular control tower. The modular communication hub 203 is connected to a display 212 to display images acquired by some of the devices 1a-1n / 2a-2m, for example, during a surgical procedure. In various embodiments, the devices 1a-1n / 2a-2m may include a variety of modules, particularly among modular devices that can be connected to the modular communication hub 203 of the surgical data network 201, such as an imaging module 138 connected to an endoscope, a generator module 140 connected to an energy-based surgical device, a smoke extraction module 126, a suction / irrigation module 128, a communication module 130, a processor module 132, a storage array 134, a surgical device connected to a display, and / or a non-contact sensor module.

[0070] In one embodiment, the surgical data network 201 may include a combination of a network hub, network switches, and network routers connecting devices 1a-1n / 2a-2m to the cloud. One or all of the devices 1a-1n / 2a-2m connected to the network hub or network switch can collect data in real time and transfer the data to a cloud computer for data processing and manipulation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to handle software applications. The term “cloud” may be used as a metaphor for “Internet,” but the term is not limited in that way. Thus, the term “cloud computing” may be used herein to refer to “one type of Internet-based computing,” in which various services such as servers, storage, and applications are delivered to a modular communication hub 203 and / or computer system 210 located in an operating room (e.g., a fixed, mobile, temporary, or on-site operating room or space), and to devices connected to the modular communication hub 203 and / or computer system 210 via the Internet. The cloud infrastructure may be maintained by a cloud service provider. In this context, the cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located in one or more operating rooms. The cloud computing service can perform numerous calculations based on data collected by smart surgical instruments, robots, and other computerized devices located in the operating room. The hub hardware enables multiple devices or connections to connect to a computer that communicates with cloud computing resources and storage.

[0071] By applying cloud computing data processing technology to data collected by devices 1a-1n / 2a-2m, the surgical data network can provide improved surgical outcomes, reduced costs, and improved patient satisfaction. At least some of devices 1a-1n / 2a-2m can be used to observe the condition of tissue after tissue sealing and cutting procedures to assess leakage or perfusion of the sealed tissue. At least some of devices 1a-1n / 2a-2m can be used to examine data, including images of body tissue samples, for diagnostic purposes using cloud-based computing to identify pathologies such as the effects of disease. Such data may include tissue localization and margin confirmation, as well as phenotype. At least some of devices 1a-1n / 2a-2m can be used to identify anatomical structures of the body using various sensors integrated with imaging devices and techniques such as overlaying images captured by multiple imaging devices. Data collected by devices 1a-1n / 2a-2m, including image data, may be transferred to the cloud 204 or the local computer system 210, or both, for data processing and manipulation, including image processing and manipulation. The data may be analyzed to improve the outcome of surgical procedures by determining whether further treatments, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robots, can be performed on tissue-specific sites and conditions. Such data analysis may further involve prognostic analysis processing, and the use of standardized methods can provide useful feedback for either confirming surgical treatment and surgeon behavior, or suggesting modifications to surgical treatment and surgeon behavior.

[0072] The operating room devices 1a-1n may be connected to the modular communication hub 203 via a wired or wireless channel, depending on the configuration of the devices 1a-1n with respect to the network hub. In one embodiment, the network hub 207 may be implemented as a local network broadcast device operating on the physical layer of the Open System Interconnection (OSI) model. The network hub can provide connectivity to the devices 1a-1n located within the same operating room network. The network hub 207 may collect data in packet form and transmit them to the router in half-duplex mode. The network hub 207 cannot store any media access control / Internet protocol (MAC / Internet Protocol, IP) for transferring device data. Only one of the devices 1a-1n can transmit data through the network hub 207 at a time. The network hub 207 cannot have a routing table or intelligence regarding the destination of information and broadcasts all network data to each connection and to the remote server 213 (Figure 4) on the cloud 204. While the Network Hub 207 can detect basic network errors such as collisions, broadcasting all information to multiple ports poses a security risk and could cause bottlenecks.

[0073] Operating room devices 2a-2m can be connected to network switch 209 via a wired or wireless channel. Network switch 209 operates within the data link layer of the OSI model. Network switch 209 may be a multicast device for connecting devices 2a-2m located within the same operating room to a network. Network switch 209 can transmit data in the form of frames to network router 211 and operates in full-duplex mode. Multiple devices 2a-2m can transmit data simultaneously through network switch 209. Network switch 209 stores and uses the MAC addresses of devices 2a-2m to transfer data.

[0074] The network hub 207 and / or network switch 209 may be connected to the network router 211 to connect to the cloud 204. The network router 211 operates within the network layer of the OSI model. The network router 211 creates a route for sending data packets received from the network hub 207 and / or network switch 211 to cloud-based computing resources for further processing and manipulation of data collected by one or all of the devices 1a-1n / 2a-2m. The network router 211 may be used to connect two or more different networks located in different locations, for example, different networks located in different operating rooms of the same medical facility or different networks located in different operating rooms of different medical facilities. The network router 211 can send data in packet form to the cloud 204 and operates in full-duplex mode. Multiple devices can send data simultaneously. The network router 211 uses IP addresses to transfer data.

[0075] In one embodiment, the network hub 207 may be implemented as a USB hub that enables the connection of multiple USB devices to a host computer. The USB hub can extend a single USB port into several layers so that there are more ports available for connecting devices to the host system computer. The network hub 207 may include wired or wireless functionality for receiving information via a wired or wireless channel. In one embodiment, a wireless USB short-range high-bandwidth wireless communication protocol may be used for communication between devices 1a-1n and devices 2a-2m located in the operating room.

[0076] In this example, the operating room devices 1a-1n / 2a-2m can exchange data over short distances from fixed and mobile devices (using short-wavelength UHF radio waves in the 2.4-2.485 GHz ISM band) and communicate with the modular communication hub 203 via the Bluetooth wireless technology standard to establish a personal area network (PAN). The surgical site devices 1a-1n / 2a-2m may communicate with the modular communication hub 203 via several wireless or wired communication standards or protocols, including, but not limited to, Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, new radio (NR), long-term evolution (LTE), and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, and their Ethernet derivatives, as well as any other wireless and wired protocols designated as 3G, 4G, 5G, and later. The computing module may include multiple communication modules. For example, the first communication module may be dedicated to shorter-range wireless communication such as Wi-Fi and Bluetooth, while the second communication module may be dedicated to longer-range wireless communication such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, and Ev-DO.

[0077] The modular communication hub 203 can function as a central connection point for one or all of the operating room devices 1a-1n / 2a-2m and can handle a data type known as a frame. Frames can carry data generated by devices 1a-1n / 2a-2m. When a frame is received by the modular communication hub 203, it is amplified and transmitted to the network router 211, which then transfers this data to cloud computing resources using a number of wireless or wired communication standards or protocols as described herein.

[0078] The modular communication hub 203 can be used as a standalone device or connected to compatible network hubs and network switches to form a larger network. Because the modular communication hub 203 is generally easy to install, configure, and maintain, it can be a good choice for networking operating room devices 1a-1n / 2a-2m.

[0079] Figure 5 shows a computer-implemented interactive surgical system 200. The computer-implemented interactive surgical system 200 is similar in many respects to the computer-implemented interactive surgical system 100. For example, the computer-implemented interactive surgical system 200 includes one or more surgical systems 202 that are similar in many respects to surgical system 102. Each surgical system 202 includes at least one surgical hub 206 that communicates with a cloud 204 which may include a remote server 213. In one embodiment, the computer-implemented interactive surgical system 200 includes a modular control tower 236 connected to multiple operating room devices, such as intelligent surgical instruments, robots, and other computerized devices located in the operating room. As shown in Figure 6, the modular control tower 236 includes a modular communication hub 203 connected to a computer system 210.

[0080] As shown in the embodiment of Figure 5, the modular control tower 236 may be connected to an imaging module 238 which can be connected to an endoscope 239, a generator module 240 which can be connected to an energy device 241, a fume exhaust module 226, a suction / irrigation module 228, a communication module 230, a processor module 232, a storage array 234, a smart device / instrument 235 optionally connected to a display 237, and a non-contact sensor module 242. Operating room devices may be connected to cloud computing resources and data storage via the modular control tower 236. The robot hub 222 may also be connected to the modular control tower 236 and cloud computing resources. In particular, the device / instrument 235 and the visualization system 208 may be connected to the modular control tower 236 via wired or wireless communication standards or protocols as described herein. The modular control tower 236 may be connected to a hub display 215 (e.g., a monitor, screen) to display and overlay images received from the imaging module, device / instrument display and / or other visualization systems 208. The hub display may also display data received from devices connected to the modular control tower, along with images and superimposed images.

[0081] Figure 6 shows a surgical hub 206 comprising multiple modules connected to a modular control tower 236. The modular control tower 236 may comprise a modular communication hub 203, such as a network connectivity device, and a computer system 210, such as for local processing, visualization, and imaging. As shown in Figure 6, the modular communication hub 203 may be connected in a hierarchical configuration to expand the number of modules (e.g., devices) that may be connected to the modular communication hub 203, and data associated with the modules may be transferred to the computer system 210, cloud computing resources, or both. As shown in Figure 6, each network hub / switch within the modular communication hub 203 may include three downstream ports and one upstream port. The upstream network hub / switch may be connected to a processor to provide communication connectivity to cloud computing resources and local displays 217. Communication to the cloud 204 can be done via either a wired communication channel or a wireless communication channel.

[0082] The surgical hub 206 may utilize a non-contact sensor module 242 to measure the dimensions of the operating room and generate a map of the operating room using either an ultrasonic non-contact measuring device or a laser-type non-contact measuring device. The ultrasonic-based non-contact sensor module can scan the operating room by transmitting bursts of ultrasound and receiving echoes as they bounce off the surrounding walls of the operating room, as described under the heading "Surgical Hub Spatial Awareness Within an Operating Room" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," which is incorporated herein by reference in its entirety. The sensor module is configured to determine the size of the operating room and adjust the Bluetooth pairing distance limits. A laser-based non-contact sensor module can, for example, scan an operating room by transmitting laser light pulses, receive the laser light pulses reflected from the outer walls of the operating room, compare the phase of the transmitted pulses with the received pulses to determine the size of the operating room, and adjust the Bluetooth pairing distance limit.

[0083] The computer system 210 may include a processor 244 and a network interface 245. The processor 244 may be connected via a system bus to a communication module 247, a storage device 248, a memory 249, a non-volatile memory 250, and an input / output interface 251. The system bus may be any of several types of bus structures, including memory buses or memory controllers, peripheral buses or external buses, and / or local buses, using any variety of available bus architectures. Examples of such architectures include, but are not limited to, 9-bit buses, Industrial Standard Architecture (ISA), Micro-Charmel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association (PCMCIA), Small Computer Systems Interface (SCSI), or any other proprietary bus.

[0084] The processor 244 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer to improve performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse-width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels. Further details are available in the product datasheet.

[0085] In one embodiment, the processor 244 may include a safety controller, including two controller-based families such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured, in particular, specifically for IEC61508 and ISO26262 safety limit applications, to provide a highly integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0086] System memory can be categorized into volatile and non-volatile memory. The Basic Input / Output System (BIOS), which includes basic routines for transferring information between elements within the computer system during startup, is stored in non-volatile memory. Examples of non-volatile memory include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Examples of volatile memory include random access memory (RAM), which functions as external cache memory. Furthermore, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), sync-link DRAM (SLDRAM), and direct rhombus RAM (DRRAM).

[0087] The computer system 210 may also include removable / non-removable volatile / non-volatile computer storage media, such as disk storage. Examples of disk storage devices include, but are not limited to, magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, the disk storage device may include the above-mentioned storage media independently or in combination with other storage media. Examples of other storage media include, but are not limited to, optical disk drives such as compact disk ROM devices (CD-ROMs), compact disk recordable drives (CD-R drives), compact disk rewritable drives (CD-RW drives), or digital multi-purpose disk ROM drives (DVD-ROMs). Removable or non-removable interfaces may be used to facilitate connection of the disk storage device to the system bus.

[0088] It should be understood that the computer system 210 may include software that acts as an intermediary between the described user and basic computer resources in a suitable operating environment. Such software may include an operating system. An operating system, which may be stored on disk storage, may function to control and allocate the resources of the computer system. System applications may leverage resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein can be implemented using various operating systems or combinations of operating systems.

[0089] The user can input commands or information to the computer system 210 via input devices connected to the I / O interface 251. Examples of input devices include, but are not limited to, pointing devices such as mice, trackballs, styluses, and touchpads; keyboards; microphones; joysticks; gamepads; satellite receivers; scanners; TV tuner cards; digital cameras; digital video cameras; and webcams. These and other input devices connect to the processor via interface ports and the system bus. Examples of interface ports include serial ports, parallel ports, game ports, and USB ports. Output devices use some of the same types of ports as the input devices. Therefore, for example, a USB port may be used to provide input to the computer system and output information from the computer system to the output device. Output adapters may be provided to indicate that some output devices, such as monitors, displays, speakers, and printers, may be present, among others, which may require special adapters. Examples of output adapters include, but are not limited to, video and sound cards that provide means of connection between the output device and the system bus. Note that other devices and / or systems of devices, such as remote computers, can provide both input and output functions.

[0090] Computer system 210 may operate in a networked environment using logical connections to one or more remote computers, such as cloud computers, or to local computers. Remote cloud computers may be personal computers, servers, routers, network PCs, workstations, microprocessor-based devices, peer devices, or other common network nodes, but typically include many or all of the elements described in relation to computer systems. For brevity, only memory storage devices are shown along with remote computers. Remote computers may be logically connected to the computer system via a network interface, and subsequently physically connected via a communication interface. Network interfaces may encompass communication networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies include fiber optic distributed data interfaces (FDDI), copper distributed data interfaces (CDDI), Ethernet / IEEE 802.3, and Token Ring / IEEE 802.5. WAN technologies include, but are not limited to, point-to-point links, integrated service digital networks (ISDN) and their variations, packet-switched networks, and digital subscriber lines (DSL).

[0091] In various embodiments, the computer system 210 in Figure 6, the imaging module 238 in Figures 5 and 6, and / or the visualization system 208, and / or the processor module 232 may include an image processor, an image processing engine, a media processor, or any dedicated digital signal processor (DSP) used for processing digital images. The image processor can be made faster and more efficient by using parallel computing with single-instruction multiple data (SIMD) or multiple-instruction multiple data (MIMD) techniques. The digital image processing engine can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0092] A communication connection(s) may refer to hardware / software used to connect a network interface to a bus. For illustrative purposes only, the communication connection(s) are shown as being inside the computer system, but they may be outside the computer system 210. For illustrative purposes only, hardware / software required for connecting to a network interface may include internal and external technologies such as standard telephone-grade modems, cable modems and DSL modems, ISDN adapters and Ethernet cards.

[0093] Figure 7 is a block diagram of an exemplary surgical simulator system. The surgical simulator system may include a simulation device 30000. The surgical simulator system may also include an application creation device 30002, a human interface device 30004, a surgeon agent device 30006, and / or a surgical data system 30008.

[0094] The simulation device 30000 may provide core simulation functions. For example, loading / running one or more simulations, receiving and processing user control information inputs, generating and transmitting visual, auditory, and / or tactile information outputs, collecting simulation operation and activity information, and primary simulation cycle processing may be performed by the simulation device 30000.

[0095] The application creation device 30002 may provide simulation creation functionality. Individual simulation applications may be stored in the simulation device 30000 as application modules 30010. Application modules 30010 may be created, modified, and / or deleted by the application creation device 30002. Application modules 30010 may contain computer-readable and / or executable instructions that direct the operation of the simulation device 30000. For example, application modules 30010 may contain any file type suitable for storing information and running surgical simulations, such as simulation scripts, programming code, structural data files such as Extensible Markup Language (XML) files, and database files.

[0096] The application creation device 30002 may include a graphical user interface with a control unit for creating application modules 30010. The application creation device 3002 can communicate with the simulation device 30000 to search for, modify, and / or load application modules 30010 for simulation operation. For example, the graphical user interface may include an interface structure that allows the user to select simulation activities, input various simulation parameters, set simulation objectives, and verify the execution of the simulation. The application creation device 30002 may be provided as a standalone device and / or integrated with one or more other devices of a surgical simulation system, such as being integrated with the simulation device 30000.

[0097] The human interface device 30004 may include any hardware, software, and / or combination thereof that enables a human user to interact with a simulation provided by the simulation device 30000. The human interface device 30004 may enable the user to provide control inputs to the simulation device 300000 and / or receive output information (such as visual, auditory, and / or tactile information) from the simulation device 30000. In one embodiment, the human interface device 30004 may include a conventional desktop computer.

[0098] The human interface device 30004 may include suitable physical equipment. For example, the human interface device 30004 may include physical equipment that physically and / or virtually mimics aspects of surgical procedures. For example, such equipment could include a desktop unit, a partial task virtual reality unit, a high-fidelity virtual reality unit, a high-fidelity full-size patient unit, a suite unit, a high-fidelity full operating room unit, a full physical virtual reality unit, or a surgical robot console unit. For example, the human interface device 30004 may include devices such as a computer-based simulator interface disclosed by Gallager et al, "Simulations for Procedural Training," Fundamentals of Surgical Simulation, Principles and Practice, Springer (2012).

[0099] Human interface devices 30004 may include physical devices that physically and / or virtually mimic surgical instruments. For example, human interface devices 30004 may include physical devices that mimic surgical instruments, appliances, and consumables such as access devices such as trocars, hand access ports, pneumoperitoneum needles, and guide sheaths; auxiliary hemostatic agents such as patches, gelatin, and powders; craniofacial instruments such as stretchers and plates; balloons and inflators; catheters such as diagnostic catheters, access catheters, vascular catheters, and therapeutic catheters; energy sealing and dissection devices such as tissue sealers, shears, blades, and forceps; and pull-down wires, compression screws, plates, etc. This may include orthopedic instruments such as plants, drills, burs, rods, and connectors; ligators such as incision and endoscopic clip applicators; microwave ablation equipment; auxiliary endoscopic instruments such as drains, sutures, ligatures, needle holders, retrievers, and suture clips; surgical stapling equipment such as open staplers, endoscopic staplers, cutter staplers, electric staplers, circular staplers, vascular staplers, linear staplers, staple cartridges, and staple line reinforcement applicators; wound closure materials such as sutures, adhesives, needles, and knotless tissue control devices; and imaging devices such as minimally invasive imaging devices. For example, human interface device 30004 may include a virtual reality handheld controller that, when operated with a virtual reality headset, mimics surgical instruments, appliances, and consumables such as those disclosed above.

[0100] The human interface device 30004 may include a display that conveys a visual representation of the simulation to the user. The human interface device 30004 may include a computer display. The human interface device 30004 may include a virtual reality headset display. For example, a virtual reality headset display may be used to display a surgical environment, such as the one disclosed in Figure 2 of this specification. A user having such a virtual reality headset display may view and / or interact with any of the elements in the surgical room 116, such as the patient, the robotic system 110, the surgeon's console 118, the surgical robot hub 122, one or more surgical tools 117, the imaging device 124, the patient-side cart 120, and one or more displays 119, 107, 109.

[0101] The human interface device 30006 can present visual information representing the surgeon's perspective. The human interface device 30006 can present visual information from simulated imaging devices such as arthroscopes, angioscopes, bronchoscopes, coredocoscopes, conoscopes, cytoscopes, duodenoscopy, enteroscopes, upper gastrointestinal endoscopes (gastroscopy), endoscopes, laryngoscopes, nasopharyngo-neproscopes, sigmoidoscopy, thoracoscopy, ureteroscopes, and their related instruments and control units. The human interface device 30006 can also present visual information from simulated auxiliary intraoperative imaging equipment such as computed tomography (CT) units, magnetic resonance imaging (MRI) units, image-guided surgery units, and intraoperative ultrasound units; and fluoroscopy units. Such perspective visual information, surgical imaging information, and supplemental intraoperative imaging information can be displayed to the user in any combination suitable for the operation of the simulation. For example, such information may be presented to the user as a single full-screen view, a tiled window view, a picture-in-picture view, or it may be registered in a simulated display unit of a virtual reality view.

[0102] The human interface device 30004 may include a physical and / or virtual reality surgical robotic surgeon console. For example, an exemplary surgeon console-like human interface device 30004 may include a display, such as a stereoscopic display, and control inputs, such as a handheld manipulator and foot pedals. For example, the surgeon console-like human interface device 30004 may include the interface of the surgeon's console 118 disclosed herein. The human interface device 30004 may enable voice control, for example, by a microphone and voice recognition function. The human interface device 30004 may provide audible feedback, for example, by a speaker. The human interface device 30004 may provide tactile feedback, for example, by vibration, force feedback, vortex rings, and ultrasonic techniques.

[0103] When implemented, the human interface device 30004 may be provided as a standalone device and / or may be integrated with one or more other devices of the surgical simulation system, such as being integrated with the simulation device 30000. The simulation device 30000 may include an interface module 30012 for communicating with the human interface device 30004. In one embodiment, the human interface device 30004 may be integrated with one or more elements of the computer-implemented interactive surgical system 100. For example, the human interface device 30004 may be integrated with the computer system 210. For example, the human interface device 30004 may be integrated with the hub 106. For example, the human interface device 30004 may be integrated with the visualization system 108. The interface module 30012 may communicate with one or more elements of the computer-implemented interactive surgical system 100, for example, via the surgical data system interface module 30014.

[0104] In one embodiment, two or more human interface devices 30004 can operate simultaneously with the simulation device 30000. For example, in a multi-person simulation application.

[0105] The surgical agent device 30006 may include any hardware and / or software suitable for providing computer-based control and response to the inputs and outputs of the simulation device 30000. The surgical agent device 30006 may include computer processing that mimics human input to the simulation device 30000. For example, the surgical agent device 30006 may be capable of recording and registering control inputs, such as basic instrument operations. The surgical agent device 30006 may include computer processing that can access the input / output application programming interface (API) of the simulation device 30000. For example, the API may reveal one or more input / output functions that can be directed according to the surgical agent device 3006. Functions may include fine-grained operation and physical-based input / output functions, such as functions that directly control the position and movement of instruments. Functions may include coarser-grained activity-based input / output functions, such as ligation activities, suturing activities, and stapling activities. The functions may include coarser-grained extrasurgical tasks such as surgical access functions and organ mobilization functions, and / or stage-based input / output functions. Each function may include parameters corresponding to the level of granularity. The parameters may provide specific details for directing the operation of the function within the simulation. The surgical agent 30006 may include functions for generating and manipulating multiple simulation runs. For example, a user may want to estimate the duration of various suturing techniques. The surgical agent device 30006 may be used to plan simulations of any number of different techniques, each of which may be run via the simulation device, and metrics collected by the simulation device may be used to estimate the differences in duration.

[0106] The surgical agent device 30006 may be provided as a standalone device and / or may be integrated with one or more other devices of a surgical simulation system, such as being integrated with the simulation device 30000. The simulation device 30000 may include an interface module 30012 for communicating with the surgical agent device 30006. For example, the surgical agent device 30006 may be integrated as a module of the simulation device 30000. For example, the surgical agent device 30006 may be integrated into the application module 30010 of the simulation device.

[0107] The surgical data system 30008 may include any hardware and / or software suitable for providing external structured surgical information and functions to the simulation device 30000. The surgical data system 30008 may include structures and / or functions described in relation to Figures 1 to 6 of this specification. For example, the surgical data system 30008 may include one or more elements of the computer-implemented interactive surgical system 100. The surgical data system 30008 may include, for example, a surgical hub 106. For example, the simulation device 30000 includes a surgical data system interface module 30014 that enables communication with the surgical hub 106 via the surgical hub's communication module 130. The surgical data system 30008 may include, for example, one or more surgical data repositories. For example, the surgical data system 30008 may include a computer system 210 located in the operating room. For example, the surgical data system 30008 may include a remote server 213 in the cloud 204.

[0108] For example, a surgical data system 30008, such as a surgical hub 106, may provide data to a simulation device 30000 and / or an application creation device 30002. For example, the data may include any surgical data collected and / or generated by the surgical hub 106. Alternatively, for example, the simulation device 30000 may directly receive similar data from any of the networked devices disclosed in Figures 1 to 6. Such data may include, for example, information about live surgical procedures. Such data may include information about past surgical procedures. Such data may include information about future scheduled surgical procedures.

[0109] Information regarding surgical procedures may include information about the patient, staff, planned procedures, procedures performed, and postoperative activities such as patient outcomes. For example, information received and used by a simulation device may include patient records, patient images, models of the patient's anatomical structure, patient test results, and patient history. For example, information received and used by a simulation device may include the staff roster for the procedure, details of past procedures for specific staff members, staff metrics, experience, recent schedules and workloads, and past surgical activities (such as instrument usage statistics and procedure duration). For example, information received and used by a simulation device may include procedure plans, equipment and inventory information, pull lists, checklists, procedure plan analyses, and recommendations. For example, information received and used by a simulation device may include any data collected or generated during a live procedure, such as procedure progress, milestones, patient information, vital signs, operating room setup, staff movements, images, instrument use, surgical techniques, such as those captured by video, manually recorded, and / or inferred from reports by smart devices, such as duration and abnormal event reports. Any data captured during a live procedure can also be stored and made available as past procedures. For example, information received and used by a simulation device may include postoperative records, patient recovery information, and postoperative diagnostic information such as patient outcomes, tests, and images.

[0110] The simulation device 30000 may include any computer or processing platform suitable for running one or more simulations. The simulation may include a computer-modeled surgical procedure environment. For example, the simulation may include a model of the patient's anatomical and / or physiological structure. For example, the simulation may include a model of the actions of one or more healthcare professionals, such as a surgeon, nurse, other physician, or technician, and / or a model of instruments.

[0111] The simulation device 30000 may include one or more additional functional modules. Each module may include hardware, software, or a combination thereof that enables the module's functionality. One or more modules working together may represent a computer framework on which simulations of medical procedures can be performed. Modules may include hardware elements such as computer processing units, graphics processing units, field-programmable gate arrays (FPGAs), communication hardware, and memory. Modules may include software elements that, when executed by the processor, cause the module to perform specific functions.

[0112] The simulation device may include, for example, a core simulation module 30016, a simulation application module directory 30018, an interface module 30012, an object properties module 30020, a physics module 30022, a physiological model 30024, a texture model 30026, a 3D graphics pipeline 30028, a surgical data system interface module 30014, a metric extraction module 30030, and a session storage and management module 30032. The simulation device may also include an operating system module 30034.

[0113] The core simulation model 30016 may provide the primary simulation functions of the simulation device 30000. For example, the core simulation module 30016 may include code for initializing the simulation, communicating and interacting with other modules of the simulation device 30000, and / or managing architecture-level simulation parameters. For example, the core simulation module 30016 may include a master event clock to provide time synchronization and / or coordination of the operation of the modules of the simulation device 30000. For example, the core simulation module 30016 may establish the overall simulation frame rate.

[0114] The core simulation module 30016 may include a core for providing a master simulation cycle. The core simulation module 30016 may execute one or more iterations of the master simulation cycle. Each iteration of the master simulation cycle may represent an individual time slice of the simulation. In one embodiment, the core simulation module 30016 may execute the master simulation cycle according to the flow disclosed in Figure 10.

[0115] The simulation application module directory 30018 may manage the storage, retrieval, and / or linking of one or more application modules 30010. Each application module 30010 may contain code that indicates an application-level aspect of the simulation. For example, an application module 30010 may include functionality that provides simulation of a specific anatomical structure, a specific teaching scope, or a specific instrument. In the exemplary simulation device 30000, the application-specific simulation device 30000 may operate with a single application module 30010, with or without the simulation application module directory 30010. The simulation application module directory 30018 may operate based on its interaction with the core simulation module 30016 and / or the application creation device 30002.

[0116] Interface module 30012 may provide functions for interacting with human interface device 30004 and / or surgical agent device 30006. For example, interface module 30012 may include one or more drivers for converting information received from human interface device 30004 into software commands, interrupts, etc. For example, interface module 30012 may include a software application programming interface (API) for interacting with surgical agent 30006. Interface module 30012 may provide information received from human interface module 30004 and / or surgical agent device 30006 to other modules of simulation device 30000. For example, interface module 30012 may receive control inputs representing the movement of simulated instruments from human interface module 30004 and / or surgical agent device 30006 and provide this information to one or more other modules of simulation device 30000 so that the movement can be represented in the simulation.

[0117] Interface module 30012 may provide an API to enable finer-grained interaction with the surgical agent device 30006. For example, the API may provide an interface for receiving simulation parameters and simulation settings from the surgical agent device 30006. Such simulation parameters and / or simulation settings may be similar to those input by the user via, for example, the application creation device 30002. For example, the surgical agent device 30006 may be capable of running one or more computer-controlled simulation trials by the simulation device 30000. For example, the surgical agent device 30006 may be capable of running multiple simulations, each with alternative interactions.

[0118] The interface module 30012 can transmit the output from the simulation device 30000 to the human interface device 30004 and / or the surgical agent device 30006. For example, the output may include visual output, tactile output, audio output, and / or structured data output.

[0119] The Object Properties module 30020 may provide functions for managing the simulated appearance and / or behavior of objects in a simulation. Examples of simulated objects include anatomical structures, instruments, equipment, consumables, and fluids. The appearance of an object can be managed by object properties such as position, dimensions, scale, material, parent / child relationships, vertices, faces, interactivity, transparency, trajectory, rendering properties, texture, surface reflectivity, motion blur, and layering. The behavior of an object can be managed by object properties such as physical properties, mass, motion, collision behavior, elasticity, viscosity, surface tension, rigging constraints, hardness, shear strength, tearing behavior, and grain size.

[0120] The physics module 30022 may provide the functionality to calculate the physical response and / or interaction of objects in a simulation. The physics module may determine such responses and / or interactions according to classical mechanics, fluid dynamics, soft body mechanics, Brownian motion, collision detection, cloth behavior, finite element analysis, etc. The physics module 30022 may include commercial and / or open-source modules such as PhysX®, Simulation Open Framework Architecture (SOFA)®, and VisSim®.

[0121] The physiological module 30024 may provide the functionality to calculate anatomical structures and / or the patient's overall physiological responses and / or interactions in a simulation. The physiological module 30024 may provide physiological models of important organs and / or systems. These physiological models may include mathematical models, statistical models, etc. For example, the physiological module 30024 may modularize the patient's vital signs to calculate the patient's responses and / or interactions to activities performed during the simulation. For example, a circulatory model may calculate blood pressure in response to severed blood vessels in the simulation. The physiological module 30024 and the physics module 30022 may cooperate with each other during the calculation of each state of the simulation. For example, blood pressure calculated by the circulatory model may be used to determine hydrodynamic properties calculated by the physics module 30022 and managed by the object properties module 30020.

[0122] The texture module 30026 may provide the functionality to determine, search for, and / or generate appropriate surfaces for objects in the simulation. The texture module 30026 may include one or more surface modalities that can be controlled according to the simulation parameters. Surface modalities may include artificially generated surfaces, surfaces based on real-world images, and combinations thereof. The texture module 30026 may coordinate its operation with the physics module 30022 to provide accurate haptic feedback to the user via the user interface module 30012.

[0123] The 3D graphics pipeline 30028 can provide functionality for the visual rendering of the simulation environment. The 3D graphics pipeline 30028 can receive object properties and depth. The 3D graphics pipeline 30028 can determine the visualization to be presented to the user, representing objects in 3D space as seen from the camera's viewpoint. The 3D graphics pipeline 30028 can determine the geometric aspects of rendering, such as lighting, projection, clipping, and view transformation. The 3D graphics pipeline 30028 can determine the rasterization aspects of rendering, such as fragmentation, pixel shading, vertex shading, geometry sharing, and texture filtering. The 3D graphics pipeline 30028, in cooperation with the texture module 30026, can provide accurate visual feedback to the user via the interface module 30012.

[0124] The surgical data system interface module 30014 can provide interactive connectivity to one or more elements of the computer-implemented interactive surgical system 100. Information from one or more elements of the computer-implemented interactive surgical system 100 can be communicated via the surgical data system interface module 30014 to one or more modules of the simulation device 30000 and may affect the operation of the simulation. For example, the surgical data system interface module 30014 may receive information about surgical procedures and communicate it to the corresponding application module 30010. For example, the surgical data system interface module 30014 may receive information about instruments and communicate it to the object properties module 30020. For example, the surgical data system interface module 30014 may receive information about patients and communicate it to the physiological module. For example, the surgical data system interface module 30014 may receive information about tissue images and communicate it to the texture module 30026.

[0125] Information from the modules of the simulation device 30000 can be provided to one or more elements of the computer-implemented interactive surgical system 100 via the surgical data system interface 30014. For example, one or more elements of the computer-implemented interactive surgical system 100 may receive statistics related to the simulated treatment plan from the metric extraction module 30030. For example, one or more elements of the computer-implemented interactive surgical system 100 may receive a reproduced simulation visualization treatment plan from the session memory and management module 30032. For example, the surgical data system interface module 30014 may provide a communication path between the interface module 30012 and one or more elements of the computer-implemented interactive surgical system 100. For example, a surgeon during a live surgical procedure may access simulation information from the operating room and / or manipulate the simulation. For example, a surgeon may use the surgeon console 118 to access and / or interact with a simulation corresponding to a live surgical procedure.

[0126] The metric extraction module 30014 may provide a function to record various parameters related to the operation of the simulation. For example, the metric extraction module 30014 may record metrics related to the simulation as a whole, such as duration, number of activities, number of movements, complexity of movements, participating staff, staff movements, and changes in equipment and / or instruments. For example, the metric extraction module 30014 may record metrics related to specific aspects of the simulation, such as simulated patient vital signs, complications, collisions, and bleeding. The metric extraction module 30014 may maintain a master log of metric-related events during the simulation. The metric extraction module 30014 may record metric-related events according to the configuration from the application module 30010 used in the simulation.

[0127] The session memory and management module 30032 may provide management functions for the main simulation execution record. For example, the session memory and management module 30032 may store information that enables the simulation to be re-executed, displayed, and / or analyzed as a whole. The session memory and management module 30032 may store information about each input, simulation state, and output, such as the inputs, simulation states, and outputs disclosed with respect to Figure 10. The session memory and management module 30032 may enable previous simulations to be recalled, copied, and initialized by new user inputs. For example, a trainee surgeon can recall a simulation performed by an experienced surgeon, pause the simulation at a critical step, and attempt that step themselves. The session memory and management module 30032 may provide an overlay function between different executions of a particular simulation. Such overlays may highlight similarities and differences and enhance training.

[0128] The operating system module 30034 can manage the hardware and / or software resources of the simulation device 30000. The operating system module 30034 can provide common computing system-level services to other modules of the simulation device 30000. For example, the operating system module 30034 may provide hardware I / O processing, memory allocation, hardware interrupt processing, software interrupt processing, thread processing, single-task processing, multi-task processing, etc. The simulation device 30000 may be a real-time computing device. The operating system module 30034 may include a real-time operating system. For example, the operating system module 30034 may be driven by events and frame rates established by the core simulation module 30016.

[0129] Figure 8 is a block diagram of an exemplary surgical simulator system. The simulation device 30000 is shown along with an exemplary hardware architecture. For example, the simulation device 30000 may include a processor 30034, memory 30036, storage 30038, display adapter 30040, operation interface adapter 30042, surgical data system adapter 30044, and / or network adapter 30046. One or more of the processor 30034, memory 30036, storage 30038, display adapter 30040, operation interface adapter 30042, surgical data system adapter 30044, and / or network adapter 30046 may be used to enable the operation of the modules of the simulation device 30000 disclosed herein.

[0130] The processor 30046 may include computer processing units, graphics processing units, any suitable microcontroller, microprocessor, field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), and / or any combination thereof that is suitable for processing and providing a 3D simulated environment for interaction with a computer agent and / or a human user. In one embodiment, the processor 30046 may include one or more processing units. The processor 30046 may be a processor of any suitable depth to carry out the digital processing requirements disclosed herein. For example, the processor 30046 may be a 32-bit processor, a 64-bit processor, a 128-bit processor, and so on.

[0131] Such a processor may include, or communicate with, a medium, such as a computer-readable medium, that can store instructions that, when executed by the processor, cause the processor to perform the steps described herein as being executed or assisted by the processor. Some embodiments of the computer-readable medium may include, but are not limited to, electronic, optical, magnetic, or other storage devices that can provide computer-readable instructions to the processor, such as a processor in a web server. Other examples of media include, but are not limited to, floppy disks, CD-ROMs, magnetic disks, memory chips, ROMs, RAMs, ASICs, configured processors, all optical media, all magnetic tapes or other magnetic media, or any other media that a computer processor can read. The processors and processes described may be one or more structures and may be distributed through one or more structures. The processor may include code for performing one or more (or part of) the methods described herein.

[0132] Memory 30036 may include any component or set of components suitable for storing data. For example, memory 30036 may include volatile memory and / or non-volatile memory. Memory 30036 may include random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, and the like.

[0133] Storage 30038 may include any component or set of components suitable for storing large amounts of data. For example, storage 30038 may include hard disk drives (HDDs), solid state drives (SSDs), network-attached storage (NAS), etc. Storage 30038 may include database structures and / or database management systems (DBMS).

[0134] The display adapter 30040 may include any components or sets of components suitable for outputting a visual representation of a 3D simulation environment. For example, the display adapter 30040 may include a graphics card, display card, graphics adapter, etc. The display adapter 30040 may be used to generate a feed of output images to a display device, such as the display of the human interface device 30004. The display adapter 30040 may include a graphics processing unit (GPU). The display adapter 30040 may include, for example, hardware for rendering a graphics pipeline. The operation interface adapter 30042 may include any components or sets of components suitable for receiving operation information from a human interface device and / or outputting feedback information to a human interface device. For example, the operation interface adapter 30042 may receive motion tracking information from a virtual reality headset and then manipulate the view displayed to the user. For example, the operation interface adapter 30042 may receive a control input indicating a user operating a surgical instrument and then output haptic feedback to the user's handheld device. For example, the operation interface adapter 30042 can receive control information from a conventional desktop keyboard and mouse.The operating interface adapter may include input / output hardware such as serial input / output ports, parallel input / output ports, universal asynchronous receiver / transmitter (UART), discrete logic input / output pins, analog-to-digital converters, digital-to-analog converters, universal serial bus (USBI) ports, USB-C ports, FireWire ports, High Performance Parallel Interface (HIPPI), Thunderbolt ports, Yapbus, Ethernet, Gigabit Ethernet, and / or any other suitable peripheral interface technologies.

[0135] The surgical data system adapter 30044 may include any components or sets of components suitable for communicating with the surgical data system 30008. The surgical data system adapter 30044 may include communication hardware for establishing a physical channel between the simulation device 30000 and the surgical data system 30008. For example, the surgical data system adapter 30044 may include a USB port, a USB-C port, a FireWire port, a HIPPI port, a Thunderbolt port, a Yapbus port, an Ethernet port, a Gigabit Ethernet port, and / or any other suitable peripheral interface. The surgical data system adapter 30044 may include hardware, software, and / or a combination thereof for establishing a logical channel between the simulation device 30000 and the surgical data system 30008 via the network adapter 30046 and the network 30048.

[0136] The network adapter 30046 may include any components or sets of components suitable for communication over a network, such as network 30048. The network adapter 30046 can enable communication over local area networks (LANs), wide area networks (WANs), and / or mobile networks. Examples of LAN technologies include fiber optic distributed data interfaces (FDDI), copper distributed data interfaces (CDDI), Ethernet / IEEE 802.3, Token Ring / IEEE 802.5, and Wi-Fi / IEEE 802.11. Examples of WAN technologies include, but are not limited to, point-to-point links, integrated service digital networks (ISDN) and their variations, packet-switched networks, and digital subscriber lines (DSL). A mobile network may include communication links based on one or more mobile communication protocols such as GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long-Term Evolution (LTE) or 4G, LTE Advanced (LTE-A), New Radio (NR), or 5G.

[0137] In one embodiment, the network adapter 30046 may include a wireless network adapter such as a 5G network adapter. Such a 5G network adapter 30046 may use 5G New Radio (NR) transceivers to provide ultra-reliable and low latency communication (URLLC) to enhanced mobile broadband (eMBB). Such a 5G network adapter 30046 may use radio bands such as 3.5GHz to 7GHz and / or higher radio bands such as 24GHz to 48GHz. The network 30048 serving such a 5G network adapter 30046 may include a public wireless network, a semi-private (e.g., network slicing-based) network, and / or a fully private wireless network.

[0138] Figure 9 is a block diagram depicting a user interface device 30004 of an exemplary surgical simulator. The human user interface device 30004 is shown together with an exemplary hardware architecture. For example, the human user interface device 30004 may include a processor 30050, memory 30052, a display subsystem 30054, and / or an operation subsystem 30056.

[0139] The processor 30050 may include a computer processing unit, a graphics processing unit, any suitable microcontroller, microprocessor, field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), and / or any combination thereof suitable for handling processing related to the display of visual information received from the simulation device 30000, processing of operation information for transmission to the simulation device, and processing of feedback information received from the simulation device 30000. The processor 30050 may include a microcontroller for interface with one or more local sensors to detect control operations from the user and / or for interface with one or more local actuators to provide feedback from the user.

[0140] Such a processor may include, or communicate with, a medium, such as a computer-readable medium, that can store instructions that, when executed by the processor, cause the processor to perform the steps described herein as being executed or assisted by the processor. Some embodiments of the computer-readable medium may include, but are not limited to, electronic, optical, magnetic, or other storage devices that can provide computer-readable instructions to the processor, such as a processor in a web server. Other examples of media include, but are not limited to, floppy disks, CD-ROMs, magnetic disks, memory chips, ROMs, RAMs, ASICs, configured processors, all optical media, all magnetic tapes or other magnetic media, or any other media that a computer processor can read. The processors and processes described may be one or more structures and may be distributed through one or more structures. The processor may include code for performing one or more (or part of) the methods described herein.

[0141] Memory 30036 may include any component or set of components suitable for storing data. For example, memory 30036 may include volatile memory and / or non-volatile memory. Memory 30036 may include random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, and the like.

[0142] The display subsystem 30054 may include any components or sets of components suitable for displaying a visual representation of a 3D simulation from the simulation device 30000 to the user. The display subsystem may include display hardware such as monitors, digital projectors, smartphones, digital headsets, virtual reality headsets, stereoscopic displays, console displays for robotic surgeons, surgical display units, and surgical microscopes.

[0143] The operation subsystem 30056 may include any components or sets of components suitable for collecting operation controls from the user and transmitting them to the simulation device 30000, and / or providing the user with feedback information received from the simulation device 30000. User operations may include any interfaces with sensors that involve the user, for example, to indicate the user's intentions in the simulation. Examples of interfaces include keyboards, mice, joysticks, physical devices that mimic the size, shape, and operation of actual surgical instruments, virtual reality handheld controllers, smart gloves, motion detection systems (e.g., hand tracking systems), console manipulators and / or control units of a robotic surgeon, and physical units that mimic the size, shape, and operation of a console manipulator and / or control unit of an actual robotic surgeon. For example, the interface may include viewpoint sensors, such as accelerometers, in a headset to indicate the user's viewpoint in the simulation.

[0144] Feedback from the simulation device 30000 may include any interface with actuators that provide sensory input to the user. For example, feedback may include tactile feedback, force feedback, temperature feedback, moisture feedback, sound feedback, and olfactory feedback. For example, force feedback and / or tactile actuators in a manipulator on a robotic surgical console can be used to simulate the feedback a user would feel when operating such a manipulator in a live procedure. For example, force feedback and / or tactile actuators in a user device that mimics the size, shape, and operation of an actual surgical stapler can be used to simulate the feedback a user would feel when operating such a device on biological tissue, such as force feedback when engaging tissue and firing the stapler.

[0145] Figure 10 is a flowchart of the operation of an exemplary surgical simulator. In 30058, a simulation application may be loaded. For example, the core simulation module 30016 may load data associated with a particular application module 30010 into memory 30036. The loaded data may include instructions for the processor 30034 to operate a particular simulation. The loaded data may include a procedure plan for the simulation. For example, the procedure plan may be structured as disclosed herein, for example with respect to Figures 11A-11B. The loaded data may include the initial state of the simulation.

[0146] In 30060, the simulation output may be determined and / or transmitted. For example, the simulation output may be determined and / or transmitted by the simulation device 30000. Here, the core simulation module 30016 may refer to the current state of the simulation (e.g., the initial state and / or subsequent states). The core simulation module 30016 may involve one or more other modules to process and output the current state. For example, the core simulation module may involve one of the following: the object properties module 30020, the texture module 30026, the application module 30010, the 3D graphics pipeline 30028, the interface module 30012, and / or the surgical data system interface module 30014 to process and output the current simulation state as information. Information related to the output may be processed and / or stored by, for example, the metric extraction module 30030 and / or the session storage and management module 30032.

[0147] For example, in a human-operated simulation session, output information may be transmitted to the display subsystem 30054 and / or operation subsystem 30056 of the human interface device 30004 via the display adapter 30040 and / or operation interface adapter 30042. For example, in a computer-controlled simulation session, output information may be transmitted to the surgical agent 30006 via the interface module 30012. Alternatively, in a computer-controlled simulation session, output information may be transmitted in the application module 30010 (for example, processed locally). For example, in a session accessed via the surgical data system 30008, output information may be transmitted by the surgical data system interface module 30014 via the surgical data system adapter 30044 and / or network adapter 30046.

[0148] In 30062, simulation inputs may be received and / or processed. For example, simulation inputs may be received and / or processed by the simulation device 30000. Here, the core simulation module may receive control inputs in conjunction with the interface device, the surgical data system interface module, and / or the application module 30010. Information related to the inputs may be processed and / or stored, for example, by the metric extraction module 30030 and / or the session storage and management module 30032.

[0149] For example, in a human-operated simulation session, input information may be transmitted from the operation subsystem 30056 of the human interface device 30004 and received via the operation interface adapter 30042. For example, in a computer-controlled simulation session, input information may be transmitted from the surgical agent 30006 and received via the interface module 30012. Also, for example, in a computer-controlled simulation session, input information may be received in the application module 30010 (for example, processed locally). For example, in a session accessed via the surgical data system 30008, input information may be received via the surgical data system adapter 30044 and / or the network adapter 30046 and first processed by the surgical data system interface module 30014.

[0150] In 30064, a subsequent simulation state can be determined. For example, a subsequent simulation state can be determined from the current simulation state and / or any received inputs. The core simulation module 30016 can determine a subsequent simulation state by engaging with one or more other modules of the simulation device 30000. For example, the code simulation module 30016 may engage with the application module, object properties module, physics module, physiological module, etc. The subsequent simulation state can be determined by the operation of the processor 30034. Information related to the input can be processed and / or stored, for example, by the metric extraction module 30030 and / or the session storage and management module 30032.

[0151] In this stage, the process may loop to receiving input at 30060. Each iteration of this flow may represent a corresponding time cycle in the simulation. The simulation frame rate may be set to a level suitable for the simulation goals and the processing power of the surgical simulation device 30000. Lower frame rates may allow processing to achieve live human interaction simulation. Higher frame rates may allow for higher simulation fidelity. For example, when running a computer-controlled simulation, for example with a surgical agent 30006, a higher frame rate may be used even if it causes the simulation processing time to exceed the real-world time being simulated.

[0152] Figures 11A and 11B illustrate the data structure of an exemplary surgical procedure plan for use in a computer-implemented interactive surgical system and / or surgical simulator. A surgical procedure plan may include information outlining the staff, equipment, techniques, and steps that may be used to perform the surgical procedure. For example, a procedure plan may include a staff roster indicating the roles and / or specific medical professionals to be involved in the procedure. A procedure plan may include a list of equipment that may be used during the procedure, such as durable surgical instruments, imaging equipment, instruments, and consumables. For example, a procedure plan may include a list of candidates for surgical technicians to be used when preparing the operating room to assemble suitable tools and materials for the surgery and for the surgeon. A procedure plan may include information about the techniques to be performed in the procedure. For example, procedure plans for the same surgical objective may include different access methods, mobilization methods, examination methods, tissue joining methods, wound closure methods, etc.

[0153] A procedure plan may reflect the surgeon's expert judgment regarding individual cases. A procedure plan may reflect the surgeon's preferences and / or experience with specific techniques. A procedure plan may map specific surgical tasks to roles and equipment. A procedure plan may provide a timeline of the planned procedure.

[0154] A treatment plan may include one or more decision points and / or branches. Such decision points and / or branches may provide surgical options available for a particular aspect of the treatment, and the selection of one of these options may be based on information from the surgery itself. For example, the selection of one or more options may be based on a specific plane of the anatomical structure of a particular patient, and the surgeon may select an option based on an assessment of the patient's tissue during live surgery.

[0155] The procedure plan may include one or more contingencies. These may include information about possible but unlikely situations that may occur during live surgery. Contingencies may include one or more surgical tasks that may be used if such situations occur. Contingencies may be used to ensure that sufficient equipment, staff, and / or supplies are available at all times during the procedure.

[0156] A treatment plan may be recorded in one or more data structures. The treatment plan data structure may be used to record data about future live surgeries, completed live surgeries, future simulated surgeries, completed simulated surgeries, etc. A treatment plan data structure for live surgeries may be used by a computer-implemented interactive surgical system 100. For example, a treatment plan data structure for live surgeries may be used by a surgical hub 106 to enhance the situational awareness and / or operational modes of the computer-implemented interactive surgical system 100. The treatment plan data structure for live surgeries may be used by the surgical hub 106 to record distinct elements of the live surgery for structured analysis.

[0157] The treatment plan data structure can be used by the simulation device 30000. For example, the treatment plan data structure can be used by the simulation device 30000 to establish the settings for a simulation session and / or one or more objectives. For example, the treatment plan data structure can be used by the simulation device 30000 to record individual elements of a simulated surgery for structured analysis.

[0158] The treatment plan data structure may include any structure suitable for capturing data elements related to the treatment. For example, the treatment plan may be recorded in a tree-like data structure, such as shown in Figure 11A, where the root of the tree structure represents the core treatment data 30066. The core treatment data 30066 may include general information about the treatment, such as the treatment name, treatment code, patient name, date, and time. In the case of a simulation, the core treatment data 30066 may include information about the simulation device, such as the device ID, software version, user, and simulation execution settings (frame rate, resolution, connected user interface device, etc.).

[0159] The treatment data may include leaves in a tree structure. The first level of leaves may include data relating to key aspects of the treatment plan, such as data relating to the treatment setup 30068, one or more treatment stages 30070, one or more contingencies 30072, and the outcome of the treatment 30074.

[0160] Setup data 30068 may include information regarding the preparation and / or initial state of the procedure. For example, setup data 30068 may include elements such as staff roster, staff roles and / or staff IDs, operating room ID, equipment list, room layout, initial operating table position, list of instruments and / or consumables prepared in the surgical field, all initial settings related to the equipment, preoperative images, and patient records. In the case of a simulation, setup data 30068 may include information regarding the simulated environment, such as records of simulated anatomical structures, records of simulated physiological functions, and preoperative images.

[0161] Stage data 30070 may include data elements related to key milestones of the procedure. For example, the stages of the procedure may include milestones such as establishing access. Stage data 30070 may include information related to the staff, equipment, techniques, and steps that may be used to perform a particular stage of the procedure. Stage data 30070 may include a stage ID.

[0162] A stage may be further detailed by one or more sub-leaves, such as one or more surgical tasks 30076. A surgical task may represent a distinct surgical step within a given stage. For example, in the access stage, positioning a trocar may be a surgical task. Surgical task data 30076 may include a task ID. Surgical task data 30076 may include information relevant to a particular task, such as the staff and / or surgeon performing the task, the equipment used, the specific technique applied, patient vital signs during task execution, other environmental information, and lists. Each task may be further detailed using objective data 30078, data related to anatomical structure-instrument interaction 30080, and outcome data 30082. Objective data 30078 may include information indicating the relative success of task execution. Objective data 30078 may include information such as planned task duration, acceptable execution specificity, efficiency modality, and avoidance of complications. Outcome data 30082 may include information related to one or more objectives. Result data 30082 may record the surgical execution (e.g., live and / or simulated) against the target.

[0163] Task data 30076 may contain one or more elements of anatomical structure-instrument interaction data 30080. Anatomical structure-instrument interaction data 30080 may represent fine-grained instructions for surgical execution. Anatomical structure-instrument interaction data 30080 may represent one or more specific activities used in the execution of a surgical task. Anatomical structure-instrument interaction data 30080 may represent observable behaviors of the surgeon.

[0164] In one embodiment, anatomical structure-instrument interaction data 30080 may include specific positions, forces, angles, etc., applied to anatomical structures by the surgeon. For example, in live surgery, data recorded from smart instruments by the surgical hub 106 may be captured as anatomical structure-instrument interaction data 30080. For example, a surgical smart stapler working in conjunction with other elements of a computer-implemented interactive surgical system 100 may record the stapler's position, angle, tip force, jaw force, staple cartridge type, closure pressure, firing speed, etc. In simulated surgery, similar data elements may be captured.

[0165] Contingency data 30072 may indicate any complications that may be associated with the procedure. Each contingency data 30072 may include one or more task data elements 30084 that address an appropriate response to a specific complication. Contingency data 30072 may indicate a deviation from the original procedure plan. Also, for example, contingency data may be cross-referenced with one or more tasks 30078 and / or anatomical structure-instrument interactions 30080. For example, if a particular execution in an anatomical structure-instrument interaction 30080 may lead to a complication, the nature of that execution and the cross-reference of the contingency may be included in the outcome data 30082 related to that anatomical structure-instrument interaction 30080.

[0166] Outcome data 30074 may show the results of the procedure. Here, overall metrics of the surgical execution may be stored, and actual and / or simulated patient recovery information, as well as / or patient outcomes, may be recorded. For example, outcome data 30074 may include efficiency information, cost information, surgical duration, workload metrics, and the percentage of planned consumable usage.

[0167] Figure 11B shows a treatment plan data structure having the elements disclosed above, which further establishes the structure of alternative steps for completing a particular treatment, task, or activity. As shown, the treatment represented by treatment data 30086 may include two alternative setups, each represented by its respective setup data (first setup data 30088, 30090, and second setup data 30092). The first setup data 30088, 30090 may include two alternative tasks 30094, 30096. The second setup data 30092 may include one task 30098. In this figure, the treatment represented by treatment data 30086 can be achieved in three different ways. The first method uses the first setup 30088 and the first task 30094. The second method uses the first setup 30090 and the second task 30096. The third method involves using the second setup 30092 and its corresponding task 30098.

[0168] Each path in the tree structure may represent a specific set of alternative methods for performing a procedure. Such a structure may be useful for assisting in the creation of specific procedure plans for particular live and / or simulated surgeries. Such a structure may be useful for simulating many possible alternatives to a procedure and evaluating the differences in outcomes.

[0169] Figure 12 is a block diagram of an exemplary machine learning framework. Machine learning is a branch of artificial intelligence that seeks to build computer systems that can learn from data without human intervention. These techniques may rely on creating analytical models that can be trained to recognize patterns in datasets, such as data sets. These models can be deployed and these patterns applied to data, such as biomarkers, to improve performance without further guidance. For example, such a model may include machine learning module 33500.

[0170] Machine learning can be supervised (e.g., supervised learning). Supervised learning algorithms can create mathematical models from training a dataset (e.g., training data). Training data can consist of a set of training examples. Training examples may include one or more inputs and one or more labeled outputs. Labeled outputs may function as monitoring feedback. In mathematical models, training examples may be represented by arrays or vectors, sometimes called feature vectors. Training data may be represented by rows of feature vectors that make up a matrix. Through iterative optimization of an objective function (e.g., a cost function), a supervised learning algorithm can learn a function (e.g., a predictive function) that can be used to predict outputs associated with one or more new inputs. A well-trained predictive function may determine outputs for one or more inputs that may not have been part of the training data. Exemplary algorithms may include linear regression, logistic regression, and neural networks. Exemplary problems that can be solved by supervised learning algorithms may include classification, regression problems, and so on.

[0171] Machine learning can be unsupervised (e.g., unsupervised learning). Unsupervised learning algorithms can be trained on datasets that may contain input and can find structures within the data. Structures within the data may be analogous to grouping or clustering of data points. Thus, algorithms can learn from training data that may be unlabeled. Instead of responding to supervised feedback, unsupervised learning algorithms can identify commonalities in the training data and react based on the presence or absence of such commonalities in each training example. Exemplary algorithms may include a priori algorithms, K-means, K-Nearest Neighbor (KNN), K-median, etc. Exemplary problems that can be solved by unsupervised learning algorithms may include clustering problems and anomaly / outlier detection problems.

[0172] Machine learning may include reinforcement learning, which is a field of machine learning that may concern how a software agent can take actions in an environment to maximize the concept of cumulative reward. Reinforcement learning algorithms may not assume knowledge of an exact mathematical model of the environment (e.g., represented by a Markov decision process, MDP), and may be used when an exact model is not feasible. Reinforcement learning algorithms may be used in autonomous vehicles or in learning to play games against human opponents.

[0173] Machine learning can be part of a technological platform called cognitive computing (CC), which can comprise various fields such as computer science and cognitive science. CC systems can learn at scale, reason purposefully, and interact naturally with humans. Through self-teaching algorithms that may utilize data mining, visual recognition, and / or natural language processing, CC systems can solve problems and optimize human processes.

[0174] In an exemplary framework, a simulation device 33502 may enable the training of a machine learning model 33500 33504. The simulation device 33502 may represent a composite system that receives setup information such as simulated procedures and / or surgical tasks, anatomical structures, physiological functions, equipment, and instruments. The simulation device 33502 may represent a composite system that receives activity information such as instrument manipulation, simulated tasks, and / or simulated surgical activity primitives. The simulation device 33502 may represent a composite system that receives setup and activity information and, in response, outputs one or more results such as final procedure status, execution score, efficiency score, and simulated patient outcome. For example, the results may include specific states, values, metrics, etc. For example, the results may include functions such as weight functions for specific states, values, metrics, etc. A dataset represented by each combination of setup information, activity information, and results associated with the simulation device 33502 may be used to train the machine learning model 33500 (33504). As disclosed herein, training (33504) can be supervised learning, unsupervised learning, reinforcement learning, cognitive computing, etc.

[0175] The output of a machine learning training process can be a model for predicting outcomes on a new dataset. For example, a linear regression learning algorithm can have a cost function that minimizes the prediction error of a linear predictor function during the training process by adjusting the coefficients and constants of the linear predictor function. When the minimum value can be reached, the linear predictor function with the adjusted coefficients is considered trained and can constitute the model generated by the training process. For example, a neural network (NN) algorithm for classification (e.g., a multilayer perceptron (MLP)) can have a hypothesis function represented by a network of layers of nodes that are biased and interconnected by weight connections. The hypothesis function can be a nonlinear function (e.g., a highly nonlinear function) that includes a linear function and a nested logistic function with an outermost layer consisting of one or more logistic functions. An NN algorithm can have a cost function that minimizes the classification error by adjusting the biases and weights through the processes of feedforward and backpropagation. When a global minimum can be reached, the optimized hypothesis function with adjusted bias and weight layers can be considered trained and constitute the model generated by the training process.

[0176] A data set can be executed for machine learning as the first stage of the machine learning lifecycle. The data set may involve processes such as identifying various data sources, collecting data from those sources, and integrating the data. For example, to train a machine learning model to predict surgical complications and / or postoperative recovery rates, data sources containing pre-surgical data, such as patient medical condition and biomarker measurement data, may be identified. Such data sources may be the patient's electronic medical record (EMR), a computing system storing the patient's pre-surgical biomarker measurement data, and / or other similar data stores. Data from such data sources may be retrieved and stored in a central location for further processing in the machine learning lifecycle. Data from such data sources may be linked (e.g., logically linked) and accessed as if they were stored centrally. Surgical data and / or postoperative data may similarly be identified and collected. Furthermore, the collected data may be integrated. In the embodiment, patient pre-surgical medical record data, pre-surgical biomarker measurement data, pre-surgical data, surgical data, and / or post-surgical data may be combined with the patient's record. The patient's record may be EMR.

[0177] Data preparation may be performed for machine learning as another stage in the machine learning lifecycle. Data preparation may include data preprocessing steps such as data formatting, data cleaning, and data sampling. For example, the data to be collected may not be in a suitable data format for training a model. In an embodiment, a patient's integrated data record of pre-surgical EMR recording data and pre-surgical biomarker measurement data, surgical data, and post-surgical data may be in a rational database. Such data records may be converted to a flat file format for model training. In an embodiment, a patient's pre-surgical EMR data may include medical data in text format, such as the patient's diagnosis of emphysema and preoperative treatment (e.g., chemotherapy, radiation, blood diluents). Such data may be mapped to numerical values ​​for model training. For example, a patient's integrated data record may include personal identifier information, or other information that can identify the patient, such as age, workplace, body mass index (BMI), demographic information, and equivalents. Such identifying data may be removed before model training. For example, identifying data may be removed for privacy reasons. As another example, there may be more available data than can be used to train the model, and therefore data may be discarded. In such cases, a subset of the available data may be randomly sampled and selected for model training, while the rest may be discarded.

[0178] Data preparation may include data transformation procedures such as scaling and aggregation (e.g., post-preprocessing). For example, preprocessed data may contain data values ​​of various scales. These values ​​may be scaled up or down to between 0 and 1 for model training, for example. For example, preprocessed data may contain data values ​​that become more meaningful when aggregated. In an embodiment, a patient may have undergone multiple previous colorectal procedures. The total number of previous colorectal procedures may be more meaningful for training a model to predict surgical complications due to adhesions. In such a case, records of previous colorectal procedures may be aggregated into a total number for model training purposes.

[0179] Model training can be another aspect of the machine learning lifecycle. The model training process described herein may depend on the machine learning algorithm used. A model may be considered preferably trained after it has been trained, cross-validated, and tested. Thus, a dataset from the data preparation stage (e.g., input dataset) may be divided into a training dataset (e.g., 60% of the input dataset), a validation dataset (e.g., 20% of the input dataset), and a test dataset (e.g., 20% of the input dataset). After the model has been trained on the training dataset, it may be run on the validation dataset to reduce overfitting. If the model's accuracy decreases when run on the validation dataset while the model's accuracy is increasing, this may indicate an overfitting problem. The test dataset may be used to test the accuracy of the final model to determine whether it is ready for deployment or if more training may be needed.

[0180] Model deployment can be another aspect of the machine learning lifecycle. A model can be deployed as part of a standalone computer program. A model can be deployed as part of a larger computing system. A model can be deployed with model performance parameters. Such performance parameters can be used to monitor model accuracy because they are used to make predictions on a running dataset. For example, such parameters can track false positives and false positives in classification models. Such parameters can further store false positives and false positives for further processing to improve model accuracy.

[0181] Post-deployment model updates can be another aspect of the machine learning cycle. For example, a deployed model may be updated when false positives and / or false positives are predicted on production data. In an example, for an MLP model deployed for classification, when false positives occur, the deployed MLP model may be updated to increase the probability cutoff for predicting positives in order to reduce false positives. In an example, for an MLP model deployed for classification, when false negatives occur, the deployed MLP model may be updated to decrease the probability cutoff for predicting positives in order to reduce false negatives. In an example, for an MLP model deployed for the classification of surgical complications, when both false positives and false negatives occur, the deployed MLP model may be updated to decrease the probability cutoff for predicting positives in order to reduce false negatives, since predicting false positives may be less critical than predicting false negatives.

[0182] For example, an expanded model may be updated as more live-production data becomes available as training data. In such cases, the expanded model may be further trained, validated, and tested using such additional live-production data. In the embodiment, the updated biases and weights of the further-trained MLP model may update the biases and weights of the expanded MLP model. Those skilled in the art will recognize that post-expand model updates may not be a one-time occurrence and may be performed at a frequency suitable for improving the accuracy of the expanded model.

[0183] In an exemplary framework, machine learning model 33500 can be used to model a solution toward a specific outcome. For example, model 33500 can be structured and / or trained to receive instances of setup information as input and output corresponding activity information. For example, model 33500 can be structured and / or trained to receive instances of activity information as input and output corresponding setup information. In one embodiment, machine learning model 33500 can be used to model a solution toward a variable outcome. For example, model 33500 can be structured and / or trained to receive instances of setup information and a desired outcome as input and output corresponding activity information. For example, model 33500 can be structured and / or trained to receive instances of activity information and a desired outcome as input and output corresponding setup information.

[0184] An exemplary training set may include the results of simulations of all setups and / or activities having results within a predetermined acceptable range. For illustrative purposes, the model may address suture placement. For example, setups, activities, and results may include elements such as suture pattern, tension, and wound healing. In one embodiment, a simulation may determine the tension on the colon during reconnection in a lower anterior resection (LAR) anastomosis, and may include the resulting predicted perfusion. In one embodiment, a simulation may determine the lung volume after dilation following segmentectomy and / or lobectomy. An exemplary training set may include the results of multiple automated simulations of a specific surgical procedure using an activity set, and the model may enable determining the results of a pre-established starting set.

[0185] In one embodiment, training data may be aggregated to provide a summary view of possible options and / or variations, such as options and / or variations within a predetermined number of variable steps. Options and / or variations may include those with an acceptable set of results. Such a summary may include a proposed approach path for the best outcome. For example, such a summary may suggest an approach based on expert review, customer satisfaction surveys, results feedback, etc. For example, such a summary may provide an adverse event risk score. The adverse event risk score may be updated progressively up or down as the options progress. The summary may include an interactive tool for comparing and / or contrasting different approaches. The interactive tool may allow comparing and / or contrasting multiple autorun simulations.

[0186] In one embodiment, the simulation device 33502 may be used to create a ground truth dataset for machine learning and / or artificial intelligence training. For example, the simulation device 33502 may create a situation for establishing boundary parameters and / or edge cases for machine learning and / or artificial intelligence training. For example, the simulation device 33502 may provide robot path guidance output. Such guidance may be direct guidance, geofencing, etc. For example, the simulation device 33502 may provide feedback data. For example, data from a database-driven simulation may be used. The data may include information about similar cases. The database may include information about machine learning and / or artificial intelligence models trained using similar cases. For example, the simulation device 33502 may predict medical risks using a machine learning model. This prediction may include information about devices, equipment, and / or consumables that may be appropriate considering the predicted risk.

[0187] In one embodiment, the machine learning model 33500 may be trained to apply annotations, comments, and / or annotations to surgical videos. For example, such notes, comments, and / or annotations may be incorporated into the training set so that future videos and / or future simulations can be processed and reviewed according to the model. The model may output notes, comments, and / or annotations at appropriate points in the processed videos and / or simulations. For example, the machine learning model 33500 may be trained to present complex simulations in real time.

[0188] In one embodiment, the simulation device 33502 may be used to create a supplemental training set (e.g., a training set to supplement a real-world training set). For example, the simulation device 33502 may be used to create a supplemental training set that includes cases that are rare in the real world, such as rare anatomical conditions, rare physiological functions, and / or rare adverse events. Such a supplemental training set may be used, for example, to train a machine learning model 33500 (33504). Similarly, the simulation device 33502 may be used to create a patient-specific hybrid dataset. For example, the simulation device 33502 (e.g., via an application module) and / or the machine learning model 33500 may be used to check input parameters for patient-specific topics. Such checks may bring modifications to the simulated anatomical structure by modifications and / or additions to specific objects and / or object properties. For example, patient-specific anatomical structures may include adding irregularities to the simulation, such as multiple pulmonary artery branches and / or intrinsic growth of tumors. For example, such patient-specific modifications may be inputs from telemedicine input and / or preoperative monitoring. Such patient-specific datasets can be included for simulation to generate, for example, a dataset for training a machine learning model. The simulation device 33502 can be used to create a supplemental training set that incorporates rare, patient-specific irregularities.

[0189] In one embodiment, the simulation device 33502 can be used to create training datasets related to treatment platforms such as robotic, laparoscopic, open, and hybrid. Here, similar surgical objectives can be simulated using various platforms and / or variations within a particular platform. Such setup and activity data can be used in conjunction with their corresponding simulation results to, for example, train a machine learning model 33500 to recommend future surgical approaches.

[0190] Figure 13 is a block diagram of an exemplary surgical agent 33506. The surgical agent 33506 may include, for example, the surgical agent 30006 disclosed herein with reference to Figure 7. The surgical agent 33506 may provide computer-based control of a simulated task. Such a surgical agent 33506 may be used to simulate many variations of a surgical task. Such simulations may be used, for example, to generate large datasets for analyzing and / or training machine learning models, such as machine learning model 33500. Such datasets may be used to analyze and / or model modifiable aspects of surgical tasks.

[0191] In one embodiment, the surgical agent 33506 may receive treatment planning task data 33508. The treatment planning task data 33508 may include information about a specific surgical task. For example, the treatment planning task data 33508 may include one or more simulation parameters related to the surgical task. Such simulation parameters may provide information about how a particular surgical task should be handled in the simulation. The simulation parameters for a surgical task may include a parameterized representation of the surgical task, each of which includes, for example, one or more specific operations and one or more reference points. For example, in the case of a suturing task, the parameterized exemplary representation may include, for example, operations such as angled initial suture insertion and reference points such as points defined for a simulated wound. The treatment planning task data 33508 may be structured as disclosed herein with respect to, for example, Figures 11A to 11B.

[0192] In 33510, the surgeon agent 33506 may receive one or more simulation parameters from the treatment plan task data 33508. The surgeon agent 33506 may include logic for translating the simulation parameters into one or more control inputs to the simulation. For example, the surgeon agent 33506 may mimic control inputs based on decisions made from the simulation parameters. For example, the surgeon agent 33506 may engage with the simulation's application programming interface (API) as a way to determine control inputs and / or control the simulation. In 33512, the surgeon agent may translate one or more simulation parameters into one or more control inputs.

[0193] In 33514, the surgical agent 33506 may output a control input to the simulation device 33516. The simulation device 33516 may perform a simulation cycle based on such a control input and then send an output to the surgical agent. The output may include a representation of the simulation state taking the control input into account. The output may include, for example, one or more result metrics.

[0194] In 33518, the surgeon agent may receive the simulation output. The surgeon agent may loop to subsequent control inputs, if any exist. Subsequent control inputs may be converted from additional simulation parameters. Such additional simulation parameters may be received from and / or from the subsequent reception of additional procedure planning task data 33508. Such a surgeon agent may enable computer-based control of the surgical simulation according to a parameterized input dataset. One or more instances of the surgeon agent may be used as a foundational element in one or more simulations to test many variations of a particular task throughout the simulation. Such foundational elements may be combined, for example, to test variations of a surgical activity of any length. For example, one or more surgeon agents may be used together with one or more elements of the corresponding procedure planning task data to enable computer-based control of many variations of an entire surgical procedure in the simulation.

[0195] Figure 14 shows an exemplary simulation activity parameterized dataset. Parameterization of surgical activity for computer-based controlled simulation may involve the operation of the surgical activity parameterization tool 33520. The surgical activity parameterization tool 33520 may be used to generate parameterized surgical activity data 33522. The surgical activity parameterization tool 33520 may be based on a simulation platform. For example, the surgical activity parameterization tool 33520 may be implemented in the application module 30010 of the simulation device 30000 disclosed herein in relation to Figure 7. The surgical activity parameterization tool 33520 may be a standalone tool.

[0196] The surgical activity parameterization tool 33520 may include a processor configured to perform the steps, functions, and / or capabilities disclosed herein. The surgical activity parameterization tool 33520 may allow a user to define one or more parameters representing a surgical activity. For example, the surgical activity parameterization tool 33520 may allow a user to define one or more parameters representing a surgical activity, and as a result, a computer-based control agent, such as the surgical agent device 30006 disclosed herein with respect to Figure 7, may perform the surgical activity. Such parameters may represent parameterized surgical activity data 33520.

[0197] In one embodiment, parameterized surgical activity data 33520 may include a target object 33522, one or more activity parameters 33524, and / or one or more constraints 33526. The target object 33522 may include information indicating a specific object, a set of objects, and / or a part of an object in the simulation environment in which the surgical activity takes place. The activity parameters 33524 may include information indicating specific data elements and / or values ​​that define the variations available when performing the activity. The constraints 33526 may include information indicating any limitations associated with the activity, such limitations may be used by the simulation to prevent certain interpolation activities from being performed.

[0198] For illustrative purposes, parameterized surgical activity data 33520 may be used to define wound closure activity. In such embodiments, the target object 33522 may include information indicating a simulated wound, such as the tissue object on which the wound is located, geometric information regarding the location, size, and shape of the wound, wound depth, and subsurface object. For example, the location of the wound in the simulation can be indicated using parameterized data such as coordinates such as the centerline 33528, upper edge 33530, and lower edge 33532. Activity parameters 33524 may include information indicating the expected operation on the target object 33522. For example, the nature and quality of suturing activity can be indicated using parameterized data such as suture width 33534, suture interval, suture pattern, suture tension, suture depth, and suture type. For example, the manner in which the activity is performed can be defined using other parameterized data such as suturing method, instrument, suture material, needle type, and knot type. Constraints 33526 may include data indicating any specific limitations to restrict the activity behavior. For example, constraint 33526 may include duration, dexterity, precision, accuracy, collision constraints, applied pressure, and adjacent tissue contact.

[0199] Parameterized surgical activities 33520 can provide structures that represent many different ways of performing a particular activity. For example, several different sutures that can be performed by applying various combinations and / or permutations of parameter values.

[0200] The surgical activity parameterization tool 33520 may include a user interface that enables the definition of data elements and / or their values. In one embodiment, the surgical activity parameterization tool 33520 may provide a “macro” recording function that records actual user input control and populates the parameterized surgical activity data 33520 with elements, values, and / or elements and values ​​based on the recorded user input. The surgical activity parameterization tool 33520 may include a media player-like user interface 33538 for recording, playing back, editing, etc., user input control.

[0201] Figure 15 shows an exemplary surgical simulation flow for machine learning training. A parameterized surgical activity data structure can enable a computationally efficient approach to simulation-based training of machine learning models. In one embodiment, one or more simulation environments may be generated in 33540. For example, one or more simulation environments may be generated by combinations and / or permutations of setup data elements such as setup data 30088, 30090, 30092, for example, as disclosed herein with respect to Figure 11B. Such generation may result in one or more training task-environment parameter sets 33542.

[0202] In 33544, for each training task-environment parameter set 33542, possible combinations of activities can be generated. For example, such generation may include various combinations and / or permutations of procedure tasks and / or activity data, such as the task data 30094, 30096, 30098 disclosed herein with respect to Figure 11B, and / or the parameterized surgical activity data 33520 disclosed herein with respect to Figure 14. Such generation may output one or more candidate action sets 33546. In one embodiment, each training task-environment parameter set 33542 may have one or more associated candidate action sets 33546. In this way, various surgical setups and / or various surgical approaches can be generated for simulation under computer-based control input. The combination of training task-environment parameter set 33542 and candidate action sets 33546 may represent the input space being simulated. The combination of the training task-environment parameter set 33542 and the candidate action set 33546 may represent a portion of the training data that can be used for a machine learning model.

[0203] In one embodiment, a training task-environment parameter set may be received from a surgical data system, such as the surgical data system 30008 disclosed herein with respect to Figure 7. The surgical data system may record real-world environmental data from the real-world execution of a surgical task. The surgical data system may use such real-world environmental data to generate multiple training task-environment parameter sets. Using this approach, the training task-environment parameter sets can better represent typical anatomical structures, physiological functions, etc. Similarly, this approach may be used so that the training task-environment parameter sets can better represent, for example, the anatomical structures, physiological functions, etc. of a particular patient population.

[0204] In 33548, a candidate action set 33546 with a corresponding training task-environment parameter set 33542 can be simulated and / or evaluated. For example, the candidate action set 33546 with a corresponding training task-environment parameter set 33542 may be provided as input to a simulation device, such as the simulation device 30000 disclosed herein with respect to Figure 7. For example, the training task-environment parameter set 33542 may be provided to the simulation as setup information. For example, the candidate action set 33546 may be driven by a surgical agent, such as the surgical agent 30006, to operate the simulation.

[0205] The evaluation of the execution of each candidate action set 33546 and training task-environment parameter set 33542 pair may yield one or more effectiveness metrics 33550. The evaluation may include any appropriate execution metric that takes into account the desired machine learning training. For example, the evaluation may include a comparison of the final simulation state with a preferred target simulation state. For example, the evaluation may include an evaluation of specific execution characteristics such as treatment duration, treatment movement, equipment changes, instrument use, and consumable use. In one embodiment, the effectiveness metric 33550 may include a function such as a weight function of one or more execution results. In one embodiment, the effectiveness metric 33550 may include a vector of one or more execution results.

[0206] In 33552, each pair of candidate action sets 33546 and training task-environment parameter sets 33542, and their respective effectiveness metrics 33550, can serve as training data for training a machine learning model. Training can include any appropriate training algorithm, taking into account the model objective and the nature of the training data. For example, training can include supervised learning, unsupervised learning, reinforcement learning, cognitive computing, etc.

[0207] The output of such training may include model 33554. In 33556, the model may be used to determine pairs of new candidate action sets 33546 and training task-environment parameter sets 33542 that match the training and effectiveness metrics 33550. In one embodiment, model 33554 may receive a "new" task-environment parameter set (e.g., target task-environment parameter set 33558). The model may then output a corresponding "new" action set (e.g., target action set 33560). In one embodiment, the model can be trained to receive a "new" action set and output a corresponding "new" task-environment parameter set. In this sense, simulation data may be used to model surgical environments, surgical activities, and their corresponding outcomes in order to determine new environments (e.g., surgical setups) and / or new activities (e.g., procedures) aimed at achieving results (e.g., specific execution objectives) (e.g., maximization, minimization, etc.).

[0208] The determined target action set 33560 and the training task-environment parameter set 33558 can be simulated in 33562. The simulation may include evaluation and corresponding effectiveness metrics. Such metrics may be used, for example, to validate model 33554.

[0209] To illustrate the process, in one embodiment, the procedure may include a suturing procedure. One or more environments may be defined. For example, a given surgical site may be flexed to establish one or more training task-environment parameter sets. For example, a single given surgical site may be used and modeled to generate a single training task-environment parameter set.

[0210] One or more suturing approaches may be defined. For example, a simulation of a needle penetrating the tissue on both sides of the suture site may be developed (e.g., using a tool such as the surgical activity parameterization tool 33520). For example, a computer vision-based approach may be applied to extract relevant geometric parameters from the suture site. For example, the suture site and / or suturing activity may be parameterized.

[0211] The suture width can be positioned in the simulation and evaluated according to effectiveness metrics such as wound contact pressure. For example, the effectiveness metric can be used to train a model to maximize the effectiveness metric across several variations, such as suture position, depth, number, and force. In one embodiment, the model may continue to learn / automate across various options and taking into account the overall suturing force to calculate the best target criterion for the entire suture site. The module can be used in a target environment to output a suturing approach that maximizes the effectiveness metric of the suture site.

[0212] Figure 16 is a flowchart of an exemplary surgical simulation operation. For example, an exemplary surgical simulation operation may include a process of determining a set of target actions to perform a simulated surgical task in a simulated environment. The simulated environment for the simulated surgical task may be represented by a target task-environment parameter set.

[0213] In 33564, multiple training task-environment parameter sets may be received. Each training task-environment parameter set may represent a simulated environment in which a simulated surgical task should be completed. For example, multiple training task-environment parameter sets may be received from a surgical data system. The surgical data system may record real-world environmental data from the execution of surgical tasks in the real world. The surgical data system may generate multiple training task-environment parameter sets from the real-world environmental data. For example, each training task-environment parameter set may include anatomical information, physiological function, and / or surgical setup information.

[0214] In 33566, multiple candidate action sets may be generated for each training task-environment parameter set. Each candidate action set may include parameterized surgical activities within its respective simulated environment. For example, a parameterized activity may include simulated dissection, parameterized by relative anatomical position, instrument selection, and instrument application direction. For example, a parameterized activity may include suturing, parameterized by suture position, suture pattern, number of sutures, suture material, and suture tension. For example, a parameterized activity may include surgical robot guidance pathways.

[0215] In 335658, each candidate action set can be executed within a simulated environment indicated by the corresponding training task-environment set. When the candidate action set is executed in this manner, it can perform a simulated surgical task. Such executions may have effectiveness indicated by the corresponding effectiveness metric. For example, each execution may be evaluated in 33570 to determine its effectiveness metric. The corresponding effectiveness metric may be stored. In one embodiment, an output may be generated. The output may be a human-readable aggregate of multiple task-environment parameter sets, each of multiple candidate action sets, and each of the corresponding effectiveness metrics.

[0216] In 33572, a model can be trained. For example, multiple task environment parameter sets, multiple candidate action sets for each, and their corresponding effectiveness metrics can be used to train the model.

[0217] In 33574, the target action set can be determined based on the target task-environment parameter set and the model. Training can be based on machine learning algorithms such as supervised learning algorithms, unsupervised learning algorithms, and reinforcement learning algorithms.

[0218] In one embodiment, an action set (e.g., a candidate action set, a target action set, or an input action set) may refer to a set of parameters that define one or more actions performed during surgery. For example, an action set may refer to several surgical choices made by an operator. For example, an action set may include decisions regarding a particular layout (both surgical staff and surgical instruments). An action set may also refer to specific instruments used during surgery. In other words, an action set may refer to a set of surgical procedures that affect the surgery in terms of both duration, surgical outcome, and effectiveness of the procedure. For example, the candidate action set, target action set, and input action set may use compatible and / or identical formats, respectively. A machine learning model may be trained, for example, on a candidate action set to help define relationships between any "input" action sets. This allows the machine learning model to be used to find a target action set with certain effectiveness within a given simulation.

[0219] Similarly, an environment parameter set (e.g., environment parameter set, target environment parameter set, or input environment parameter set) can refer to a set of parameters that define the overall environment of a surgical simulation. This can refer to parameters that initialize and define the constraints and physical properties of the surgical simulation. Target and environment parameter sets may use, for example, a format compatible with and / or identical to the format of the environment parameter set. A machine learning model can be trained, for example, on the environment parameter set to help define relationships between any "input" environment parameter sets. This allows the machine learning model to be used to find a set of target actions that have certain effectiveness within a given simulation.

[0220] In one embodiment, a machine learning model may be trained in a supervised manner using a labeled dataset. For example, a combination of an environment parameter set and an action set may be considered input data (as it refers to a specific set of actions performed within a particular simulation environment). This data for training purposes may be "labeled" with its associated effectiveness metric (i.e., the simulation's output regarding how effective a particular action was within that simulation). By feeding this labeled data to a machine learning model, the model may generate relationships between effectiveness associated with different actions / environments.

[0221] The simulated surgical task and effectiveness metrics may include any surgical activity that undergoes simulation with objective goals subject to computer evaluation. For example, the simulated surgical task may include a lower anterior resection anastomosis. The effectiveness metrics may include perfusion resulting from the performance of the simulated surgical task. For example, the simulated surgical task may include either a segmentectomy or a lobectomy. The effectiveness metrics may also include indications of output lung pneumopulmonary volume resulting from the performance of the simulated surgical task.

[0222] The following non-exclusive enumeration of embodiments also forms part of this disclosure.

[0223] Embodiment 1. A computer implementation method for determining a target action set, wherein the method is: This involves receiving multiple environmental parameter sets, each of which corresponds to a simulated environment in which a simulated surgical task is to be completed. The method involves generating multiple candidate action sets for each environmental parameter set, where each candidate action set, when executed within its respective simulated environment, performs its respective simulated surgical task with the effectiveness indicated by the corresponding effectiveness metric. Training a machine learning model using multiple environmental parameter sets, multiple candidate action sets, and corresponding effectiveness metrics, such that the trained machine learning model may include a functional relationship between the model input and the model output, wherein the model input includes an input environmental parameter set and an input action set, and the output includes an output effectiveness metric. A method comprising determining a target action set based on a target environment parameter set and a corresponding effectiveness metric output by a trained machine learning model.

[0224] It will be understood that an action set (e.g., a candidate action set, a target action set, or an input action set) can refer to a set of parameters that define one or more actions performed during surgery. In some embodiments, an action set can refer to several surgical choices made by the operator. In some embodiments, an action set can include decisions regarding a particular layout (both surgical staff and surgical instruments). An action set can also refer to specific instruments used during surgery. In other words, an action set can refer to a set of surgical procedures that affect the surgery in terms of both duration, surgical outcome, and effectiveness of the procedure. It should be understood that candidate action sets, target action sets, and input action sets are all datasets having the same format. That is, a machine learning model can be trained on a candidate action set to help define relationships between any "input" action sets. This allows the machine learning model to be used to find a target action set with certain effectiveness within a given simulation.

[0225] The environmental parameter set (e.g., environmental parameter set, target environmental parameter set, or input environmental parameter set) can also be understood as a set of parameters that define the overall environment of a surgical simulation. In other words, the environmental parameter set refers to the parameters that initialize and define the constraints and physical properties of the surgical simulation. It should be understood that both the target parameter set and the environmental parameter set are datasets with the same format as the environmental parameter set. That is, a machine learning model can be trained on the environmental parameter set to help define the relationships between any "input" environmental parameter sets. This allows the machine learning model to be used to find a target action set that has some effectiveness within a given simulation.

[0226] In one embodiment, “training a machine learning model” may refer to training a model in a supervised manner using a labeled dataset. In this case, a combination of environment parameter sets and action sets can be considered input data (as it refers to a specific set of actions performed within a particular simulation environment). This data for training purposes may be “labeled” with its associated effectiveness metric (i.e., the simulation output regarding how effective a particular action was within that simulation). By feeding this labeled data to a machine learning model, the model can generate relationships between effectiveness associated with different actions / environments.

[0227] To avoid misunderstanding, when this embodiment refers to the “environmental parameter set,” it may be the same parameter set that the application refers to as the “training task-environmental parameter set” (i.e., the parameter set that may be used to train a machine learning model).

[0228] Both the initialization of the surgical simulation (defined by the environment parameter set) and the specific choices / actions performed within the simulation (defined by the candidate action set) affect the overall effectiveness of the surgery (defined by the effectiveness metric).

[0229] In one embodiment, "determining the target action set" may refer to selecting a set of action sets from a given set of action sets within a simulation (as defined by the environmental parameter set). The target action set may be selected based on the target action set that produces the best output efficiency metric from the machine learning model (as input to the machine learning model).

[0230] In other words, the present invention uses simulation output data to improve the training of a machine learning model, and then uses the improved trained artificial intelligence to make improved conclusions regarding surgical actions / selections that can be performed within a particular surgical setup.

[0231] One advantage of this is that machine learning models (such as neural networks) can be trained with additional data that would not normally occur during live surgical procedures. In other words, simulations can provide machine learning models with data on "higher-risk" surgical choices and their outcomes. The intelligence derived from machine learning models can then inform future surgical decisions by highlighting decisions that have a truly beneficial effect, without requiring actual patients to take risks to collect the necessary data.

[0232] The result of improving the training of machine learning models using the method of this embodiment is that improved surgical outcomes can be achieved by the determined target action set.

[0233] Embodiment 2. The computer implementation method according to Embodiment 1, wherein receiving multiple environmental parameter sets includes receiving multiple environmental parameter sets from a surgical data system that records real-world environmental data from the execution of a surgical task in the real world and generates multiple environmental parameter sets from the real-world environmental data.

[0234] Embodiment 3. A computer-aided method according to either Embodiment 1 or 2, wherein each environmental parameter set includes anatomical information, physiological information, and surgical setup information.

[0235] Embodiment 4. A computer-aided method according to any one of Embodiments 1 to 3, wherein each candidate action set includes parameterized surgical activities within their respective simulated environments.

[0236] Embodiment 5. The computer-aided method according to Embodiment 4, wherein the parameterized activity includes a simulated dissection parameterized by either the relative anatomical position, the selection of the instrument, or the direction of application of the instrument.

[0237] Embodiment 6. The computer-aided method according to Embodiment 4, wherein the parameterized activity includes simulated suturing, which is parameterized by one of the following: suture position, suture pattern, number of sutures, suture material, and suture tension.

[0238] Embodiment 7. The computer-assisted method according to Embodiment 4, wherein the parameterized activity includes a surgical robot guidance path.

[0239] Embodiment 8. A computer-aided method according to any one of Embodiments 1 to 7, wherein the simulated surgical task includes a lower anterior resection anastomosis, and each candidate action set is performed within a simulated environment indicated by the corresponding environment set, and the simulated surgical task is performed with the effectiveness indicated by the corresponding effectiveness metric, where the corresponding effectiveness metric is an indicator of perfusion resulting from the execution of the simulated surgical task.

[0240] Embodiment 9. A computer-aided method according to any one of Embodiments 1 to 8, wherein the simulated surgical task includes either a segmentectomy or a lobectomy, and each candidate action set is executed within a simulated environment indicated by a corresponding environment set, and the simulated surgical task is executed with the effectiveness indicated by a corresponding effectiveness metric, the corresponding effectiveness metric being an indicator of the output lung volume resulting from the execution of the simulated surgical task.

[0241] Embodiment 10. A computer implementation method according to any one of Embodiments 1 to 9, further comprising generating an output which is a human-readable aggregate of multiple task environment parameter sets, multiple candidate action sets for each, and their respective corresponding effectiveness metrics.

[0242] Embodiment 11. A computer implementation method according to any one of Embodiments 1 to 10, wherein training a machine learning model includes training the machine learning model using one of a supervised learning algorithm, an unsupervised learning algorithm, or a reinforcement learning algorithm.

[0243] Embodiment 12. A device for determining a target action set, wherein the device is Equipped with a processor, the processor, The process involves receiving multiple environmental parameter sets from a surgical data system, where each environmental parameter set corresponds to a simulated environment in which a simulated surgical task is to be completed. The method involves generating multiple candidate action sets for each environmental parameter set, where each candidate action set, when executed within the respective simulated environment corresponding to the corresponding environmental set, performs the respective simulated surgical task with the effectiveness indicated by the corresponding effectiveness metric. Training a machine learning model using multiple environmental parameter sets, multiple candidate action sets, and corresponding effectiveness metrics, such that the trained machine learning model may include a functional relationship between the model input and the model output, wherein the model input includes an input environmental parameter set and an input action set, and the output includes an output effectiveness metric. A device configured to determine a target action set based on a target environment parameter set and the corresponding effectiveness metrics output by a trained machine learning model.

[0244] Embodiment 13. The surgical data system is configured to record real-world environmental data from the execution of a surgical task in the real world, and is further configured to generate a set of environmental parameters from the real-world environmental data, as described in Embodiment 12.

[0245] Embodiment 14. The device according to either Embodiment 12 or 13, wherein each environmental parameter set includes anatomical information, physiological information, and surgical setup information.

[0246] Embodiment 15. The device according to any one of Embodiments 12 to 14, wherein each candidate action set includes a parameterized surgical activity within its respective simulated environment.

[0247] Embodiment 16. The device according to Embodiment 15, wherein the parameterized activity includes simulated dissection, which is parameterized by either the relative anatomical position, the selection of the instrument, and the direction of application of the instrument.

[0248] Embodiment 17. The device according to Embodiment 15, wherein the parameterized activity includes simulated suturing, which is parameterized by any of the following: suture location, suture pattern, number of sutures, suture material, and suture tension.

[0249] Embodiment 18. The device according to Embodiment 15, wherein the parameterized activity includes a surgical robot guidance path.

[0250] Embodiment 19. The device according to any one of Embodiments 12 to 18, wherein the simulated surgical task includes a lower anterior resection anastomosis, and each candidate action set is performed within a simulated environment indicated by the corresponding environment set, and the simulated surgical task is performed with the effectiveness indicated by the corresponding effectiveness metric, where the corresponding effectiveness metric is an indicator of perfusion resulting from the execution of the simulated surgical task.

[0251] Embodiment 20. The device according to any one of Embodiments 12 to 18, wherein the simulated surgical task includes either a segmentectomy or a lobectomy, and each candidate action set is executed within a simulated environment indicated by the corresponding environment set, and the simulated surgical task is executed with the effectiveness indicated by the corresponding effectiveness metric, the corresponding effectiveness metric being an indicator of the output lung volume resulting from the execution of the simulated surgical task.

[0252] Embodiment 21. The device according to any one of Embodiments 12 to 20, wherein the processor is further configured to produce an output which is a human-readable aggregate of multiple task environment parameter sets, multiple candidate action sets for each, and their respective corresponding effectiveness metrics.

[0253] Embodiment 22. The device according to any one of Embodiments 12 to 21, wherein the processor is configured to train a machine learning model using one of the supervised learning algorithms, the unsupervised learning algorithms, and the reinforcement learning algorithms.

[0254] Embodiment 23. A device for determining a target action set, wherein the device is Equipped with a processor, the processor, The process involves receiving multiple environmental parameter sets from a surgical data system, where each environmental parameter set corresponds to a simulated environment in which a simulated surgical task is to be completed. The method involves generating multiple candidate action sets for each environmental parameter set, where each candidate action set, when executed within its respective simulated environment, performs a simulated surgical task with effectiveness indicated by the corresponding effectiveness metric. Training a machine learning model using multiple environmental parameter sets, multiple candidate action sets, and corresponding effectiveness metrics, such that the trained machine learning model includes a functional relationship between the model input and the model output, wherein the model input includes an input environmental parameter set and an input action set, and the output includes an output effectiveness metric. A device configured to determine a target action set based on a target environment parameter set and the corresponding effectiveness metrics output by a trained machine learning model.

[0255] Embodiment 24. The surgical data system is configured to record real-world environmental data from the execution of a surgical task in the real world, and is further configured to generate a set of environmental parameters from the real-world environmental data, as described in Embodiment 23.

[0256] Embodiment 25. The device according to either Embodiment 23 or 24, wherein each environmental parameter set includes anatomical information, physiological information, and surgical setup information.

[0257] Embodiment 26. The device according to any one of Embodiments 23 to 25, wherein each candidate action set includes parameterized surgical activities within their respective simulated environments.

[0258] Embodiment 27. A device for determining a target action set, wherein the device is Equipped with a processor, the processor, The trained machine learning model receives a machine learning model trained with multiple environmental parameter sets, each with multiple candidate action sets, and corresponding effectiveness metrics, such that the trained machine learning model includes a functional relationship between the model input and the model output, wherein the model input includes an input environmental parameter set and an input action set, and the output includes an output effectiveness metric. It is configured to determine a target action set based on an environmental parameter set and a machine learning model, Each environmental parameter set corresponds to the respective simulated environment in which the simulated surgical task is to be completed. Multiple environmental parameter sets are from the surgical data system. For each environmental parameter set, multiple candidate action sets are executed within the corresponding simulated environment, allowing the device to perform a simulated surgical task with effectiveness indicated by the corresponding effectiveness metric.

[0259] Embodiment 28. The device according to Embodiment 27, wherein each environmental parameter set includes anatomical information, physiological information, and surgical setup information, and each candidate action set includes parameterized surgical activities within their respective simulated environments.

[0260] Embodiment 29. A computer-readable medium that, when executed by a computer, includes instructions causing the computer to perform the method described in any one of Embodiments 1 to 11.

[0261] The following non-exclusive enumeration of embodiments also forms part of this disclosure.

[0262] Embodiment 1. A device for determining a target action set for executing a simulated surgical task in a simulated environment indicated by a target task-environment parameter set, wherein the device comprises a processor, the processor being configured to receive a plurality of training task-environment parameter sets from a surgical data system, each training task-environment parameter set corresponding to a respective simulated environment in which a simulated surgical task is to be completed, and to generate a plurality of candidate action sets for each training task-environment parameter set, each candidate action set, when executed in the simulated environment indicated by the corresponding training task-environment set, execute the simulated surgical task with the effectiveness indicated by the corresponding effectiveness metric, train a model using the plurality of task-environment parameter sets, each plurality of candidate action sets, and each corresponding effectiveness metric, and determine a target action set based on the target task-environment parameter set and the model.

[0263] Embodiment 2. The device according to Embodiment 1, wherein the processor is further configured to record real-world environmental data from the execution of a surgical task in the real world and to receive multiple training task-environment parameter sets from a surgical data system that generates multiple training task-environment parameter sets from the real-world environmental data.

[0264] Embodiment 3. The device according to Embodiment 1, wherein each training task-environment parameter set includes anatomical information, physiological information, and surgical setup information.

[0265] Embodiment 4. The device according to Embodiment 1, wherein each candidate action set includes parameterized surgical activities within their respective simulated surgical environments.

[0266] Embodiment 5. The device according to Embodiment 4, wherein the parameterized activity includes simulated dissection, which is parameterized by either the relative anatomical position, the selection of the instrument, or the direction of application of the instrument.

[0267] Embodiment 6. The device according to Embodiment 1, further comprising generating an output which is a human-readable aggregate of multiple task environment parameter sets, multiple candidate action sets for each, and their respective corresponding effectiveness metrics.

[0268] Embodiment 7. The device according to Embodiment 1, wherein training a model includes training the model using any of the supervised learning algorithm, the unsupervised learning algorithm, and the reinforcement learning algorithm.

[0269] Embodiment 8. A method for determining a target action set for performing a simulated surgical task in a simulated environment indicated by a target task-environment parameter set, the method comprising: receiving a plurality of training task-environment parameter sets, each training task-environment parameter set corresponding to a respective simulated environment in which a simulated surgical task is to be completed; generating a plurality of candidate action sets for each training task-environment parameter set, each candidate action set, when executed in the simulated environment indicated by the corresponding training task-environment set, causing the simulated surgical task to be performed with the effectiveness indicated by the corresponding effectiveness metric; training a model using the plurality of task-environment parameter sets, each plurality of candidate action sets, and each corresponding effectiveness metric; and determining a target action set based on a target task-environment parameter set and the model.

[0270] Embodiment 9. The method according to Embodiment 8, wherein receiving multiple training task-environment parameter sets includes receiving multiple training task-environment parameter sets from a surgical data system that records real-world environmental data from the execution of a surgical task in the real world and generates multiple training task-environment parameter sets from the real-world environmental data.

[0271] Embodiment 10. The method according to Embodiment 8, wherein each training task-environment parameter set includes anatomical information, physiological information, and surgical setup information.

[0272] Aspect 11. The method according to aspect 8, wherein each candidate action set includes parameterized surgical activities within each simulated surgical environment.

[0273] Aspect 12. The method according to aspect 11, wherein the parameterized activities include simulated dissections parameterized by any one of relative anatomical position, instrument selection, and instrument application direction.

[0274] Aspect 13. The method according to aspect 11, wherein the parameterized activities include simulated suturing parameterized by any one of suture position, suture pattern, number of sutures, suture material, and suture tension.

[0275] Aspect 14. The method according to aspect 11, wherein the parameterized activities include a surgical robot guidance path.

[0276] Aspect 15. The simulated surgical task includes a lower anterior resection anastomosis. When each candidate action set is executed within the simulated environment indicated by the corresponding training task - environment set, the simulated surgical task is executed with an effectiveness indicated by the corresponding effectiveness metric, and the corresponding effectiveness metric is an indicator of perfusion resulting from the execution of the simulated surgical task. The method according to aspect 8.

[0277] Aspect 16. The simulated surgical task includes either a segmentectomy or a lobectomy. When each candidate action set is executed within the simulated environment indicated by the corresponding training task - environment set, the simulated surgical task is executed with an effectiveness indicated by the corresponding effectiveness metric, and the corresponding effectiveness metric is an indicator of the output lung volume resulting from the execution of the simulated surgical task. The method according to aspect 8.

[0278] Aspect 17. The method according to aspect 8, further including generating an output that is a human - readable aggregation of a plurality of task environment parameter sets, each of the plurality of candidate action sets, and each corresponding effectiveness metric.

[0279] Embodiment 18. The method according to Embodiment 8, wherein training the model includes training the model using one of the supervised learning algorithm, the unsupervised learning algorithm, and the reinforcement learning algorithm.

[0280] Embodiment 19. A device for determining a target action set for executing a simulated surgical task in a simulated environment indicated by a target task-environment parameter set, the device comprising a processor, the processor receiving a plurality of training task-environment parameter sets, a plurality of candidate action sets for each, and a model trained using a corresponding effectiveness metric, each training task-environment parameter set corresponding to a simulated environment in which a simulated surgical task is to be completed, the plurality of training task-environment parameter sets being from a surgical data system, and for each training task-environment parameter set, the plurality of candidate action sets causing the simulated surgical task to be executed with effectiveness indicated by the corresponding effectiveness metric when executed in the simulated environment indicated by the corresponding training task-environment set, and the device determining a target action set based on the target task-environment parameter set and the model.

[0281] Embodiment 20. The device according to Embodiment 19, wherein each training task-environment parameter set includes anatomical information, physiological information, and surgical setup information, and each candidate action set includes parameterized surgical activities within their respective simulated surgical environments.

[0282] [Implementation Method] (1) A computer implementation method for determining a target action set, wherein the method is This involves receiving multiple environmental parameter sets, each of which corresponds to a simulated environment in which a simulated surgical task is to be completed. The method involves generating multiple candidate action sets for each environmental parameter set, wherein each candidate action set, when executed within the respective simulated environment, performs the respective simulated surgical task with the effectiveness indicated by the corresponding effectiveness metric. Training a machine learning model using a plurality of environment parameter sets, a plurality of candidate action sets, and a corresponding effectiveness metric, such that the trained machine learning model may include a functional relationship between the model input and the model output, wherein the model input includes an input environment parameter set and an input action set, and the output includes an output effectiveness metric. A method comprising determining a target action set based on a target environment parameter set and a corresponding effectiveness metric output by the trained machine learning model. (2) The computer implementation method according to Embodiment 1, wherein receiving the plurality of environmental parameter sets includes receiving the plurality of environmental parameter sets from a surgical data system that records real-world environmental data from the execution of the surgical task in the real world and generates the plurality of environmental parameter sets from the real-world environmental data. (3) The computer-aided method according to Embodiment 1, wherein each environmental parameter set includes anatomical information, physiological information, and surgical setup information. (4) Each candidate action set includes a parameterized surgical activity within the respective simulated environment, as described in Embodiment 1, computer-aided method. (5) The computer-aided method according to Embodiment 4, wherein the parameterized activity includes simulated dissection, which is parameterized by either the relative anatomical position, the selection of the instrument, or the direction of application of the instrument.

[0283] (6) The computer-aided method according to Embodiment 4, wherein the parameterized activity includes simulated suturing, which is parameterized by any of the following: suture position, suture pattern, number of sutures, suture material, and suture tension. (7) The computer-aided method according to Embodiment 4, wherein the parameterized activity includes a surgical robot guidance pathway. (8) The computer-aided method according to Embodiment 1, wherein the simulated surgical task includes a lower anterior resection anastomosis, and each candidate action set is performed within the simulated environment indicated by the corresponding environment set, causing the simulated surgical task to be performed with the effectiveness indicated by the corresponding effectiveness metric, the corresponding effectiveness metric being an indicator of perfusion resulting from the execution of the simulated surgical task. (9) The computer-aided method according to Embodiment 1, wherein the simulated surgical task includes either a segmentectomy or a lobectomy, and each candidate action set, when executed within the simulated environment indicated by the corresponding environment set, causes the simulated surgical task to be executed with the effectiveness indicated by the corresponding effectiveness metric, the corresponding effectiveness metric being an indicator of the output lung volume resulting from the execution of the simulated surgical task. (10) A computer implementation method according to Embodiment 1, further comprising generating an output which is a human-readable aggregate of the plurality of task environment parameter sets, each of the plurality of candidate action sets, and each of the corresponding effectiveness metrics.

[0284] (11) The computer implementation method according to Embodiment 1, wherein training the machine learning model includes training the machine learning model using any of the supervised learning algorithm, the unsupervised learning algorithm, and the reinforcement learning algorithm. (12) A device for determining a target action set, the device is The processor comprises, The process involves receiving multiple environmental parameter sets from a surgical data system, where each environmental parameter set corresponds to a simulated environment in which a simulated surgical task is to be completed. The method involves generating multiple candidate action sets for each environmental parameter set, wherein each candidate action set, when executed within the respective simulated environment corresponding to the corresponding environmental set, performs the respective simulated surgical task with the effectiveness indicated by the corresponding effectiveness metric. Training a machine learning model using a plurality of environment parameter sets, a plurality of candidate action sets, and a corresponding effectiveness metric, such that the trained machine learning model may include a functional relationship between the model input and the model output, wherein the model input includes an input environment parameter set and an input action set, and the output includes an output effectiveness metric. A device configured to determine a target action set based on a target environment parameter set and a corresponding effectiveness metric output by the trained machine learning model. (13) The device according to Embodiment 12, wherein the surgical data system is configured to record real-world environmental data from the execution of the surgical task in the real world, and is further configured to generate the plurality of environmental parameter sets from the real-world environmental data. (14) The device according to Embodiment 12, wherein each environmental parameter set includes anatomical information, physiological information, and surgical setup information. (15) The device according to Embodiment 12, wherein each candidate action set includes a parameterized surgical activity within the respective simulated environment.

[0285] (16) The device according to Embodiment 15, wherein the parameterized activity includes simulated dissection, which is parameterized by either the relative anatomical position, the selection of the instrument, and the direction of application of the instrument. (17) The device according to Embodiment 15, wherein the parameterized activity includes simulated suturing, which is parameterized by any of the following: suture location, suture pattern, number of sutures, suture material, and suture tension. (18) The device according to Embodiment 15, wherein the parameterized activity includes a surgical robot guidance pathway. (19) The device according to Embodiment 12, wherein the simulated surgical task includes a lower anterior resection anastomosis, and each candidate action set is performed within the simulated environment indicated by the corresponding environment set, causing the simulated surgical task to be performed with the effectiveness indicated by the corresponding effectiveness metric, the corresponding effectiveness metric being an indicator of perfusion resulting from the execution of the simulated surgical task. (20) The device according to Embodiment 12, wherein the simulated surgical task includes either a segmentectomy or a lobectomy, and each candidate action set, when performed within the simulated environment indicated by the corresponding environment set, causes the simulated surgical task to be performed with the effectiveness indicated by the corresponding effectiveness metric, the corresponding effectiveness metric being an indicator of the output lung volume resulting from the execution of the simulated surgical task.

[0286] (21) The device according to Embodiment 12, wherein the processor is further configured to produce an output which is a human-readable aggregate of the plurality of task environment parameter sets, each of the plurality of candidate action sets, and each of the corresponding effectiveness metrics. (22) The device according to Embodiment 12, wherein the processor is configured to train the machine learning model using any of the supervised learning algorithm, the unsupervised learning algorithm, and the reinforcement learning algorithm. (23) A device for determining a target action set, wherein the device is The processor comprises, The process involves receiving multiple environmental parameter sets from a surgical data system, where each environmental parameter set corresponds to a simulated environment in which a simulated surgical task is to be completed. Generating a plurality of candidate action sets for each environmental parameter set, wherein each candidate action set, when executed within the respective simulation environment, has an effectiveness indicated by a corresponding effectiveness metric and causes the simulation surgical task to be executed. Training the machine learning model using the plurality of environmental parameter sets, the respective plurality of candidate action sets, and their respective corresponding effectiveness metrics such that the trained machine learning model includes a functional relationship between a model input and an output, wherein the model input includes an input environmental parameter set and an input action set, and the output includes an output effectiveness metric. A device configured to determine a target action set based on a target environmental parameter set and a corresponding effectiveness metric output by the trained machine learning model. (24) The surgical data system according to embodiment 23, configured to record real-world environmental data from the real-world execution of the surgical task and further configured to generate the plurality of environmental parameter sets from the real-world environmental data. (25) The device according to embodiment 23, wherein each environmental parameter set includes anatomical information, physiological information, and surgical setup information.

[0287] (26) The device according to embodiment 2, wherein each candidate action set includes parameterized surgical activities within the respective simulation environment. (27) A device for determining a target action set, the device comprising: A processor, the processor being: Receiving a machine learning model trained with a plurality of environment parameter sets, a plurality of candidate action sets, and a corresponding effectiveness metric, such that the trained machine learning model includes a functional relationship between the model input and the model output, wherein the model input includes an input environment parameter set and an input action set, and the output includes an output effectiveness metric. The system is configured to determine a target action set based on the aforementioned environmental parameter set and the aforementioned machine learning model, Each environmental parameter set corresponds to the respective simulated environment in which the simulated surgical task is to be completed. The aforementioned sets of environmental parameters are from the surgical data system. For each set of environmental parameters, multiple candidate action sets, when executed within the simulated environment corresponding to the corresponding environmental set, cause the simulated surgical task to be performed with the effectiveness indicated by the corresponding effectiveness metric. (28) The device according to Embodiment 27, wherein each environmental parameter set includes anatomical information, physiological information, and surgical setup information, and each candidate action set includes parameterized surgical activities within the respective simulated environment. (29) A computer-readable medium that, when executed by a computer, includes instructions causing the computer to perform the method described in Embodiment 1.

Claims

1. A device for determining a target action set, wherein the device is The processor comprises, The process involves receiving multiple environmental parameter sets from a surgical data system, where each environmental parameter set corresponds to a simulated environment in which a simulated surgical task is to be completed. The method involves generating multiple candidate action sets for each environmental parameter set, wherein each candidate action set, when executed within the respective simulated environment corresponding to the corresponding environmental set, performs the respective simulated surgical task with the effectiveness indicated by the corresponding effectiveness metric. Training a machine learning model using a plurality of environmental parameter sets, a plurality of candidate action sets corresponding to each of the environmental parameter sets, and a corresponding effectiveness metric, such that the trained machine learning model may include a functional relationship between the model input and the model output, wherein the model input includes an input environmental parameter set, and the output includes the plurality of candidate action sets corresponding to the input environmental parameter set and the output effectiveness metric corresponding to the plurality of candidate action sets. A device configured to determine a target action set based on a target environment parameter set and an effectiveness metric corresponding to the multiple candidate action sets output by the trained machine learning model.

2. The device according to claim 1, wherein the surgical data system is configured to record real-world environmental data from the execution of the surgical task in the real world, and is further configured to generate the plurality of environmental parameter sets from the real-world environmental data.

3. The device according to claim 1, wherein each environmental parameter set includes anatomical information, physiological information, and surgical setup information.

4. The device according to claim 1, wherein each candidate action set includes a parameterized surgical activity within the respective simulated environment.

5. The device according to claim 4, wherein the parameterized activity includes simulated dissection, which is parameterized by either the relative anatomical position, the selection of the instrument, or the direction of application of the instrument.

6. The device according to claim 4, wherein the parameterized activity includes simulated suturing, which is parameterized by any of the following: suture location, suture pattern, number of sutures, suture material, and suture tension.

7. The device according to claim 4, wherein the parameterized activity includes a surgical robot guidance pathway.

8. The device according to claim 1, wherein the simulated surgical task includes a lower anterior resection anastomosis, and each candidate action set, when performed within the simulated environment indicated by the corresponding environment set, causes the simulated surgical task to be performed with the effectiveness indicated by the corresponding effectiveness metric, the corresponding effectiveness metric being an indicator of perfusion resulting from the execution of the simulated surgical task.

9. The device according to claim 1, wherein the simulated surgical task includes either a segmentectomy or a lobectomy, and each candidate action set, when performed within the simulated environment indicated by the corresponding environment set, causes the simulated surgical task to be performed with the effectiveness indicated by the corresponding effectiveness metric, the corresponding effectiveness metric being an indicator of the output lung volume resulting from the execution of the simulated surgical task.

10. The device according to claim 1, wherein the processor is further configured to produce an output which is a human-readable aggregate of the plurality of environmental parameter sets, each of the plurality of candidate action sets, and each of the corresponding effectiveness metrics.

11. The device according to claim 1, wherein the processor is configured to train the machine learning model using any of the supervised learning algorithm, the unsupervised learning algorithm, and the reinforcement learning algorithm.

12. A device for determining a target action set, wherein the device is The processor comprises, The process involves receiving multiple environmental parameter sets from a surgical data system, where each environmental parameter set corresponds to a simulated environment in which a simulated surgical task is to be completed. The method involves generating multiple candidate action sets for each environmental parameter set, wherein each candidate action set, when executed within the respective simulated environment, performs the simulated surgical task with effectiveness indicated by the corresponding effectiveness metric. Training a machine learning model using a plurality of environmental parameter sets, a plurality of candidate action sets corresponding to each of the environmental parameter sets, and a corresponding effectiveness metric, such that the trained machine learning model includes a functional relationship between the model input and the model output, wherein the model input includes an input environmental parameter set, and the output includes the plurality of candidate action sets corresponding to the input environmental parameter set and the output effectiveness metric corresponding to the plurality of candidate action sets. A device configured to determine a target action set based on a target environment parameter set and an effectiveness metric corresponding to the multiple candidate action sets output by the trained machine learning model.

13. The device according to claim 12, wherein the surgical data system is configured to record real-world environmental data from the execution of the surgical task in the real world, and is further configured to generate the plurality of environmental parameter sets from the real-world environmental data.

14. The device according to claim 12, wherein each environmental parameter set includes anatomical information, physiological information, and surgical setup information.

15. The device according to claim 12, wherein each candidate action set includes a parameterized surgical activity within the respective simulated environment.

16. A device for determining a target action set, wherein the device is The processor comprises, Receiving a machine learning model trained with a plurality of environmental parameter sets, a plurality of candidate action sets corresponding to each of the environmental parameter sets, and a corresponding effectiveness metric, such that the trained machine learning model includes a functional relationship between the model input and the model output, wherein the model input includes an input environmental parameter set, and the output includes the plurality of candidate action sets corresponding to the input environmental parameter set and the output effectiveness metric corresponding to the plurality of candidate action sets. The system is configured to determine a target action set based on the environmental parameter set and the effectiveness metrics corresponding to the multiple candidate action sets output by the machine learning model, Each environmental parameter set corresponds to the respective simulated environment in which the simulated surgical task is to be completed. The aforementioned sets of environmental parameters are from the surgical data system. For each set of environmental parameters, multiple candidate action sets, when executed within the simulated environment corresponding to the corresponding environmental set, cause the simulated surgical task to be performed with the effectiveness indicated by the corresponding effectiveness metric.

17. The device according to claim 16, wherein each environmental parameter set includes anatomical information, physiological information, and surgical setup information, and each candidate action set includes parameterized surgical activity within the respective simulated environment.

18. A computer-based method for determining a target action set, wherein the method is: This involves receiving multiple environmental parameter sets, each of which corresponds to a simulated environment in which a simulated surgical task is to be completed. The method involves generating multiple candidate action sets for each environmental parameter set, wherein each candidate action set, when executed within the respective simulated environment, performs the respective simulated surgical task with the effectiveness indicated by the corresponding effectiveness metric. Training a machine learning model using a plurality of environmental parameter sets, a plurality of candidate action sets corresponding to each of the environmental parameter sets, and a corresponding effectiveness metric, such that the trained machine learning model may include a functional relationship between the model input and the model output, wherein the model input includes an input environmental parameter set, and the output includes the plurality of candidate action sets corresponding to the input environmental parameter set and the output effectiveness metric corresponding to the plurality of candidate action sets. A method comprising determining a target action set based on a target environment parameter set and an effectiveness metric corresponding to the plurality of candidate action sets output by the trained machine learning model.

19. The computer implementation method according to claim 18, wherein receiving the plurality of environmental parameter sets includes receiving the plurality of environmental parameter sets from a surgical data system that records real-world environmental data from the execution of the surgical task in the real world and generates the plurality of environmental parameter sets from the real-world environmental data.

20. The computer-aided method according to claim 18, wherein each environmental parameter set includes anatomical information, physiological information, and surgical setup information.

21. The computer-aided method according to claim 18, wherein each candidate action set includes parameterized surgical activities within the respective simulated environments.

22. The computer-aided method according to claim 21, wherein the parameterized activity includes simulated dissection, which is parameterized by either the relative anatomical position, the selection of an instrument, or the direction of application of the instrument.

23. The computer-aided method according to claim 21, wherein the parameterized activity includes simulated suturing, which is parameterized by any of the following: suture position, suture pattern, number of sutures, suture material, and suture tension.

24. The computer-assisted method according to claim 21, wherein the parameterized activity includes a surgical robot guidance pathway.

25. The computer-aided method according to claim 18, wherein the simulated surgical task includes a lower anterior resection anastomosis, and each candidate action set is performed within the simulated environment indicated by the corresponding environment set, causing the simulated surgical task to be performed with the effectiveness indicated by the corresponding effectiveness metric, the corresponding effectiveness metric being an indicator of perfusion resulting from the execution of the simulated surgical task.

26. The computer-aided method according to claim 18, wherein the simulated surgical task includes either a segmentectomy or a lobectomy, and each candidate action set, when executed within the simulated environment indicated by the corresponding environment set, causes the simulated surgical task to be executed with the effectiveness indicated by the corresponding effectiveness metric, the corresponding effectiveness metric being an indicator of the output lung volume resulting from the execution of the simulated surgical task.

27. The computer implementation method according to claim 18, further comprising generating an output which is a human-readable aggregate of the plurality of environmental parameter sets, each of the plurality of candidate action sets, and each of the corresponding effectiveness metrics.

28. The computer implementation method according to claim 18, wherein training the machine learning model includes training the machine learning model using any of a supervised learning algorithm, an unsupervised learning algorithm, and a reinforcement learning algorithm.

29. A computer-readable medium that, when executed by a computer, includes instructions causing the computer to perform the method described in claim 18.

Images