Adjusting automated collaborative operations based on situation-derived constraints

Abstract

The systems, methods, and / or instrumentalities disclosed herein can involve adjusting an automated cooperative operation based on a situation-derived constraint. A system can be configured to receive a first data stream and / or a second data stream from a first surgical element and / or a second surgical element. The second data stream can be indicative of a second output parameter associated with a complementary task. The system can determine a relationship between the first surgical element and the second surgical element based on the first data stream and / or the second data stream. The system can determine a boundary parameter associated with the second output parameter based on the first output parameter and / or the relationship. The system can communicate an indication of the boundary parameter to the second surgical element. The system can cause the second surgical element to perform the complementary task based on the boundary parameter.

Description

Cross-references to related applications

[0001] This application relates to the following concurrently filed patent applications, the contents of each of which are incorporated herein by reference: • Case file END9638USNP1, entitled “Progresses Advancement of Authorized Level Based on Learned Complimentary Assistance”. • Case file END9638USNP3, titled “ASSISTANCE ADVANCEMENT MULTI-SYSTEM INTERACTION”. • Case file END9638USNP4, titled "MONITORING AND IDENTIFYING SURGEON CONTROL AND SUGGESTING ATASK THAT MAY BE DONE AUTONOMOUSLY". • Case file END9638USNP5, entitled “CONTROL OF INFORMATION FLOW, PRIORITIZATION AND MANIFESTATION OF DATA ASSOCIATED WITH AN ACTIVE HCP INTERACTION SPACE”. • Case file END9638USNP6, titled "ADAPTIVE RETRACTION FORCE CONTROL". • The agent's case file END9638USNP7 entitled "ADJUSTMENT OR DISPLAY OF OPTIONS OF POSITIONAL ORORIENTATION IMPLICATIONS ON SURGICAL TOOL USAGE", and • Case file END9638USNP8, titled “ADJUSTMENT OF PHYSIOLOGIC FUNCTION SUPPLEMENTATION CONTROL”.

[0002] The following are all cited and incorporated into this article: • U.S. Patent Application No. 18 / 810,323, filed on August 20, 2024, entitled “METHOD FOR MULTI-SYSTEM INTERACTION”; • U.S. Patent Application No. 18 / 960,006, filed November 26, 2024, entitled “METHOD FOR SMART SURGICAL SYSTEMS”; and • U.S. Patent Application No. 18 / 954,186, filed on November 20, 2024, entitled “METHOD FOR MULTI-SYSTEM INTERACTION”. Background Technology

[0003] Surgical procedures are typically performed in the operating room or surgical chamber of a medical facility, such as a hospital. Various surgical devices and systems are utilized in the performance of surgical procedures. In the digital and information age, due to patient safety and the general expectation of maintaining traditional practices, medical systems and facilities are often slower to adopt systems or procedures utilizing newer and more advanced technologies. Summary of the Invention

[0004] To optimize a task autonomously performed by a second intelligent device based on input from a first intelligent device controlled by a healthcare professional (HCP), a surgical computing system can be configured to adjust the automated collaborative operation (e.g., automated task) of the second intelligent device based on situation-derived constraints (e.g., boundary parameters). For example, the surgical computing system may receive a first data stream from the first intelligent device indicating a first output parameter. The output parameter may be associated with a task performed by the HCP (e.g., using the first intelligent device). The first intelligent device may be manually operated by the HCP. The surgical computing system may receive a second data stream from the second intelligent device indicating a second output parameter associated with a complementary task. The complementary task may be associated with the first task.

[0005] The surgical computing system can determine the relationship between a first intelligent device and a second intelligent device based on a first data stream and a second data stream. The surgical computing system can determine boundary parameters associated with the second intelligent device. These boundary parameters can indicate constraints and / or adjustments to the second intelligent device during complementary tasks. The surgical computing system can determine the boundary parameters based on output parameters as described herein (e.g., the first and / or second intelligent devices), the determined relationship, and / or the data streams (e.g., the first or second intelligent device). The surgical computing system can transmit the indication of the boundary parameters to the second intelligent device, causing the second intelligent device to perform complementary tasks based on the boundary parameters.

[0006] The systems, methods, and / or instruments disclosed herein may relate to adjusting automated collaborative operations based on situation-derived constraints. In an example, a system for presenting assistance to a user during surgery based on adaptive recognition of capabilities may include a processor. The system may be configured to receive a first data stream from a first surgical element. The first data stream may indicate a first output parameter associated with a first task. The system may receive a second data stream from a second surgical element. The second data stream may indicate a second output parameter associated with a complementary task. The second surgical element may be configured to perform the complementary task based on the first task. The system may determine a relationship between the first and second surgical elements based on the first and / or second data streams. This relationship may indicate that the complementary task is associated with the first task. The system may determine a boundary parameter associated with the second output parameter based on the first output parameter and the relationship. The boundary parameter may indicate adjustments to the second surgical element during the complementary task. The system may transmit the indication of the boundary parameter to the second surgical element. The system may enable the second surgical element to perform the complementary task based on the boundary parameter.

[0007] This may include one or more features. For example, the first surgical element may be a first surgical robot configured to be user-controllable. The first task may be associated with applying radiofrequency (RF) energy to the patient's tissue during tissue ablation. The second surgical element may be a second surgical robot configured to be autonomously controllable. A complementary task may be associated with maintaining tension applied to the tissue to fix it during the first task. A boundary parameter may be the energy density applied to the tissue during tissue ablation. The system may determine the tension to be applied by the second surgical robot during the application of RF energy to the tissue by the first surgical robot. The tension to be applied may be maintained by the energy density applied to the tissue.

[0008] The system can receive the patient's physiological parameters from a first data stream. The system can determine a second boundary parameter associated with a second output parameter. The system can determine if the patient's physiological parameters meet a threshold. The system can select the second boundary parameter based on the determined physiological parameter meets the threshold. The system can transmit an indication of the second boundary parameter to a second surgical element. The system can enable the second surgical element to perform a complementary task based on the boundary parameter and the second boundary parameter. A first surgical element can be configured to perform a first task based on input from a user. The second surgical element can be configured to autonomously perform a complementary task. The system can determine if the second output parameter meets a threshold. The threshold can be associated with the boundary parameter. The system can transmit an alert message to the first surgical element indicating that the output parameter exceeds the boundary parameter. The second surgical element can be configured to perform a complementary task in synchronized motion based on the movement of the first surgical element while performing the first task. The boundary parameter can be further determined based on the patient's safety risk during surgery, historical data, or quality of service (QoS). The relationship between the first and second surgical elements can be determined based on a lookup table. The second output parameter can be at least one of the following: position, force, configuration, or treatment type. The system can feed a first data stream, a second data stream, instructions for a first task, and / or instructions for a complementary task as input to a machine learning (ML) model. The system can receive boundary parameters associated with a second output parameter as a response from the ML model.

[0009] A method for presenting assistance to a user during surgery based on adaptive recognition of capabilities may include receiving a first data stream from a first surgical element. The first data stream may indicate a first output parameter associated with a first task. The method may include receiving a second data stream from a second surgical element. The second data stream may indicate a second output parameter associated with a complementary task. The second surgical element may be configured to perform the complementary task based on the first task. The method may include determining a relationship between the first and second surgical elements based on the first and second data streams. This relationship may indicate that the complementary task is associated with the first task. The method may include determining a boundary parameter associated with the second output parameter based on the first output parameter and / or the relationship. The boundary parameter may indicate adjustments to the second surgical element during the complementary task. The method may include transmitting an indication of the boundary parameter to the second surgical element. The method may include causing the second surgical element to perform the complementary task based on the boundary parameter.

[0010] In the example, the first surgical element may be a first surgical robot configured to be user-controllable, and the first task may be associated with applying radiofrequency (RF) energy to the patient's tissue during tissue ablation. The second surgical element may be a second surgical robot configured to be autonomously controllable, and the complementary task may be associated with maintaining tension applied to the tissue to fix the tissue during the first task. The boundary parameter may be the energy density applied to the tissue during tissue ablation. The method may include determining the tension to be applied by the second surgical robot during the application of RF energy to the tissue by the first surgical robot. The tension to be applied may maintain the energy density applied to the tissue.

[0011] The method may include receiving physiological parameters of a patient from a first data stream. The method may include determining a second boundary parameter associated with a second output parameter. The method may include determining that the patient's physiological parameters meet a threshold. The method may include selecting the second boundary parameter based on determining that the physiological parameters meet the threshold. The method may include transmitting an indication of the second boundary parameter to a second surgical element. The method may include causing the second surgical element to perform a complementary task based on the boundary parameter and the second boundary parameter.

[0012] A first surgical element may be configured to perform a first task based on input from a user. A second surgical element may be configured to autonomously perform a complementary task. The method may include determining that a second output parameter satisfies a threshold. The threshold may be associated with boundary parameters. The method may include transmitting an alert message to the first surgical element. The alert message may indicate that the output parameter exceeds the boundary parameters.

[0013] The second surgical element may be configured to perform a complementary task in synchronized motion based on the movement of the first surgical element while performing the first task. The method may include feeding a first data stream, a second data stream, an instruction for the first task, and / or an instruction for the complementary task as input to a machine learning (ML) model. The method may include receiving boundary parameters associated with a second output parameter as a response from the ML model.

[0014] A system for presenting assistance to a user based on adaptive recognition of capabilities during surgery can be described. The system may include a first surgical element configured to transmit a first data stream. The system may include a second surgical element configured to receive a second data stream. The system receives the first data stream. The first data stream may indicate output parameters associated with a first task and a complementary task. The system may determine a relationship between the first and second surgical elements. This relationship may indicate that the complementary task is associated with the first task. The system may determine boundary parameters associated with the second surgical element based on the output parameters and the relationship. The system may transmit indications of the boundary parameters to the second surgical element. The system may enable the second surgical element to perform the complementary task based on the boundary parameters.

[0015] The system can receive the patient's physiological parameters from a first data stream. The system can determine a second boundary parameter associated with a second surgical element. The system can determine if the patient's physiological parameters meet a threshold. The system can select the second boundary parameter based on the determined threshold. The system can transmit an indication of the second boundary parameter to the second surgical element. The system can enable the second surgical element to perform complementary tasks based on the boundary parameter and the second boundary parameter.

[0016] A first surgical element may be configured to perform a first task based on input from a user, and / or a second surgical element may be configured to autonomously perform a complementary task. The system may transmit an alert message to the first surgical element. The alert message may indicate that the output parameters have exceeded boundary parameters. The system may feed a first data stream and an instruction for the first task as input to a machine learning (ML) model. The system may receive boundary parameters as a response from the ML model. Attached Figure Description

[0017] Figure 1 This is a block diagram of a computer-implemented surgical system.

[0018] Figure 2 An example surgical system in an operating room is shown.

[0019] Figure 3 Example surgical hubs paired with various systems are shown.

[0020] Figure 4 An example situational awareness surgical system is shown.

[0021] Figure 5 Example surgical instruments are shown.

[0022] Figure 6 This is an example operating environment in which a surgical computing system can receive data streams to determine the optimal boundary parameters of surgical components.

[0023] Figure 7 This is a flowchart of an example optimization boundary parameter determination routine performed by a surgical computing system.

[0024] Figure 8A This is an example operating environment in which the surgical computing device can determine boundary parameters.

[0025] Figure 8B This is a graph illustrating example boundary parameters that can be used by surgical components to perform complementary tasks. Detailed Implementation

[0026] The operating room has become more complex with the introduction of intelligent devices (e.g., referred to interchangeably herein as "surgical elements"). These intelligent devices can be used and / or adjusted by healthcare professionals (HCPs) during surgery. Intelligent devices may include one or more advanced capabilities to significantly enhance the precision, safety, and efficiency of surgeries (e.g., surgical procedures, diagnostic procedures, therapeutic procedures, preventative procedures, etc.) while reducing the risk of patient complications. Examples of intelligent devices may include robotic surgical systems, navigation systems, intelligent imaging systems, endoscopic and / or laparoscopic systems, harmonic scalpels, anesthesia machines, patient monitoring systems (pulse oximeters, blood pressure monitors, EKG monitors, EEG monitors, etc.), energy devices (e.g., electrosurgical units, laser surgical systems, etc.), infusion pumps, etc.

[0027] Several smart devices may be connected (e.g., via a network) to generate and / or acquire data during surgery. Examples of data may include real-time and / or historical data associated with environmental data (e.g., temperature, humidity, airflow rate, differential pressure, and / or air filtration associated with the operating room), the number and location of HCPs during surgery, surgical plans (e.g., patient data associated with the surgery, HCP data, tasks, supplies, smart devices, staff workflows, etc.), the functionality of associated smart devices (e.g., sensor data, output parameters, smart device information, etc.) and / or patient data (e.g., physiological parameters, patient health data, etc.).

[0028] In the example, the first intelligent device may be controlled by the HCP, and the second intelligent device may operate autonomously during the procedure. The first and / or second intelligent devices may perform complementary tasks. Complementary tasks may be tasks associated with the first task during the procedure. In the example, complementary tasks may include sub-tasks, support tasks, contributing tasks, collaborative tasks, auxiliary tasks, supplementary tasks, and / or tasks associated with another task. Complementary tasks may be performed together with the first task to achieve a target outcome. As an illustrative example, the first task may include ablating tissue, while the complementary task may include applying tension to the tissue during HCP ablation (e.g., via an intelligent device).

[0029] Autonomously operating intelligent devices can perform complementary tasks; however, the intelligent device may not determine how best to perform the complementary task based on inputs from the HCP controlling the first intelligent device and / or on data streams generated by the first and / or second intelligent devices. For example, the (e.g., autonomously operating) second intelligent device may not provide optimized control of the energy density applied to the tissue during the ablation process based on the HCP operating the first intelligent device. Energy density can be the sum of the energy output of the ablation device, the force applied to the tissue by the device, and / or the tissue tension applied by the second robotic device. The second intelligent device may not control the output of the tensioner, resulting in inconsistent energy density at the tissue. This may lead to excessive damage to non-target tissue and / or insufficient energy to destroy the target tissue. In the example, the second intelligent device may not be configured to adjust output parameters to maintain the target energy density based on inputs received from the first intelligent device (e.g., manual operation of the first intelligent device based on the HCP).

[0030] Furthermore, an autonomously operating intelligent device may not limit or constrain its operation (e.g., task) based on input from (e.g., by HCP) a first intelligent device. For example, in the event that the HCP applies excessive pressure and / or energy to the target tissue, the second intelligent device may not determine that the maximum energy density limit has been reached based on the actions of the HCP (e.g., manual control of the first intelligent device by the HCP), may not generate an alert based on the actions of the HCP, may not perform one or more steps to pause ablation to prevent further tissue damage, and / or may not adjust output parameters to maintain the energy density at the target tissue.

[0031] To optimize a task autonomously performed by a second intelligent device based on input from a first intelligent device controlled by the HCP, the surgical computing system can be configured to adjust the automated collaborative operation (e.g., automated task) of the second intelligent device based on situation-derived constraints (e.g., boundary parameters). For example, the surgical computing system may receive a first data stream from the first intelligent device indicating a first output parameter. The output parameter may be associated with a task performed by the HCP (e.g., using the first intelligent device). The first intelligent device may be manually operated by the HCP. The surgical computing system may receive a second data stream from the second intelligent device indicating a second output parameter associated with a complementary task. The complementary task may be associated with the first task.

[0032] The surgical computing system can determine the relationship between a first intelligent device and a second intelligent device based on a first data stream and a second data stream. The surgical computing system can determine boundary parameters associated with the second intelligent device. These boundary parameters can indicate constraints and / or adjustments to the second intelligent device during complementary tasks. The surgical computing system can determine the boundary parameters based on output parameters as described herein (e.g., the first and / or second intelligent devices), the determined relationship, and / or the data streams (e.g., the first or second intelligent device). The surgical computing system can transmit the indication of the boundary parameters to the second intelligent device, causing the second intelligent device to perform complementary tasks based on the boundary parameters.

[0033] As an illustrative example, the first intelligent device may be a first surgical robot configured to be controlled by the HCP (e.g., manually operated by the HCP), with a first task associated with applying radiofrequency (RF) energy to a target tissue of the patient during tissue ablation. The second intelligent device may be a second surgical robot configured to autonomously perform a complementary task, which may be associated with maintaining tension applied to the tissue to fix the tissue during the first task. Boundary parameters may be associated with controlling the energy density applied to the target tissue during tissue ablation. A surgical computational system may determine the tension to be applied by the second surgical robot during the application of RF energy to the tissue by the first surgical robot, such that the determined tension maintains the energy density applied to the tissue. Advantageously, the second intelligent device may optimize the tissue ablation process by contributing to and / or compensating for one or more movements of the HCP to maintain optimal energy density at the target tissue site.

[0034] A more detailed understanding can be obtained by referring to the following description, which is given by way of example in conjunction with the accompanying drawings. Example aspects of surgical systems

[0035] Figure 1 An example computer-implemented surgical system 20000 is illustrated. The example surgical system 20000 may include one or more surgical systems (e.g., surgical subsystems) 20002, 20003, and 20004. For example, surgical system 20002 may include a computer-implemented interactive surgical system. For example, surgical system 20002 may include a surgical hub 20006 and / or a computing device 20016 communicating with a cloud computing system 20008, such as... Figure 2The cloud computing system 20008 may include at least one remote cloud server 20009 and at least one remote cloud storage unit 20010. Example surgical systems 20002, 20003, or 20004 may include one or more wearable sensing systems 20011, one or more environmental sensing systems 20015, one or more robotic systems 20013, one or more intelligent instruments 20014, one or more human-machine interface systems 20012, etc. The human-machine interface system is also referred to herein as a human-machine interface device. The wearable sensing system 20011 may include one or more HCP sensing systems and / or one or more patient sensing systems. The environmental sensing system 20015 may include one or more devices, for example, for measuring one or more environmental properties, such as... Figure 2 Further described. The robotic system 20013 may include multiple devices for performing surgical procedures, such as... Figure 2 Further described.

[0036] Surgical system 20002 can communicate with remote server 20009, which may be part of cloud computing system 20008. In one example, surgical system 20002 can communicate with remote server 20009 via a cable / FIOS networking node of an Internet service provider. In one example, patient sensing system can communicate directly with remote server 20009. Surgical system 20002 (and / or the various subsystems, intelligent surgical instruments, robots, sensing systems, and other computerized devices described herein) can collect data in real time and transmit the data to a cloud computer for data processing and manipulation. It should be understood that cloud computing may rely on shared computing resources rather than using local servers or personal devices to process software applications.

[0037] Surgical system 20002 and / or components thereof may communicate with remote server 20009 via a cellular transmit / receive point (TRP) or base station using one or more of the following cellular protocols: GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long Term Evolution (LTE) or 4G, LTE-Advanced (LTE-A), New Radio (NR) or 5G, and / or other wired or wireless communication protocols. Various examples of cloud-based analytics performed by cloud computing system 20008 and applicable to use with this disclosure are described in U.S. Patent Application Publication No. US 2019-0206569 A1 (U.S. Patent Application No. 16 / 209,403), filed December 4, 2018, entitled “METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB,” the entire disclosure of which is incorporated herein by reference.

[0038] The surgical hub 20006 can collaboratively interact with one of a plurality of devices displaying images from a laparoscopy and information from one or more other intelligent devices and one or more sensing systems 20011. The surgical hub 20006 can interact with one or more sensing systems 20011, one or more intelligent devices, and multiple displays. The surgical hub 20006 can be configured to collect measurement data from the sensing systems and transmit notification or control messages to the one or more sensing systems 20011. The surgical hub 20006 can transmit and / or receive information including notification information to and / or from a human-machine interface system 20012. The human-machine interface system 20012 may include one or more human-machine interface devices (HIDs). The surgical hub 20006 can transmit and / or receive notification or control information to be converted into audio, display, and / or control information for various devices communicating with the surgical hub.

[0039] For example, the sensing system may include a wearable sensing system 20011 (which may include one or more HCP sensing systems and / or one or more patient sensing systems) and / or an environmental sensing system 20015, such as Figure 1 As shown. The sensing system can measure data associated with various biomarkers. The sensing system can use one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc., to measure biomarkers. The sensor can use one or more of the following sensing techniques to measure biomarkers as described herein: photoplethysmography, electrocardiography, electroencephalography, colorimetry, impedance spectroscopy, potentiometry, current measurement, etc.

[0040] Biomarkers measured by the sensing system may include, but are not limited to, sleep, core body temperature, maximum oxygen uptake, physical activity, alcohol consumption, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, blood pH, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, cough and sneezing, gastrointestinal motility, gastrointestinal imaging, respiratory bacteria, edema, psychological factors, sweat, circulating tumor cells, autonomic tone, circadian rhythm and / or menstrual cycle.

[0041] Biomarkers can relate to physiological systems, including but not limited to behavioral and psychological systems, cardiovascular systems, renal systems, dermal systems, nervous systems, gastrointestinal systems, respiratory systems, endocrine systems, immune systems, tumors, musculoskeletal systems, and / or reproductive systems. For example, information from biomarkers can be determined and / or used by a computer-implemented patient and surgical system 20000. This information from biomarkers can be determined and / or used by the computer-implemented patient and surgical system 20000 to improve said systems and / or improve patient outcomes.

[0042] The sensing system can transmit data to the surgical hub 20006. The sensing system can communicate with the surgical hub 20006 using one or more of the following RF protocols: Bluetooth, Bluetooth Low Energy (BLE), Bluetooth Smart, Zigbee, Z-Wave, IPv6 Low Power Wireless Personal Area Network (6LoWPAN), and Wi-Fi.

[0043] The sensing system, biomarkers, and physiological system are described in more detail in U.S. Application No. 17 / 156,287 (Attorney’s File END9290USNP1), filed on January 22, 2021, entitled “METHOD OF ADJUSTING A SURGICAL PARAMETERBASED ON BIOMARKER MEASUREMENTS”, the entire disclosure of which is incorporated herein by reference.

[0044] The sensing system described herein can be used to assess the physiological condition of a surgeon performing surgery on a patient, a patient preparing for surgery, or a patient recovering after surgery. The cloud-based computing system 20008 can be used to monitor biomarkers associated with the surgeon or patient in real time, and can be used to generate surgical plans based at least on measurement data collected prior to surgery, provide control signals to surgical instruments during surgery, and notify the patient of complications during the postoperative period.

[0045] A cloud-based computing system 20008 can be used to analyze surgical data. Surgical data can be obtained via one or more intelligent instruments 20014, wearable sensing systems 20011, environmental sensing systems 20015, robotic systems 20013, etc., within the surgical system 20002. Surgical data may include tissue status to assess leakage or perfusion of sealed tissue following tissue sealing and surgical pathology data, including images of samples of body tissue, anatomical structures of the body using various sensors integrated with imaging devices, and techniques such as overlaying images captured by multiple imaging devices, image data, etc. Surgical data can be analyzed to improve surgical outcomes by determining whether further treatment can proceed (such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robotics to tissue-specific sites and conditions). Such data analysis can employ outcome analysis processing, and using standardized methods can provide beneficial feedback to validate surgical treatment and surgeon behavior, or to suggest modifications to surgical treatment and surgeon behavior.

[0046] Figure 2 An example surgical system 20002 in an operating room is shown. Figure 2 As illustrated, the patient undergoes surgery performed by one or more healthcare professionals (HCPs). The HCP is monitored by one or more HCP sensing systems 20020 worn by the HCP. The HCP and the environment surrounding the HCP may also be monitored by one or more environmental sensing systems, including, for example, a collection of cameras 20021, a collection of microphones 20022, and other sensors that can be deployed in the operating room. The HCP sensing systems 20020 and the environmental sensing systems may communicate with a surgical hub 20006, which in turn may communicate with one or more cloud servers 20009 of a cloud computing system 20008, such as... Figure 1 As shown. Environmental sensing systems can be used to measure one or more environmental properties, such as the location of the HCP in the surgical room, HCP movement, environmental noise in the surgical room, temperature / humidity in the surgical room, etc.

[0047] like Figure 2As illustrated, a main display 20023 and one or more audio output devices (e.g., speakers 20019) are positioned within a sterile area to be visible to the operator at the operating table 20024. Furthermore, a visualization / notification tower 20026 is positioned outside the sterile area. The visualization / notification tower 20026 may include a first non-sterile human-machine interface (HID) 20027 and a second non-sterile HID 20029 that are mutually exclusive. The HID may be a display or a display with a touchscreen that allows direct human-machine interface with the HID. The HID system guided by the surgical hub 20006 can be configured to utilize HIDs 20027, 20029, and 20023 to coordinate information flow to the operator both inside and outside the sterile area. In one example, the surgical hub 20006 may enable the HID (e.g., the main HID 20023) to display notifications and / or information about the patient and / or surgical procedures. In one example, the surgical hub 20006 may prompt and / or receive input from personnel in a sterile or non-sterile area. In another example, the surgical hub 20006 may enable the HID to display a snapshot of the surgical site recorded by the imaging device 20030 on a non-sterile HID 20027 or 20029, while maintaining a real-time feed of the surgical site on the main HID 20023. For example, the snapshot on the non-sterile display 20027 or 20029 may allow a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0048] The surgical hub 20006 can be configured to route diagnostic inputs or feedback entered by a non-sterile operator at the visualization tower 20026 to the main display 20023 within the sterile area, where a sterile operator at the operating table can view the diagnostic inputs or feedback. In one example, the input may be a modified form of a snapshot displayed on a non-sterile display 20027 or 20029, which can be routed to the main display 20023 via the surgical hub 20006.

[0049] See Figure 2Surgical instrument 20031 is used in surgical procedures as part of surgical system 20002. Hub 20006 can be configured to coordinate the flow of information to the display of surgical instrument 20031. For example, it is described in U.S. Patent Application Publication No. US 2019-0200844 A1 (U.S. Patent Application No. 16 / 209,385), filed December 4, 2018, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” the entire disclosure of which is incorporated herein by reference. Diagnostic inputs or feedback entered by a non-aseptic operator at visualization tower 20026 can be routed by hub 20006 to the surgical instrument display within the aseptic area, where the operator of surgical instrument 20031 can view the diagnostic inputs or feedback. For example, an example surgical instrument suitable for use with the surgical system 20002 is described in U.S. Patent Application Publication No. 2019-0200844 A1 (U.S. Patent Application No. 16 / 209,385), filed December 4, 2018, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” under the heading “Surgical Instrument Hardware,” the entire disclosure of which is incorporated herein by reference.

[0050] like Figure 2 As shown, surgical system 20002 can be used to perform surgery on a patient lying supine on operating table 20024 in operating room 20035. Robotic system 20034 can be used as part of surgical system 20002 during surgery. Robotic system 20034 may include surgeon's console 20036, patient-side trolley 20032 (surgical robot), and surgical robot hub 20033. While the surgeon views the surgical site through surgeon's console 20036, patient-side trolley 20032 can manipulate at least one removably attached surgical tool 20037 through a minimally invasive incision within the patient's body. Images of the surgical site can be obtained via medical imaging device 20030, which can be manipulated by patient-side trolley 20032 to orient the imaging device 20030. Robotic hub 20033 can be used to process images of the surgical site for subsequent display to the surgeon via surgeon's console 20036.

[0051] Other types of robotic systems can be readily adapted for use with surgical system 20002. Various examples of robotic systems and surgical tools applicable to this disclosure are described herein and in U.S. Patent Application No. US 2019-0201137 A1 (U.S. Patent Application No. 16 / 209,407), filed December 4, 2018, entitled “METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL,” the entire disclosure of which is incorporated herein by reference.

[0052] In various aspects, the imaging device 20030 may include at least one image sensor and one or more optical components. Suitable image sensors may include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0053] The optical components of the imaging device 20030 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate multiple portions of the surgical site. One or more image sensors may receive light reflected or refracted from the surgical site, including light reflected or refracted from tissue and / or surgical instruments.

[0054] Light sources can be configured to radiate electromagnetic energy in both the visible and invisible spectra. The visible spectrum (sometimes referred to as the optical spectrum or emission spectrum) is the portion of the electromagnetic spectrum that is visible to the human eye (e.g., detectable by it) and can be referred to as "visible light" or simply "light." The typical human eye responds to wavelengths in the range of approximately 380 nm to approximately 750 nm in air.

[0055] The invisible spectrum (e.g., the non-luminescent spectrum) is the portion of the electromagnetic spectrum that lies below and above the visible spectrum (i.e., wavelengths below about 380 nm and above about 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum and become invisible infrared (IR), microwave, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum and become invisible ultraviolet, X-ray, and gamma-ray electromagnetic radiation.

[0056] In various respects, the imaging device 20030 is configured to be used in minimally invasive surgery. Examples of imaging devices suitable for use in this disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cytoscopes, duodenoscopes, colonoscopes, esophagoduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-renal endoscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0057] Imaging devices can employ multispectral monitoring to distinguish morphology and underlying structures. A multispectral image is an image that captures image data across a specific wavelength range of the electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including light from frequencies outside the visible light range, such as IR and ultraviolet. Spectral imaging allows the extraction of additional information that the human eye fails to capture with its red, green, and blue receptors. The use of multispectral imaging is described in more detail under the title “Advanced Imaging Acquisition Module” in U.S. Patent Application No. US 2019-0200844 A1 (U.S. Patent Application No. 16 / 209,385), filed December 4, 2018, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE ANDDISPLAY,” the entire disclosure of which is incorporated herein by reference. After completing a surgical task to perform one or more of the previously described tests on the treated tissue, multispectral monitoring can be a useful tool for repositioning the surgical site. It goes without saying that rigorous sterilization of the operating room and surgical equipment is required during any surgical procedure. The stringent hygienic and sterilization conditions required in a “surgical room” (e.g., operating room or treatment room) necessitate the highest possible sterility for all medical devices and apparatus. Part of this sterilization process requires the sterilization of any material that comes into contact with the patient or penetrates the sterile area, including the imaging device 20030 and its attachments and components. It should be understood that a sterile area can be considered a designated area deemed free of microorganisms, such as within a tray or sterile towel, or can be considered the area surrounding the patient prepared for surgical procedures. A sterile area may include properly dressed scrubbed team members, as well as all equipment and fixtures within that area.

[0058] Figure 1 The illustrated wearable sensing system 20011 may include, for example: Figure 2One or more HCP sensing systems 20020 are shown. The HCP sensing system 20020 may include sensing systems for monitoring and detecting a set of physical and / or physiological states of an HCP. An HCP can typically be a surgeon or one or more healthcare professionals or other healthcare providers assisting the surgeon. In one example, the HCP sensing system 20020 may measure a set of biomarkers to monitor the heart rate of the HCP. In one example, the HCP sensing system 20020 worn on the surgeon's wrist (e.g., a watch or wristband) may use an accelerometer to detect hand movements and / or tremors and determine the amplitude and frequency of the tremors. The sensing system 20020 may transmit measurement data associated with the set of biomarkers and data associated with the surgeon's physical state to a surgical hub 20006 for further processing.

[0059] Figure 1 The illustrated environmental sensing system 20015 can transmit environmental information to the surgical hub 20006. For example, the environmental sensing system 20015 may include a camera 20021 for detecting the hand / body position of the HCP. The environmental sensing system 20015 may include a microphone 20022 for measuring ambient noise in the surgical room. Other environmental sensing systems 20015 may include devices such as a thermometer for measuring temperature and a hygrometer for measuring the humidity of the surrounding environment in the surgical room. Surgeon biomarkers may include one or more of the following: pressure, heart rate, etc. Environmental measurements from the surgical room may include ambient noise levels associated with the surgeon or patient, surgeon and / or staff movement, surgeon and / or staff attention levels, etc. The surgical hub 20006 (either independently or in communication with a cloud computing system) can use surgeon biomarker measurement data and / or environmental sensing information to modify the control algorithms of handheld instruments or the average latency of robot interfaces, for example, to minimize tremors.

[0060] The surgical hub 20006 can use surgeon biomarker measurements associated with HCP to adaptively control one or more surgical instruments 20031. For example, the surgical hub 20006 can transmit control programs to the surgical instrument 20031 to control its actuators to limit or compensate for fatigue and the use of fine motor skills. The surgical hub 20006 can transmit control programs based on situational awareness and / or context regarding the importance or criticality of the task. When control is needed, the control program can instruct the instrument to change its operation to provide more control.

[0061] Figure 3An example surgical system 20002 with a surgical hub 20006 is shown. The surgical hub 20006 can be paired with a wearable sensing system 20011, an environmental sensing system 20015, a human-machine interface system 20012, a robotic system 20013, and a smart instrument 20014 via modular controls. The hub 20006 includes a display 20048, an imaging module 20049, a generator module 20050 (e.g., an energy generator), a communication module 20056, a processor module 20057, a storage array 20058, and an operating room mapping module 20059. In some aspects, such as Figure 3 As illustrated, hub 20006 also includes smoke extraction module 20054 and / or suction / flushing module 20055. Various modules and systems can be connected directly to the modular control via a router or via communication module 20056. The operating room device can be coupled to cloud computing resources and data storage devices via the modular control. Human-machine interface system 20012 may include a display subsystem and a notification subsystem.

[0062] Modular controls can be coupled to a non-contact sensor module. The non-contact sensor module can use ultrasonic, laser-based, and / or similar non-contact measuring devices to measure the size of the operating room and generate a mapping of the surgical room. Other distance sensors can be used to determine the boundaries of the operating room. An ultrasonic-based non-contact sensor module can scan the operating room by emitting a burst of ultrasound and receiving the echo as it bounces back from the walls of the operating room, as described under the title “Surgical Hub Spatial Awareness Within an Operating Room” in U.S. Provisional Patent Application Serial No. 62 / 611,341, filed December 28, 2017, entitled “INTERACTIVE SURGICAL PLATFORM”, the entire contents of which are incorporated herein by reference. The sensor module can be configured to determine the size of the operating room and adjust Bluetooth pairing distance limits. A laser-based non-contact sensor module can scan the operating room by emitting laser pulses, receiving laser pulses bouncing back from the walls of the operating room, and comparing the phase of the emitted pulse with the received pulse to determine the size of the operating room and adjust Bluetooth pairing distance limits.

[0063] During surgery, the application of energy to tissue for sealing and / or cutting can be associated with fumigation, aspiration of excess fluid, and / or tissue flushing. Fluid lines, power lines, and / or data lines from different sources can become entangled during surgery. Resolving this issue during surgery wastes valuable time. Disconnecting lines may require disconnecting them from their respective modules, which may necessitate module resets. The Hub Modular Housing 20060 provides a unified environment for managing power lines, data lines, and fluid lines, reducing the frequency of entanglement between such lines.

[0064] Energy can be applied to tissue at a surgical site. The surgical hub 20006 may include a hub housing 20060 and a combined generator module slidably received in a docking base within the hub housing 20060. The docking base includes data contacts and power contacts. The combined generator module may include two or more of an ultrasonic energy generator component, a bipolar RF energy generator component, or a monopolar RF energy generator component housed in a single unit. The combined generator module may include a smoke extraction component, at least one energy delivery cable for connecting the combined generator module to a surgical instrument, at least one smoke extraction component configured to extract smoke, fluid, and / or particles generated by applying therapeutic energy to tissue, and a fluid line extending from a remote surgical site to the smoke extraction component. The fluid line may be a first fluid line, and a second fluid line may extend from a remote surgical site to a suction and flushing module 20055 slidably housed in the hub housing 20060. The hub housing 20060 may include a fluid interface.

[0065] The combined generator module can generate multiple energy types for application to tissue. One energy type may be more advantageous for cutting tissue, while another different energy type may be more advantageous for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Aspects of this disclosure present a solution in which the hub modular housing 20060 is configured to accommodate different generators and facilitate interactive communication between them. The hub modular housing 20060 allows for the rapid removal and / or replacement of various modules.

[0066] The modular surgical housing may include: a first energy generator module configured to generate a first energy for application to tissue; and a first docking base including a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and wherein the first energy generator module is slidably movable to no longer electrically engage with the first power and data contacts. The modular surgical housing may also include: a second energy generator module configured to generate a second energy, different from the first energy, for application to tissue; and a second docking base including a second docking port including second data and power contacts, wherein the second energy generator module is slidably movable to electrically engage with the power and data contacts, and wherein the second energy generator module is slidably movable to no longer electrically engage with the second power and data contacts. Furthermore, the modular surgical housing also includes a communication bus between the first and second docking ports, configured to facilitate communication between the first and second energy generator modules.

[0067] See Figure 3 The hub modular housing 20060 allows for modular integration of the generator module 20050, the smoke extraction module 20054, and the suction / flushing module 20055. The hub modular housing 20060 facilitates interactive communication between modules 20059, 20054, and 20055. The generator module 20050 may have integrated monopolar, bipolar, and ultrasonic components supported in a single housing unit slidably inserted into the hub modular housing 20060. The generator module 20050 can be connected to the monopolar device 20051, the bipolar device 20052, and the ultrasonic device 20053. The generator module 20050 may include a series of monopolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact through the hub modular housing 20060. The hub modular housing 20060 facilitates the insertion of multiple generators and interactive communication between generators connected to the hub modular housing 20060, allowing the generator to function as a single generator.

[0068] A surgical data network with a set of communication hubs can connect sensing systems and modular devices located in one or more operating rooms, patient recovery rooms, or rooms in a medical facility specifically equipped for surgical procedures to a cloud computing system 20008.

[0069] Figure 4A diagram illustrating a situation-aware surgical system 5100 is provided. Data source 5126 may include, for example, a modular device 5102, a database 5122 (e.g., an EMR database containing patient records), a patient monitoring device 5124 (e.g., a blood pressure (BP) monitor and an electrocardiogram (EKG) monitor), an HCP monitoring device 35510, and / or an environmental monitoring device 35512. Modular device 5102 may include sensors configured to detect parameters associated with the patient, HCP, and environment, and / or the modular device itself. Modular device 5102 may include one or more intelligent instruments 20014. Surgical hub 5104 may derive surgical context information from the data, for example, based on a specific combination of received data or a specific order in which data is received from data source 5126. The context information inferred from the received data may include, for example, the type of surgical procedure being performed, the specific steps of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity of the surgical object. The ability of the surgical hub 5104 to derive or infer surgical-related information from received data can be termed "situational awareness." For example, the surgical hub 5104 may incorporate a situational awareness system, which could be hardware and / or surgical planning associated with the surgical hub 5104 that derives surgical-related background information from received data, and / or surgical planning information received from edge computing system 35514 or enterprise cloud server 35516. Background information derived from data source 5126 may include, for example, the steps of the surgical procedure being performed, whether and how a specific modular device 5102 is being used, and the patient's condition.

[0070] Surgical hub 5104 can connect to various databases 5122 to retrieve data from them regarding surgical procedures being performed or to be performed. In one example of surgical system 5100, database 5122 may include a hospital's EMR database. Data that can be received from database 5122 by the situational awareness system of surgical hub 5104 may include, for example, start (or setup) time or operational information about a procedure (e.g., a segmental resection in the upper right thoracic region). Surgical hub 5104 can derive background information about the surgical procedure from this data alone or from this data in combination with data from other data sources 5126.

[0071] The surgical hub 5104 can be connected to (e.g., paired with) various patient monitoring devices 5124. In one example of the surgical system 5100, the patient monitoring devices 5124 that can be paired with the surgical hub 5104 may include a pulse oximeter (SpO2 monitor) 5114, a BP monitor 5116, and an EKG monitor 5120. Perioperative data that can be received by the situational awareness system of the surgical hub 5104 from the patient monitoring devices 5124 may include, for example, the patient's oxygen saturation, blood pressure, heart rate, and other physiological parameters. Background information that can be derived by the surgical hub 5104 from the perioperative data sent by the patient monitoring devices 5124 may include, for example, whether the patient is in the operating room or under anesthesia. The surgical hub 5104 may derive these inferences individually from data from the patient monitoring devices 5124 or in combination with data from other data sources 5126 (e.g., a ventilator 5118).

[0072] The surgical hub 5104 can be connected to (e.g., paired with) various modular devices 5102. In one example of the surgical system 5100, the modular device 5102 paired with the surgical hub 5104 may include a fume extractor, medical imaging devices (such as...) Figure 2 The imaging device 20030 shown includes an inhaler, a combined energy generator (for providing power to ultrasound surgical instruments and / or RF electrosurgical instruments), and a ventilator.

[0073] Perioperative data received by the surgical hub 5104 from the medical imaging device may include, for example, whether the medical imaging device is activated and video or image feeds. Background information derived by the surgical hub 5104 from the perioperative data transmitted by the medical imaging device may include, for example, whether the surgery is a VATS procedure (based on whether the medical imaging device is activated or paired with the surgical hub 5104 at the start of the surgery or during the procedure). Image or video data (or a data stream representing video for a digital medical imaging device) from the medical imaging device may be processed by a pattern recognition system or a machine learning system to, for example, identify features (e.g., organ or tissue type) in the field of view (FOY) of the medical imaging device. Background information derived by the surgical hub 5104 from the identified features may include, for example, the type of surgical procedure (or its steps) being performed, the organ being operated on, or the body cavity in which the operation is being performed.

[0074] The situational awareness system of the surgical hub 5104 can derive contextual information from data received from the data source 5126 in a variety of different ways. For example, the situational awareness system may include a pattern recognition system or a machine learning system (e.g., an artificial neural network) trained on training data to associate various inputs (e.g., data from the database 5122, patient monitoring device 5124, modular device 5102, HCP monitoring device 35510, and / or environmental monitoring device 35512) with corresponding contextual information about the surgical procedure. For example, the machine learning system can accurately derive contextual information about the surgical procedure from the provided inputs. In an example, the situational awareness system may include a lookup table that stores pre-represented environmental information about the surgical procedure associated with one or more inputs (or ranges of inputs) corresponding to environmental information. In response to a query using one or more inputs, the lookup table can return corresponding contextual information used by the situational awareness system to control the modular device 5102. In the example, the contextual information received by the situational awareness system of the surgical hub 5104 may be associated with a specific control adjustment or a set of control adjustments for one or more modular devices 5102. In the example, the situational awareness system may include a machine learning system, lookup table, or other such system that can generate or retrieve one or more control adjustments for one or more modular devices 5102 when provided with contextual information as input.

[0075] For example, based on data source 5126, the situational-aware surgical hub 5104 can determine the type of tissue being operated on. The situational-aware surgical hub 5104 can infer whether the surgery being performed is thoracic or abdominal, thus allowing the surgical hub 5104 to determine whether the tissue held by the end effector of the surgical suture and cutting instruments is lung tissue (for thoracic surgery) or stomach tissue (for abdominal surgery). The situational-aware surgical hub 5104 can determine whether the surgical site is under pressure (by determining that the surgery is utilizing airflow) and determine the type of surgery to achieve a consistent amount of smoke extraction for both thoracic and abdominal surgeries. Based on data source 5126, the situational-aware surgical hub 5104 can determine which step of the surgery is being performed or will be performed subsequently.

[0076] The situation-aware surgical hub 5104 can determine the type of surgical procedure being performed and customize energy levels based on the expected tissue profile of the procedure. The situation-aware surgical hub 5104 can adjust the energy levels of ultrasonic surgical instruments or RF electrosurgical instruments throughout the entire surgical procedure, rather than just on a per-procedure basis.

[0077] In the example, data can be extracted from an additional data source 5126 to improve the conclusions drawn by the surgical hub 5104 from one data source 5126. The situation-aware surgical hub 5104 can supplement the data received from the modular device 5102 with the background information about the surgery that it has built from other data sources 5126.

[0078] The situational awareness system of the surgical hub 5104 can take physiological measurement data into account to provide additional contextual information when analyzing visualization data. This additional context can be useful when the visualization data itself may be uncertain or incomplete.

[0079] The situational awareness surgical hub 5104 can determine whether a surgeon (or other HCP) is making an error or otherwise deviating from the intended procedure during a surgical operation. For example, the surgical hub 5104 can determine the type of surgery being performed, retrieve a corresponding list of steps or the order of equipment use (e.g., from memory), and compare the steps being performed or the equipment being used during the surgical procedure with the expected steps or equipment determined by the surgical hub 5104 for that type of surgery. The surgical hub 5104 can provide alerts indicating that a specific step in the surgical procedure is performing an unexpected action or utilizing an unexpected device.

[0080] Surgical instruments (and other modular devices 5102) can be adapted to the specific context of each surgical procedure (such as adaptation to different tissue types) and verification actions during the surgical procedure. Subsequent steps, data, and display adjustments can be provided to the surgical instruments (and other modular devices 5102) in the operating room according to the specific context of the surgery.

[0081] Figure 5An example surgical system 20280 is illustrated, which may include a surgical instrument 20282. The surgical instrument 20282 may communicate with a console 20294 and / or a portable device 20296 via a wired and / or wireless connection through a local area network 20292 and / or a cloud network 20293. The console 20294 and the portable device 20296 may be any suitable computing device. The surgical instrument 20282 may include a handle 20297, an adapter 20285, and a loading unit 20287. The adapter 20285 is releasably coupled to the handle 20297, and the loading unit 20287 is releasably coupled to the adapter 20285, such that the adapter 20285 transmits force from a drive shaft to the loading unit 20287. The adapter 20285 or the loading unit 20287 may include a force gauge (not explicitly shown) disposed therein to measure the force applied to the loading unit 20287. The loading unit 20287 may include an end effector 20289 having a first jaw 20291 and a second jaw 20290. The loading unit 20287 may be an in-situ loading or multiple-fire loading unit (MFLU), which allows clinicians to fire multiple fasteners multiple times without removing the loading unit 20287 from the surgical site to reload it.

[0082] The first jaw 20291 and the second jaw 20290 may be configured to clamp tissue therebetween, fire a fastener through the clamped tissue, and cut the clamped tissue. The first jaw 20291 may be configured to fire at least one fastener multiple times, or may be configured to include a replaceable multiple-fire fastener cartridge containing multiple fasteners (e.g., nails, clamps, etc.) that can be fired more than once before being replaced. The second jaw 20290 may include an anvil that deforms or otherwise secures the fastener when it is ejected from the multiple-fire fastener cartridge.

[0083] The handle 20297 may include a motor coupled to a drive shaft to influence its rotation. The handle 20297 may include a control interface for selectively activating the motor. The control interface may include buttons, switches, levers, sliders, touchscreens, and any other suitable input mechanisms or user interfaces that can be engaged by a clinician to activate the motor.

[0084] The control interface of the handle 20297 can communicate with the controller 20298 of the handle 20297 to selectively activate the motor to affect the rotation of the drive shaft. The controller 20298 may be located within the handle 20297 and is configured to receive input from the control interface and adapter data from the adapter 20285 or loading unit data from the loading unit 20287. The controller 20298 can analyze the input from the control interface and the data received from the adapter 20285 and / or the loading unit 20287 to selectively activate the motor. The handle 20297 may also include a display that a clinician can view during use of the handle 20297. The display may be configured to show portions of the adapter or loading unit data before, during, or after the firing instrument 20282.

[0085] Adapter 20285 may include an adapter identification device 20284 disposed therein, and loading unit 20287 may include a loading unit identification device 20288 disposed therein. Adapter identification device 20284 may communicate with controller 20298, and loading unit identification device 20288 may communicate with controller 20298. It should be understood that loading unit identification device 20288 may communicate with adapter identification device 20284, and the adapter identification device relays or transmits communications from loading unit identification device 20288 to controller 20298.

[0086] The adapter 20285 may also include a plurality of sensors 20286 (one shown) disposed around it to detect various conditions of the adapter 20285 or the environment (e.g., whether the adapter 20285 is connected to the loading unit, whether the adapter 20285 is connected to the handle, whether the drive shaft rotates, the torque of the drive shaft, the strain of the drive shaft, the temperature within the adapter 20285, the number of times the adapter 20285 is fired, the peak force of the adapter 20285 during firing, the total force applied to the adapter 20285, the peak retraction force of the adapter 20285, the number of pauses of the adapter 20285 during firing, etc.). The plurality of sensors 20286 may provide input to the adapter identification device 20284 in the form of data signals. The data signals of the plurality of sensors 20286 may be stored in the adapter identification device 20284 or may be used to update adapter data stored in the adapter identification device. The data signals of the plurality of sensors 20286 may be analog or digital. Multiple sensors 20286 may include force gauges to measure the force applied to the loading unit 20287 during firing.

[0087] The handle 20297 and adapter 20285 can be configured to interconnect the adapter identification device 20284 and the loading unit identification device 20288 with the controller 20298 via an electrical interface. The electrical interface can be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals therebetween). Additionally or alternatively, the electrical interface can be a contactless electrical interface for wirelessly transmitting energy and signals therebetween (e.g., inductive transmission). It is also conceivable that the adapter identification device 20284 and the controller 20298 can wirelessly communicate with each other via a wireless connection separate from the electrical interface.

[0088] The handle 20297 may include a transceiver 20283 configured to transmit instrument data from the controller 20298 to other components of the system 20280 (e.g., LAN 20292, cloud 20293, console 20294, or portable device 20296). The controller 20298 may also transmit instrument data and / or measurement data associated with one or more sensors 20286 to the surgical hub. The transceiver 20283 may receive data (e.g., pod data, loading unit data, adapter data, or other notifications) from the surgical hub 20270. The transceiver 20283 may also receive data (e.g., pod data, loading unit data, or adapter data) from other components of the system 20280. For example, controller 20298 can send instrument data to console 20294, including the serial number of the attachment adapter (e.g., adapter 20285) attached to handle 20297, the serial number of the loading unit (e.g., loading unit 20287) attached to adapter 20285, and the serial number of the multi-fire fastener cartridge loaded onto the loading unit. Console 20294 can then send data associated with the attached cartridge, loading unit, and adapter (e.g., cartridge data, loading unit data, or adapter data), respectively, back to controller 20298. Controller 20298 can display the message on a local instrument display or send the message via transceiver 20283 to console 20294 or portable device 20296 for display on display 20295 or portable device screen, respectively. Example aspects of surgical computing systems in operating environments

[0089] Figure 6This is an example operating environment 56300. Operating environment 56300 may include a patient 56302, surgical elements 56310 and 56320, a system 56330, a human medical computing (HCP) 56308, and a network 56301. As part of operating environment 56300, surgical computing system 56330 may receive data streams from surgical elements 56310 and 56320 via network 56301. System 56330 may receive data streams during surgery on patient 56302 (where HCP 56308 performs a first task by operating a first surgical element 56310 and / or where a second surgical element 56320 autonomously performs a second task (e.g., a complementary task)) and determine boundary parameters to optimize the surgery. System 56330 may transmit the boundary parameters to second surgical element 56320 to enable second surgical element 56320 to optimize the second task associated with the surgery.

[0090] The operating environment 56300 may include a surgical computing system 56330 (referred to as "system 56330" below). Among other features, system 56330 may receive data streams from surgical elements 56310, 56320 and / or user input from HCP 56308 via network 56301.

[0091] System 56330 may include a data flow analyzer 56332, a historical data repository 56334, and / or an ML model trainer 56336. In the example, data flow analyzer 56332 may generate training data based on historical data received from historical data repository 56334. Data flow analyzer 56332 may feed the training data to ML model trainer 56336, whereby ML model trainer may train one or more ML models to generate boundary parameters as described herein, and so on.

[0092] The dataflow analyzer 56332 can be a processor, microprocessor, microcontroller, field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), system-on-chip (SOIC), digital signal processing (DSP) platform, real-time computing system, etc. For example, the processor can be configured to implement computing functions and / or modules as disclosed herein.

[0093] The data stream analyzer 56332 can send and / or receive data streams from surgical elements 56310 and 56320 (e.g., during surgery). The data streams may include data generated and / or associated with the system 56330, surgical elements 56310 and 56320, patient 56302, and / or HCP 56308. For example, the data streams may include boundary parameters as described herein, output parameters (e.g., for controlling outputs 56315 and 56325), data generated by sensors 56313 and 56323, data generated by the system 56330, training data, data stream configuration information, data received from the HCP 56308, and / or historical data.

[0094] In the example, boundary parameters may indicate adjustments and / or constraints on the second intelligent device during complementary tasks. In the example, boundary parameters may indicate adjustments to a setpoint or constraints on the range of motion (e.g., of a robotic system). In the example, boundary parameters may indicate the level of automation (e.g., the number of tasks for which surgery can be automated by the second surgical element 56320). In the example, boundary parameters may indicate a mapping path that the second (e.g., autonomous) surgical element 56320 can traverse. The mapping path may be determined based on HCP 56308 while the first surgical element 56310 is being manually operated.

[0095] Output parameters can control the output of surgical elements 56310 and 56320. In the example, output parameters may indicate (e.g., setpoints for control variables associated with motor speed, temperature, flow rate, etc.), current and / or voltage associated with electrosurgical tools, commands for generating images from cameras, etc., and / or power levels (e.g., heating pads, laser ablation tools, etc.). As an illustrative example, output parameters may include setpoints for the speed of cutting tools, voltage and / or current setpoints for electrosurgical tools, flow rates of infusion pumps, and positions of robotic arms, etc.

[0096] In the example, the data generated by sensors 56313 and 56323 may include signals indicative of one or more physiological parameters of the patient (e.g., oxygen saturation, blood pressure, respiratory rate, blood glucose, heart rate, core body temperature, hydration status, etc.), environmental characteristics of the operating room (e.g., room temperature, number of HCPs during surgery, location of HCPs during surgery, humidity level, air quality, lighting level, CO2 level, etc.), and / or one or more characteristics of surgical components 56310 and 56320 (e.g., current, voltage, pressure, applied force, distance, vibration, orientation, flow rate, state, etc.). As an illustrative example, the HVAC system may include temperature, humidity, and / or air quality sensors. In the example, the robotic surgical system may include force and / or pressure sensors, depth sensors, temperature sensors, blood flow and / or oxygen sensors, pH sensors (e.g., for measuring blood gases), electrocardiogram (ECG) sensors, position sensors (e.g., for measuring the position of the robotic arm), etc.

[0097] In the example, the data generated by system 56330 may include surgical plans, morbidity data associated with one or more patients, the skill level of HCP 56308, preferred tools used by HCP 56308, tasks, supplies, and whether surgical elements 56310, 56320 are available for surgical and / or staff workflows (e.g., the number of HCP 56308 and / or surgical elements 56310, 56320 required for the task). The data generated by system 56330 may include indications of relationships (e.g., between the first surgical element 56310 and the second surgical element 56320), indications of tasks that can be automated and / or complementary tasks, thresholds associated with patient physiological parameters (e.g., life-threatening thresholds such as core body temperature limits, SpO2 limits, etc.) and / or thresholds associated with boundary parameters (e.g., areas outside the mapping paths instructed to be traversed by surgical elements 56310, 56320).

[0098] In the example, training data may include indications of boundary parameters, output parameters, data generated by sensors 56313 and 56323, data generated by system 56330, data received by HCP 56308, historical data, and / or data flow configuration information for training the ML model. Data flow analyzer 56332 may generate training data and / or receive training data from surgical elements 56310 and 56320. Data flow analyzer 56332 may transmit the training data to ML model trainer 56336 to determine relationships between surgical elements 56310 and 56320 and / or determine boundary parameters as described herein, etc.

[0099] In the example, data stream configuration information may indicate the surgical element ID, data stream availability, measurement unit, communication protocol (RS-232, Ethernet, TCP / IP, Bluetooth, etc.), scheduling and / or frequency information (e.g., transmission time and / or frequency of sensor data), destination (e.g., port information associated with surgical element controllers 56311, 56321, and / or system 56330), and / or security and / or access control credentials. In the example, the data stream configuration information can be used to configure system 56330 to receive output parameters, boundary parameters, sensor data, etc., from the first surgical element 56310 and / or the second surgical element 56320.

[0100] In the example, historical data may include past information associated with the surgery. For example, historical data may include data streams (e.g., boundary parameters, output parameters, data generated by sensors 56313 and 56323, data generated by system 56330, training data, data stream configuration information, data received by HCP 56308, etc.). Data analyzer 65332 can use historical data to determine relationships between surgical elements 56310 and 56320, and so on.

[0101] In the example, data received from HCP 56308 may include indications of surgery, tasks, complementary tasks, relationships, thresholds (e.g., associated with boundary parameters, life-threatening thresholds, etc.), instructions to automate tasks, etc. Data flow analyzer 56332 may receive user input from HCP 56308 and / or generate output to the HCP (e.g., via user interfaces 56317, 56327). In the example, data flow analyzer 56332 may transmit indications for alerts to HCP 56308. Alerts may be generated in response to determining that output parameters meet thresholds (e.g., associated with boundary parameters and / or physiological parameters of patient 56302). In the example, data flow analyzer 56332 may receive user input from HCP 56308, including indications of relationships between surgical elements 56310 and 56320, surgery, tasks, complementary tasks, etc. (e.g., tasks to be performed autonomously by surgical elements 56310, 56320).

[0102] The data flow analyzer 56332 can determine complementary tasks that can be automated, and then allow a user (e.g., HCP 56308 via user interface 56317) to choose whether to automate those tasks. In the example, HCP 56308 can choose to manually perform the indicated task and / or request system 56330 to obtain additional data (e.g., via data flow) based on the manual execution of the task. In the example, system 56330 can receive confirmation that the first task has been successfully automated before determining the second task to be automated. As an illustrative example, system 56330 can perform stitching multiple times to develop a basic level of capability in the stitching (e.g., as determined by HCP 56308). Before determining the second automated task, system 56330 can instruct corresponding elements for arm positioning, etc.

[0103] System 56330 may provide a level of automation associated with supporting HCP 56308 during surgery based on the surgeon's perspective (e.g., a defined number of tasks that may be automated by one or more surgical elements 56310, 56320, as indicated in the automation assistance parameters). For example, for opening and closing the patient 56302, the robotic system (e.g., surgical element 56320) may provide a lower level of automation. However, for intraoperative laparoscopic anatomy, the robotic system may provide a higher level of automation (e.g., system 56330 may determine a second automation assistance parameter based on one or more tasks associated with the surgery), and then for suturing actions, the robotic system may provide a lower level of support.

[0104] In the example, automation assist parameters can indicate instructions associated with routine and / or complex tasks. For instance, in suturing, automation assist parameters can indicate simple suturing scenarios as well as more complex suturing scenarios based on tissue, anatomical pathways, and other constraints (e.g., based on operational data).

[0105] Data stream analyzer 56332 can determine and / or infer one or more relationships (e.g., determine relationship chains) between components of operating environment 56300. In an example, data stream analyzer 56332 can determine relationships between surgical elements 56310, 56320. Relationships can be determined and / or inferred based on, for example, compatible sensors (e.g., sensors 56313, 56323) between surgical elements 56310, 56320, determined data streams that may be needed to resolve issues during surgery, and / or based on user input (e.g., by HCP 56308 via user interfaces 56317, 56327). As an illustrative example, data stream analyzer 56332 can receive a first data stream (e.g., a data stream indicating the temperature of an inflatable cavity measured via sensor 56313) from a laparoscopic tool and a second data stream (e.g., a data stream indicating the temperature near the inflatable cavity measured via sensor 56323) from a heating pad. Data stream analyzer 56332 can determine whether two data streams are related based on configuration information, sensor data, and / or output parameters associated with each data stream (e.g., a change detected by the first temperature sensor of the second surgical element 56320 may be related to the response and / or change of the output parameters of the first surgical element 56310). In the example, sensor data may be related if, for example, sensor data from the first surgical element 56310 and the second surgical element 56320 share similar units of measurement, measure similar physiological parameters of the patient's body, are close to each other during surgery, etc. After data stream analyzer 56332 determines that one or more aspects of surgical elements 56310 and 56320 are related, data stream analyzer 56332 may store indications of the relationship in, for example, a historical data repository 56334 (e.g., in a lookup table (LUT) that defines one or more relationships). Additionally and / or alternatively, data stream analyzer 56332 provides training data (e.g., including historical data) to ML model trainer 56336 to train an ML model based on one or more relationships.

[0106] In the example, data stream analyzer 56332 can determine presence relationships based on data stream configuration information of surgical elements 56310 and 56320 (e.g., based on surgical element ID, communication protocol, scheduling and frequency information, destination, security and access control credentials, and / or measurement unit). As an illustrative example, data stream analyzer 56332 can receive data streams from laparoscopic tools (e.g., data streams indicating the temperature of the inflated cavity measured via sensor 56313) and data streams from a heating pad (e.g., data streams indicating the temperature near the inflated cavity measured via sensor 56323 and / or the power level associated with heat generated by the heating pad (e.g., via output 56325)). Data stream analyzer 56332 can determine that a temperature increase, as measured by the laparoscopic tool, is caused by the power output associated with the heating pad. Data stream analyzer 56332 can indicate that the temperature sensor associated with the laparoscopic tool can be used to control the power output of the heating pad, and so on.

[0107] The data stream analyzer 56332 can determine boundary parameters associated with output parameters (e.g., those of surgical elements 56310, 56320). Boundary parameters can indicate spatial boundaries, forces, velocity and / or acceleration, proximity, range of motion, tissue type, image, user control limitations, etc. As an illustrative example, boundary parameters (e.g., spatial boundary parameters) can enable the second surgical element 56320 to maintain a safe distance to avoid interference or collision with the first surgical element 56310. Boundary parameters (e.g., force boundary parameters) can limit excessive pressure and / or strain on tissue to prevent tissue damage and / or maintain target energy density (e.g., during mobile ablation procedures). Velocity and / or acceleration boundaries can limit the speed at which the surgical device can move (e.g., limiting output parameters such as motor speed when moving through high-risk areas). Proximity boundary parameters can limit movement to prevent unintentional contact with the patient 56302, surgical instruments, HCP 56308, etc. Tissue type boundary parameters can limit automated tasks (e.g., complementary tasks) based on tissue characteristics (e.g., tissue density, elasticity, highly vascularized tissue, and / or tissue fragility).

[0108] The data flow analyzer 56332 can determine a second boundary parameter based on a first boundary parameter. As an illustrative example, the data flow analyzer 56332 can determine a tissue type boundary parameter (e.g., a first boundary parameter associated with tissue density, elasticity, high vascularity, or tissue fragility). The tissue type boundary parameter can be used to determine the second boundary parameter. The second boundary parameter can be a spatial boundary parameter and / or a velocity boundary parameter (e.g., to prevent unintentional contact or reduce the likelihood of tissue damage). As an example, the data flow analyzer 56332 can use image boundary parameters to adjust the spatial boundary parameter and / or adjust the range of motion (e.g., to limit catheter movement, etc.).

[0109] In the example, boundary parameters may be associated with the operational envelope of the second surgical element 56320. The boundary parameters may indicate adjustments to the operational envelope of the second surgical element 56310 based on actions associated with the first surgical element (e.g., movement of the first surgical element 56320 when controlled by user input from HCP 56308 via user interfaces 56317, 56327).

[0110] In the example, boundary parameters may include a safety envelope. The data flow analyzer 56332 may determine the safety envelope of the patient 56302 based on data flow, historical data, and / or user input from the HCP 56308. In the example, boundary parameters may be determined based on the patient's safety risk during surgery, historical data, or Quality of Service (QoS). For example, the data flow analyzer 56332 may determine, based on the determined boundary parameters, that tasks and / or complementary tasks can be performed during a first time period (e.g., tasks and / or complementary tasks can be performed within a time period associated with the determined safety envelope). As an illustrative example, the data flow analyzer 56332 may determine a blood pressure range. Tasks and / or complementary tasks can be performed when the patient's blood pressure is within a blood pressure range (e.g., a safety envelope). In the example, for a patient prone to hypothermia, the data flow analyzer 56332 may determine a safety envelope that can maintain body temperature within a certain range during the task and / or complementary task. In the example, if a safety envelope threshold is met (e.g., the parameter falls outside the range, etc.), the data flow analyzer 56332 can determine the boundary parameters for reducing actions associated with complementary tasks, increasing the level of automation to fully automate the task, stopping complementary tasks, and / or warning HCP 56308 (e.g., via user interfaces 56317, 56327).

[0111] The data flow analyzer 56332 can determine boundary parameters based on a complementary task. Based on a first task performed by the first surgical element 56310 (e.g., during surgery), a complementary task can be performed by the second surgical element 56320. As described herein, a complementary task can be a subtask, supporting task, contributing task, collaborative task, auxiliary task, supplementary task, and / or task related to another task. The complementary task can be performed together with the first task to achieve a target outcome. As an illustrative example, the first task may include applying energy to ablate tissue (e.g., via the first surgical element 56310), while the complementary task may include applying tension to the tissue during energy application (e.g., via the autonomously operated second surgical element 56320) to achieve a target energy density. In this example, boundary parameters may include instructions to perform the complementary task in synchronized motion based on the movement of the first surgical element 56310 (e.g., when the first surgical element 56310 performs the first task). For example, the boundary parameters can instruct the second surgical element 56320 to change the amount of force applied to the tissue relative to the amount of force applied by the HCP 56308 (e.g., via the first surgical element 56310) and / or relative to the energy output determined by the HCP 56308, in order to achieve a target energy density.

[0112] The data stream analyzer 56332 can determine boundary parameters based on the patient's physiological parameters (e.g., received from surgical elements 56310, 56320). For example, the data stream analyzer 56332 can determine boundary parameters based on perfusion level, electrophysiological signals, blood pressure, heart rate, respiratory rate, tissue temperature, SpO2, CO2 levels, electroencephalogram (EEG) readings, etc. As an illustrative example, the data stream analyzer 56332 can receive a data stream including an indication of tissue temperature and determine boundary parameters that limit the movement of the second surgical element 56320 during ablation (e.g., boundary parameters that limit robot movement to the tissue if the tissue temperature exceeds a threshold). As a second illustrative example, if the patient's SpO2 level meets a threshold, the data stream analyzer 56332 can determine boundary parameters that limit robot movement to the patient's lungs and / or airway.

[0113] Data stream analyzer 56332 can send alerts to surgical elements 56310 and 56320. Data stream analyzer 56332 can generate an alert if at least one of a physiological parameter, a boundary parameter, or an output parameter meets a threshold. In an example, if HCP 56308 (e.g., via the first surgical element 56310) performs an action that causes the output parameter (e.g., of the first surgical element 56310 or the second surgical element 56320) to meet a threshold, data stream analyzer 56332 can generate an alert. As an illustrative example, if HCP 56308 applies excessive energy during a mobile ablation procedure, causing the second surgical element 56320 to be unable to move and / or unable to apply sufficient force to the tissue to maintain the target energy density (e.g., due to boundary parameters limiting the speed of the second surgical element 56320 to prevent tissue damage), data stream analyzer 56332 can generate an alert.

[0114] The historical data repository 56334 may store and / or include data associated with one or more past surgeries. The historical data repository 56334 may store data generated by surgical elements 56310, 56320, HCP 56308 (e.g., via user interfaces 56317, 56327) and / or by system 56330.

[0115] In the example, the historical data repository 56334 may store boundary parameters, output parameters (e.g., for controlling outputs 56315, 56325), data generated by sensors 56313, 56323, data generated by system 56330, training data, data flow configuration information, data received by HCP 56308, and / or historical data. The historical data repository 56334 may store ML models (e.g., generated by ML model trainer 56336). In the example, the historical data repository 56334 may store data used by data flow analyzer 56332 to determine one or more relationships associated with surgical elements 56310, 56320. In the example, the data stored in the historical data repository 56334 may include LUTs indicating one or more relationships between surgical elements 56310, 56320. As an illustrative example, the historical data repository 56334 may store information associated with (e.g., based on data flow configuration information) identifying relationships between data flows, such as surgical element IDs, communication protocols, scheduling and frequency information, destinations, security and access control credentials, units of measurement, etc.

[0116] System 56330 may include an ML model trainer 56336. The ML model trainer 56336 may receive training data from a data stream analyzer 56332. The training data may include information associated with the data stream (e.g., boundary parameters, output parameters, data generated by sensors 56313 and 56323, data generated by system 56330, data received by HCP 56308, historical data, and / or data stream configuration information). In response to receiving the training data, the ML model trainer 56336 may train an ML model to determine the relationships between surgical elements 56310 and 56320, and / or determine boundary parameters as described herein, etc.

[0117] ML model trainer 56336 can train an ML model to determine a chain of relationships (e.g., relations) associated with the first surgical element 56310 and / or the second surgical element 56320. The chain of relationships can be determined in real time (e.g., during surgery) or at another time (e.g., based on data stored in a historical data repository 56334 as described herein). Relationships can be determined based on one or more aspects of the surgical elements 56310, 56320 (e.g., information associated with data flows, historical data, configuration information, etc.). ML model trainer 56336 can store indications in the historical data repository 56334 that one or more aspects of the surgical elements 56310, 56320 are related (e.g., via a LUT).

[0118] In the example, ML model trainer 56336 can train an ML model to determine a relationship between sensor data associated with the first surgical element 56310 (e.g., sensor 56313) and sensor data associated with the second surgical element 56320 (e.g., sensor 56323). As described herein, sensor data can be correlated if, for example, the sensor data shares similar units of measurement, the sensor data measures similar physiological parameters of the patient's body, the sensors are close to each other during surgery, etc.

[0119] In the example, ML model trainer 55363 can train an ML model to determine a relationship between sensor data associated with the first surgical element 56310 (e.g., sensor 56313) and / or output parameters generated for the second surgical element 56320 (e.g., output parameters for adjusting output 56325). If, for example, sensor 56313 measures output characteristics associated with the second surgical element 56320 (as described herein), then the sensor data associated with the first surgical element 56310 (e.g., sensor 56313) can be correlated with the output parameters generated for the second surgical element 56320.

[0120] In the example, the ML model trainer 56336 can train an ML model to determine presence relationships based on data flow configuration information of surgical elements 56310 and 56320 (e.g., based on surgical element ID, communication protocol, scheduling and frequency information, destination, security and access control credentials, and / or measurement units, etc.). As an illustrative example, the data flow analyzer 56332 can send a first data flow from the laparoscopic tool (e.g., a data flow indicating the temperature of the inflated cavity measured via sensor 56313) and a second data flow from the heating pad (e.g., a data flow indicating the temperature near the inflated cavity measured via sensor 56323 and / or the power level associated with the heat generated by the heating pad (e.g., via output 56315)). The ML model trainer 56336 can determine that a temperature increase, as measured by the laparoscopic tool, is caused by the power output associated with the heating pad. The ML model trained by the ML model trainer 56336 can indicate to the data stream analyzer 56332 that the temperature sensor associated with the laparoscopic tool can be used to control the power output of the heating pad (e.g., the relationship between the first surgical element 56310 and the second surgical element 56320).

[0121] The ML model trainer 56336 can train ML models to determine boundary parameters. As described herein, boundary parameters can indicate spatial boundaries, forces, velocities and / or accelerations, proximity, range of motion, tissue type, image, user-controlled constraints, etc. The ML model trainer 56336 can train ML models to determine one or more boundary parameters based on data streams, historical data, complementary tasks, patient physiological parameters, first boundary parameters, etc.

[0122] As an illustrative example, the ML model trainer 56336 can train an ML model to determine tissue type boundary parameters (e.g., associated with tissue density, elasticity, high vascularity, or tissue fragility) and spatial boundary parameters and / or velocity boundary parameters (e.g., to prevent unintentional contact or reduce the likelihood of tissue damage). The ML model can receive images as input and adjust the spatial boundary parameters and / or adjust the range of motion (e.g., to limit catheter movement, etc.).

[0123] The ML model trainer 56336 can train an ML model to determine the operational envelope of the second surgical element 56320. The ML model can output boundary parameters that indicate adjustments to the operational envelope of the second surgical element 56320 based on inputs including actions associated with the first surgical element 56310 (e.g., movement of the first surgical element 56310 when controlled by user input from the HCP 56308 via user interfaces 56317, 56327).

[0124] The ML model trainer 56336 can train an ML model to determine boundary parameters associated with a safety envelope. For example, the ML model can be trained to determine, based on the determined boundary parameters, that a task and / or complementary task can be performed during a first time period (e.g., the task and / or complementary task can be performed within a time period associated with the determined safety envelope). As an illustrative example, the ML model trainer 56336 can train an ML model to determine a blood pressure range (e.g., a safety envelope). The task and / or complementary task can be performed when the patient's blood pressure is within the blood pressure range (e.g., the safety envelope determined by the ML model). In the example, for a patient prone to hypothermia, the ML model trainer 56336 can train an ML model to determine a safety envelope that maintains body temperature within a certain range during the task and / or complementary task. In the example, the ML model can be trained to determine boundary parameters for reducing actions associated with the complementary task, increasing the level of automation to fully automate the task, and / or stopping the complementary task and warning HCP 56308.

[0125] Operating environment 56300 may include surgical elements 56310 and 56320. Although two surgical elements are included as part of operating environment 56300, this is not intended to be limiting, as multiple surgical elements may be included and / or communicate with one or more components of operating environment 56300. Surgical element 56310 may include surgical element controller 56311, sensor 56313, output 56315, and / or user interface 56317. Surgical element 56320 may include surgical element controller 56321, sensor 56323, output 56325, and / or user interface 56327. One or more components of surgical elements 56310 and 56320 may be referenced. Figure 5 The features described in 20282 are similar to and / or identical.

[0126] Surgical elements 56310 and 56320 may communicate with patient 56302 (e.g., during surgery via sensors 56313, 56323 and / or outputs 56315 and 56325). As an illustrative example, during open-heart surgery, a ventilator (e.g., surgical element 56310) and / or a pulse oximeter (e.g., surgical element 56320) may be attached to the patient. HCP 56308 and / or system 56330 may interact with one or more surgical elements 56310 and 56320 to monitor patient 56302's oxygen saturation (e.g., via pulse oximeter) and / or determine the ventilator's flow rate. In the example, HCP 56308 can interact with system 56330 and / or surgical elements 56310 and 56320 via user interfaces 56317 and 56327 (e.g., graphical user interface (GUI), knob, button, smart device, wearable electronic device, etc.).

[0127] Surgical element 56310 may be operated by HCP 56308 and / or autonomously operated (e.g., to perform complementary tasks associated with surgery). As an illustrative example, surgical element 56310 may include robotic surgical systems, navigation systems, intelligent imaging systems, endoscopic and / or laparoscopic systems, harmonic scalpels, anesthesia machines, patient monitoring systems (pulse oximeters, blood pressure monitors, EKG monitors, EEG monitors, etc.), energy devices (e.g., electrosurgical units, laser surgical systems, etc.), infusion pumps, etc.

[0128] In the example, surgical element 56310 may be wirelessly and / or physically connected (e.g., wired) to network 56301, system 56330, and / or another surgical element 56320. Surgical element 56310 may transmit / receive data streams from system 56330 (e.g., to determine boundary parameters). For example, surgical element 56310 may transmit / receive data streams including boundary parameters, output parameters (e.g., for controlling outputs 56315, 56325), data generated by sensors 56313, 56323, data generated by system 56330, training data, data stream configuration information, data received by HCP 56308, etc.

[0129] As an illustrative example, surgical element 56310 may be a heating blanket. The heating blanket may include a temperature measuring device (e.g., sensor 56313) and / or a heating element (e.g., output 56315). The heating blanket may be configured to activate the heating element based on user input (e.g., from HCP 56308) and / or based on boundary parameters to maintain the patient's core body temperature during open-heart surgery. The heating blanket may transmit core body temperature measurements (e.g., as part of a data stream) to system 56330. A second surgical element 56320 may receive boundary parameters (e.g., based on determined relationships and / or based on output parameters of surgical elements 56310, 56320). Boundary parameters may include setpoints and / or limitations for complementary tasks, such as the output power level of the heating blanket, the duration of maintaining the power level, indications that the heating blanket will operate automatically, etc. Advantageously, the heating blanket can receive instructions to automatically control the patient's core body temperature based on the determination of the task performed by the first surgical element 56310, so that the HCP56308 can eliminate human error and / or optimize one or more tasks during surgery.

[0130] Sensor 56313 can be any instrument, transducer, etc., configured to measure one or more environmental conditions. In the example, sensor 56313 can measure physiological parameters of patient 56302, such as oxygen saturation, blood pressure, respiratory rate, blood glucose, heart rate, core body temperature, and / or hydration status. Sensor 56313 can measure environmental conditions of the operating room (e.g., operating environment 56300), such as room temperature, the number of HCPs 56308 present during surgery, the location of HCPs 56308 during surgery, humidity level, air quality, lighting level, CO2 level, etc. Sensor 56313 can measure one or more environmental conditions associated with surgical element 56310, such as current, voltage, pressure, applied force, distance, vibration, orientation, flow rate, state, etc. As an illustrative example, an HVAC system may include temperature, humidity, and / or air quality sensors, and a robotic surgical system may include force and / or pressure sensors, depth sensors, temperature sensors, blood flow and / or oxygen sensors, pH sensors (e.g., for measuring blood gases), electrocardiogram (ECG) sensors, position sensors (e.g., for measuring the position of a robotic arm), and so on.

[0131] Sensor 56313 can transmit sensor data (e.g., information associated with measured environmental conditions) to surgical element controller 56311. Sensor data can be transmitted based on data stream configuration information. As described herein, data stream configuration information may include an indication of the surgical element ID, an indication of data stream availability, updates and / or acknowledgments of the data stream configuration information, units of measurement, communication protocols (RS-232, Ethernet, TCP / IP, Bluetooth, etc.), scheduling and / or frequency information (e.g., the time and / or frequency of sensor data transmission), destination (e.g., port information associated with surgical element controllers 56311, 56321, and / or system 56330), and / or security and / or access control credentials.

[0132] Output 56315 can be any number of devices and / or instruments that generate output characteristics based on received signals (e.g., based on output parameters) as part of surgical element 56310. In the example, output 56315 may include motors and / or actuators (e.g., as part of robotic devices, cutting tools, ventilators, etc.), lighting output devices, GUIs, cameras (e.g., as part of endoscopes, IR cameras, ultrasound cameras, etc.), pumps (e.g., as part of fumigation devices, aspiration devices, anesthesia delivery devices, IV infusion devices, etc.), and / or lasers (e.g., as part of tissue ablation devices), etc.

[0133] Output 56315 may receive data from surgical element controller 56311. In this example, output 56315 may adjust output characteristics, such as the speed of the cutting tool, the power intensity of the electrosurgical tool, the flow rate of the infusion pump, the position of the robotic arm, etc., based on the data received from surgical element controller 56311. Although one output 56315 is included in surgical element 56310, it should be understood that multiple outputs may exist, each receiving data and generating output characteristics. As an illustrative example, a robotic system may include multiple outputs 56315 that generate any number of outputs, including positioning the robot, performing measurements, capturing images, making incisions, etc.

[0134] Surgical element 56310 may include a user interface 56317. User interface services 56213 may include buttons, switches, levers, sliders, touchscreens, GUIs, user devices, etc., to enable surgical element 56310 to interact with a user (e.g., HCP 56308). User interface 56317 may display information received from surgical element controller 56311 and / or from system 56330. User interface 56317 may generate a GUI to display data from one or more components of operating environment 56300. In this example, user interface 56317 may display data associated with a data flow (e.g., patient physiological parameters), indications of relationships, boundary parameters, output parameters, tasks, complementary tasks, etc. User interface 56317 may request user input from HCP 56308. In the example, the user interface 56317 may display requests for information associated with the surgical element 56310 (e.g., identification, function, capability, output, input, etc.), and / or requests associated with tasks (e.g., approving determined boundary parameters, modifying boundary parameters, selecting tasks and / or complementary tasks, confirming relationships, etc.).

[0135] Surgical element 56310 may include surgical element controller 56311. Surgical element controller 56311 may receive sensor data from sensor 56313. Surgical element controller 56311 may send data to system 56330, output 56315, and / or user interface 56317 (including and / or indicating output parameters). Surgical element controller 56311 may communicate data streams to / from system 56330 (e.g., sensor data, boundary parameters, output parameters, data stream configuration information, etc.). In an example, surgical element controller 56311 may generate and / or transmit a first data stream and / or a second data stream to / from system 56330. In an example, surgical element controller 56311 may receive boundary parameters as described herein as part of the data stream. Surgical element controller 56311 may perform a task (e.g., a complementary task) based on the received boundary parameters.

[0136] It should be understood that surgical element 56320 and / or one or more associated components (e.g., sensor 56323, output 56325, user interface 56327, and / or surgical element controller 56321) may include interactions, features, functions, etc., that are the same as and / or similar to those described herein with reference to surgical element 56310 and / or one or more components of surgical element 56310 (e.g., sensor 56313, output 56315, user interface 56317, and / or surgical element controller 56311). Surgical element 56320 may be a second surgical element (e.g., separate from surgical element 56310). In the example, communication between one or more components as described with reference to surgical element 56310 may be the same as and / or similar to that between one or more components of surgical element 56320. As an illustrative example, surgical element 56320 may transmit and / or receive data streams to / from system 56330 and / or surgical element 56310, including sensor data, data stream configuration information, output parameters, etc.

[0137] Operating environment 56300 may include network 56301. Network 56301 may include one or more communication networks, such as the Internet. Network 56301 may be any combination of a local area network (“LAN”) and / or a wireless local area network (“WAN”). Therefore, various components of operating environment 56300 (including system 56330, surgical elements 56310, 56320 and / or HCP 56308 (e.g., via user equipment)) may communicate directly or indirectly with each other via any suitable communication link and / or network (such as network 56301 (e.g., one or more communication links, one or more computer networks, one or more wired or wireless connections, the Internet, etc.)). Example routines for determining boundary parameters for optimization

[0138] Figure 7 Example routine 56370 for determining boundary parameters for optimization is illustrated. Example routine 56370 can be executed by system 56330 (e.g., data flow analyzer 56332) of operating environment 56300. Example routine 56370 begins at box 56371.

[0139] At box 56371, data stream analyzer 56332 may receive data streams from the first surgical element and / or the second surgical element 56310, 56320. As described herein, data stream analyzer 56332 may receive data streams including data generated and / or associated with system 56330, surgical element 56310, 56320, patient 56302, and / or HCP 56308. For example, the data stream may include boundary parameters, output parameters (e.g., for controlling outputs 56315, 56325), data generated by sensors 56313, 56323, data generated by system 56330, training data, data stream configuration information, data received by HCP 56308, and / or historical data. In this example, boundary parameters may indicate adjustments and / or constraints associated with the second intelligent device during complementary tasks. Output parameters may indicate control variable setpoints, current and / or voltage, command and / or power levels. The data generated by sensors 56313 and 56323 may include signals indicating one or more physiological parameters of the patient, environmental characteristics of the operating room, and / or one or more characteristics of surgical components 56310 and 56320.

[0140] In the example, the data generated by system 56330 may include surgical plans, morbidity data associated with one or more patients, the skill level of HCP 56308, preferred tools, tasks, supplies used by HCP 56308, whether surgical elements 56310 and 56320 are available for surgery, staff workflows, etc. The data generated by system 56330 may include indications of the relationship between the first surgical element 56310 and the second surgical element 56320, indications of tasks that can be automated and / or complementary tasks, and / or thresholds associated with patient physiological parameters, thresholds associated with boundary parameters, etc.

[0141] In the example, training data may include indications of boundary parameters, output parameters, data generated by sensors 56313 and 56323, data generated by system 56330, data received by HCP 56308, historical data, and / or data flow configuration information for training the ML model. In the example, data flow configuration information may include indications of surgical element IDs, indications of data flow availability, units of measurement, communication protocols as described herein, scheduling and / or frequency information, destination and / or security and / or access control credentials. In the example, historical data may include past information associated with surgery. For example, historical data may include data flows. Data analyzer 65332 may use historical data to determine relationships between surgical elements 56310 and 56320 as described herein, and so on.

[0142] In the example, data received from HCP 56308 may include indications of surgery, tasks, complementary tasks, relationships, thresholds, instructions to automate tasks, etc. Data stream analyzer 56332 may receive user input from HCP 56308 and / or generate output to the HCP (e.g., via data streams transmitted to user interfaces 56317, 56327). In the example, data stream analyzer 56332 may receive sensor data from first surgical element 56310 and / or second surgical element 56320 and transmit indications of alerts to HCP 56308 (e.g., via data streams).

[0143] At box 56372, data flow analyzer 56332 may determine a complementary task. As described herein, a complementary task may be associated with a first task during surgery. In examples, a complementary task may be a subtask, supporting task, contributing task, collaborative task, auxiliary task, supplementary task, and / or a task related to another task. The complementary task may be performed together with the first task to achieve a target outcome. As an illustrative example, the first task may include applying energy to ablate tissue (e.g., via a first surgical element 56310). Data flow analyzer 56332 may determine that the complementary task includes applying tension to the tissue while the first surgical element 56310 applies energy to the tissue (e.g., via a autonomously operating second surgical element 56320) to achieve a target energy density.

[0144] At box 56373, data flow analyzer 56332 determines the relationship between first surgical element 56310 and second surgical element 56320. As described herein, data flow analyzer 56332 may receive user input from HCP 56308, including surgery, tasks, complementary tasks, etc. (e.g., to be performed autonomously by surgical elements 56310, 56320). The relationship may be determined and / or inferred based on, for example, compatible sensors between surgical elements 56310, 56320 (e.g., sensors 56313, 56323), the determined data flow that may be needed to resolve issues during surgery, and / or based on user input (e.g., by HCP 56308 via user interfaces 56317, 56327). As an illustrative example, data stream analyzer 56332 may receive a first data stream (e.g., a data stream indicating the temperature of an inflatable cavity measured via sensor 56313) from a laparoscopic tool and a second data stream (e.g., a data stream indicating the temperature near the inflatable cavity measured via sensor 56323) from a heating pad. Data stream analyzer 56332 may determine that two data streams are related based on configuration information, sensor data, and / or output parameters associated with each data stream (e.g., a change detected by the first temperature sensor of the second surgical element 56320 may be related to a response and / or change in the output parameters of the first surgical element 56310). In this example, sensor data may be related if, for example, the sensor data from the first surgical element 56310 and the second surgical element 56320 share similar units of measurement, measure similar physiological parameters of the patient's body, are close to each other during surgery, etc.

[0145] After the data flow analyzer 56332 determines that one or more aspects of surgical elements 56310, 56320 are related, the data flow analyzer 56332 may store indications of the relationships in, for example, a historical data repository 56334 (e.g., in a LUT that defines one or more relationships). Additionally and / or alternatively, the data flow analyzer 56332 provides training data (e.g., including historical data) to the ML model trainer 56336 to train the ML model based on one or more relationships.

[0146] At box 56374, data flow analyzer 56332 determines boundary parameters associated with the output parameters. As described herein, boundary parameters may indicate spatial boundaries, forces, velocity and / or acceleration, proximity, range of motion, tissue type, image, user-controlled limitations, etc. As an illustrative example, spatial boundary parameters may enable the second surgical element 56320 to maintain a safe distance to avoid interference or collision with the first surgical element 56310. Boundary parameters (e.g., force boundary parameters) may limit excessive pressure and / or strain on tissue to prevent tissue damage or maintain target energy density (e.g., during mobile ablation procedures). Velocity and / or acceleration boundaries may limit the speed at which the surgical device can move (e.g., limiting output parameters such as motor speed when moving through high-risk areas). Proximity boundaries may limit movement to prevent unintentional contact with the patient 56302, surgical instruments, HCP 56308, etc. Tissue type boundary parameters may limit movement based on tissue characteristics (e.g., based on tissue density, elasticity, high vascularity, and / or tissue fragility).

[0147] The data flow analyzer 56332 can determine a second boundary parameter based on a first boundary parameter. As an illustrative example, the data flow analyzer 56332 can determine tissue type boundary parameters (e.g., the data flow analyzer 56332 can limit movements associated with a complementary task based on determined tissue density, elasticity, high vascularity, or tissue fragility). Tissue type boundary parameters can be used to determine spatial boundary parameters and / or velocity boundary parameters (e.g., to prevent unintentional contact or reduce the likelihood of tissue damage). The data flow analyzer 56332 can use image boundary parameters to adjust spatial boundary parameters and / or adjust the range of motion (e.g., to limit catheter movement, etc.).

[0148] In the example, boundary parameters may be associated with the operational envelope of the second surgical element 56320. The boundary parameters may indicate adjustments to the operational envelope of the second surgical element 56320 based on actions associated with the first surgical element 56310.

[0149] In the example, boundary parameters may include a safety envelope. The data flow analyzer 56332 may determine the safety envelope of the patient 56302 based on data flow, historical data, and / or user input from the HCP 56308. For example, the data flow analyzer 56332 may determine, based on the determined boundary parameters, that the task and / or complementary task can be performed during a first time period (e.g., the task and / or complementary task can be performed within a time period associated with the determined safety envelope). As an illustrative example, the data flow analyzer 56332 may determine a blood pressure range. The task and / or complementary task can be performed when the patient's blood pressure is within the blood pressure range (e.g., within the safety envelope). In the example, for a patient prone to hypothermia, the data flow analyzer 56332 may determine a safety envelope that can maintain body temperature within a certain range during the task and / or complementary task. In the example, if the safety envelope threshold is met (e.g., the parameter falls outside the range, etc.), the data flow analyzer 56332 can determine the boundary parameters for reducing actions associated with complementary tasks, increasing the level of automation to fully automate the task, and / or stopping complementary tasks and / or warning HCP56308.

[0150] The data flow analyzer 56332 can determine boundary parameters based on a complementary task. The complementary task may be associated with a first task during surgery. In examples, the complementary task may be a subtask, supporting task, contributing task, collaborative task, auxiliary task, supplementary task, and / or a task related to another task. The complementary task may be performed together with the first task to achieve a target outcome. As an illustrative example, the first task may include applying energy to ablate tissue (e.g., via a first surgical element 56310), while the complementary task may include applying tension to the tissue during energy application (e.g., via a autonomously operating second surgical element 56320) to achieve a target energy density. In examples, boundary parameters may include instructions to perform the complementary task in synchronized motion based on the movement of the first surgical element 56310 (e.g., when the first surgical element 56310 performs the first task). For example, the boundary parameters can instruct the second surgical element 56320 to change the amount of force applied to the tissue relative to the amount of force applied by the HCP 56308 (e.g., via the first surgical element 56310) and / or relative to the energy output determined by the HCP 56308, in order to achieve a target energy density.

[0151] The data stream analyzer 56332 can determine boundary parameters based on the patient's physiological parameters (e.g., received from surgical elements 56310, 56320). For example, the data stream analyzer 56332 can determine boundary parameters based on perfusion level, electrophysiological signals, blood pressure, heart rate, respiratory rate, tissue temperature, SpO2, CO2 levels, electroencephalogram (EEG) readings, etc. As an illustrative example, the data stream analyzer 56332 can receive a data stream including an indication of tissue temperature and determine boundary parameters that limit the movement of the second surgical element 56320 during ablation (e.g., if the tissue temperature exceeds a threshold, the boundary parameters can limit the robot's movement towards the tissue to reduce the tissue temperature). As a second illustrative example, if the patient's SpO2 level meets a threshold, the data stream analyzer 56332 can determine boundary parameters that limit the robot's movement towards the patient's lungs and / or airway.

[0152] At box 56375, data stream analyzer 56332 can transmit indications of boundary parameters to second surgical element 56320. Indications of boundary parameters can be transmitted as part of, for example, a data stream between second surgical element 56320 and data stream analyzer 56332.

[0153] At block 56376, dataflow analyzer 56332 enables second surgical element 56320 to perform a complementary task. The complementary task can be performed based on boundary parameters. In an example, boundary parameters may include instructions to perform the complementary task in synchronized motion based on the movement of first surgical element 56310 (e.g., when first surgical element 56310 performs a first task based on user input). For example, boundary parameters may instruct second surgical element 56320 to vary the amount of force applied to tissue relative to the amount of force applied by HCP 56308 (e.g., via first surgical element 56310) and based on the energy output determined by HCP 56308 to achieve a target energy density. After dataflow analyzer 56332 enables second surgical element 56320 to perform the complementary task, routine 56370 may terminate.

[0154] The exemplary examples described herein can provide (e.g., bounded, fully autonomous operation of a first system (e.g., a first surgical element) that collaborates with the operation of another system (e.g., a second surgical element) directly controlled by a surgeon.

[0155] The procedure may include cardiac ablation and remodeling to treat AFIB. An automated RF catheter motion control system (e.g., a second surgical element controlled by the surgeon) can ablate predefined tissue areas at predefined depths, pressures, or power intensity levels. The surgeon can then define the area to be treated and / or provide the desired ablation depth after cardiac mapping.

[0156] The system (e.g., system 56330) can display a mapped path of catheter movement and receive confirmation that a robot (e.g., a first surgical element) sweeps the catheter through the path, thereby maintaining pressure and / or energy balance while compensating for cardiac movement during the process. In the example, automated definition (e.g., defining the surgeon's movement) enables the intelligent system (e.g., 56330) to actively interact with and support the surgeon-controlled instrument movement, while including some restrictions on the movement to prevent collateral damage or unintentional collisions with nearby structures caused by cooperative or antagonistic actions or forces (e.g., system 56330 can determine automated auxiliary parameters to define the robot's movement requested by the surgeon).

[0157] To create sufficient contact and / or pressure for the mobile ablation system to induce cauterization and / or ablation (e.g., depending on pressure, power, and energy mode), a user (e.g., a surgeon) can control the ablation electrodes. Using the ablation electrodes, the user can define the electrode's path and / or thrust capability (e.g., to create the required energy density, resulting in sufficient tissue welding or tissue death that an auxiliary system (e.g., a robot) would have to provide) to maintain the required energy density. In the example, the heart can be a sensitive and interconnected structure that may be easily damaged unintentionally if too much force or too much differential motion is applied. The robot can monitor surgical images of adjacent structures from system 56330 and / or can monitor its own applied forces relative to forces applied by the user. The sum of forces fixed relative to the heart to surrounding anatomical structures can be limited to the applicable forces and movements (e.g., the robot can apply more and / or less force based on the surgeon's movement of the electrodes to maintain contact during tissue modification, resulting in less and / or more automation in the operating environment 56300, depending on the surgeon's actions). Example aspects related to determining the boundary parameters of surgery

[0158] Figure 8A Example operating environment 56380, wherein as a mobile ablation system, surgical element 56381 can be operated by HCP 56308 and / or surgical element 56382 can be operated autonomously.

[0159] Surgical element 56381 may include features similar to and / or identical to those of surgical element 56310 of operating environment 56300. For example, a console may be similar to the user interface 56317 of operating environment 56300. Surgical element 56381 and / or 56382 may include one or more components and / or features associated with a surgical computing system (e.g., a system 56330 similar to operating environment 56300). Operating environment 56380 may include boundary parameters 56383 determined by surgical element 56381 and / or surgical element 56382.

[0160] As shown in the figure, surgical element 56381 includes a robotic system manually operated by HCP 56308. Surgical element 56381 includes electrodes defining a path for ablation of tissue and / or means for generating thrust. The electrodes and / or thrust can be used to generate a target energy density to produce sufficient tissue welding or tissue death.

[0161] Surgical element 56382 can operate autonomously (e.g., as an auxiliary system to surgical element 56381). Surgical element 56382 can perform complementary tasks in response to one or more inputs from HCP 56308 via surgical element 56381. As shown, surgical element 56382 can provide reactive and / or additive complementary forces or displacements (e.g., via boundary parameters 56383) to maintain a target energy density (e.g., based on determined boundary parameters). The total force that can be applied by the second surgical element 56382 can be limited based on boundary parameters 56383 (e.g., the limit can be indicated by solid lines and arrows). The total amount of force that can be applied by surgical element 56382 can be greater (e.g., as shown by dashed lines terminating at the two endpoints).

[0162] The determined boundary parameters may include instructions to apply force and / or tension to tissue relative to the force applied by HCP 56308 via surgical element 56381 (e.g., via surgical element 56382) (e.g., an automated robotic system may perform complementary tasks with synchronized motion based on a robotic system controlled by HCP). Because the heart is a sensitive and interconnected structure, it may be prone to unintentional injury if too much force or too much differential motion is applied. Therefore, surgical element 56382 may, for example, monitor surgical images of adjacent structures, and / or the force applied by surgical element 56382 relative to the force applied by HCP 56308 (e.g., via surgical element 56381), and / or the sum of fixed forces (e.g., those of surgical elements 56381 and 56382) relative to the heart to surrounding anatomical structures, to limit the total applied force and / or limit the total motion (e.g., to achieve a target energy density during mobile ablation).

[0163] Figure 8B This is an example of what can be determined by surgical elements 56381 and / or 56382 to perform. Figure 8AThe figure 56390 shows an example of boundary parameter curves for complementary tasks. As shown, the system (e.g., as part of surgical elements 56381 and / or 56382) determines a target energy density as a function of time, location, etc., during a mobile ablation procedure. The target energy density may vary based on one or more factors, such as the type of tissue being ablated (e.g., tissue characteristics), the location of the tissue being ablated, the speed of the energy device (e.g., output parameters), changes in the contact angle of the energy device, energy output, etc. Boundary parameters may be determined based on the characteristics of the first surgical element 56381 and / or the second surgical element 56382. For example, boundary parameters may indicate and / or limit forces, tension, pressure, energy output, energy device angle, speed, etc.

[0164] As an example, the target energy density can be achieved based on the sum of the determined forces. Boundary parameters can indicate a limit (e.g., a set point) on the forces that can be applied by (e.g., autonomously operated) the second surgical element 56382 based on the forces applied by (e.g., operated via input from HCP 56308) the first surgical element 56381. Although graph 56390 is described as having example boundary parameters related to force, this does not imply a limitation, as multiple inputs (e.g., force, tension, pressure, energy output, energy device angle, velocity, etc.) can be considered to determine the boundary parameters to achieve the target energy density (e.g., to achieve synchronized movement of the second surgical element 56382 based on the action of the first surgical element 56381).

Claims

1. A system for presenting assistance to a user during surgery based on adaptive recognition of ability, the system comprising: Processor, the processor being configured to: Receive a first data stream from a first surgical element, wherein the first data stream indicates a first output parameter associated with a first task; A second data stream is received from a second surgical element, wherein the second data stream indicates a second output parameter associated with a complementary task, and wherein the second surgical element is configured to perform the complementary task based on the first task; The relationship between the first surgical element and the second surgical element is determined based on the first data stream and the second data stream, wherein the relationship indicates that the complementary task is associated with the first task; A boundary parameter associated with the second output parameter is determined based on the first output parameter and the relationship, wherein the boundary parameter indicates the adjustment of the second surgical element during the complementary task; The indication of the boundary parameters is transmitted to the second surgical element; and The second surgical element performs the complementary task based on the boundary parameters.

2. The system according to claim 1, in, The first surgical element is a first surgical robot configured to be controllable by a user; The first task is associated with applying radiofrequency (RF) energy to the patient's tissue during tissue ablation; The second surgical element is a second surgical robot configured to be autonomously controlled. The complementary task is associated with maintaining tension applied to the tissue to fix the tissue during the first task; Wherein, the boundary parameter is the energy density applied to the tissue during tissue ablation; and The processor is further configured to: The tension to be applied by the second surgical robot during the application of RF energy to the tissue by the first surgical robot is determined, wherein the tension to be applied maintains the energy density applied to the tissue.

3. The system according to claim 1, wherein, The processor is also configured to: Receive the patient's physiological parameters from the first data stream; Determine the second boundary parameter associated with the second output parameter; Determine that the patient's physiological parameters meet the threshold; The second boundary parameter is selected based on determining that the physiological parameter satisfies the threshold. The indication of the second boundary parameter is transmitted to the second surgical element; as well as The second surgical element performs the complementary task based on the boundary parameter and the second boundary parameter.

4. The system according to claim 1, wherein, The first surgical element is configured to perform the first task based on input from the user, and the second surgical element is configured to perform the complementary task autonomously.

5. The system according to claim 1, wherein, The processor is also configured to: Determine that the second output parameter satisfies a threshold, wherein the threshold is associated with the boundary parameter; and A warning message is transmitted to the first surgical element, wherein the warning message indicates that the output parameter exceeds the boundary parameter.

6. The system according to claim 1, wherein, The second surgical element is configured to perform the complementary task in a synchronized motion based on the movement of the first surgical element while performing the first task.

7. The system according to claim 1, wherein, The relationship between the first surgical element and the second surgical element is determined based on a lookup table.

8. The system according to claim 1, wherein, The processor is also configured to: The first data stream, the second data stream, the instruction for the first task, and the instruction for the complementary task are fed as input to the machine learning (ML) model; and The boundary parameters associated with the second output parameters are received as a response from the ML model.

9. A method for presenting assistance to a user during surgery based on adaptive recognition of ability, the method comprising: Receive a first data stream from a first surgical element, wherein the first data stream indicates a first output parameter associated with a first task; A second data stream is received from a second surgical element, wherein the second data stream indicates a second output parameter associated with a complementary task, and wherein the second surgical element is configured to perform the complementary task based on the first task; The relationship between the first surgical element and the second surgical element is determined based on the first data stream and the second data stream, wherein the relationship indicates that the complementary task is associated with the first task; A boundary parameter associated with the second output parameter is determined based on the first output parameter and the relationship, wherein the boundary parameter indicates the adjustment of the second surgical element during the complementary task; The indication of the boundary parameters is transmitted to the second surgical element; and The second surgical element performs the complementary task based on the boundary parameters.

10. The method according to claim 9, in, The first surgical element is a first surgical robot configured to be controllable by a user; The first task is associated with applying radiofrequency (RF) energy to the patient's tissue during tissue ablation; The second surgical element is a second surgical robot configured to be autonomously controlled. The complementary task is associated with maintaining tension applied to the tissue to fix the tissue during the first task; Wherein, the boundary parameter is the energy density applied to the tissue during tissue ablation; and The method further includes: The tension to be applied by the second surgical robot during the application of RF energy to the tissue by the first surgical robot is determined, wherein the tension to be applied maintains the energy density applied to the tissue.

11. The method according to claim 9, further comprising: Receive the patient's physiological parameters from the first data stream; Determine the second boundary parameter associated with the second output parameter; Determine that the patient's physiological parameters meet the threshold; The second boundary parameter is selected based on determining that the physiological parameter satisfies the threshold. The indication of the second boundary parameter is transmitted to the second surgical element; as well as The second surgical element performs the complementary task based on the boundary parameter and the second boundary parameter.

12. The method according to claim 9, wherein, The first surgical element is configured to perform the first task based on input from the user, and the second surgical element is configured to perform the complementary task autonomously.

13. The method according to claim 9, wherein, The method further includes: Determine that the second output parameter satisfies a threshold, wherein the threshold is associated with the boundary parameter; and A warning message is transmitted to the first surgical element, wherein the warning message indicates that the output parameter exceeds the boundary parameter.

14. The method according to claim 9, wherein, The second surgical element is configured to perform the complementary task in a synchronized motion based on the movement of the first surgical element while performing the first task.

15. The method according to claim 9, wherein, The method further includes: The first data stream, the second data stream, the instruction for the first task, and the instruction for the complementary task are fed as input to the machine learning (ML) model; and The boundary parameters associated with the second output parameters are received as a response from the ML model.

16. A system for presenting assistance to a user during surgery based on adaptive recognition of ability, the system comprising: A first surgical element, the first surgical element being configured to transmit a first data stream; A second surgical element, configured to receive a second data stream; and Processor, the processor being configured to: Receive the first data stream, wherein the first data stream indicates output parameters associated with the first task and the complementary task; Determine the relationship between the first surgical element and the second surgical element, wherein the relationship indicates that the complementary task is associated with the first task; The boundary parameters associated with the second surgical element are determined based on the output parameters and the relationship. The indication of the boundary parameters is transmitted to the second surgical element; and The second surgical element performs the complementary task based on the boundary parameters.

17. The system according to claim 16, wherein, The processor is also configured to: Receive the patient's physiological parameters from the first data stream; Determine the second boundary parameter associated with the second surgical element; Determine that the patient's physiological parameters meet the threshold; The second boundary parameter is selected based on determining that the physiological parameter satisfies the threshold. The indication of the second boundary parameter is transmitted to the second surgical element; as well as The second surgical element performs the complementary task based on the boundary parameter and the second boundary parameter.

18. The system according to claim 16, wherein, The first surgical element is configured to perform the first task based on input from the user, and the second surgical element is configured to perform the complementary task autonomously.

19. The system according to claim 16, wherein, The processor is also configured to transmit an alert message to the first surgical element, wherein the alert message indicates that the output parameter exceeds the boundary parameter.

20. The system of claim 16, wherein, The processor is also configured to: The first data stream and the instruction for the first task are fed as input to the machine learning (ML) model; and The boundary parameters are received as a response from the ML model.

Images