progressive advancement of the level of automation based on the learned supplemental assistance

Abstract

A system can adaptively identify a surgical element's ability to adjust automation strategies during a procedure. The system can be configured to obtain operational data of a first surgical element and a second surgical element. The system can determine an automation assistance parameter. The system can detect an adaptive identification event. The adaptive identification event can be at least one of a determination that a relationship between the first surgical element and the second surgical element, or that the physiological parameter of the patient meets a life-threatening threshold. The system can select a second automation assistance parameter of the first surgical element. The system can send an indication of the second automation assistance parameter to the first surgical element. The system can cause the first surgical element to perform a second set of automated tasks based on the second automation assistance parameter.

Description

Cross-references to related applications

[0001] This application relates to the following concurrently filed patent applications, the contents of each of which are incorporated herein by reference: • Case file END9638USNP2, titled "ADJUSTING AUTOMATED COOPERATIVE OPERATIONS BASED ONSITUATIONALLY DERIVED CONSTRAINTS". • Case file END9638USNP3 for the agent named ASSISTANCE ADVANCEMENT MULTI-SYSTEM INTERACTION • Case file END9638USNP4, titled "MONITORING AND IDENTIFYING SURGEON CONTROL AND SUGGESTING ATASK THAT MAY BE DONE AUTONOMOUSLY". • Case file END9638USNP5, entitled "Control of Information Flow, Prioritization and Manifestation of Data Assaulted with an Active HCP Interaction Space". • Case file END9638USNP6 for the agent named ADAPTIVE RETRACTION FORCE CONTROL • The agent's case file END9638USNP7, entitled "ADJUSTMENT OR DISPLAY OF OPTIONS OF POSITIONAL OR ORIENTATIONIMPLICATIONS ON SURGICAL TOOL USAGE", and • Case file END9638USNP8 for the agent named ADJUSTMENT OF PHYSIOLOGIC FUNCTION SUPPLEMENTATION CONTROL.

[0002] The following are all cited and incorporated into this article: • U.S. Patent Application No. 18 / 810,323, filed on August 20, 2024, entitled METHOD FOR MULTI-SYSTEM INTERACTION; • U.S. Patent Application No. 18 / 960,006, filed November 26, 2024, entitled "METHOD FOR SMART SURGICAL SYSTEMS"; and • U.S. Patent Application No. 18 / 954,186, filed on November 20, 2024, entitled METHOD FOR MULTI-SYSTEM INTERACTION. Background Technology

[0003] Surgical procedures are typically performed in the operating room or surgical chamber of a medical facility, such as a hospital. Various surgical devices and systems are utilized in the performance of surgical procedures. In the digital and information age, due to patient safety and the general expectation of maintaining traditional practices, medical systems and facilities are often slower to adopt systems or procedures utilizing newer and more advanced technologies. Summary of the Invention

[0004] The level of automation can be progressively advanced based on learned supplementary aids to reduce manual tasks performed by the HCP, improve efficiency and patient safety, and further optimize the procedure. An automation strategy selector system (referred to herein as the "System") can obtain operational data from one or more smart devices, the HCP, and / or a server. Operational data may include indications of the patient's physiological parameters during the procedure and / or multiple automated tasks associated with the procedure. The System can determine one or more tasks that can be automated during the procedure based on the operational data. The System can detect adaptively identified events. Adaptively identified events may include the determination of relationships between smart devices and / or the fulfillment of thresholds (e.g., life-threatening thresholds) by the patient's physiological parameters.

[0005] Based on adaptive event recognition, the system can select a second automation assistance parameter for the intelligent device. This automation assistance parameter can indicate a task from a set of automated tasks that the intelligent device can perform during surgery. For example, if the system detects that a physiological parameter meets a life-threatening threshold, it can select automation assistance parameters that increase the level of automation and / or fully automate the tasks associated with the surgery. In the example, if the system detects a relationship between surgical components, it can select automation assistance parameters that optimize the surgery based on the determined relationship (e.g., automating selected tasks to reduce interference, focusing the workload on critical tasks, and / or eliminating the possibility of erroneous communication during surgery). The system can send indications of the automation assistance parameters to the intelligent device (e.g., and / or instructions) to cause the intelligent device to perform the tasks associated with the automation assistance parameters.

[0006] The systems, methods, and / or instruments disclosed herein may relate to a progressive advancement of automation levels based on learned supplementary assistance. The system may be configured to adaptively identify automation strategies during surgery. The system may include a processor. The system may acquire operational data of a first surgical element and / or a second surgical element. The operational data may include indications of physiological parameters of a patient during surgery and / or multiple automated tasks associated with surgery. The system may determine automation assistance parameters for the first surgical element. The automation assistance parameters may be associated with a set of automated tasks performed by the first surgical element during surgery, among multiple automated tasks. The system may detect adaptive identification events. The adaptive identification event may be at least one of the following: a relationship between the first and second surgical elements, or a determination that the patient's physiological parameter meets a life-threatening threshold. The system may select a second automation assistance parameter for the first surgical element based on the adaptive identification event. The second automation assistance parameter may indicate a second set of automated tasks performed by the first surgical element during surgery. The system may send an indication of the second automation assistance parameter to the first surgical element. The system may cause the first surgical element to perform the second set of automated tasks based on the second automation assistance parameter.

[0007] This may include one or more features. For example, the first surgical element may be a ventilator. The second surgical element may be a heating device. The patient's physiological parameters may be the CO2 percentage from the ventilator or the core body temperature from the heating device. The patient's procedure may include atrial fibrillation (AFIB) neuroablation. Adaptive event recognition may include determining that the patient's CO2 to O2 or core body temperature meets a life-threatening threshold. The second automated assist parameter may include instructions to reduce the ventilator's O2 supplementation level or instructions to increase the ventilator's tidal volume.

[0008] For example, if the first or second surgical element determines that: the heart rate exceeds the operating window of 40-120 beats per minute, the patient's oxygen saturation decreases to below 90%, or the patient's core body temperature is below 89 degrees Fahrenheit. Adaptive event recognition may include determining that the patient's physiological parameters meet a life-threatening threshold. The selected second set of automated assistance parameters may indicate that the second set of automated tasks includes multiple automated tasks. Adaptive event recognition may include the relationship between the first and second surgical elements. The selected second set of automated assistance parameters may include instructions for performing the second set of automated tasks in response to user input.

[0009] For example, the system may transmit an automation strategy message to the user. The automation strategy message may include a request to perform a second set of automated tasks. The system may receive user input including instructions for performing at least one of the automated tasks in the second set. The system may receive location data associated with the user's movement during surgery and / or control data associated with the first and / or second surgical elements. The system may determine the relationship between the first and second surgical elements based on location data, operational data, and / or the patient's physiological parameters. The system may select automation assistance parameters for the second surgical element based on this adaptive recognition event. The automation assistance parameters of the second surgical element may indicate a set of automated tasks to be performed by the second surgical element during surgery. The system may send instructions for the automation assistance parameters of the second surgical element to the second surgical element. The system may cause the second surgical element to perform a set of automated tasks based on the automation assistance parameters of the second surgical element.

[0010] A method for adaptively identifying automated strategies during surgery can be described. The method may include obtaining operational data of a first surgical element and / or a second surgical element. The operational data may include indications of physiological parameters of a patient during surgery and / or multiple automated tasks associated with surgery. The method may include determining automated auxiliary parameters of the first surgical element. The automated auxiliary parameters may be associated with a set of automated tasks performed by the first surgical element during surgery, among multiple automated tasks. The method may include detecting an adaptive identification event. The adaptive identification event may be at least one of the following: the relationship between the first and second surgical elements, or the determination that the patient's physiological parameter meets a life-threatening threshold. The method may include selecting a second automated auxiliary parameter of the first surgical element based on the adaptive identification event. The second automated auxiliary parameter may indicate a second set of automated tasks performed by the first surgical element during surgery. The method may include sending an indication of the second automated auxiliary parameter to the first surgical element. The method may include causing the first surgical element to perform the second set of automated tasks based on the second automated auxiliary parameter.

[0011] The method may include one or more features. For example, the first surgical element may be a ventilator. The second surgical element may be a heating device. The patient's physiological parameters may be the CO2 percentage from the ventilator or the core body temperature from the heating device. The patient's procedure may include cardiac AFIB nerve ablation. Adaptive event recognition may include determining that the patient's CO2 to O2 or core body temperature meets a life-threatening threshold. The second automated assist parameter may include instructions for reducing the ventilator's O2 supplementation level or instructions for increasing the ventilator's tidal volume.

[0012] For example, adaptive event recognition may include determining that a patient's physiological parameters meet a life-threatening threshold. The selected second automation assist parameter may indicate that the second set of automated tasks includes multiple automated tasks. Adaptive event recognition may include a relationship between a first surgical element and a second surgical element. The selected second automation assist parameter may include instructions for performing the second set of automated tasks in response to user input. The method may include transmitting an automation strategy message to the user. The automation strategy message may include a request to perform the second set of automated tasks. The method may include receiving user input including instructions for performing at least one of the automated tasks in the second set of automated tasks. The method may include receiving location data associated with the user's movement during surgery and / or control data associated with the first and / or second surgical elements. The method may include determining the relationship between the first and second surgical elements based on location data, operational data, and / or the patient's physiological parameters.

[0013] For example, the method may include selecting automated auxiliary parameters of the second surgical element based on the adaptive recognition event. The automated auxiliary parameters of the second surgical element may indicate a set of automated tasks to be performed by the second surgical element during surgery. The method may include sending an indication of the automated auxiliary parameters of the second surgical element to the second surgical element. The method may include causing the second surgical element to perform a set of automated tasks based on the automated auxiliary parameters of the second surgical element.

[0014] A system for adaptively identifying automated strategies can acquire operational data of a first surgical element and / or a second surgical element. The operational data may include indications of physiological parameters of a patient. The system can detect adaptive identification events based on the operational data. Based on the adaptive identification events, the system can determine automation assistance parameters for the first surgical element. These automation assistance parameters may indicate a set of automated tasks to be performed by the first surgical element. The system can send indications of these automation assistance parameters to the first surgical element. Based on these automation assistance parameters, the system can cause the first surgical element to perform the second set of automated tasks.

[0015] The system may include one or more features. For example, the adaptive recognition event may include determining that the patient's physiological parameters meet a life-threatening threshold or determining the relationship between the first surgical element and the second surgical element. The system may transmit an automation strategy message to the user. The automation strategy message may include a request to perform an automated task associated with the automation assist parameter. The system may receive user input including instructions for performing the automated task. The system may select a second automation assist parameter based on the adaptive recognition event. The second automation assist parameter may indicate a second automation task. The system may send an instruction for the second automation assist parameter to the second surgical element. The system may cause the second surgical element to perform the second automation task. Attached Figure Description

[0016] Figure 1 This is a block diagram of a computer-implemented surgical system.

[0017] Figure 2 An example surgical system in an operating room is shown.

[0018] Figure 3 Example surgical hubs paired with various systems are shown.

[0019] Figure 4 An example situational awareness surgical system is shown.

[0020] Figure 5 Example surgical instruments are shown.

[0021] Figure 6 This is an example operating environment in which an automation strategy selector system can determine whether to automate one or more tasks of a surgery.

[0022] Figure 7A This is an example of... Figure 6 The flowchart illustrates the process of executing components within the operating environment to determine whether to automate one or more tasks.

[0023] Figure 7B This is an example of... Figure 6 The flowchart shows an example operation performed by components of the operating environment to train a machine learning (ML) model.

[0024] Figure 8 This is a flowchart depicting an example adaptive recognition detector routine.

[0025] Figure 9 This is an example implementation for determining automation strategies. Detailed Implementation

[0026] The operating room has become more complex with the introduction of intelligent devices (e.g., referred to interchangeably herein as "surgical elements"). These intelligent devices can be used and / or adjusted by healthcare professionals (HCPs) during surgery. Intelligent devices may include one or more advanced capabilities to significantly enhance the precision, safety, and efficiency of surgeries (e.g., surgical procedures, diagnostic procedures, therapeutic procedures, preventative procedures, etc.) while reducing the risk of patient complications. Examples of intelligent devices may include robotic surgical systems, navigation systems, intelligent imaging systems, endoscopic and / or laparoscopic systems, harmonic scalpels, anesthesia machines, patient monitoring systems (pulse oximeters, blood pressure monitors, EKG monitors, EEG monitors, etc.), energy devices (e.g., electrosurgical units, laser surgical systems, etc.), infusion pumps, etc.

[0027] Several smart devices may be connected (e.g., via a network) to generate and / or acquire operational data during surgery. Examples of operational data may include real-time and / or historical data associated with environmental data (e.g., temperature, humidity, airflow rate, differential pressure, and / or air filtration associated with the operating room), the number and location of HCPs during surgery, surgical plans (e.g., patient data associated with the surgery, HCP data, tasks, supplies, smart devices, staff workflows, etc.), the functionality of associated smart devices (e.g., automation auxiliary parameters indicating tasks associated with surgical plans that can be automated by smart devices), and / or patient data (e.g., physiological parameters, patient-associated life-threatening thresholds, patient health data, etc.).

[0028] Although smart devices can communicate via a network and generate operational data, they may not be optimized to reduce the workload of the HCP during surgery. The HCP may manually adjust one or more smart devices to, for example, maintain the patient's vital signs (e.g., physiological parameters) within safe operating levels (e.g., the HCP can adjust the patient's core body temperature, SpO2 level, heart rate, etc. by adjusting the setpoints on the ventilator, infusion pump, heating blanket, etc.).

[0029] Each time an HCP performs a manual task, its focus may shift away from critical parts of the procedure, potentially disrupting the workflow and / or leading to erroneous communication. For example, if the HCP is constantly adjusting the position of the ablation catheter, it may be unable to adequately focus on analyzing the tissue's response to the applied energy.

[0030] During high-risk surgeries such as open-heart surgery, repetitive manual tasks can disrupt the natural workflow of the hemococcal partner (HCP) by delaying the HCP's decision-making process, which can ultimately affect patient safety by increasing the risk of complications such as failure to quickly resolve bleeding. As another example, instructing support staff (e.g., nurses and assistant surgeons) to manage various aspects of the surgery (e.g., aspiration, tool positioning, etc.) can lead to miscommunication, potentially delaying the team's response and / or causing further disruption during the procedure.

[0031] To reduce manual tasks performed by the HCP, improve efficiency and patient safety, and further optimize surgery, an automated strategy selector system (referred to herein as the "System") may obtain operational data from one or more intelligent devices, the HCP, and / or a server. Operational data may include indications of the patient's physiological parameters during surgery and / or multiple automated tasks associated with the surgery. The System may determine one or more tasks that can be automated during surgery based on the operational data. The System may determine automation assistance parameters for one or more intelligent devices (e.g., as part of an automation assistance strategy). Automation assistance parameters may be associated with a set of automated tasks that can be performed by the intelligent devices during surgery. The System may detect adaptive recognition events. Adaptive recognition events may include the determination of relationships between intelligent devices and / or the fulfillment of thresholds (e.g., life-threatening thresholds) by the patient's physiological parameters.

[0032] Based on adaptive event recognition, the system can select a second automation assistance parameter for the intelligent device. For example, if the system detects that a physiological parameter meets a life-threatening threshold, it can select a second automation assistance parameter that includes multiple automated tasks (e.g., to increase the level of automation and / or fully automate the remaining tasks associated with the surgery). In the example, if the system detects a relationship between surgical components, it can select automation assistance parameters that optimize the surgery based on the determined relationship (e.g., to reduce interference, focus the workload on critical tasks, and / or eliminate the possibility of erroneous communication during surgery). The system can send indications of the second automation assistance parameters (e.g., and / or instructions) to the intelligent device to cause the intelligent device to perform the tasks associated with the second automation assistance parameters.

[0033] A more detailed understanding can be obtained by referring to the following description, which is given by way of example in conjunction with the accompanying drawings. Example aspects of surgical systems

[0034] Figure 1An example computer-implemented surgical system 20000 is illustrated. The example surgical system 20000 may include one or more surgical systems (e.g., surgical subsystems) 20002, 20003, and 20004. For example, surgical system 20002 may include a computer-implemented interactive surgical system. For example, surgical system 20002 may include a surgical hub 20006 and / or a computing device 20016 communicating with a cloud computing system 20008, such as... Figure 2 The cloud computing system 20008 may include at least one remote cloud server 20009 and at least one remote cloud storage unit 20010. Example surgical systems 20002, 20003, or 20004 may include one or more wearable sensing systems 20011, one or more environmental sensing systems 20015, one or more robotic systems 20013, one or more intelligent devices 20014, one or more human-machine interface systems 20012, etc. Human-machine interface systems are also referred to herein as human-machine interface devices. Wearable sensing system 20011 may include one or more healthcare professional (HCP) sensing systems and / or one or more patient sensing systems. Environmental sensing system 20015 may include one or more devices, for example, for measuring one or more environmental properties, such as... Figure 2 Further described. The robotic system 20013 may include multiple devices for performing surgical procedures, such as... Figure 2 Further described.

[0035] Surgical system 20002 can communicate with remote server 20009, which may be part of cloud computing system 20008. In one example, surgical system 20002 can communicate with remote server 20009 via a cable / FIOS networking node of an Internet service provider. In one example, patient sensing system can communicate directly with remote server 20009. Surgical system 20002 (and / or the various subsystems, intelligent surgical instruments, robots, sensing systems, and other computerized devices described herein) can collect data in real time and transmit the data to a cloud computer for data processing and manipulation. It should be understood that cloud computing may rely on shared computing resources rather than using local servers or personal devices to process software applications.

[0036] Surgical system 20002 and / or components thereof may communicate with remote server 20009 via a cellular transmit / receive point (TRP) or base station using one or more of the following cellular protocols: GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long Term Evolution (LTE) or 4G, LTE-Advanced (LTE-A), New Radio (NR) or 5G, and / or other wired or wireless communication protocols. Various examples of cloud-based analytics performed by cloud computing system 20008 and applicable to use with this disclosure are described in U.S. Patent Application Publication No. US 2019-0206569 A1 (U.S. Patent Application No. 16 / 209,403), filed December 4, 2018, entitled “METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB,” the entire disclosure of which is incorporated herein by reference.

[0037] The surgical hub 20006 can collaboratively interact with one of a plurality of devices displaying images from a laparoscopy and information from one or more other intelligent devices and one or more sensing systems 20011. The surgical hub 20006 can interact with one or more sensing systems 20011, one or more intelligent devices, and multiple displays. The surgical hub 20006 can be configured to collect measurement data from the sensing systems and transmit notification or control messages to the one or more sensing systems 20011. The surgical hub 20006 can transmit and / or receive information including notification information to and / or from a human-machine interface system 20012. The human-machine interface system 20012 may include one or more human-machine interface devices (HIDs). The surgical hub 20006 can transmit and / or receive notification or control information to be converted into audio, display, and / or control information for various devices communicating with the surgical hub.

[0038] For example, the sensing system may include a wearable sensing system 20011 (which may include one or more HCP sensing systems and / or one or more patient sensing systems) and / or an environmental sensing system 20015, such as Figure 1 As shown. The sensing system can measure data associated with various biomarkers. The sensing system can use one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc., to measure biomarkers. The sensor can use one or more of the following sensing techniques to measure biomarkers as described herein: photoplethysmography, electrocardiography, electroencephalography, colorimetry, impedance spectroscopy, potentiometry, current measurement, etc.

[0039] Biomarkers measured by the sensing system may include, but are not limited to, sleep, core body temperature, maximum oxygen uptake, physical activity, alcohol consumption, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, blood pH, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, cough and sneezing, gastrointestinal motility, gastrointestinal imaging, respiratory bacteria, edema, psychological factors, sweat, circulating tumor cells, autonomic tone, circadian rhythm and / or menstrual cycle.

[0040] Biomarkers can relate to physiological systems, including but not limited to behavioral and psychological systems, cardiovascular systems, renal systems, dermal systems, nervous systems, gastrointestinal systems, respiratory systems, endocrine systems, immune systems, tumors, musculoskeletal systems, and / or reproductive systems. For example, information from biomarkers can be determined and / or used by a computer-implemented patient and surgical system 20000. This information from biomarkers can be determined and / or used by the computer-implemented patient and surgical system 20000 to improve said systems and / or improve patient outcomes.

[0041] The sensing system can transmit data to the surgical hub 20006. The sensing system can communicate with the surgical hub 20006 using one or more of the following RF protocols: Bluetooth, Bluetooth Low Energy (BLE), Bluetooth Smart, Zigbee, Z-Wave, IPv6 Low Power Wireless Personal Area Network (6LoWPAN), and Wi-Fi.

[0042] The sensing system, biomarkers, and physiological system are described in more detail in U.S. Application No. 17 / 156,287 (Attorney’s File END9290USNP1), filed on January 22, 2021, entitled “METHOD OF ADJUSTING A SURGICAL PARAMETERBASED ON BIOMARKER MEASUREMENTS”, the entire disclosure of which is incorporated herein by reference.

[0043] The sensing system described herein can be used to assess the physiological condition of a surgeon performing surgery on a patient, a patient preparing for surgery, or a patient recovering after surgery. The cloud-based computing system 20008 can be used to monitor biomarkers associated with the surgeon or patient in real time, and can be used to generate surgical plans based at least on measurement data collected prior to surgery, provide control signals to surgical instruments during surgery, and notify the patient of complications during the postoperative period.

[0044] A cloud-based computing system 20008 can be used to analyze surgical data. Surgical data can be obtained via one or more intelligent instruments 20014, wearable sensing systems 20011, environmental sensing systems 20015, robotic systems 20013, etc., within the surgical system 20002. Surgical data may include tissue status to assess leakage or perfusion of sealed tissue following tissue sealing and surgical pathology data, including images of samples of body tissue, anatomical structures of the body using various sensors integrated with imaging devices, and techniques such as overlaying images captured by multiple imaging devices, image data, etc. Surgical data can be analyzed to improve surgical outcomes by determining whether further treatment can proceed (such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robotics to tissue-specific sites and conditions). Such data analysis can employ outcome analysis processing, and using standardized methods can provide beneficial feedback to validate surgical treatment and surgeon behavior, or to suggest modifications to surgical treatment and surgeon behavior.

[0045] Figure 2 An example surgical system 20002 in an operating room is shown. Figure 2 As illustrated, the patient undergoes surgery performed by one or more healthcare professionals (HCPs). The HCP is monitored by one or more HCP sensing systems 20020 worn by the HCP. The HCP and the environment surrounding the HCP may also be monitored by one or more environmental sensing systems, including, for example, a collection of cameras 20021, a collection of microphones 20022, and other sensors that can be deployed in the operating room. The HCP sensing systems 20020 and the environmental sensing systems may communicate with a surgical hub 20006, which in turn may communicate with one or more cloud servers 20009 of a cloud computing system 20008, such as... Figure 1 As shown. Environmental sensing systems can be used to measure one or more environmental properties, such as the location of the HCP in the surgical room, HCP movement, environmental noise in the surgical room, temperature / humidity in the surgical room, etc.

[0046] like Figure 2As illustrated, a main display 20023 and one or more audio output devices (e.g., speakers 20019) are positioned within a sterile area to be visible to the operator at the operating table 20024. Furthermore, a visualization / notification tower 20026 is positioned outside the sterile area. The visualization / notification tower 20026 may include a first non-sterile human-machine interface (HID) 20027 and a second non-sterile HID 20029 that are mutually exclusive. The HID may be a display or a display with a touchscreen that allows direct human-machine interface with the HID. The HID system guided by the surgical hub 20006 can be configured to utilize HIDs 20027, 20029, and 20023 to coordinate the flow of information to the operator both inside and outside the sterile area. In one example, the surgical hub 20006 may enable the HID (e.g., the main HID 20023) to display notifications and / or information about the patient and / or surgical procedures. In one example, the surgical hub 20006 may prompt and / or receive input from personnel in a sterile or non-sterile area. In another example, the surgical hub 20006 may enable the HID to display a snapshot of the surgical site recorded by the imaging device 20030 on a non-sterile HID 20027 or 20029, while maintaining a real-time feed of the surgical site on the main HID 20023. For example, the snapshot on the non-sterile display 20027 or 20029 may allow a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0047] The surgical hub 20006 can be configured to route diagnostic inputs or feedback entered by a non-sterile operator at the visualization tower 20026 to the main display 20023 within the sterile area, where a sterile operator at the operating table can view the diagnostic inputs or feedback. In one example, the input may be a modified form of a snapshot displayed on a non-sterile display 20027 or 20029, which can be routed to the main display 20023 via the surgical hub 20006.

[0048] See Figure 2Surgical instrument 20031 is used in surgical procedures as part of surgical system 20002. Hub 20006 can be configured to coordinate the flow of information to the display of surgical instrument 20031. For example, it is described in U.S. Patent Application Publication No. US 2019-0200844 A1 (U.S. Patent Application No. 16 / 209,385), filed December 4, 2018, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” the entire disclosure of which is incorporated herein by reference. Diagnostic inputs or feedback entered by a non-aseptic operator at visualization tower 20026 can be routed by hub 20006 to the surgical instrument display within the aseptic area, where the operator of surgical instrument 20031 can view the diagnostic inputs or feedback. For example, an example surgical instrument suitable for use with the surgical system 20002 is described in U.S. Patent Application Publication No. US 2019-0200844 A1 (U.S. Patent Application No. 16 / 209,385), filed December 4, 2018, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” under the heading “Surgical Instrument Hardware,” the entire disclosure of which is incorporated herein by reference.

[0049] like Figure 2 As shown, surgical system 20002 can be used to perform surgery on a patient lying supine on operating table 20024 in operating room 20035. Robotic system 20034 can be used as part of surgical system 20002 during surgery. Robotic system 20034 may include surgeon's console 20036, patient-side trolley 20032 (surgical robot), and surgical robot hub 20033. While the surgeon views the surgical site through surgeon's console 20036, patient-side trolley 20032 can manipulate at least one removably attached surgical tool 20037 through a minimally invasive incision within the patient's body. Images of the surgical site can be obtained via medical imaging device 20030, which can be manipulated by patient-side trolley 20032 to orient the imaging device 20030. Robotic hub 20033 can be used to process images of the surgical site for subsequent display to the surgeon via surgeon's console 20036.

[0050] Other types of robotic systems can be readily adapted for use with surgical system 20002. Various examples of robotic systems and surgical tools applicable to this disclosure are described herein and in U.S. Patent Application No. US 2019-0201137 A1 (U.S. Patent Application No. 16 / 209,407), filed December 4, 2018, entitled “METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL,” the entire disclosure of which is incorporated herein by reference.

[0051] In various aspects, the imaging device 20030 may include at least one image sensor and one or more optical components. Suitable image sensors may include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0052] The optical components of the imaging device 20030 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate multiple portions of the surgical site. One or more image sensors may receive light reflected or refracted from the surgical site, including light reflected or refracted from tissue and / or surgical instruments.

[0053] Light sources can be configured to radiate electromagnetic energy in both the visible and invisible spectra. The visible spectrum (sometimes referred to as the optical spectrum or emission spectrum) is the portion of the electromagnetic spectrum that is visible to the human eye (e.g., detectable by it) and can be called "visible light" or simply "light." The typical human eye responds to wavelengths in the range of approximately 380 nm to approximately 750 nm in air.

[0054] The invisible spectrum (e.g., the non-luminescent spectrum) is the portion of the electromagnetic spectrum that lies below and above the visible spectrum (i.e., wavelengths below about 380 nm and above about 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum and become invisible infrared (IR), microwave, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum and become invisible ultraviolet, X-ray, and gamma-ray electromagnetic radiation.

[0055] In various respects, the imaging device 20030 is configured to be used in minimally invasive surgery. Examples of imaging devices suitable for use in this disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopes, colonoscopes, cytoscopes, duodenoscopes, colonoscopes, esophagoduodenoscopes (gastroscopes), endoscopes, laryngoscopes, nasopharyngeal-renal endoscopes, sigmoidoscopes, thoracoscopes, and ureteroscopes.

[0056] Imaging devices can employ multispectral monitoring to distinguish morphology and underlying structures. A multispectral image is an image that captures image data across a specific wavelength range of the electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including light from frequencies outside the visible light range, such as IR and ultraviolet. Spectral imaging allows the extraction of additional information that the human eye fails to capture with its red, green, and blue receptors. The use of multispectral imaging is described in more detail under the title “Advanced Imaging Acquisition Module” in U.S. Patent Application No. US 2019-0200844 A1 (U.S. Patent Application No. 16 / 209,385), filed December 4, 2018, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE ANDDISPLAY,” the entire disclosure of which is incorporated herein by reference. After completing a surgical task to perform one or more of the previously described tests on the treated tissue, multispectral monitoring can be a useful tool for repositioning the surgical site. It goes without saying that rigorous sterilization of the operating room and surgical equipment is required during any surgical procedure. The stringent hygienic and sterilization conditions required in a “surgical room” (e.g., operating room or treatment room) necessitate the highest possible sterility for all medical devices and apparatus. Part of this sterilization process requires the sterilization of any material that comes into contact with the patient or penetrates the sterile area, including the imaging device 20030 and its attachments and components. It should be understood that a sterile area can be considered a designated area deemed free of microorganisms, such as within a tray or sterile towel, or can be considered the area surrounding the patient prepared for surgical procedures. A sterile area may include properly dressed scrubbed team members, as well as all equipment and fixtures within that area.

[0057] Figure 1 The illustrated wearable sensing system 20011 may include, for example: Figure 2One or more HCP sensing systems 20020 are shown. The HCP sensing system 20020 may include sensing systems for monitoring and detecting a set of physical and / or physiological states of a healthcare professional (HCP). An HCP may typically be a surgeon or one or more healthcare professionals or other healthcare providers assisting a surgeon. In one example, the HCP sensing system 20020 may measure a set of biomarkers to monitor the HCP's heart rate. In one example, the HCP sensing system 20020 worn on the surgeon's wrist (e.g., a watch or wristband) may use an accelerometer to detect hand movements and / or tremors and determine the amplitude and frequency of the tremors. The sensing system 20020 may transmit measurement data associated with the set of biomarkers and data associated with the surgeon's physical state to a surgical hub 20006 for further processing.

[0058] Figure 1 The illustrated environmental sensing system 20015 can transmit environmental information to the surgical hub 20006. For example, the environmental sensing system 20015 may include a camera 20021 for detecting the hand / body position of the HCP. The environmental sensing system 20015 may include a microphone 20022 for measuring ambient noise in the surgical room. Other environmental sensing systems 20015 may include devices such as a thermometer for measuring temperature and a hygrometer for measuring the humidity of the surrounding environment in the surgical room. Surgeon biomarkers may include one or more of the following: pressure, heart rate, etc. Environmental measurements from the surgical room may include ambient noise levels associated with the surgeon or patient, surgeon and / or staff movement, surgeon and / or staff attention levels, etc. The surgical hub 20006 (either independently or in communication with a cloud computing system) can use surgeon biomarker measurement data and / or environmental sensing information to modify the control algorithms of handheld instruments or the average latency of robot interfaces, for example, to minimize tremors.

[0059] The surgical hub 20006 can use surgeon biomarker measurements associated with HCP to adaptively control one or more surgical instruments 20031. For example, the surgical hub 20006 can transmit control programs to the surgical instrument 20031 to control its actuators to limit or compensate for fatigue and the use of fine motor skills. The surgical hub 20006 can transmit control programs based on situational awareness and / or context regarding the importance or criticality of the task. When control is needed, the control program can instruct the instrument to change its operation to provide more control.

[0060] Figure 3An example surgical system 20002 with a surgical hub 20006 is shown. The surgical hub 20006 can be paired with a wearable sensing system 20011, an environmental sensing system 20015, a human-machine interface system 20012, a robotic system 20013, and a smart instrument 20014 via modular controls. The hub 20006 includes a display 20048, an imaging module 20049, a generator module 20050 (e.g., an energy generator), a communication module 20056, a processor module 20057, a storage array 20058, and an operating room mapping module 20059. In some aspects, such as Figure 3 As illustrated, hub 20006 also includes smoke extraction module 20054 and / or suction / flushing module 20055. Various modules and systems can be connected directly to the modular control via a router or via communication module 20056. The operating room device can be coupled to cloud computing resources and data storage devices via the modular control. Human-machine interface system 20012 may include a display subsystem and a notification subsystem.

[0061] Modular controls can be coupled to a non-contact sensor module. The non-contact sensor module can use ultrasonic, laser-based, and / or similar non-contact measuring devices to measure the size of the operating room and generate a mapping of the surgical room. Other distance sensors can be used to determine the boundaries of the operating room. An ultrasonic-based non-contact sensor module can scan the operating room by emitting a burst of ultrasound and receiving the echo as it bounces back from the walls of the operating room, as described under the title “Surgical Hub Spatial Awareness Within an Operating Room” in U.S. Provisional Patent Application Serial No. 62 / 611,341, filed December 28, 2017, entitled “INTERACTIVE SURGICAL PLATFORM”, the entire contents of which are incorporated herein by reference. The sensor module can be configured to determine the size of the operating room and adjust Bluetooth pairing distance limits. A laser-based non-contact sensor module can scan the operating room by emitting laser pulses, receiving laser pulses bouncing back from the walls of the operating room, and comparing the phase of the emitted pulse with the received pulse to determine the size of the operating room and adjust Bluetooth pairing distance limits.

[0062] During surgery, the application of energy to tissue for sealing and / or cutting can be associated with fumigation, aspiration of excess fluid, and / or tissue flushing. Fluid lines, power lines, and / or data lines from different sources can become entangled during surgery. Resolving this issue during surgery can waste valuable time. Disconnecting lines may require disconnecting them from their respective modules, which may necessitate module resets. The Hub Modular Housing 20060 provides a unified environment for managing power lines, data lines, and fluid lines, reducing the frequency of entanglement between such lines.

[0063] Energy can be applied to tissue at a surgical site. The surgical hub 20006 may include a hub housing 20060 and a combined generator module slidably received in a docking base within the hub housing 20060. The docking base includes data contacts and power contacts. The combined generator module may include two or more of an ultrasonic energy generator component, a bipolar RF energy generator component, or a monopolar RF energy generator component housed in a single unit. The combined generator module may include a smoke extraction component, at least one energy delivery cable for connecting the combined generator module to a surgical instrument, at least one smoke extraction component configured to extract smoke, fluid, and / or particles generated by applying therapeutic energy to tissue, and a fluid line extending from a remote surgical site to the smoke extraction component. The fluid line may be a first fluid line, and a second fluid line may extend from a remote surgical site to a suction and flushing module 20055 slidably housed in the hub housing 20060. The hub housing 20060 may include a fluid interface.

[0064] The combined generator module can generate multiple energy types for application to tissue. One energy type may be more advantageous for cutting tissue, while another different energy type may be more advantageous for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Aspects of this disclosure present a solution in which the hub modular housing 20060 is configured to accommodate different generators and facilitate interactive communication between them. The hub modular housing 20060 allows for the rapid removal and / or replacement of various modules.

[0065] The modular surgical housing may include: a first energy generator module configured to generate a first energy for application to tissue; and a first docking base including a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and wherein the first energy generator module is slidably movable to no longer electrically engage with the first power and data contacts. The modular surgical housing may also include: a second energy generator module configured to generate a second energy, different from the first energy, for application to tissue; and a second docking base including a second docking port including second data and power contacts, wherein the second energy generator module is slidably movable to electrically engage with the power and data contacts, and wherein the second energy generator module is slidably movable to no longer electrically engage with the second power and data contacts. Furthermore, the modular surgical housing also includes a communication bus between the first and second docking ports, configured to facilitate communication between the first and second energy generator modules.

[0066] See Figure 3 The hub modular housing 20060 allows for modular integration of the generator module 20050, the smoke extraction module 20054, and the suction / flushing module 20055. The hub modular housing 20060 facilitates interactive communication between modules 20059, 20054, and 20055. The generator module 20050 may have integrated monopolar, bipolar, and ultrasonic components supported in a single housing unit slidably inserted into the hub modular housing 20060. The generator module 20050 can be connected to the monopolar device 20051, the bipolar device 20052, and the ultrasonic device 20053. The generator module 20050 may include a series of monopolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact through the hub modular housing 20060. The hub modular housing 20060 facilitates the insertion of multiple generators and interactive communication between generators connected to the hub modular housing 20060, allowing the generator to function as a single generator.

[0067] A surgical data network with a set of communication hubs can connect sensing systems and modular devices located in one or more operating rooms, patient recovery rooms, or rooms in a medical facility specifically equipped for surgical procedures to a cloud computing system 20008.

[0068] Figure 4A diagram illustrating a situation-aware surgical system 5100 is provided. Data source 5126 may include, for example, a modular device 5102, a database 5122 (e.g., an EMR database containing patient records), a patient monitoring device 5124 (e.g., a blood pressure (BP) monitor and an electrocardiogram (EKG) monitor), an HCP monitoring device 35510, and / or an environmental monitoring device 35512. Modular device 5102 may include sensors configured to detect parameters associated with the patient, HCP, and environment, and / or the modular device itself. Modular device 5102 may include one or more intelligent instruments 20014. Surgical hub 5104 may derive contextual information related to the surgical procedure from the data, for example, based on a specific combination of received data or a specific order in which data is received from data source 5126. The contextual information inferred from the received data may include, for example, the type of surgical procedure being performed, the specific steps of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity of the object of the surgery. The ability of the surgical hub 5104 to derive or infer surgical-related information from received data can be termed "situational awareness." For example, the surgical hub 5104 may incorporate a situational awareness system, which could be hardware and / or surgical planning associated with the surgical hub 5104 that derives surgical-related background information from received data, and / or surgical planning information received from edge computing system 35514 or enterprise cloud server 35516. Background information derived from data source 5126 may include, for example, the steps of the surgical procedure being performed, whether and how a specific modular device 5102 is being used, and the patient's condition.

[0069] Surgical hub 5104 can connect to various databases 5122 to retrieve data from them regarding surgical procedures being performed or to be performed. In one example of surgical system 5100, database 5122 may include a hospital's EMR database. Data that can be received from database 5122 by the situational awareness system of surgical hub 5104 may include, for example, start (or setup) time or operational information about a procedure (e.g., a segmental resection in the upper right thoracic region). Surgical hub 5104 can derive background information about the surgical procedure from this data alone or from this data in combination with data from other data sources 5126.

[0070] The surgical hub 5104 can be connected to (e.g., paired with) various patient monitoring devices 5124. In one example of the surgical system 5100, the patient monitoring devices 5124 that can be paired with the surgical hub 5104 may include a pulse oximeter (SpO2 monitor) 5114, a BP monitor 5116, and an EKG monitor 5120. Perioperative data that can be received by the situational awareness system of the surgical hub 5104 from the patient monitoring devices 5124 may include, for example, the patient's oxygen saturation, blood pressure, heart rate, and other physiological parameters. Background information that can be derived by the surgical hub 5104 from the perioperative data sent by the patient monitoring devices 5124 may include, for example, whether the patient is in the operating room or under anesthesia. The surgical hub 5104 may derive these inferences individually from data from the patient monitoring devices 5124 or in combination with data from other data sources 5126 (e.g., a ventilator 5118).

[0071] The surgical hub 5104 can be connected to (e.g., paired with) various modular devices 5102. In one example of the surgical system 5100, the modular device 5102 paired with the surgical hub 5104 may include a fume extractor, medical imaging devices (such as...) Figure 2 The imaging device 20030 shown includes an inhaler, a combined energy generator (for providing power to ultrasound surgical instruments and / or RF electrosurgical instruments), and a ventilator.

[0072] Perioperative data received by the surgical hub 5104 from the medical imaging device may include, for example, whether the medical imaging device is activated and video or image feeds. Background information derived by the surgical hub 5104 from the perioperative data transmitted by the medical imaging device may include, for example, whether the surgery is a VATS procedure (based on whether the medical imaging device is activated or paired with the surgical hub 5104 at the start of the surgery or during the procedure). Image or video data (or a data stream representing video for a digital medical imaging device) from the medical imaging device may be processed by a pattern recognition system or a machine learning system to, for example, identify features (e.g., organ or tissue type) in the field of view (FOY) of the medical imaging device. Background information derived by the surgical hub 5104 from the identified features may include, for example, the type of surgical procedure (or its steps) being performed, the organ being operated on, or the body cavity in which the operation is being performed.

[0073] The situational awareness system of the surgical hub 5104 can derive contextual information from data received from the data source 5126 in a variety of different ways. For example, the situational awareness system may include a pattern recognition system or a machine learning system (e.g., an artificial neural network) trained on training data to associate various inputs (e.g., data from the database 5122, patient monitoring device 5124, modular device 5102, HCP monitoring device 35510, and / or environmental monitoring device 35512) with corresponding contextual information about the surgical procedure. For example, the machine learning system can accurately derive contextual information about the surgical procedure from the provided inputs. In an example, the situational awareness system may include a lookup table that stores pre-represented environmental information about the surgical procedure associated with one or more inputs (or ranges of inputs) corresponding to environmental information. In response to a query using one or more inputs, the lookup table can return corresponding contextual information used by the situational awareness system to control the modular device 5102. In the example, the contextual information received by the situational awareness system of the surgical hub 5104 may be associated with a specific control adjustment or a set of control adjustments for one or more modular devices 5102. In the example, the situational awareness system may include a machine learning system, lookup table, or other such system that can generate or retrieve one or more control adjustments for one or more modular devices 5102 when provided with contextual information as input.

[0074] For example, based on data source 5126, the situational-aware surgical hub 5104 can determine the type of tissue being operated on. The situational-aware surgical hub 5104 can infer whether the surgery being performed is thoracic or abdominal, thus allowing the surgical hub 5104 to determine whether the tissue held by the end effector of the surgical suture and cutting instruments is lung tissue (for thoracic surgery) or stomach tissue (for abdominal surgery). The situational-aware surgical hub 5104 can determine whether the surgical site is under pressure (by determining that the surgery is utilizing airflow) and determine the type of surgery to achieve a consistent amount of smoke extraction for both thoracic and abdominal surgeries. Based on data source 5126, the situational-aware surgical hub 5104 can determine which step of the surgery is being performed or will be performed subsequently.

[0075] The situation-aware surgical hub 5104 can determine the type of surgical procedure being performed and customize energy levels based on the expected tissue profile of the procedure. The situation-aware surgical hub 5104 can adjust the energy levels of ultrasonic surgical instruments or RF electrosurgical instruments throughout the entire surgical procedure, rather than just on a per-procedure basis.

[0076] In the example, data can be extracted from an additional data source 5126 to improve the conclusions drawn by the surgical hub 5104 from one data source 5126. The situation-aware surgical hub 5104 can supplement the data received from the modular device 5102 with the background information about the surgery that it has built from other data sources 5126.

[0077] The situational awareness system of the surgical hub 5104 can take physiological measurement data into account to provide additional contextual information when analyzing visualization data. This additional context can be useful when the visualization data itself may be uncertain or incomplete.

[0078] The situational awareness surgical hub 5104 can determine whether a surgeon (or other HCP) is making an error or otherwise deviating from the intended procedure during surgery. For example, the surgical hub 5104 can determine the type of surgery being performed, retrieve a corresponding list of steps or the order of equipment use (e.g., from memory), and compare the steps being performed or the equipment being used during the surgery with the expected steps or equipment determined by the surgical hub 5104 for that type of surgery. The surgical hub 5104 can provide alerts indicating that a particular step in the surgery is performing an unexpected action or utilizing an unexpected device.

[0079] Surgical instruments (and other modular devices 5102) can be adapted to the specific context of each surgical procedure (such as adaptation to different tissue types) and verification actions during the surgical procedure. Subsequent steps, data, and display adjustments can be provided to the surgical instruments (and other modular devices 5102) in the operating room according to the specific context of the surgery.

[0080] Figure 5An example surgical system 20280 is illustrated, which may include a surgical instrument 20282. The surgical instrument 20282 may communicate with a console 20294 and / or a portable device 20296 via a wired and / or wireless connection through a local area network 20292 and / or a cloud network 20293. The console 20294 and the portable device 20296 may be any suitable computing device. The surgical instrument 20282 may include a handle 20297, an adapter 20285, and a loading unit 20287. The adapter 20285 is releasably coupled to the handle 20297, and the loading unit 20287 is releasably coupled to the adapter 20285, such that the adapter 20285 transmits force from a drive shaft to the loading unit 20287. The adapter 20285 or the loading unit 20287 may include a force gauge (not explicitly shown) disposed therein to measure the force applied to the loading unit 20287. The loading unit 20287 may include an end effector 20289 having a first jaw 20291 and a second jaw 20290. The loading unit 20287 may be an in-situ loading or multiple-fire loading unit (MFLU), which allows clinicians to fire multiple fasteners multiple times without removing the loading unit 20287 from the surgical site to reload it.

[0081] The first jaw 20291 and the second jaw 20290 may be configured to clamp tissue therebetween, fire a fastener through the clamped tissue, and cut the clamped tissue. The first jaw 20291 may be configured to fire at least one fastener multiple times, or may be configured to include a replaceable multiple-firing fastener cartridge containing multiple fasteners (e.g., nails, clamps, etc.) that can be fired more than once before being replaced. The second jaw 20290 may include an anvil that deforms or otherwise secures the fastener when it is ejected from the multiple-firing fastener cartridge.

[0082] The handle 20297 may include a motor coupled to a drive shaft to influence its rotation. The handle 20297 may include a control interface for selectively activating the motor. The control interface may include buttons, switches, levers, sliders, touchscreens, and any other suitable input mechanisms or user interfaces that can be engaged by a clinician to activate the motor.

[0083] The control interface of the handle 20297 can communicate with the controller 20298 of the handle 20297 to selectively activate the motor to affect the rotation of the drive shaft. The controller 20298 may be located within the handle 20297 and is configured to receive input from the control interface and adapter data from the adapter 20285 or loading unit data from the loading unit 20287. The controller 20298 can analyze the input from the control interface and the data received from the adapter 20285 and / or loading unit 20287 to selectively activate the motor. The handle 20297 may also include a display that a clinician can view during use of the handle 20297. The display may be configured to show portions of the adapter or loading unit data before, during, or after the firing instrument 20282.

[0084] Adapter 20285 may include an adapter identification device 20284 disposed therein, and loading unit 20287 may include a loading unit identification device 20288 disposed therein. Adapter identification device 20284 may communicate with controller 20298, and loading unit identification device 20288 may communicate with controller 20298. It should be understood that loading unit identification device 20288 may communicate with adapter identification device 20284, and the adapter identification device relays or transmits communications from loading unit identification device 20288 to controller 20298.

[0085] The adapter 20285 may also include a plurality of sensors 20286 (one shown) disposed around it to detect various conditions of the adapter 20285 or the environment (e.g., whether the adapter 20285 is connected to the loading unit, whether the adapter 20285 is connected to the handle, whether the drive shaft rotates, the torque of the drive shaft, the strain of the drive shaft, the temperature within the adapter 20285, the number of times the adapter 20285 is fired, the peak force of the adapter 20285 during firing, the total force applied to the adapter 20285, the peak retraction force of the adapter 20285, the number of pauses of the adapter 20285 during firing, etc.). The plurality of sensors 20286 may provide input to the adapter identification device 20284 in the form of data signals. The data signals of the plurality of sensors 20286 may be stored in the adapter identification device 20284 or may be used to update adapter data stored in the adapter identification device. The data signals of the plurality of sensors 20286 may be analog or digital. Multiple sensors 20286 may include force gauges to measure the force applied to the loading unit 20287 during firing.

[0086] The handle 20297 and adapter 20285 may be configured to interconnect the adapter identification device 20284 and the loading unit identification device 20288 with the controller 20298 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals therebetween). Additionally or alternatively, the electrical interface may be a contactless electrical interface for wirelessly transmitting energy and signals therebetween (e.g., inductive transmission). It is also conceivable that the adapter identification device 20284 and the controller 20298 may communicate wirelessly with each other via a wireless connection separate from the electrical interface.

[0087] The handle 20297 may include a transceiver 20283 configured to transmit instrument data from the controller 20298 to other components of the system 20280 (e.g., LAN 20292, cloud 20293, console 20294, or portable device 20296). The controller 20298 may also transmit instrument data and / or measurement data associated with one or more sensors 20286 to the surgical hub. The transceiver 20283 may receive data (e.g., pod data, loading unit data, adapter data, or other notifications) from the surgical hub 20270. The transceiver 20283 may also receive data (e.g., pod data, loading unit data, or adapter data) from other components of the system 20280. For example, controller 20298 can send instrument data to console 20294, including the serial number of the attachment adapter (e.g., adapter 20285) attached to handle 20297, the serial number of the loading unit (e.g., loading unit 20287) attached to adapter 20285, and the serial number of the multi-fire fastener cartridge loaded onto the loading unit. Console 20294 can then send data associated with the attached cartridge, loading unit, and adapter (e.g., cartridge data, loading unit data, or adapter data), respectively, back to controller 20298. Controller 20298 can display the message on a local instrument display or send the message via transceiver 20283 to console 20294 or portable device 20296 for display on display 20295 or portable device screen, respectively. Example operating environment for determining automation strategies

[0088] Figure 6 Example operating environment 56200, wherein automation strategy selector system 56210 (hereinafter referred to as "system 56210") can receive data from surgical elements 56222, 56224, operating data storage device 56203 and / or HCP 56208, and determine automation strategies, including one or more automation tasks (e.g., actions, functions, steps, methods, activities, goals, etc.) associated with a surgery that can be performed by surgical elements 56222, 56224.

[0089] Operating environment 56200 may include HCP 56208, patient 56209, operating data storage device 56203, surgical elements 56222 and 56224, system 56210, and network 56201. System 56210 may receive instructions for surgery from HCP 56208 (e.g., via graphical user interface (GUI)) and, in response, send a first request to surgical elements 56222 and 56224 (e.g., via network 56201) to obtain physiological parameters of patient 56209, and send a second request to operating data storage device 56203 to send operating data associated with the surgery. System 56210 may receive operating data (e.g., real-time or historical operating data) from surgical elements 56222 and 56224 and / or operating data storage device 56203. System 56210 may determine one or more tasks, etc., that can be automated by surgical elements 56222 and / or 56224 based on the operating data.

[0090] Operating environment 56200 may include system 56210. System 56210 may include adaptive recognition detector 56211, automated auxiliary controller 56212, user interface service 56213, auxiliary data storage device 56214, and / or ML model trainer 56215. Although system 56210 is depicted as separate from one or more components of operating environment 56200, system 56210 and / or one or more components of system 56210 may reside on a device, server, etc., such as, for example, surgical elements 56222, 56224, or a surgical hub.

[0091] The system may include an adaptive identification detector 56211 (hereinafter referred to as "detector 56211"). Detector 56211 may be, for example, a processor, controller, field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), etc., configured to perform one or more actions as described herein. Detector 56211 may receive data from one or more components of the operating environment 56200 (e.g., via network 56201). For example, detector 56211 may receive operational data from surgical elements 56222, 56224, operational data storage device 56203, and / or from HCP 56208 (e.g., via user interface service 56213). Detector 56211 may send data to one or more components. In the example, detector 56211 may send requests for information to surgical elements 56222, 56224, requests for operational data to operational data storage device 56203, requests for data to HCP 56208 (e.g., via user interface service 56213), etc.

[0092] Operational data may be generated by one or more components of the operating environment 56200. Operational data may include past and / or current information associated with the operating environment 56200. Operational data may be generated based on one or more surgeries (e.g., past and / or current surgeries). Operational data may include: patient physiological parameters (e.g., patient's heart rate, blood pressure, SpO2, core body temperature, etc.); environmental data (e.g., the number of HCPs in the operating room, historical records of HCPs 56208 performing surgeries, location data associated with the movement of HCPs 56208 during surgery, temperature, humidity, airflow rate, hospital-related data such as the number of surgeries performed and / or the results of surgeries, or functions associated with surgical elements 56222, 56224 available during surgery, etc.); data associated with surgical elements 56222, 56224 (e.g., setpoints, output characteristics, measured variables, communication data, etc.), user data (e.g., data generated based on user input from HCPs 56208 via user interface service 56213), etc.

[0093] As an illustrative example, operational data associated with surgical elements 56222 and 56224 may include control data such as the speed of the cutting tool, the selected control loop, the voltage and / or current of the electrosurgical tool, the flow rate of the infusion pump, the RPM of the fan, the setpoint of the heating blanket, etc. In the example, operational data may include system-generated data and / or other data based on the patient's physiological parameters. As an illustrative example, operational data may include thresholds (e.g., life-threatening thresholds) associated with the patient's physiological parameters (e.g., as determined by detector 56211). Examples of thresholds may include an operational window for heart rate (e.g., 40-120 heartbeats per minute), oxygen saturation thresholds (e.g., oxygen saturation of 92%, 90%, 88%, etc.), or core body temperature thresholds (e.g., core body temperature of 90, 89, 88, 87 degrees Fahrenheit), etc.

[0094] Detector 56211 can determine (e.g., detect) that an adaptive recognition event (which may be interchangeably referred to herein as an "event") has occurred. Detector 56211 can determine events during surgery. Events may occur based on the determination of relationships between surgical elements 56222, 562224, patient 56209, HCP 56208, and / or another component of the operating environment 56200. An event may occur if the patient's physiological parameters have met a threshold. Detector 56211 can send an indication that an event has occurred (e.g., including indications of relationships and / or indications that physiological parameters have met thresholds) to one or more components of the operating environment 56200 (e.g., automated assist controller 56212, user interface service 56213, assist data storage device 56214, ML model trainer 56215, operating data storage device 56203, and / or surgical elements 56222, 56224). In the example, detector 56211 may send an indication that an event has occurred to automation assistance controller 56212 (e.g., automation assistance controller 56212 may use this indication to determine whether to automate one or more tasks associated with the surgery). In the example, detector 56211 may send an indication that an event has occurred to user interface service 56213, causing user interface service 56213 to generate an alert for the event to HCP 56208. In the example, detector 56211 may send an indication that an event has occurred (e.g., along with operational data) to ML model trainer 56215 as training data for training an ML model to detect one or more events.

[0095] Adaptive recognition events may include the determination of relationships (e.g., as determined by detector 56211). Detector 56211 may determine relationships between the first surgical element 56222, the second surgical element 56224, the patient 56209, and / or the HCP 56208. Detector 56211 may determine relationships based on operational data (e.g., data generated during surgery and / or data generated based on historical operational data). Detector 56211 may determine relationships, for example, in response to user input (e.g., user input from the HCP 56208 via user interface service 56213). Detector 56211 may generate a lookup table including one or more determined relationships and / or determine relationships based on analysis of the lookup table (e.g., by referencing the lookup table based on the surgery selected by the HCP 56208).

[0096] As an illustrative example, detector 56211 may determine a relationship between core body temperature (e.g., a set point measured by surgical element 56222), the ventilator (e.g., surgical element 56224), and / or a heating blanket (e.g., one or more additional surgical elements not illustrated as part of the operating environment 56200) (e.g., based on analysis of operational data). Detector 56211 may determine the relationship, for example, based on data indicating that if the tidal volume of air supplied to patient 56209 by the ventilator is increased, the power output of the heating blanket may be increased to maintain the patient's core body temperature.

[0097] Adaptive event recognition may include determining whether physiological parameters meet a threshold (e.g., as determined by detector 56211). Detector 56211 may receive data associated with the patient's physiological parameters from surgical elements 56222, 56224 and compare that data with a threshold. The threshold may be determined by system 56210, by HCP 56208 (e.g., received via user input), by an ML model, and / or based on operational data associated with patient 56209. In the example, detector 56211 may determine that an event has occurred based on the condition that one, two, three, four, and / or more physiological parameters meet the threshold. Detector 56211 may determine one or more aspects of the threshold based on operational data (e.g., data associated with the surgery, aspects of surgical elements 56222, 56224 available during the surgery, HCP 56208, and / or patient 56209). Aspects of the threshold may include the type of threshold (e.g., heart rate, SpO2, etc.), the threshold (e.g., 90 degrees Fahrenheit for core body temperature, range for heart rate, etc.), and / or the number of thresholds (1, 2, 3, 4, etc.) which may vary depending on the procedure.

[0098] System 56210 may include an automation assistance controller 56212 (referred to herein as "controller 56212"). Controller 56212 may be, for example, a processor, controller, FPGA, ASIC, etc., configured to perform one or more actions as described herein. Controller 56212 may receive data from one or more components of operating environment 56200 (e.g., via network 56201). For example, controller 56212 may receive operational data from surgical elements 56222, 56224, operational data storage device 56203, and / or receive indications of adaptive recognition events from detector 56211. Controller 56212 may send data to one or more components. In the example, controller 56212 may send requests for information to surgical elements 56222, 56224, requests for operational data to operational data storage device 56203, requests for data to HCP 56208 (e.g., via user interface service 56213), etc.

[0099] Controller 56212 can determine automation strategies to reduce manual tasks performed by the HCP, improve efficiency and patient safety, and / or further optimize the procedure. Controller 56212 can determine automation strategies before and / or during the procedure (e.g., during the procedure itself, based on requests from HCP 56208, system-generated requests, and / or based on actions, events, etc.). As part of the automation assistance strategy, controller 56212 can instruct (e.g., via automation assistance parameters) a first set of tasks that can be automated by surgical elements 56222, 56224 during the first part of the procedure, and / or can instruct (e.g., via second automation assistance parameters) a second set of tasks that can be automated during the second part of the procedure.

[0100] The controller 56212 can determine an automation strategy based on operational data and / or auxiliary data. In the example, the controller 56212 can use operational data and / or auxiliary data to determine whether tasks associated with the surgery can be automated (e.g., based on the availability of surgical components 56222, 56224, etc.).

[0101] As described herein, auxiliary data may include an ML model, one or more tasks associated with the surgery, attributes associated with the tasks, and / or instructions for performing the tasks. Controller 56212 may send operational data and / or auxiliary data as input to the ML model to determine an automation strategy (e.g., to determine automation auxiliary parameters as described herein). Controller 56212 may transmit instructions to surgical elements 56222, 56224 to automate the tasks associated with the surgery. Instructions may include setpoints, output characteristics (e.g., power levels, RPM of the cutting tool, position of the robotic arm, etc.), instructions for performing the tasks, etc.

[0102] An automation strategy can be determined in response to receiving an indication of an adaptive recognition event (e.g., from detector 56211). For example, controller 56212 can determine the strategy based on an indication of a determined relationship and / or based on an assessment of the risk to patient safety (e.g., an indication that the physiological parameters of patient 56209 have met a life-threatening threshold). In the example, controller 56212 can determine a first strategy for the surgery that automates a first part of the surgery (e.g., a task, multiple tasks, etc.). The first strategy may include a first set of tasks to be automated by system 56210 based on user input from HCP 56208 (e.g., HCP 56208 may choose to perform the first set of tasks manually while choosing to automate a second set of tasks during the surgery). Controller 56212 can instruct (e.g., display the automation assistance strategy via user interface service 56213) that HCP 56208 is performing the first set of tasks.

[0103] Controller 56212 may receive instructions for one or more relationships as described herein. Relationships may be determined based on data received, for example, from surgical elements 56222 and 56224. If controller 56212 receives instructions for relationships during surgery (e.g., surgical element 56222 may be unreliable and / or malfunctioning), controller 56212 may dynamically determine an automation strategy that enables HCP 56208 to continue performing tasks associated with a first automation strategy and / or transmit instructions to HCP 56208 that previously automated tasks may not be automatable based on relationships.

[0104] Controller 56212 can receive indications that physiological parameters have met thresholds (e.g., events) as described herein. Based on the received events, controller 56212 can determine an automation strategy (e.g., that increases the level of automation associated with the surgery) such that HCP 56208 can direct its focus, attention, and / or attention toward the strategic portion of the surgery. As an illustrative example, if the patient's core body temperature decreases below a threshold (e.g., if a life-threatening event is detected), controller 56212 can determine (e.g., select) an automation strategy that fully automates one or more tasks (e.g., to stabilize and / or correct the patient's core body temperature) associated with the control of the output characteristics of the ventilator and / or heating blanket. Due to the determined automation strategy, the surgery can be performed more effectively because the surgeon can continue to focus their attention on removing tissue from the tumor located on the patient's heart, rather than stopping the surgery, and direct their attention toward stabilizing and / or maintaining the patient's core body temperature.

[0105] Controller 56212 may determine an automation strategy based on one or more ML models. In the example, controller 56212 may send operational data, instructions for events, and / or auxiliary data as input to the ML model (e.g., stored on auxiliary data storage device 56214). In response, controller 56212 may receive output from the ML model, including automation auxiliary parameters (e.g., instructions such as those for surgical element 56222 to automate a task). Controller 56212 may transmit data (e.g., operational data, automation auxiliary parameters, instructions for events, etc.) to ML model trainer 56215 for use as training data. The data may be used to generate training data to train one or more ML models (e.g., as part of ML model trainer 56215 as described herein).

[0106] Automation auxiliary parameters may indicate the task and / or attributes associated with the task to be automated during surgery. For example, automation auxiliary parameters may indicate a task identifier, surgery, start / end time of the task, duration, surgical elements 56222 and 56224 assigned to perform the task, task priority, task risk value, automation level identifier, etc. Controller 56212 may (e.g., via user interface service 56213) display the automation auxiliary parameters to HCP 56208, transmit the automation auxiliary parameters to surgical elements 56222 and 56224, and / or store the automation auxiliary parameters (e.g., in auxiliary data storage device 56214 and / or operational data storage device 56203). As an illustrative example, automation auxiliary parameters may include surgical element identifiers (serial number, port number, model, etc.), instructions (e.g., instructions for the surgical element to automate the task), task start / stop time, setpoint, and / or output characteristics (e.g., power level, RPM of the cutting tool, position of the robotic arm, etc.).

[0107] In the example, auxiliary parameters may include an automation level identifier. In the example, controller 56212 may instruct surgical elements 56222, 56224 to automate tasks including the automation level identifier. In the example, the number of automated tasks increases as the automation level identifier increases. The automation level identifier may be a number associated with a set of tasks to be automated by the surgery (e.g., 1, 2, 3, 4, etc., where 1 includes only a few automated tasks, while 4 fully automates the tasks associated with the surgery). In the example, the automation level identifier may include a name (e.g., assistant, partner, interviewer, acting as a substitute, etc.).

[0108] Controller 56212 may determine tasks that can be automated and then allow a user (e.g., HCP 56208) to select whether to automate the task via user interface service 56213. In an example, HCP 56208 may choose to manually perform the indicated task and / or request system 56210 to obtain additional operational data based on the manual execution of the task. In an example, system 56210 may receive confirmation that the first task has been successfully automated before determining a second task to be automated. As an illustrative example, system 56210 may perform suturing multiple times to develop a basic level of capability in the suturing (e.g., as determined by HCP 56208). Before determining the second automated task, system 56210 may instruct corresponding elements for arm positioning, etc.

[0109] System 56210 may provide a level of automation associated with supporting HCP 56208 during surgery (e.g., as indicated in automation assistance parameters) based on the surgeon's perspective. For example, a robotic system (e.g., surgical element 56222) may provide a lower level of automation for opening and closing the patient. However, for intraoperative laparoscopic anatomy, the robotic system may provide a higher level of automation (e.g., system 56210 may determine a second automation assistance parameter based on one or more tasks associated with the surgery), and then for suturing actions, the robotic system may provide a lower level of support.

[0110] In the example, automation assist parameters can indicate instructions associated with routine and / or complex tasks. For instance, in suturing, automation assist parameters can indicate simple suturing scenarios as well as more complex suturing scenarios based on tissue, anatomical pathways, and other constraints (e.g., based on operational data).

[0111] This document describes monitoring user-controlled actions to suggest automating actions based on situational awareness. In an example, system 56210 (e.g., controller 56212) can monitor HCP 52608 utilization, surgical procedures, decisions, etc., to detect behavioral patterns, enabling system 56210 to suggest tasks (e.g., secondary operations) that system 56210 can perform (e.g., generating surgical aid parameters as described herein to enable surgical elements 56222, 56224 to perform surgery-related tasks). The suggested tasks can be automatically performed by surgical elements 56222, 56224 to alleviate repetitive tasks, preparations, or setups from HCP 56208.

[0112] In the example, system 56210 can determine learning based on automation aid parameters across surgeons (e.g., determining relationships between data associated with surgeon actions). As an example, system 56210 can adapt (e.g., determine relationships) based on data collected from right-handed and / or left-handed surgeons (e.g., system 56210 can determine relationships based on operational data including whether HCP 56208 is right-handed or left-handed). By learning across multiple surgeons, system 56210 can determine general terms (e.g., relationships between HCP 56208, surgical elements 56222, 56224) and / or surgeon-specific preferences or nuances. The tasks that system 56210 can automate may be surgeon-agnostic. In the example, one or more levels of automation can be described and / or assigned based on automation aid parameters. In the example, lower levels of automation can be shared without posing additional risks to the patient and / or during surgery. In the example, operational data may include surgeon-specific data. To minimize the conflict and / or risk of differences from numerous surgeons, learning may not be shared across multiple surgeons (e.g., if system 56210 determines a relationship that may pose a risk to the patient and / or the surgery, the relationship may not be determined across one or more surgeons).

[0113] As described herein, a database of existing surgeon cases to be built for training the model may be included (e.g., in an auxiliary data storage device, an operational data storage device, etc.). Existing surgical videos and / or surgeries may be included and / or utilized (e.g., as part of operational data) to allow system 56210 to view and / or understand how to improve responses (e.g., determine automated auxiliary parameters). In the example, if the surgeon is using multiple systems (e.g., a first surgical element 56222 and a second surgical element 56224), preferences and insights (e.g., operational data) from the first surgical element 56222 may be shared with the second surgical element 56224.

[0114] The controller 56212 can determine automation assistance parameters based on surgical plans, the sequence of operations or instruments, the order of steps in using instruments, and / or responses to repetitive problems and / or constraints, as well as other inputs described herein. Examples described herein can be used to identify and / or highlight the actions of surgical elements 56222, 56224, which may have sequence or event triggers, so that the system 56210 can then initiate and complete the task based on its recognition of the triggers or sequences. (For example, the system 56210 can determine / detect adaptive recognition events based on operational data described herein.)

[0115] As an illustrative example, the skeletalization or orientation of the mesentery surrounding the colon during sigmoid colectomy is used to release the colon from its anchorage into the body. This involves inserting a double-clamping device into translucent connective tissue to form an opening, and then using a spring to enlarge the opening. Once sufficient openings have been formed around arteries and / or capillaries (e.g., which are opaque and red), the device is clamped onto the tissue. Once clamped, energy is used to coagulate and transect the arteries. Advanced visualization systems (e.g., surgical element 56222) can be used to monitor the tissue with which the double-clamping system (e.g., surgical element 56224) is interacting. The double-clamping system can use tissue impedance to determine whether the double-clamping system is interacting with tissue outside the clamps or between the clamps. System 56210 can monitor surgical planning and / or instruments in use to communicate that this is a collection procedure and that the surgeon is at the planned mesenteric anatomical step. Energy generation may not occur automatically when the clamps are closed for insertion and there is no tissue between the clamps. When the jaws are closed with tissue present and the visualization system identifies the artery, the generator can be automatically activated to cauterize the artery. In the example, system 56210 can determine that if the jaws compress connective tissue, the energy may not be activated. Operational data from system 56210 and the advanced visualization system (e.g., surgical element 56222) can provide the energy generator (e.g., surgical element 56224) with data on the tissue type, which can be combined with data identifying the tissue location (e.g., tissue impedance) derived from the generator itself. The operational data (e.g., tissue type, tissue impedance, tissue location data, etc.) can then be used to trigger automatic activation of the energy (e.g., to generate automated auxiliary parameters including an indication to trigger energy activation), rather than waiting for the surgeon to activate control (e.g., controlling energy activation via a manual process).

[0116] System 56210 may include user interface service 56213. User interface service 56213 allows system 56210 to interact with a user (e.g., HCP 56208). User interface service 56213 can generate a GUI displayed by system 56210 and / or by surgical elements 56222, 56224. User interface service 56213 can receive data from system 56210 and / or surgical elements 56222, 56224 and / or send data to various other components of operating environment 56200. User interface service 56213 can generate a GUI to display data from one or more components of operating environment 56200, including, for example, operational data (e.g., patient physiological parameters), indications that an event has occurred, criteria associated with the event (e.g., relationships and / or thresholds), a description of the task, attributes associated with the task, etc. User interface service 56213 can request user input from HCP 56208. In the example, user interface service 56213 may display requests for information associated with surgical elements 56222, 56224 (e.g., identification, function, capability, output, input, etc.), requests associated with automation strategies (e.g., approval of determined automation strategies, modification of automation strategies, selection of automation auxiliary parameters, etc.), requests for tasks to be performed and / or surgery, requests for information related to patient 56209, etc.

[0117] System 56210 may include auxiliary data storage device 56214. Auxiliary data storage device 56214 may store operational data, training data, automated auxiliary parameters, ML models, etc., as described herein. In the example, system 56210 may send data to auxiliary data storage device 56214 in response to user input from HCP 56208 (e.g., via user interface service 56213).

[0118] System 56210 may include an ML model trainer 56215. The ML model trainer 56215 may receive training data from controller 56212. The training data may include operational data, automation auxiliary parameters, indications of events, auxiliary data, etc. In response to receiving the training data, the ML model trainer 56215 may train an ML model to detect events, determine one or more tasks that can be automated by surgical elements 56222, 56224, and / or determine automation auxiliary parameters, etc.

[0119] ML model trainer 56215 can train an ML model to detect adaptive recognition events. The ML model can be used to determine whether physiological parameters meet thresholds and / or to determine relationships between surgical elements 56222 and 56224. As an illustrative example, the ML model can receive instructions for surgery and / or identification of the first surgical element 56222 (e.g., a heating blanket) and the second surgical element 56224 (e.g., a ventilator). The ML model can determine the relationship, for example, based on data indicating that if / when the ventilator increases the tidal volume of air to the patient, the power output of the heating blanket is increased to maintain the patient's core body temperature. The ML model can send instructions on the relationship to controller 56212 (e.g., and / or controller 56212 can adjust automation strategies based on the relationship). For example, controller 56212 can instruct surgical element 56222 to automate one or more tasks based on the relationship received from the ML model.

[0120] ML model trainer 56215 can train an ML model to determine one or more tasks that can be automated by surgical elements 56222, 56224. As an illustrative example, the ML model can receive operational data (e.g., historical data and / or real-time data during surgery) from surgical elements 56222, 56224, HCP 56208, auxiliary data storage device 56214, etc. Operational data may include, for example, position data associated with the movement of the robotic arm during the task (e.g., position data output from surgical element 56222 controlled by HCP 56208). The ML model can determine instructions based on the received position data that enable the robotic arm to automatically perform one or more movements associated with the task. Instructions may be stored, for example, in auxiliary data storage device 56214. Instructions may be associated with automation auxiliary parameters (e.g., automation auxiliary parameters may include instructions to cause surgical element 56222 to automatically perform the task).

[0121] The ML model trainer 56215 can train an ML model to determine automation assistance parameters. As described herein, automation assistance parameters can indicate the task to be automated during surgery and / or one or more attributes associated with the task (e.g., task identifier, surgery, start / end time of the task, duration, surgical elements 56222, 56224 used to perform the task, task priority, task risk value, automation level identifier, etc.).

[0122] Operating environment 56200 may include surgical element 56222 and / or surgical element 56224. Although both surgical elements are depicted as part of operating environment 56200, this is not intended to be limiting, as multiple surgical elements may be operable and / or communicate with one or more components of operating environment 56200. Surgical elements 56222 and 56224 may include ( Figure 6(Not shown in the example) For reference Figure 5 The sensors, instruments, and / or tools described in 20282. Surgical elements 56222 and 56224 may be operated by HCP 56208 and / or autonomously operated (e.g., based on automation strategies) to perform one or more tasks associated with surgery. As illustrative examples, surgical elements 56222 and 56224 may include robotic surgical systems, navigation systems, intelligent imaging systems, endoscopic and / or laparoscopic systems, harmonic scalpels, anesthesia machines, patient monitoring systems (pulse oximeters, blood pressure monitors, EKG monitors, EEG monitors, etc.), energy devices (e.g., electrosurgical units, laser surgical systems, etc.), infusion pumps, etc.

[0123] In the example, surgical elements 56222 and 56224 may be wirelessly and / or physically connected (e.g., wired) to network 56201, system 56210, and / or another surgical element 56224. Surgical elements 56222 and 56224 may generate operational data. Surgical elements 56222 and 56224 may transmit operational data to system 56210 (e.g., to determine automation strategies), operational data storage device 56203, etc. For example, surgical elements 56222 and 56224 may transmit / receive environmental data, setpoints, output characteristics, measured variables, communication data, and / or user data, as described herein.

[0124] As an illustrative example, surgical element 56222 may be a heating blanket. The heating blanket may include a temperature measuring device and / or a heating element. The heating blanket may be configured to activate the heating element based on user input (e.g., in an open loop) and / or based on automated auxiliary parameters (e.g., in a closed loop) to maintain the patient's core body temperature during open-heart surgery. The heating blanket may send core body temperature measurements (e.g., as operational data) to system 56210. Surgical element 56222 may receive automated auxiliary parameters (e.g., if system 56210 determines that an event has occurred, such as an established relationship and / or that the core body temperature has met a life-threatening threshold). Automated auxiliary parameters may include the heating blanket's setpoint, the heating blanket's output power level, the duration for maintaining the power level, and instructions for the heating blanket to operate automatically. Advantageously, the heating blanket may receive instructions to automatically control the patient's core body temperature based on the occurrence of an event, allowing HCP 56208 to direct its focus, concentration, and / or attention to the strategic aspects of the surgery.

[0125] Operating environment 56200 may include operating data storage device 56203. Operating data storage device 56203 may be an external data storage device (e.g., located on a third-party server). Operating data storage device 56203 may store operating data, training data, automation auxiliary parameters, auxiliary data, etc. In the example, system 56210 may send data to operating data storage device 56203 in response to user input from HCP 56208 (e.g., via user interface service 56213). Although operating data storage device 56203 is depicted as external to system 56210, this is not intended as a limitation. In the example, operating data storage device 56203 may be a database residing on surgical elements 56222, 56224. In the example, data associated with the operational data storage device 56203 may be transmitted from surgical elements 56222, 56224 to system 56210 (e.g., detector 56211, controller 56212, user interface service 56213, auxiliary data storage 56214 and / or ML model trainer 56215) via network 56201.

[0126] Operating environment 56200 may include network 56201. Network 56201 may include one or more communication networks, such as the Internet. Network 56201 may be any combination of a local area network (“LAN”) and / or a wireless local area network (“WAN”). Therefore, various components of operating environment 56200 (including system 56210, surgical components 56222, 56224 and / or operating data storage device 56203) may communicate directly or indirectly with each other via any suitable communication link and / or network (such as network 56201 (e.g., one or more communication links, one or more computer networks, one or more wired or wireless connections, the Internet, etc.)). Example flowcharts for identifying tasks to be automated

[0127] Figure 7A This is an example of... Figure 6 The flowchart illustrates an example operation in which components of the operating environment 56200 perform actions to determine whether to automate one or more tasks based on inputs from surgical elements 56222, 56224, HCP 56208, and / or operating data storage device 56203.

[0128] like Figure 7AAs illustrated, at (1), HCP 56208, surgical elements 56222, 56224, and / or operational data storage device 56203 transmit operational data and / or automated auxiliary parameters to detector 56211. As described herein, operational data may include past and / or current information associated with the operational environment 56200, such as data generated based on one or more surgeries (e.g., past and / or current surgeries). In the example, operational data may include: patient physiological parameters (e.g., patient's heart rate, blood pressure, SpO2, core body temperature, etc.); environmental data (e.g., the number of HCPs in the operating room, historical records of HCPs 56208 performing surgery, location data associated with the movement of HCPs 56208 during surgery, hospital-related data such as the number of surgeries performed and / or the results of surgeries, or the functions associated with surgical elements 56222, 56224 available during surgery, etc.); data associated with surgical elements 56222, 56224 (e.g., setpoints, output characteristics, measured variables, communication data, location data, etc.); user data (e.g., data generated based on user input from HCPs 56208 via user interface service 56213), etc.

[0129] Automation auxiliary parameters may include surgical component identification (serial number, port number, model, etc.), instructions (e.g., instructions that enable the surgical component to automate a task and / or instructions associated with performing a task), task start / stop time, setpoint and / or output characteristics (e.g., power level, RPM of the cutting tool, position of the robotic arm, etc.).

[0130] As an illustrative example, HCP 56208 may send automated auxiliary parameters via user interface service 56213, which instruct surgical element 56222 to maintain the patient’s core body temperature during a portion of the procedure (e.g., surgical element 56222 will operate in automatic mode while the surgeon removes tissue from the patient’s heart during open-heart surgery).

[0131] Once detector 56211 receives operational data and / or automated auxiliary parameters, detector 56211 can determine the relationship between the first surgical element and the second surgical element at (2), and / or determine at (3) whether physiological parameters meet a life-threatening threshold (e.g., determine an adaptive recognition event). For example, detector 56211 can determine the relationship between the first surgical element 56222, the second surgical element 56224, the patient 56209, and / or the HCP 56208. Detector 56211 can generate a lookup table including one or more determined relationships and / or determine the relationships based on analysis of the lookup table (e.g., by referencing the lookup table based on the surgery selected by HCP 56208). As an illustrative example, detector 56211 can determine the relationship between core body temperature (e.g., a setpoint measured by surgical element 56222), a ventilator (e.g., surgical element 56224), and / or a heating blanket (e.g., one or more additional surgical elements not illustrated as part of the operating environment 56200) (e.g., based on analysis of operational data). The detector 56211 can determine a relationship based on data indicating that if the tidal volume of air supplied to the patient 56209 by the ventilator is increased, the power output of the heating blanket can be increased to maintain the patient's core body temperature.

[0132] Detector 56211 can determine that an event has occurred based on whether physiological parameters meet a threshold (e.g., a life-threatening threshold). The threshold can be determined by system 56210, by HCP 56208 (e.g., received via user input), by an ML model, by detector 56211, and / or based on operational data associated with patient 56209. In the example, detector 56211 can determine that an event has occurred based on the condition that one, two, three, four, and / or more physiological parameters meet the threshold. Detector 56211 can determine one or more aspects of the threshold based on operational data (e.g., data associated with the surgery, aspects of surgical components 56222 and 56224 available during the surgery, HCP 56208, and / or patient 56209). Aspects of the threshold may include threshold type (e.g., heart rate, SpO2, etc.), threshold (e.g., 90 degrees Fahrenheit for core body temperature, range for heart rate, etc.), and / or the number of thresholds (1, 2, 3, 4, etc.) which may vary depending on the surgery.

[0133] Based on the determination of the relationship between the first surgical element 56222 and the second surgical element 56224 (e.g., and / or HCP 56208, patient 56209, etc.) at (2), or based on the determination that the physiological parameters meet the threshold at (3), the detector 56211 may determine at (4) that an adaptive recognition event has occurred.

[0134] After detector 56211 has determined that an adaptive recognition event has occurred, detector 56211 may send an indication of the adaptive recognition event to controller 56212 at (5). The indication of the adaptive recognition event may include information associated with the determined relationship and / or physiological parameters of patient 56209 that meet the thresholds described herein.

[0135] Once the controller 56212 receives an indication of an adaptive recognition event, it can retrieve auxiliary data from the auxiliary data storage device 56214 at (6). As described herein, the auxiliary data may include an ML model, one or more tasks associated with the surgery, attributes associated with the tasks, and / or instructions for performing the tasks. Optionally, the controller 56212 may send operational data, indications of adaptive recognition events, and / or automation auxiliary parameters to the auxiliary data storage device 56214 as input to the ML model (the ML model stored in the auxiliary data storage device) to determine automation strategies (e.g., to determine automation auxiliary parameters as described herein).

[0136] Controller 56212 may determine an automation strategy at (7). The automation strategy may be determined in response to receiving an indication of an adaptive recognition event (e.g., from detector 56211), automation assistance parameters, and / or operational data. For example, controller 56212 may determine a strategy (e.g., including automation assistance parameters) based on an indication of a determined relationship and / or based on an assessment of the risk to patient safety (e.g., an indication that the physiological parameters of patient 56209 have met a life-threatening threshold). In the example, controller 56212 may determine a first strategy for the procedure that automates a first part of the procedure (e.g., a task, multiple tasks, etc.). The first strategy may include a first set of tasks that can be automated by system 56210 based on user input from HCP 56208 (e.g., HCP 56208 may choose to perform the first set of tasks manually while simultaneously choosing to automate a second set of tasks during the procedure).

[0137] The controller 56212 may send a determined automation strategy (e.g., automation auxiliary parameters) to surgical elements 56222, 56224 at (8). As described herein, the sent strategy may include automation auxiliary parameters. Automation auxiliary parameters provide instructions for performing tasks associated with the surgery. In the example, automation auxiliary parameters may instruct surgical element 56222 to fully automate the surgery. In the example, automation auxiliary parameters may include setpoints, output characteristics (e.g., power levels, RPM of cutting tools, position of robotic arms, etc.), instructions / commands for performing tasks, etc. Automation auxiliary parameters may indicate tasks and / or attributes associated with tasks to be automated during surgery. For example, automation auxiliary parameters may indicate task identifiers, surgery, start / end time of task execution, duration, surgical elements 56222, 56224 assigned to perform the task, task priority, task risk value, automation level identifier, etc.

[0138] The controller 56212 may send an automation policy message to the user interface service 56213 at (9). The automation policy message may instruct and / or direct the HCP 56208 to perform a first set of tasks and instruct the surgical element 56222 to perform a second set of tasks. Optionally, the user interface service 56213 may receive user input authorizing the selected automation tasks (e.g., tasks to be performed manually by the HCP 56208 and / or automatically by the surgical elements 56222, 56224) in response to displaying the automation policy message to the HCP 56208. Example flowchart for training an ML model

[0139] Figure 7B This is an example of... Figure 6 The flowchart illustrates an example operation in which components of the operating environment 56200 execute to train an ML model to determine automated auxiliary parameters and / or automated strategies during surgery based on inputs from HCP56208, surgical elements 56222, 56224 and / or operating data storage device 56203.

[0140] like Figure 7B As illustrated, controller 56212 may obtain operational data and / or automation auxiliary parameters from HCP 56208, surgical elements 56222, 56224 and / or operational data storage device 56203 at (1). As described herein, controller 56212 may receive operational data and / or automation auxiliary parameters from surgical elements 56222, 56224, operational data storage device 56203, HCP 56208, etc.

[0141] The controller may generate training data at (2). Training data may be generated based on operational data and / or automated auxiliary parameters. Optionally, the training data may include operational data, automated auxiliary parameters, indications of events (e.g., adaptive event recognition), auxiliary data, etc. In response to receiving the training data, the ML model trainer 56215 may train an ML model to detect events, determine one or more tasks that can be automated by surgical elements 56222, 56224, and / or determine automated auxiliary parameters, etc.

[0142] Once the training data has been generated, the controller 56212 can send the training data to the ML model trainer 56215 at (3).

[0143] ML model trainer 56215 may receive training data and train a model at (4) to generate instructions for automating a task. As used herein, ML model trainer 56215 may train an ML model to determine one or more tasks that can be automated by surgical elements 56222, 56224. As an illustrative example, ML model trainer 56215 may receive training data including position data associated with the movement of the robotic arm during a task (e.g., position data output from surgical element 56222 controlled by HCP 56208). ML model trainer 56215 may train an ML model to determine, based on the received position data, instructions that enable the robotic arm to automate one or more movements associated with the task.

[0144] The ML model trainer 56215 can receive training data and train an ML model at (5) to select an automation strategy. For example, the ML model trainer 56215 can train an ML model to determine automation auxiliary parameters as part of the automation strategy. As described herein, automation auxiliary parameters may indicate the task to be automated during surgery and / or one or more attributes associated with the task (e.g., task identifier, surgery, start / end time of the task, duration, surgical elements 56222, 56224 used to perform the task, task priority, task risk value, automation level identifier, etc.).

[0145] Optionally, the ML model trainer 56215 can train an ML model based on training data to detect adaptive recognition events. The ML model can be used to determine whether physiological parameters meet thresholds and / or to determine the relationship between surgical elements 56222 and 56224. As an illustrative example, the ML model can be trained to receive instructions for surgery and / or identification of the first surgical element 56222 (e.g., a heating blanket) and the second surgical element 56224 (e.g., a ventilator). The ML model can be trained to determine the relationship, for example, based on data indicating that if the ventilator increases the tidal volume of air supplied to the patient, the power output of the heating blanket can be increased to maintain the patient's core body temperature. The output of the first ML model (e.g., trained to detect adaptive recognition events) can be passed to a second ML model trained to determine an automation strategy. For example, the ML model trainer 56215 can train the second ML model to receive instructions for adaptive recognition events and instruct the surgical element 56222 to automate one or more tasks based on the detected events.

[0146] After the ML model has been trained by the ML model trainer 56215, the ML model trainer 56215 may send the trained ML model to the controller 56212 at (6). Additionally and / or optionally, the ML model trainer 56215 may send the trained ML model to another component of the operating environment 56200 (e.g., auxiliary data storage device 56214, operating data storage device 56203, etc.).

[0147] Once the controller 56212 receives the trained ML model from the ML model trainer 56215, the controller 56212 can apply the operation data, auxiliary data, etc. as input to the trained ML model at (7), which causes the trained ML model to output automated auxiliary parameters and / or instructions for automating tasks. Example aspects related to adaptive recognition detector routines

[0148] Figure 8 This is a flowchart of the adaptive recognition detector routine 56280, which is exemplarily implemented by system 56210. As an example, Figure 6 The system 56210 (e.g., detector 56211 and / or controller 56212) can be configured to execute adaptive identification detector routine 56280. Routine 56280 begins at block 56281.

[0149] At box 56281, system 56210 can obtain operational data. As described herein, system 56210 can obtain operational data from surgical elements 56222, 56224, HCP 56208, and / or operational data storage device 56203. Operational data may include past and / or current information associated with the operational environment 56200. In this example, operational data may include patient physiological parameters, environmental data, data associated with surgical elements 56222, 56224, user data, etc.

[0150] At box 56282, system 56210 may obtain automation auxiliary parameters for the first surgical element 56222. System 56210 may obtain automation auxiliary parameters from surgical elements 56222, 56224, from HCP 56208 (e.g., via user interface service 56213), from operational data storage device 56203, from auxiliary data storage device 56214, and / or from another component of the operating environment 56200. In the example, system 56210 may determine the first automation auxiliary parameters based on operational data. As described herein, automation auxiliary parameters may indicate a task and / or attributes associated with the task to be automated during surgery. For example, automation auxiliary parameters may indicate a task identifier, surgery, start / end time of task execution, duration, surgical elements 56222, 56224 assigned to perform the task, task priority, task risk value, automation level identifier, etc. In the example, controller 56212 may (e.g., via user interface service 56213) display automated auxiliary parameters to HCP 56208, transmit automated auxiliary parameters to surgical elements 56222, 56224, and / or store automated auxiliary parameters (e.g., in auxiliary data storage device 56214 and / or operational data storage device 56203).

[0151] As an illustrative example, automation assistance parameters may include surgical component identification (serial number, port number, model, etc.), instructions (e.g., instructions for the surgical component to automate a task), task start / stop time, setpoint, and / or output characteristics (e.g., power level, RPM of the cutting tool, position of the robotic arm, etc.).

[0152] In the example, auxiliary parameters may include an automation level identifier. In the example, controller 56212 may instruct surgical elements 56222, 56224 to automate tasks including the automation level identifier. In the example, the number of automated tasks increases as the automation level identifier increases.

[0153] At box 56283, system 56210 can detect adaptive recognition events. As described herein, adaptive recognition events may include the determination of relationships and / or the determination that physiological parameters meet thresholds (e.g., as determined by detector 56211).

[0154] Optionally, at box 56283a, system 56210 (e.g., detector 56211) may determine relationships between the first surgical element 56222, the second surgical element 56224, the patient 56209, and / or the HCP 56208. Detector 56211 may determine relationships based on operational data (e.g., data generated during surgery and / or data generated based on historical operational data). System 56210 may determine relationships, for example, in response to user input (e.g., user input from the HCP 56208 via user interface service 56213). System 56210 may generate a lookup table including one or more determined relationships and / or determine relationships based on analysis of the lookup table (e.g., by referencing the lookup table based on the surgery selected by the HCP 56208).

[0155] As an illustrative example, detector 56211 may determine a relationship between core body temperature (e.g., a set point measured by surgical element 56222), the ventilator (e.g., surgical element 56224), and / or a heating blanket (e.g., one or more additional surgical elements not illustrated as part of the operating environment 56200) (e.g., based on analysis of operational data). Detector 56211 may determine the relationship, for example, based on data indicating that if / when the tidal volume of the air supplied to patient 56209 by the ventilator is increased, the power output of the heating blanket increases to maintain the patient's core body temperature.

[0156] Optionally, at block 56283b, system 56210 (e.g., detector 56211) may determine that a physiological parameter meets a threshold (e.g., as determined by detector 56211). As described herein, system 56210 may receive data associated with a patient's physiological parameters from surgical elements 56222, 56224 and compare that data with a threshold. The threshold may be determined by system 56210, by HCP 56208 (e.g., received via user input), by an ML model, and / or based on operational data associated with patient 56209. In the example, system 56210 may determine that an event has occurred based on the condition that one, two, three, four, and / or more physiological parameters meet the threshold.

[0157] System 56210 can determine one or more aspects of a threshold based on operational data (e.g., data associated with the surgery, data from surgical components 56222, 56224, HCP 56208, and / or patient 56209 available during the surgery) and / or automated auxiliary parameters. In the example, aspects of the threshold may include threshold type (e.g., heart rate, SpO2, etc.), threshold (e.g., 90 degrees Fahrenheit for core body temperature, range for heart rate, etc.), and / or the number of thresholds (1, 2, 3, 4, etc.) which may vary for the specific surgery.

[0158] At box 56284, system 56210 may determine a second automation assistance parameter. The second automation assistance parameter may be determined based on operational data, adaptively recognized events, assistance data, user input (e.g., by HCP 56208 via user interface service 56213), and / or based on an ML model (e.g., trained by ML model trainer 56215). The automation assistance parameter may include instructions for performing tasks associated with the surgery. In the example, an automation strategy may instruct surgical element 56222 to fully automate the surgery. In the example, the automation assistance parameter may include a setpoint, output characteristics (e.g., power level, RPM of the cutting tool, position of the robotic arm, etc.), instructions / commands for performing the task, etc. The automation assistance parameter may indicate the task and / or attributes associated with the task to be automated during the surgery. For example, the automation assistance parameter may indicate a task identifier, surgery, start / end time of task execution, duration, surgical elements 56222, 56224 assigned to perform the task, task priority, task risk value, automation level identifier, etc.

[0159] In the example, controller 56212 may determine first automation aid parameters for the surgery, which automate a first part of the surgery (e.g., tasks, multiple tasks, etc.). The first parameter may include a first set of tasks that can be automated by system 56210 based on user input from HCP 56208 (e.g., HCP 56208 may choose to perform the first set of tasks manually while simultaneously automating a second set of tasks during the surgery).

[0160] Optionally, automation auxiliary parameters can be determined by the ML model. As described herein, the ML model trainer 56215 can receive training data and train the ML model to select an automation strategy (e.g., determine a second automation auxiliary parameter). The system 56210 can receive the trained ML model from the ML model trainer 56215 and apply operational data, auxiliary data, etc., to the trained ML model. The trained ML model can output automation auxiliary parameters and / or instructions for automating the task.

[0161] At box 56285, system 56210 may send instructions for a second automation assistance parameter to surgical elements 56222, 56224. As described herein, the instructions for the second automation assistance parameter may include a task identifier, surgery, start / end time of the task, duration, surgical elements 56222, 56224 assigned to perform the task, task priority, task risk value, automation level identifier, etc.

[0162] As an illustrative example, surgical element 56222 may be a heating blanket. The heating blanket may include a temperature measuring device and / or a heating element. The heating blanket may be configured to activate the heating element based on user input (e.g., in an open loop) and / or based on automated auxiliary parameters (e.g., in a closed loop) to maintain the patient's core body temperature during open-heart surgery. Surgical element 56222 may receive automated auxiliary parameters (e.g., if system 56210 determines that an event has occurred, such as an established relationship and / or that the core body temperature has met a life-threatening threshold). The automated auxiliary parameters may indicate the heating blanket's setpoint, the heating blanket's output power level, the duration for maintaining the power level, an indication that the heating blanket will operate automatically, etc.

[0163] Optionally, system 56210 may send indications of automation assistance parameters and / or indications of adaptive recognition events to HCP 56208 (e.g., via user interface service 56213).

[0164] At block 56286, system 56210 enables surgical elements 56222, 56224 to automate tasks based on a second automation auxiliary parameter. In the example, surgical elements 56222, 56224 may receive automation auxiliary parameters that cause surgical elements 56222, 56224 to operate in a fully automatic mode. As an illustrative example, surgical element 56222 (e.g., a smoke extraction component) may receive instructions to automatically extract smoke, fluid, and / or particles generated by the application of therapeutic energy from second surgical element 56224 (e.g., operated by HCP 56208).

[0165] The exemplary examples described herein can provide (e.g., bounded, fully autonomous operation of a first system (e.g., a first surgical element) that collaborates with the operation of another system (e.g., a second surgical element) directly controlled by a surgeon.

[0166] The procedure may include cardiac ablation and remodeling to treat AFIB. An automated RF catheter motion control system (e.g., a second surgical element controlled by the surgeon) can ablate predefined tissue areas at predefined depths, pressures, or power intensity levels. The surgeon can then define the area to be treated and / or provide the desired ablation depth after cardiac mapping.

[0167] The system (e.g., system 56210) can display a mapped path of catheter movement and receive confirmation that a robot (e.g., a first surgical element) sweeps the catheter through the path, thereby maintaining pressure and / or energy balance while compensating for cardiac movement during the process. In the example, automated definition (e.g., defining the surgeon's movement) enables the intelligent system (e.g., 56210) to actively interact with and support the surgeon-controlled instrument movement, while including some restrictions on the movement to prevent collateral damage or unintentional collisions with nearby structures caused by cooperative or antagonistic actions or forces (e.g., system 56210 can determine automated auxiliary parameters to define the robot's movement requested by the surgeon).

[0168] To create sufficient contact and / or pressure for moving the ablation system to induce cauterization and / or ablation (e.g., depending on pressure, power, and energy mode), a user (e.g., a surgeon) can control the ablation electrodes. Using the ablation electrodes, the user can define the electrode's path and / or thrust capability (e.g., to create the required energy density, resulting in sufficient tissue welding or tissue death that an auxiliary system (e.g., a robot) would have to provide) to maintain the required energy density. In the example, the heart can be a sensitive and interconnected structure that may be easily and unintentionally damaged if too much force or differential motion is applied. The robot can monitor surgical images of adjacent structures from system 56210 and / or can monitor its own applied forces relative to forces applied by the user. The sum of forces fixed relative to the heart to surrounding anatomical structures can be limited to the applicable forces and movements (e.g., the robot can apply more and / or less force based on the surgeon's movement of the electrodes to maintain contact during tissue modification, resulting in less and / or more automation in the operating environment 56200, depending on the surgeon's actions). Examples of aspects related to determining automation strategies

[0169] Figure 9 This is an exemplary example of an operating environment 56290 configured to determine an automation strategy. Operating environment 56290 may include a procedure 56291 for atrial fibrillation (AFIB) neuroablation to eliminate the arrhythmia using RF ablation. A smart ventilator 56292 (e.g., a first surgical element) may have preset tidal volume and / or O2 replenishment percentage from initial surgeon settings (e.g., as set by HCP 56208). A smart patient heating system 56294 (e.g., a second surgical element) may have a preset heat transfer rate to the patient to create a slightly hypothermic (34°C–33°C) thermoneutral condition.

[0170] The intelligent ventilator 56292 can be configured to use closed-loop control with a finger-based transcutaneous oxygen sensor to correlate O2 blood gas with O2 replenishment rate. The heating system 56294 can be closed-loop (e.g., operated based on first automated assist parameters) based on a core body temperature measurement of the patient 56209 (e.g., as determined by the heating system 56294). To mitigate potential collateral thermal damage from the RF generator 56296 (e.g., an RF monopolar catheter) during ablation and remodeling of local organs, cooling can be applied to the heart under a controlled cooling thermal load (e.g., via the heating system 56294), which is controlled in a closed-loop manner by the energy density and activation timing of the RF generator 56296.

[0171] The ventilator 56292 can monitor degassing CO2, enabling the system 56210 to measure the percentage of exhaled CO2 in each controlled breath of the patient 56209. During surgery, sedation and mild hypothermia may reduce the patient's O2 metabolism. CO2-to-O2 measurements may drift apart (e.g., as measured by the ventilator 56292 and / or another surgical component). The system 56210 can detect adaptive recognition events if / when at least one measurement (CO2-to-O2 and / or core body temperature) meets a threshold (e.g., a life-threatening threshold for the patient's physiological parameters). The system 56210 can generate an automated strategy including automated support parameters based on core body temperature, RF-generated energy, and / or the setpoint of the heating system 56294 (e.g., operational data). The system 56210 can send automated support parameters to the ventilator 56292, including indications that one of the two measurements is out of sync, instructions for automatically reducing O2 supplementation levels, and / or instructions for automatically increasing tidal volume.

[0172] Advantageously, system 56210 can determine automation strategies based on operational data and / or adaptive recognition events associated with ventilator 56292, heating system 56294, and / or RF generator 56296 to automatically instruct ventilator 56292 to control O2 and / or tidal volume, and allow HCP 56208 to continue ablation and remodeling of the local organ without delaying and / or interrupting the procedure. System 56210 determines automation strategies to effectively reduce procedure time, rather than the conventional method of sending requests to HCP 56208 and waiting for confirmation from HCP 56208 (e.g., for automatic adjustment of O2 supplementation levels by surgical components) and / or having O2 supplementation levels manually adjusted by HCP 56208.

Claims

1. A system for adaptively identifying automation strategies during surgery, the system comprising: Processor, the processor being configured to: Obtain operational data for a first surgical element and a second surgical element, wherein the operational data includes indications of physiological parameters of the patient during the procedure, and multiple automated tasks associated with the procedure; Determine the automation assistance parameters of the first surgical element, wherein the automation assistance parameters are associated with a set of automation tasks performed by the first surgical element during the operation, among the plurality of automation tasks; Detect adaptive recognition events, wherein the adaptive recognition event is at least one of the following: The relationship between the first surgical element and the second surgical element; or The determination that the patient's physiological parameters meet the life-threatening threshold; Based on the adaptive recognition event, a second automated assistance parameter of the first surgical element is selected, wherein the second automated assistance parameter indicates a second set of automated tasks performed by the first surgical element during the surgery; Send the instruction for the second automated assistance parameter to the first surgical element; and The first surgical element performs the second set of automated tasks based on the second set of automated auxiliary parameters.

2. The system according to claim 1, wherein: The first surgical component is a ventilator; The second surgical element is a heating device; The patient's physiological parameters are either the CO2 percentage from the ventilator or the core body temperature from the heating device; The patient's procedure included atrial fibrillation (AFIB) neuroablation; The adaptive recognition event includes determining that the patient's CO2 to O2 or core body temperature meets the life-threatening threshold; and The second automated auxiliary parameter includes instructions for reducing the O2 supplementation level of the ventilator or instructions for increasing the tidal volume of the ventilator.

3. The system according to claim 1, wherein, If the first surgical element or the second surgical element determines that the patient's heart rate exceeds the operating window of 40-120 heartbeats per minute, the patient's oxygen saturation decreases to below 90%, or the patient's core body temperature is below 89 degrees Fahrenheit, then the patient's physiological parameters meet the life-threatening threshold.

4. The system according to claim 1, wherein, The adaptive recognition event includes the determination that the patient's physiological parameters meet the life-threatening threshold, and wherein the selected second automated auxiliary parameter indicates that the second set of automated tasks includes the plurality of automated tasks.

5. The system according to claim 1, wherein, The adaptive recognition event includes the relationship between the first surgical element and the second surgical element, and wherein the selected second automation assistance parameter further includes instructions for performing the second set of automated tasks in response to user input.

6. The system according to claim 5, wherein, The processor is further configured to be able to: Sending an automation policy message to the user, wherein the automation policy message includes a request to execute the second set of automation tasks; and Receive user input including instructions for performing at least one of the automated tasks in the second group of automated tasks.

7. The system according to claim 1, wherein, The processor is further configured to be able to: Receive location data associated with the user's movements during the procedure and control data associated with the first and second surgical elements; as well as The relationship between the first surgical element and the second surgical element is determined based on the location data, the operational data, and the patient's physiological parameters.

8. The system according to claim 1, wherein, The processor is further configured to be able to: Based on the adaptive recognition event, the automated auxiliary parameters of the second surgical element are selected, wherein the automated auxiliary parameters of the second surgical element indicate a set of automated tasks to be performed by the second surgical element during the operation; Sending instructions to the second surgical element regarding the automated auxiliary parameters of the second surgical element; and The second surgical element performs the set of automated tasks based on the automated auxiliary parameters of the second surgical element.

9. A method for adaptively identifying automated strategies during surgery, the method comprising: Obtain operational data for a first surgical element and a second surgical element, wherein the operational data includes indications of physiological parameters of the patient during the procedure, and multiple automated tasks associated with the procedure; Determine the automation assistance parameters of the first surgical element, wherein the automation assistance parameters are associated with a set of automation tasks performed by the first surgical element during the operation, among the plurality of automation tasks; Detect adaptive recognition events, wherein the adaptive recognition event is at least one of the following: The relationship between the first surgical element and the second surgical element; or The determination that the patient's physiological parameters meet the life-threatening threshold; Based on the adaptive recognition event, a second automated assistance parameter of the first surgical element is selected, wherein the second automated assistance parameter indicates a second set of automated tasks performed by the first surgical element during the surgery; Send the instruction for the second automated assistance parameter to the first surgical element; and The first surgical element performs the second set of automated tasks based on the second set of automated auxiliary parameters.

10. The method according to claim 9, wherein: The first surgical component is a ventilator; The second surgical element is a heating device; The patient's physiological parameters are either the CO2 percentage from the ventilator or the core body temperature from the heating device; The patient's procedure included atrial fibrillation (AFIB) neuroablation; The adaptive recognition event includes determining that the patient's CO2 to O2 or core body temperature meets the life-threatening threshold; and The second automated auxiliary parameter includes instructions for reducing the O2 supplementation level of the ventilator or instructions for increasing the tidal volume of the ventilator.

11. The method according to claim 9, wherein, The adaptive recognition event includes the determination that the patient's physiological parameters meet the life-threatening threshold, and wherein the selected second automated auxiliary parameter indicates that the second set of automated tasks includes the plurality of automated tasks.

12. The method according to claim 9, wherein, The adaptive recognition event includes the relationship between the first surgical element and the second surgical element, and wherein the selected second automation assistance parameter includes instructions for performing the second set of automated tasks in response to user input.

13. The method of claim 12, wherein, The method further includes: Sending an automation policy message to the user, wherein the automation policy message includes a request to execute the second set of automation tasks; and Receive user input including instructions for performing at least one of the automated tasks in the second group of automated tasks.

14. The method of claim 9, wherein, The method further includes: Receive location data associated with the user's movements during the procedure and control data associated with the first and second surgical elements; and The relationship between the first surgical element and the second surgical element is determined based on the location data, the operational data, and the patient's physiological parameters.

15. The method according to claim 9, wherein, The method further includes: Based on the adaptive recognition event, the automated auxiliary parameters of the second surgical element are selected, wherein the automated auxiliary parameters of the second surgical element indicate a set of automated tasks to be performed by the second surgical element during the operation; Sending instructions to the second surgical element regarding the automated auxiliary parameters of the second surgical element; and The second surgical element performs the set of automated tasks based on the automated auxiliary parameters of the second surgical element.

16. A system for adaptively identifying automation strategies, the system comprising: Processor, the processor being configured to: Obtain operational data for a first surgical element and a second surgical element, wherein the operational data includes indications of the patient's physiological parameters; Based on the operational data, adaptive recognition events are detected; Based on the adaptive recognition event, automated auxiliary parameters of the first surgical element are determined, wherein the automated auxiliary parameters indicate a set of automated tasks performed by the first surgical element; Send the instruction for the automated auxiliary parameters to the first surgical element; and The first surgical element performs the second set of automated tasks based on the automated auxiliary parameters.

17. The system according to claim 16, wherein, The adaptive recognition event includes the determination that the patient's physiological parameters meet a life-threatening threshold or the determined relationship between the first surgical element and the second surgical element.

18. The system according to claim 16, wherein, The processor is further configured to be able to: Sending an automation policy message to the user, wherein the automation policy message includes a request to execute an automation task associated with the automation assist parameters; and Receive user input including instructions for performing the automated task.

19. The system according to claim 16, wherein, The processor is further configured to be able to: Based on the adaptive recognition event, a second automated auxiliary parameter is selected, wherein the second automated auxiliary parameter indicates a second automated task.

20. The system according to claim 19, wherein, The processor is further configured to be able to: Sending instructions for the second automated assistance parameters to the second surgical element; and The second surgical element performs the second automated task.

Images