Controls associated with information flow, priority, and display of data within the interaction space with the active HCP

Abstract

A surgical system can include a processor configured to receive data from a medical instrument during a surgical procedure. The system can analyze the instrument data to determine a current surgical task being performed, and can further reference surgical data from a database of prior procedures. Based on the analysis, the system can identify surgeon-specific preferences, surgical context, or patient factors to adjust a level of information displayed on a screen. The system can prioritize critical data based on factors such as risk to the patient, task complexity, and importance of particular information for documentation or decision making.

Description

Cross-references to related applications

[0001] This application relates to the following concurrently filed patent applications, the contents of each of which are incorporated herein by reference: • Case file number END9638USNP1, entitled “Progresses Advancement of Authorized Level Based on Learned Complimentary Assistance”. • Case number END9638USNP2, titled "ADJUSTING AUTOMATED COOPERATIVE OPERATIONS BASED ONSITUATIONALLY DERIVED CONSTRAINTS". • Case file number END9638USNP3, entitled “ASSISTANCE ADVANCEMENT MULTI-SYSTEM INTERACTION”. • Case No. END9638USNP4 concerning the agent's case titled "MONITORING AND IDENTIFYING SURGEON CONTROL AND SUGGESTING ATASK THAT MAY BE DONE AUTONOMOUSLY". • Case number END9638USNP6, concerning the agent's case titled "ADAPTIVE RETRACTION FORCE CONTROL". • The agent's case file number END9638USNP7, entitled "ADJUSTMENT OR DISPLAY OF OPTIONS OF POSITIONAL ORORIENTATION IMPLICATIONS ON SURGICAL TOOL USAGE", and • Case number END9638USNP8, concerning the agent's case titled "ADJUSTMENT OF PHYSIOLOGIC FUNCTION SUPPLEMENTATION CONTROL".

[0002] The following are all cited and incorporated into this article: • U.S. Patent Application No. 18 / 810,323, filed on August 20, 2024, entitled “METHOD FOR MULTI-SYSTEM INTERACTION”; • U.S. Patent Application No. 18 / 960,006, filed November 26, 2024, entitled “METHOD FOR SMART SURGICAL SYSTEMS”; and • U.S. Patent Application No. 18 / 954,186, filed on November 20, 2024, entitled “METHOD FOR MULTI-SYSTEM INTERACTION”. Background Technology

[0003] Surgical procedures are typically performed in the operating room or surgical chamber of a medical facility, such as a hospital. Various surgical devices and systems are utilized in the performance of surgical procedures. In the digital and information age, due to patient safety and the general expectation of maintaining traditional practices, medical systems and facilities are often slower to adopt systems or procedures utilizing newer and more advanced technologies. Summary of the Invention

[0004] A surgical system may include a processor configured to receive data from a medical device during surgical procedures. The system can analyze the device data to determine the current surgical task being performed and may further reference surgical data from a database of previous surgeries. Based on the analysis, the system can identify surgeon-specific preferences, surgical context, or patient factors to adjust the level of information displayed on the screen. The system can prioritize key data based on factors such as patient risk, task complexity, and the importance of specific information for documentation or decision-making.

[0005] The system can dynamically adjust the information displayed during surgery to reduce the operator's cognitive load by prioritizing higher-risk information and de-prioritizing less critical data. The display can be configured based on real-time input from the operator, allowing user preferences to influence the hierarchy of the displayed information. The system can compare the current surgery with (e.g., previous) surgeries to identify differences, generate recommendations for improving surgical techniques or device performance, and / or adjust the displayed information accordingly.

[0006] In the example, the system can combine patient-specific data, such as anatomical structures, comorbidities, and intraoperative measurements, to refine the prioritization of information. When patient-specific data indicates an increased risk, the system can generate an alert, thus alerting the operator (e.g., immediately) to a critical situation. Attached Figure Description

[0007] Figure 1 This is a block diagram of a computer-implemented surgical system.

[0008] Figure 2 An example surgical system in an operating room is shown.

[0009] Figure 3 Example surgical hubs paired with various systems are shown.

[0010] Figure 4 An example situational awareness surgical system is shown.

[0011] Figure 5 An example surgical system that may include surgical instruments is illustrated.

[0012] Figure 6 An example is shown of a smart system display that can provide an interface for presenting categories of information during surgical procedures.

[0013] Figure 7 A flowchart illustrating actions that can be performed by an intelligent system is provided.

[0014] Figure 8 A schematic diagram illustrates an AI / ML-enabled system framework capable of performing actions related to surgical data processing and decision support. Detailed Implementation

[0015] A more detailed understanding can be obtained by referring to the following description, which is given by way of example in conjunction with the accompanying drawings.

[0016] Figure 1 An example computer-implemented surgical system 20000 is illustrated. The example surgical system 20000 may include one or more surgical systems (e.g., surgical subsystems) 20002, 20003, and 20004. As described herein, a system may refer to multiple systems (e.g., a clustered system). For example, surgical system 20002 may include a computer-implemented interactive surgical system. For instance, surgical system 20002 may include a surgical hub 20006 and / or a computing device 20016 communicating with a cloud computing system 20008, such as... Figure 2 The cloud computing system 20008 may include at least one remote cloud server 20009 and at least one remote cloud storage unit 20010. Example surgical systems 20002, 20003, or 20004 may include one or more wearable sensing systems 20011, one or more environmental sensing systems 20015, one or more robotic systems 20013, one or more intelligent devices 20014, one or more human-machine interface systems 20012, etc. Human-machine interface systems are also referred to herein as human-machine interface devices. Wearable sensing system 20011 may include one or more healthcare professional (HCP) sensing systems and / or one or more patient sensing systems. Environmental sensing system 20015 may include one or more devices, for example, for measuring one or more environmental properties, such as... Figure 2Further described. The robotic system 20013 may include multiple devices for performing surgical procedures, such as... Figure 2 Further described.

[0017] Surgical system 20002 can communicate with remote server 20009, which may be part of cloud computing system 20008. In one example, surgical system 20002 can communicate with remote server 20009 via a cable / FIOS networking node of an Internet service provider. In one example, patient sensing system can communicate directly with remote server 20009. Surgical system 20002 (and / or the various subsystems, intelligent surgical instruments, robots, sensing systems, and other computerized devices described herein) can collect data in real time and transmit the data to a cloud computer for data processing and manipulation. It should be understood that cloud computing may rely on shared computing resources rather than using local servers or personal devices to process software applications.

[0018] Surgical system 20002 and / or components thereof may communicate with remote server 20009 via a cellular transmit / receive point (TRP) or base station using one or more of the following cellular protocols: GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long Term Evolution (LTE) or 4G, LTE-Advanced (LTE-A), New Radio (NR) or 5G, and / or other wired or wireless communication protocols. Various examples of cloud-based analytics performed by cloud computing system 20008 and applicable to use with this disclosure are described in U.S. Patent Application Publication No. US 2019-0206569 A1 (U.S. Patent Application No. 16 / 209,403), filed December 4, 2018, entitled “METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB,” the entire disclosure of which is incorporated herein by reference.

[0019] The surgical hub 20006 can collaboratively interact with one of a plurality of devices displaying images from a laparoscopy and information from one or more other intelligent devices and one or more sensing systems 20011. The surgical hub 20006 can interact with one or more sensing systems 20011, one or more intelligent devices, and multiple displays. The surgical hub 20006 can be configured to collect measurement data from the sensing systems and transmit notification or control messages to the one or more sensing systems 20011. The surgical hub 20006 can transmit and / or receive information including notification information to and / or from a human-machine interface system 20012. The human-machine interface system 20012 may include one or more human-machine interface devices (HIDs). The surgical hub 20006 can transmit and / or receive notification or control information to be converted into audio, display, and / or control information for various devices communicating with the surgical hub.

[0020] For example, the sensing system may include a wearable sensing system 20011 (which may include one or more HCP sensing systems and / or one or more patient sensing systems) and / or an environmental sensing system 20015, such as Figure 1 As shown. The sensing system can measure data associated with various biomarkers. The sensing system can use one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc., to measure biomarkers. The sensor can use one or more of the following sensing techniques to measure biomarkers as described herein: photoplethysmography, electrocardiography, electroencephalography, colorimetry, impedance spectroscopy, potentiometry, current measurement, etc.

[0021] Biomarkers measured by the sensing system may include, but are not limited to, sleep, core body temperature, maximum oxygen uptake, physical activity, alcohol consumption, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, blood pH, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, cough and sneezing, gastrointestinal motility, gastrointestinal imaging, respiratory bacteria, edema, psychological factors, sweat, circulating tumor cells, autonomic tone, circadian rhythm and / or menstrual cycle.

[0022] Biomarkers can relate to physiological systems, including but not limited to behavioral and psychological systems, cardiovascular systems, renal systems, dermal systems, nervous systems, gastrointestinal systems, respiratory systems, endocrine systems, immune systems, tumors, musculoskeletal systems, and / or reproductive systems. For example, information from biomarkers can be determined and / or used by a computer-implemented patient and surgical system 20000. This information from biomarkers can be determined and / or used by the computer-implemented patient and surgical system 20000 to improve said systems and / or improve patient outcomes.

[0023] The sensing system can transmit data to the surgical hub 20006. The sensing system can communicate with the surgical hub 20006 using one or more of the following RF protocols: Bluetooth, Bluetooth Low Energy (BLE), Bluetooth Smart, Zigbee, Z-Wave, IPv6 Low Power Wireless Personal Area Network (6LoWPAN), and Wi-Fi.

[0024] The sensing system, biomarkers, and physiological system are described in more detail in U.S. Application No. 17 / 156,287 (Attorney’s File No. END9290USNP1), filed on January 22, 2021, entitled “METHOD OF ADJUSTING A SURGICAL PARAMETERBASED ON BIOMARKER MEASUREMENTS”, the entire disclosure of which is incorporated herein by reference.

[0025] The sensing system described herein can be used to assess the physiological condition of a surgeon performing surgery on a patient, a patient preparing for surgery, or a patient recovering after surgery. The cloud-based computing system 20008 can be used to monitor biomarkers associated with the surgeon or patient in real time, and can be used to generate surgical plans based at least on measurement data collected prior to surgery, provide control signals to surgical instruments during surgery, and notify the patient of complications during the postoperative period.

[0026] A cloud-based computing system 20008 can be used to analyze surgical data. Surgical data can be obtained via one or more intelligent instruments 20014, wearable sensing systems 20011, environmental sensing systems 20015, robotic systems 20013, etc., within the surgical system 20002. Surgical data may include tissue status to assess leakage or perfusion of sealed tissue following tissue sealing and surgical pathology data, including images of samples of body tissue, anatomical structures of the body using various sensors integrated with imaging devices, and techniques such as overlaying images captured by multiple imaging devices, image data, etc. Surgical data can be analyzed to improve surgical outcomes by determining whether further treatment can proceed (such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robotics to tissue-specific sites and conditions). Such data analysis can employ outcome analysis processing, and using standardized methods can provide beneficial feedback to validate surgical treatment and surgeon behavior, or to suggest modifications to surgical treatment and surgeon behavior.

[0027] Figure 2 An example surgical system 20002 in an operating room is shown. Figure 2 As illustrated, the patient undergoes surgery performed by one or more healthcare professionals (HCPs). The HCP is monitored by one or more HCP sensing systems 20020 worn by the HCP. The HCP and the environment surrounding the HCP may also be monitored by one or more environmental sensing systems, including, for example, a collection of cameras 20021, a collection of microphones 20022, and other sensors that can be deployed in the operating room. The HCP sensing systems 20020 and the environmental sensing systems may communicate with a surgical hub 20006, which in turn may communicate with one or more cloud servers 20009 of a cloud computing system 20008, such as... Figure 1 As shown. Environmental sensing systems can be used to measure one or more environmental properties, such as the location of the HCP in the surgical room, HCP movement, environmental noise in the surgical room, temperature / humidity in the surgical room, etc.

[0028] like Figure 2As illustrated, a main display 20023 and one or more audio output devices (e.g., speakers 20019) are positioned within a sterile area to be visible to the operator at the operating table 20024. Furthermore, a visualization / notification tower 20026 is positioned outside the sterile area. The visualization / notification tower 20026 may include a first non-sterile human-machine interface (HID) 20027 and a second non-sterile HID 20029 that are mutually exclusive. The HID may be a display or a display with a touchscreen that allows direct human-machine interface with the HID. The HID system guided by the surgical hub 20006 can be configured to utilize HIDs 20027, 20029, and 20023 to coordinate information flow to the operator both inside and outside the sterile area. In one example, the surgical hub 20006 may enable the HID (e.g., the main HID 20023) to display notifications and / or information about the patient and / or surgical procedures. In one example, the surgical hub 20006 may prompt and / or receive input from personnel in a sterile or non-sterile area. In another example, the surgical hub 20006 may enable the HID to display a snapshot of the surgical site recorded by the imaging device 20030 on a non-sterile HID 20027 or 20029, while maintaining a real-time feed of the surgical site on the main HID 20023. For example, the snapshot on the non-sterile display 20027 or 20029 may allow a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0029] The surgical hub 20006 can be configured to route diagnostic inputs or feedback entered by a non-sterile operator at the visualization tower 20026 to the main display 20023 within the sterile area, where a sterile operator at the operating table can view the diagnostic inputs or feedback. In one example, the input may be a modified form of a snapshot displayed on a non-sterile display 20027 or 20029, which can be routed to the main display 20023 via the surgical hub 20006.

[0030] See Figure 2Surgical instrument 20031 is used in surgical procedures as part of surgical system 20002. Hub 20006 can be configured to coordinate the flow of information to the display of surgical instrument 20031. For example, it is described in U.S. Patent Application Publication No. US 2019-0200844 A1 (U.S. Patent Application No. 16 / 209,385), filed December 4, 2018, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” the entire disclosure of which is incorporated herein by reference. Diagnostic inputs or feedback entered by a non-aseptic operator at visualization tower 20026 can be routed by hub 20006 to the surgical instrument display within the aseptic area, where the operator of surgical instrument 20031 can view the diagnostic inputs or feedback. For example, an example surgical instrument suitable for use with the surgical system 20002 is described in U.S. Patent Application Publication No. US 2019-0200844 A1 (U.S. Patent Application No. 16 / 209,385), filed December 4, 2018, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY,” under the heading “Surgical Instrument Hardware,” the entire disclosure of which is incorporated herein by reference.

[0031] like Figure 2 As shown, surgical system 20002 can be used to perform surgery on a patient lying supine on operating table 20024 in operating room 20035. Robotic system 20034 can be used as part of surgical system 20002 during surgery. Robotic system 20034 may include surgeon's console 20036, patient-side trolley 20032 (surgical robot), and surgical robot hub 20033. While the surgeon views the surgical site through surgeon's console 20036, patient-side trolley 20032 can manipulate at least one removably attached surgical tool 20037 through a minimally invasive incision within the patient's body. Images of the surgical site can be obtained via medical imaging device 20030, which can be manipulated by patient-side trolley 20032 to orient the imaging device 20030. Robotic hub 20033 can be used to process images of the surgical site for subsequent display to the surgeon via surgeon's console 20036.

[0032] Other types of robotic systems can be readily adapted for use with surgical system 20002. Various examples of robotic systems and surgical tools applicable to this disclosure are described herein and in U.S. Patent Application No. US 2019-0201137 A1 (U.S. Patent Application No. 16 / 209,407), filed December 4, 2018, entitled “METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL,” the entire disclosure of which is incorporated herein by reference.

[0033] In various aspects, the imaging device 20030 may include at least one image sensor and one or more optical components. Suitable image sensors may include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0034] The optical components of the imaging device 20030 may include one or more illumination sources and / or one or more lenses. The one or more illumination sources may be directed to illuminate multiple portions of the surgical site. One or more image sensors may receive light reflected or refracted from the surgical site, including light reflected or refracted from tissue and / or surgical instruments.

[0035] Light sources can be configured to radiate electromagnetic energy in both the visible and invisible spectra. The visible spectrum (sometimes referred to as the optical spectrum or emission spectrum) is the portion of the electromagnetic spectrum that is visible to the human eye (e.g., detectable by it) and can be referred to as "visible light" or simply "light." The typical human eye responds to wavelengths in the range of approximately 380 nm to approximately 750 nm in air.

[0036] The invisible spectrum (e.g., the non-luminescent spectrum) is the portion of the electromagnetic spectrum that lies below and above the visible spectrum (i.e., wavelengths below about 380 nm and above about 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths greater than about 750 nm are longer than the red visible spectrum and become invisible infrared (IR), microwave, and radio electromagnetic radiation. Wavelengths less than about 380 nm are shorter than the violet spectrum and become invisible ultraviolet, X-ray, and gamma-ray electromagnetic radiation.

[0037] In various respects, the imaging device 20030 is configured to be used in minimally invasive surgery. Examples of imaging devices suitable for use in this disclosure include, but are not limited to, arthroscopy, angioscopy, bronchoscopy, cholangioscopy, colonoscopy, cytology endoscopy, duodenoscope, colonoscope, esophagoduodenoscope (gastroscopy), endoscope, laryngoscope, nasopharyngeal-renal endoscopy, sigmoidoscopy, thoracoscopy, and ureteroscopy.

[0038] Imaging devices can employ multispectral monitoring to distinguish morphology and underlying structures. A multispectral image is an image that captures image data across a specific wavelength range of the electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including light from frequencies outside the visible light range, such as IR and ultraviolet. Spectral imaging allows the extraction of additional information that the human eye fails to capture with its red, green, and blue receptors. The use of multispectral imaging is described in more detail under the title “Advanced Imaging Acquisition Module” in U.S. Patent Application No. US 2019-0200844 A1 (U.S. Patent Application No. 16 / 209,385), filed December 4, 2018, entitled “METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE ANDDISPLAY,” the entire disclosure of which is incorporated herein by reference. After completing a surgical task to perform one or more of the previously described tests on the treated tissue, multispectral monitoring can be a useful tool for repositioning the surgical site. It goes without saying that rigorous sterilization of the operating room and surgical equipment is required during any surgical procedure. The stringent hygienic and sterilization conditions required in a “surgical room” (e.g., operating room or treatment room) necessitate the highest possible sterility for all medical devices and apparatus. Part of this sterilization process requires the sterilization of any material that comes into contact with the patient or penetrates the sterile area, including the imaging device 20030 and its attachments and components. It should be understood that a sterile area can be considered a designated area deemed free of microorganisms, such as within a tray or sterile towel, or can be considered the area surrounding the patient prepared for surgical procedures. A sterile area may include properly dressed scrubbed team members, as well as all equipment and fixtures within that area.

[0039] Figure 1 The illustrated wearable sensing system 20011 may include, for example: Figure 2One or more HCP sensing systems 20020 are shown. The HCP sensing system 20020 may include sensing systems for monitoring and detecting a set of physical and / or physiological states of a healthcare professional (HCP). An HCP may typically be a surgeon or one or more healthcare professionals or other healthcare providers assisting a surgeon. In one example, the HCP sensing system 20020 may measure a set of biomarkers to monitor the HCP's heart rate. In one example, the HCP sensing system 20020 worn on the surgeon's wrist (e.g., a watch or wristband) may use an accelerometer to detect hand movements and / or tremors and determine the amplitude and frequency of the tremors. The sensing system 20020 may transmit measurement data associated with the set of biomarkers and data associated with the surgeon's physical state to a surgical hub 20006 for further processing.

[0040] Figure 1 The illustrated environmental sensing system 20015 can transmit environmental information to the surgical hub 20006. For example, the environmental sensing system 20015 may include a camera 20021 for detecting the hand / body position of the HCP. The environmental sensing system 20015 may include a microphone 20022 for measuring ambient noise in the surgical room. Other environmental sensing systems 20015 may include devices such as a thermometer for measuring temperature and a hygrometer for measuring the humidity of the surrounding environment in the surgical room. Surgeon biomarkers may include one or more of the following: pressure, heart rate, etc. Environmental measurements from the surgical room may include ambient noise levels associated with the surgeon or patient, surgeon and / or staff movement, surgeon and / or staff attention levels, etc. The surgical hub 20006 (either independently or in communication with a cloud computing system) can use surgeon biomarker measurement data and / or environmental sensing information to modify the control algorithms of handheld instruments or the average latency of robot interfaces, for example, to minimize tremors.

[0041] The surgical hub 20006 can use surgeon biomarker measurements associated with HCP to adaptively control one or more surgical instruments 20031. For example, the surgical hub 20006 can transmit control programs to the surgical instrument 20031 to control its actuators to limit or compensate for fatigue and the use of fine motor skills. The surgical hub 20006 can transmit control programs based on situational awareness and / or context regarding the importance or criticality of the task. When control is needed, the control program can instruct the instrument to change its operation to provide more control.

[0042] Figure 3An example surgical system 20002 with a surgical hub 20006 is shown. The surgical hub 20006 can be paired with a wearable sensing system 20011, an environmental sensing system 20015, a human-machine interface system 20012, a robotic system 20013, and a smart instrument 20014 via modular controls. The hub 20006 includes a display 20048, an imaging module 20049, a generator module 20050 (e.g., an energy generator), a communication module 20056, a processor module 20057, a storage array 20058, and an operating room mapping module 20059. In some aspects, such as Figure 3 As illustrated, hub 20006 also includes smoke extraction module 20054 and / or suction / flushing module 20055. Various modules and systems can be connected directly to the modular control via a router or via communication module 20056. The operating room device can be coupled to cloud computing resources and data storage devices via the modular control. Human-machine interface system 20012 may include a display subsystem and a notification subsystem.

[0043] Modular controls can be coupled to a non-contact sensor module. The non-contact sensor module can use ultrasonic, laser-based, and / or similar non-contact measuring devices to measure the size of the operating room and generate a mapping of the surgical room. Other distance sensors can be used to determine the boundaries of the operating room. An ultrasonic-based non-contact sensor module can scan the operating room by emitting a burst of ultrasound and receiving the echo as it bounces back from the walls of the operating room, as described under the title “Surgical Hub Spatial Awareness Within an Operating Room” in U.S. Provisional Patent Application Serial No. 62 / 611,341, filed December 28, 2017, entitled “INTERACTIVE SURGICAL PLATFORM”, the entire contents of which are incorporated herein by reference. The sensor module can be configured to determine the size of the operating room and adjust Bluetooth pairing distance limits. A laser-based non-contact sensor module can scan the operating room by emitting laser pulses, receiving laser pulses bouncing back from the walls of the operating room, and comparing the phase of the emitted pulse with the received pulse to determine the size of the operating room and adjust Bluetooth pairing distance limits.

[0044] During surgery, the application of energy to tissue for sealing and / or cutting can be associated with fumigation, aspiration of excess fluid, and / or tissue flushing. Fluid lines, power lines, and / or data lines from different sources can become entangled during surgery. Resolving this issue during surgery wastes valuable time. Disconnecting lines may require disconnecting the lines from their respective modules, which may necessitate module resets. The Hub Modular Housing 20060 provides a unified environment for managing power lines, data lines, and fluid lines, reducing the frequency of entanglement between such lines.

[0045] Energy can be applied to tissue at a surgical site. The surgical hub 20006 may include a hub housing 20060 and a combined generator module slidably received in a docking base within the hub housing 20060. The docking base includes data and power contacts. The combined generator module may include two or more of an ultrasonic energy generator component, a bipolar RF energy generator component, or a monopolar RF energy generator component housed in a single unit. The combined generator module may include a smoke extraction component, at least one energy delivery cable for connecting the combined generator module to a surgical instrument, at least one smoke extraction component configured to extract smoke, fluid, and / or particles generated by applying therapeutic energy to tissue, and a fluid line extending from a remote surgical site to the smoke extraction component. The fluid line may be a first fluid line, and a second fluid line may extend from a remote surgical site to a suction and flushing module 20055 slidably housed in the hub housing 20060. The hub housing 20060 may include a fluid interface.

[0046] The combined generator module can generate multiple energy types for application to tissue. One energy type may be more advantageous for cutting tissue, while another different energy type may be more advantageous for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Aspects of this disclosure present a solution in which the hub modular housing 20060 is configured to accommodate different generators and facilitate interactive communication between them. The hub modular housing 20060 allows for the rapid removal and / or replacement of various modules.

[0047] The modular surgical housing may include: a first energy generator module configured to generate a first energy for application to tissue; and a first docking base including a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and wherein the first energy generator module is slidably movable to no longer electrically engage with the first power and data contacts. The modular surgical housing may also include: a second energy generator module configured to generate a second energy, different from the first energy, for application to tissue; and a second docking base including a second docking port including second data and power contacts, wherein the second energy generator module is slidably movable to electrically engage with the power and data contacts, and wherein the second energy generator module is slidably movable to no longer electrically engage with the second power and data contacts. Furthermore, the modular surgical housing also includes a communication bus between the first and second docking ports, configured to facilitate communication between the first and second energy generator modules.

[0048] See Figure 3 The hub modular housing 20060 allows for modular integration of the generator module 20050, the smoke extraction module 20054, and the suction / flushing module 20055. The hub modular housing 20060 facilitates interactive communication between modules 20059, 20054, and 20055. The generator module 20050 may have integrated monopolar, bipolar, and ultrasonic components supported in a single housing unit slidably inserted into the hub modular housing 20060. The generator module 20050 can be connected to the monopolar device 20051, the bipolar device 20052, and the ultrasonic device 20053. The generator module 20050 may include a series of monopolar generator modules, bipolar generator modules, and / or ultrasonic generator modules that interact through the hub modular housing 20060. The hub modular housing 20060 facilitates the insertion of multiple generators and interactive communication between generators connected to the hub modular housing 20060, allowing the generator to function as a single generator.

[0049] A surgical data network with a set of communication hubs can connect sensing systems and modular devices located in one or more operating rooms, patient recovery rooms, or rooms in a medical facility specifically equipped for surgical procedures to a cloud computing system 20008.

[0050] Figure 4A diagram illustrating a situation-aware surgical system 5100 is provided. Data source 5126 may include, for example, a modular device 5102, a database 5122 (e.g., an EMR database containing patient records), a patient monitoring device 5124 (e.g., a blood pressure (BP) monitor and an electrocardiogram (EKG) monitor), an HCP monitoring device 35510, and / or an environmental monitoring device 35512. Modular device 5102 may include sensors configured to detect parameters associated with the patient, HCP, and environment, and / or the modular device itself. Modular device 5102 may include one or more intelligent instruments 20014. Surgical hub 5104 may derive surgical context information from the data, for example, based on a specific combination of received data or a specific order in which data is received from data source 5126. The context information inferred from the received data may include, for example, the type of surgical procedure being performed, the specific steps of the surgical procedure being performed by the surgeon, the type of tissue being operated on, or the body cavity of the surgical object. The ability of the surgical hub 5104 to derive or infer surgical-related information from received data can be termed "situational awareness." For example, the surgical hub 5104 may incorporate a situational awareness system, which could be hardware and / or surgical planning associated with the surgical hub 5104 that derives surgical-related background information from received data, and / or surgical planning information received from edge computing system 35514 or enterprise cloud server 35516. Background information derived from data source 5126 may include, for example, the steps of the surgical procedure being performed, whether and how a specific modular device 5102 is being used, and the patient's condition.

[0051] Surgical hub 5104 can connect to various databases 5122 to retrieve data from them regarding surgical procedures being performed or to be performed. In one example of surgical system 5100, database 5122 may include a hospital's EMR database. Data that can be received from database 5122 by the situational awareness system of surgical hub 5104 may include, for example, start (or setup) time or operational information about a procedure (e.g., a segmental resection in the upper right thoracic region). Surgical hub 5104 can derive background information about the surgical procedure from this data alone or from this data in combination with data from other data sources 5126.

[0052] The surgical hub 5104 can be connected to (e.g., paired with) various patient monitoring devices 5124. In one example of the surgical system 5100, the patient monitoring devices 5124 that can be paired with the surgical hub 5104 may include a pulse oximeter (SpO2 monitor) 5114, a BP monitor 5116, and an EKG monitor 5120. Perioperative data that can be received by the situational awareness system of the surgical hub 5104 from the patient monitoring devices 5124 may include, for example, the patient's oxygen saturation, blood pressure, heart rate, and other physiological parameters. Background information that can be derived by the surgical hub 5104 from the perioperative data sent by the patient monitoring devices 5124 may include, for example, whether the patient is in the operating room or under anesthesia. The surgical hub 5104 may derive these inferences individually from data from the patient monitoring devices 5124 or in combination with data from other data sources 5126 (e.g., a ventilator 5118).

[0053] The surgical hub 5104 can be connected to (e.g., paired with) various modular devices 5102. In one example of the surgical system 5100, the modular device 5102 paired with the surgical hub 5104 may include a fume extractor, medical imaging devices (such as...) Figure 2 The imaging device 20030 shown includes an insufflator, a combined energy generator (for providing power to ultrasound surgical instruments and / or RF electrosurgical instruments), and a ventilator.

[0054] Perioperative data received by the surgical hub 5104 from the medical imaging device may include, for example, whether the medical imaging device is activated and video or image feeds. Background information derived by the surgical hub 5104 from the perioperative data transmitted by the medical imaging device may include, for example, whether the surgery is a VATS procedure (based on whether the medical imaging device is activated or paired with the surgical hub 5104 at the start of the surgery or during the procedure). Image or video data (or a data stream representing video for a digital medical imaging device) from the medical imaging device may be processed by a pattern recognition system or a machine learning system to, for example, identify features (e.g., organ or tissue type) in the field of view (FOY) of the medical imaging device. Background information derived by the surgical hub 5104 from the identified features may include, for example, the type of surgical procedure (or its steps) being performed, the organ being operated on, or the body cavity in which the operation is being performed.

[0055] The situational awareness system of the surgical hub 5104 can derive contextual information from data received from the data source 5126 in a variety of different ways. For example, the situational awareness system may include a pattern recognition system or a machine learning system (e.g., an artificial neural network) trained on training data to associate various inputs (e.g., data from the database 5122, patient monitoring device 5124, modular device 5102, HCP monitoring device 35510, and / or environmental monitoring device 35512) with corresponding contextual information about the surgical procedure. For example, the machine learning system can accurately derive contextual information about the surgical procedure from the provided inputs. In an example, the situational awareness system may include a lookup table that stores pre-represented environmental information about the surgical procedure associated with one or more inputs (or ranges of inputs) corresponding to environmental information. In response to a query using one or more inputs, the lookup table can return corresponding contextual information used by the situational awareness system to control the modular device 5102. In the example, the contextual information received by the situational awareness system of the surgical hub 5104 may be associated with a specific control adjustment or a set of control adjustments for one or more modular devices 5102. In the example, the situational awareness system may include a machine learning system, lookup table, or other such system that can generate or retrieve one or more control adjustments for one or more modular devices 5102 when provided with contextual information as input.

[0056] For example, based on data source 5126, the situational-aware surgical hub 5104 can determine the type of tissue being operated on. The situational-aware surgical hub 5104 can infer whether the surgery being performed is thoracic or abdominal, thus allowing the surgical hub 5104 to determine whether the tissue held by the end effector of the surgical suture and cutting instruments is lung tissue (for thoracic surgery) or stomach tissue (for abdominal surgery). The situational-aware surgical hub 5104 can determine whether the surgical site is under pressure (by determining that the surgery is utilizing airflow) and determine the type of surgery to achieve a consistent amount of smoke extraction for both thoracic and abdominal surgeries. Based on data source 5126, the situational-aware surgical hub 5104 can determine which step of the surgery is being performed or will be performed subsequently.

[0057] The situation-aware surgical hub 5104 can determine the type of surgical procedure being performed and customize energy levels based on the expected tissue profile of the procedure. The situation-aware surgical hub 5104 can adjust the energy levels of ultrasonic surgical instruments or RF electrosurgical instruments throughout the entire surgical procedure, rather than just on a per-procedure basis.

[0058] In the example, data can be extracted from an additional data source 5126 to improve the conclusions drawn by the surgical hub 5104 from one data source 5126. The situation-aware surgical hub 5104 can supplement the data received from the modular device 5102 with the background information about the surgery that it has built from other data sources 5126.

[0059] The situational awareness system of the surgical hub 5104 can take physiological measurement data into account to provide additional contextual information when analyzing visualization data. This additional context can be useful when the visualization data itself may be uncertain or incomplete.

[0060] The situational awareness surgical hub 5104 can determine whether a surgeon (or other HCP) is making an error or otherwise deviating from the intended procedure during surgery. For example, the surgical hub 5104 can determine the type of surgery being performed, retrieve a corresponding list of steps or the order of equipment use (e.g., from memory), and compare the steps being performed or the equipment being used during the surgical procedure with the expected steps or equipment determined by the surgical hub 5104 for that type of surgery. The surgical hub 5104 can provide alerts indicating that a specific step in the surgery is performing an unexpected action or utilizing an unexpected device.

[0061] Surgical instruments (and other modular devices 5102) can be adapted to the specific context of each surgical procedure (such as adaptation to different tissue types) and verification actions during the surgical procedure. Subsequent steps, data, and display adjustments can be provided to the surgical instruments (and other modular devices 5102) in the operating room according to the specific context of the surgery.

[0062] Figure 5An example surgical system 20280 is illustrated, which may include a surgical instrument 20282. The surgical instrument 20282 may communicate with a console 20294 and / or a portable device 20296 via a wired and / or wireless connection through a local area network 20292 and / or a cloud network 20293. The console 20294 and the portable device 20296 may be any suitable computing device. The surgical instrument 20282 may include a handle 20297, an adapter 20285, and a loading unit 20287. The adapter 20285 is releasably coupled to the handle 20297, and the loading unit 20287 is releasably coupled to the adapter 20285, such that the adapter 20285 transmits force from a drive shaft to the loading unit 20287. The adapter 20285 or the loading unit 20287 may include a force gauge (not explicitly shown) disposed therein to measure the force applied to the loading unit 20287. The loading unit 20287 may include an end effector 20289 having a first jaw 20291 and a second jaw 20290. The loading unit 20287 may be an in-situ loading or multiple-fire loading unit (MFLU), which allows clinicians to fire multiple fasteners multiple times without removing the loading unit 20287 from the surgical site to reload it.

[0063] The first jaw 20291 and the second jaw 20290 may be configured to clamp tissue therebetween, fire a fastener through the clamped tissue, and cut the clamped tissue. The first jaw 20291 may be configured to fire at least one fastener multiple times, or may be configured to include a replaceable multiple-fire fastener cartridge containing multiple fasteners (e.g., nails, clamps, etc.) that can be fired more than once before being replaced. The second jaw 20290 may include an anvil that deforms or otherwise secures the fastener when it is ejected from the multiple-fire fastener cartridge.

[0064] The handle 20297 may include a motor coupled to a drive shaft to influence its rotation. The handle 20297 may include a control interface for selectively activating the motor. The control interface may include buttons, switches, levers, sliders, touchscreens, and any other suitable input mechanisms or user interfaces that can be engaged by a clinician to activate the motor.

[0065] The control interface of the handle 20297 can communicate with the controller 20298 of the handle 20297 to selectively activate the motor to affect the rotation of the drive shaft. The controller 20298 may be located within the handle 20297 and is configured to receive input from the control interface and adapter data from the adapter 20285 or loading unit data from the loading unit 20287. The controller 20298 can analyze the input from the control interface and the data received from the adapter 20285 and / or the loading unit 20287 to selectively activate the motor. The handle 20297 may also include a display that a clinician can view during use of the handle 20297. The display may be configured to show portions of the adapter or loading unit data before, during, or after the firing instrument 20282.

[0066] Adapter 20285 may include an adapter identification device 20284 disposed therein, and loading unit 20287 may include a loading unit identification device 20288 disposed therein. Adapter identification device 20284 may communicate with controller 20298, and loading unit identification device 20288 may communicate with controller 20298. It should be understood that loading unit identification device 20288 may communicate with adapter identification device 20284, and the adapter identification device relays or transmits communications from loading unit identification device 20288 to controller 20298.

[0067] The adapter 20285 may also include a plurality of sensors 20286 (one shown) disposed around it to detect various conditions of the adapter 20285 or the environment (e.g., whether the adapter 20285 is connected to the loading unit, whether the adapter 20285 is connected to the handle, whether the drive shaft rotates, the torque of the drive shaft, the strain of the drive shaft, the temperature within the adapter 20285, the number of times the adapter 20285 is fired, the peak force of the adapter 20285 during firing, the total force applied to the adapter 20285, the peak retraction force of the adapter 20285, the number of pauses of the adapter 20285 during firing, etc.). The plurality of sensors 20286 may provide input to the adapter identification device 20284 in the form of data signals. The data signals of the plurality of sensors 20286 may be stored in the adapter identification device 20284 or may be used to update adapter data stored in the adapter identification device. The data signals of the plurality of sensors 20286 may be analog or digital. Multiple sensors 20286 may include force gauges to measure the force applied to the loading unit 20287 during firing.

[0068] The handle 20297 and adapter 20285 can be configured to interconnect the adapter identification device 20284 and the loading unit identification device 20288 with the controller 20298 via an electrical interface. The electrical interface can be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals therebetween). Additionally or alternatively, the electrical interface can be a contactless electrical interface for wirelessly transmitting energy and signals therebetween (e.g., inductive transmission). It is also conceivable that the adapter identification device 20284 and the controller 20298 can wirelessly communicate with each other via a wireless connection separate from the electrical interface.

[0069] The handle 20297 may include a transceiver 20283 configured to transmit instrument data from the controller 20298 to other components of the system 20280 (e.g., LAN 20292, cloud 20293, console 20294, or portable device 20296). The controller 20298 may also transmit instrument data and / or measurement data associated with one or more sensors 20286 to the surgical hub. The transceiver 20283 may receive data (e.g., pod data, loading unit data, adapter data, or other notifications) from the surgical hub 20270. The transceiver 20283 may also receive data (e.g., pod data, loading unit data, or adapter data) from other components of the system 20280. For example, controller 20298 can send instrument data to console 20294, including the serial number of the attachment adapter (e.g., adapter 20285) attached to handle 20297, the serial number of the loading unit (e.g., loading unit 20287) attached to adapter 20285, and the serial number of the multi-fire fastener cartridge loaded onto the loading unit. Console 20294 can then send data associated with the attached cartridge, loading unit, and adapter (e.g., cartridge data, loading unit data, or adapter data), respectively, back to controller 20298. Controller 20298 can display the message on a local instrument display or send the message via transceiver 20283 to console 20294 or portable device 20296 for display on display 20295 or portable device screen, respectively.

[0070] Figure 6 An example is illustrated by an intelligent system display 56600 that provides an interface for presenting information categories during surgical procedures. The display can adjust the visual salience of information categories based on the surgical context, user preferences, and data relevance. For example, high-priority information 56602 may occupy a central or visible position, making it accessible to the surgical team. High-priority information may correspond to data such as patient vital signs or task-specific metrics that are relevant to decisions made during high-risk phases of the surgery, for example.

[0071] Standard priority information 56604 can be displayed alongside high priority information and, for example, has reduced visual salience. Standard priority information may include surgical updates, instrument status, or medium-priority notifications that can be indirectly monitored by the surgical team. The system can use algorithms to determine which datasets fall into the standard priority category, thereby balancing relevance and cognitive load to avoid overwhelming the user and keeping contextual information accessible.

[0072] Low-priority information 56606 may include data that, while not directly related to the ongoing surgical task, may have utility for broader surgical supervision or postoperative analysis. For example, low-priority information may include ergonomic feedback, assistive device performance metrics, or ongoing documentation tasks. The system can locate such information in areas of the display that allow for unobtrusive monitoring without diminishing the impact on higher-priority factors.

[0073] Alarm 56608 can be visually distinct and is designed to draw attention to significant or unexpected events during surgery. Alarm 56608 can correspond to an anomaly associated with (e.g., immediate) intervention, patient-specific risk, or instrument malfunction. The system can prioritize alarms throughout the information hierarchy, progressively escalating their prominence based on severity, potential impact on the patient, or stage of surgery.

[0074] The Intelligent System Display 56600 supports information management through adaptive configuration consistent with surgical workflows. The layout can change in response to user input, evolving surgical use cases, or predetermined hierarchical structures established by the system. This adaptability is linked to the distribution of information among various stakeholders in the operating room, such as surgeons, anesthesiologists, and support staff.

[0075] Control over the flow, prioritization, and display of data within the active HCP interaction space can include managing how data is displayed and escalated based on its context and relevance. The system can organize data according to the devices in use, the tasks those devices are performing, the automated physical actions, or the results the healthcare provider (HCP) is working to achieve. A hierarchical structure can define display levels, which can be escalated or degraded based on factors such as the risks associated with patient data, the complexity of the task or device generating the data, or the importance of recorded or consented information. The relevance of the technology or smart device's performance can guide how information is presented.

[0076] To modify focus of attention during laparoscopic and robotic surgery, systems can incorporate mechanisms to detect declines in attention or suboptimal ergonomics. For example, a real-time display of a "mini avatar" on a laparoscopic monitor can help surgeons remain aware of their posture and movements. This visual feedback can serve as a reminder to correct body positioning or adjust proximity, thereby reducing fatigue and increasing endurance. By addressing the influencing factors of both physical and cognitive factors, such tools support surgeons in maintaining optimal focus throughout the procedure.

[0077] Figure 7 A flowchart illustrating actions that can be performed by an intelligent system is provided. At 56630, the system may receive instrument data from a medical device during surgery on a patient. At 56632, the system may determine the current surgical task being performed based on the received instrument data. At 56634, the system may determine user preferences for information display based on surgical data from multiple surgical procedures, and the surgical data may include one or more of surgeon-specific preferences, surgical background, or patient factors. At 56636, the system may adjust the display level of the information based on at least one or more of the current surgical task, the surgical instruments being used, the determined user preferences, or a hierarchical structure of multiple display levels. This hierarchy may be based on one or more of the risk to the patient, the complexity of the current surgical task, or the importance level associated with the data document. At 56638, the system may generate control signals for the display based on the adjusted display level of the information.

[0078] The display level can be adjusted by prioritizing information related to the patient's higher risk over information related to the patient's lower risk on the monitor.

[0079] The display priority can be adjusted by lowering the priority of information related to lower patient risk on the monitor compared to information related to higher patient risk. This lower priority can be associated with a reduced cognitive burden on the operator of the surgical system.

[0080] This system can receive input from the operator of the surgical system during surgical procedures, indicating preferences for the display level of information. The system can store user preferences based on the received input for use in information display.

[0081] This system compares the adjusted display level of information and the current surgical task with surgical data from multiple surgical procedures. It can identify differences between the current surgery and previous surgeries from multiple procedures. Based on the identified differences, the system can determine recommendations for one or more of the following: technological improvements or changes in the performance of intelligent devices. The system can update control signals to adjust the display level of information, thereby prioritizing the determined recommendations.

[0082] This system can identify patient-specific data, including one or more of the patient's anatomy, comorbidities, or intraoperative physiological measurements. The system can update a hierarchical structure across multiple display levels to prioritize information relevant to the patient-specific data. Based on the updated hierarchy, the system can update control signals to adjust the display level of information, and the updated hierarchy can be associated with the display level of information that prioritizes the patient-specific data.

[0083] This system can identify patient-specific data indicating an increased risk for the patient. It can generate alerts based on this increased risk, and prioritizing patient-specific data can be correlated with prioritizing alerts.

[0084] Figure 8 A schematic diagram illustrates an AI / ML-enabled system framework capable of performing actions related to surgical data processing and decision support. The framework can be organized as follows: Input 56640, Processing 56642, and Output 56644. At Input 56640, the system receives data streams from sources, including instrument data generated by medical devices, surgeon-specific preferences stored in a database, historical surgical data from previous surgeries, and patient-specific information such as anatomical details, comorbidities, or physiological measurements. The inputs provide the basis for the system to analyze the surgical environment and accordingly standardize processing and adjust the output.

[0085] Information and data management in a surgical context can include organizing and guiding data based on its source, destination, and relevance to the ongoing surgery. The system can process subtypes of information, such as patient vital signs, equipment status, and surgical progress metrics. Patient vital signs can include (e.g., critical) parameters such as heart rate and blood pressure, and equipment status can include current alarms and potential risks for future alarms. Surgical progress data can track surgical steps and milestones, allowing the system to adjust its output accordingly.

[0086] The integration of in-situ devices with external imaging systems allows instruments to provide a variety of functions, such as marking anatomical structures or assisting in imaging overlays. For example, during laparoscopic or robotic surgery, instruments can be used to identify (e.g., critical) structures in semi-rigid organs such as the liver or lungs. Highlighting these structures can reduce the risk of accidental injury by providing the surgical team with real-time visual feedback and anatomical context. The ability to mark specific points on anatomical features using handheld tools enables precise triangulation when combined with CT scans or X-rays, ensuring surgical accuracy.

[0087] Handheld tools can be used to control settings on other smart devices, providing a centralized interface for managing different systems. For example, instrument ends can be used to navigate graphical user interfaces (GUIs) displayed on screens, allowing device settings to be adjusted without disrupting aseptic workflows. This interaction may include tracking instrument movement or recording points within the field of view to modify system behavior. Such integration can impact surgical efficiency and reduce the cognitive burden associated with managing multiple devices simultaneously.

[0088] Medical imaging data can be accessed and interacted with directly from a sterile environment, allowing HCPs to manipulate preoperative scans during surgery. For example, devices equipped with internal accelerometers or externally attached tracking components can transmit orientation and motion data to other systems. This capability allows instruments to act as motion input devices, facilitating real-time adjustments to 3D imaging, or to be used as virtual pointers to highlight specific areas on a digital display. Such features can be used to analyze anatomical changes or plan intraoperative strategies.

[0089] FollowMeMovement and position tracking technologies can support these capabilities by providing dimensional tracking and motion data from instruments lacking internal capabilities. Accelerometers can wirelessly transmit data to other systems, enabling the monitoring and control of device movement. For example, such systems can detect end-effector movement, focus shift, or auditory interference to determine focus and surgical efficiency. These tools can be used as laser pointers for digital screens, impacting surgeons' ability to interact with visual aids while maintaining a sterile environment.

[0090] Monitoring the operating room (OR), staff, and surgical users can include tracking surgical progress, staff interactions, and available instruments to effectively control the flow of information. The system can utilize data on job, outcome, and limitation (JOC) and surgical plans or procedures to dynamically adjust how information is prioritized. For example, changes in data flow priority can reflect a user's role, the task at hand, and the performance of instruments or workflows that appear to deviate from the expected course.

[0091] The system tracks surgical steps to monitor and measure the achievement of surgical goals. An automated display updates the surgical checklist in real time to align with the predetermined steps of the procedure. For example, during laparoscopic or robotic surgery, the system provides visual cues to guide the surgical team through the process, ensuring that (e.g., critical) tasks are completed in the correct sequence. Efficiency monitoring tools assess the smoothness of surgical progress and identify bottlenecks or delays that may indicate opportunities for process improvement.

[0092] Processing 56642 may include determining the current surgical task based on instrument data and calculations to align display adjustments with a hierarchical structure of information priorities. Processing 56642 may include comparing real-time surgical data with historical surgical records to identify patterns, discrepancies, or potential areas for improvement. Patient-specific information may be integrated into the decision-making process, enabling the system to tailor its output to reflect individual patient, surgical stage, and operator preferences.

[0093] The hierarchical structure of display levels managed in processing 56642 can take into account the risk level associated with patient data, the complexity of the surgical task, and the importance of document or technology adjustments. For example, higher-risk data, such as abnormal vital signs or devices, can be salienced on the display, while lower-priority information can be filtered or routed to other relevant parties in the operating room. This stage may include generating recommendations for the performance of surgical techniques or devices based on comparisons of historical and real-time data.

[0094] Technology can aid in decision support for surgical tasks to achieve key objectives, such as accessing specific anatomical structures or performing transverse incisions. Data flow control and prioritization can operate at the interaction level, which can be pre-defined by the system or navigated based on escalating tasks. For example, notifications can be categorized by importance, type, or source, such as critical datasets (e.g., patient vital signs, blood pressure, blood oxygenation) that are given higher priority. Medium-priority data (such as tissue tension or resistance observable by the surgeon) can be processed differently from low-priority information (such as ergonomic feedback or reporting metrics).

[0095] In the example, data upgrades and management can be environment-sensitive, adapting to the surgical stage, the role of the HCP, and the activity level of instruments. By appropriately routing and prioritizing notifications, the system can improve the efficiency of surgical workflows while keeping the most relevant data focused on for decision-making. For example, while ergonomic data can be used for postoperative analysis, it may not interrupt the surgeon's attention during (e.g., critical) tasks, and a balance can be established between situational awareness and cognitive needs.

[0096] Notification buffering and prioritization can manage situations where the number of received or created notifications exceeds available display capacity. Notifications can be buffered until they are acknowledged in some form. Acknowledgment methods can include automatic acknowledgment, where the message is considered addressed after a specified timeout; manual acknowledgment, where the user actively acknowledges receipt of the notification; or semi-automatic acknowledgment, where the system detects secondary signals (such as visual tracking) to determine user acknowledgment. Such buffering can allow the creation of notification history, enabling users to review previous messages for situational awareness or surgical analysis.

[0097] The ordering of buffered notifications can follow established priority ranking methods. For example, notifications can be organized based on a First-In-First-Out (FIFO) approach, which prioritizes messages according to the order they are received. The system can apply a hierarchical priority model, where messages are ordered based on predetermined importance levels. Ranking allows the most relevant and urgent notifications to be presented quickly, reducing the risk of information overload and enabling efficient decision-making at crucial moments in the process.

[0098] Notification separation and flow control can include routing messages to users based on their respective roles and responsibilities. For example, notifications about patient vital signs might be routed to anesthesiologists, and messages about equipment status might be routed to circulating nurses or technicians. This separation can minimize the burden on users who do not need the information to view unnecessary messages.

[0099] Notifications can be tailored and manipulated based on coexisting messages and the surgical context. For example, if a low-priority notification about an ultrasonic scalpel issue is received simultaneously with a high-priority message about an insufflator, the system can lower the priority of the scalpel notification. Instead of highlighting the scalpel notification, the system can provide an icon or indicator in a less prominent location, such as a corner of the surgeon's monitor, to signal the issue without distracting from the urgent insufflator message. This allows for the integration of multiple notifications while maintaining clarity and focus of the displayed information.

[0100] The integration of notification buffering, sorting, separation, and adjustment enables dynamic and context-sensitive information flow during surgical procedures. By aligning notification presentation with the user's role and the surgical context, the system can determine a balance between situational awareness and cognitive load, ensuring that secondary or less urgent information flows are effectively managed without overlooking (e.g., critical) outputs.

[0101] Information routing within the operating room can depend on the context in which the data is generated and the intended recipient. For example, relevant information can be directed based on the surgical line of sight and the user of the active device. In the example involving a smart-connected endoscopic cutter, an alarm generated when the device is reloaded by a surgical technician may not be directly relevant to the surgeon. The alarm could be routed to a staff monitor, where the surgical technician can address the issue, utilizing geofencing or other contextual tools to determine the most appropriate recipient.

[0102] Routing can be based on message type. Maintenance messages can be delivered to relevant personnel at a customized level of detail. For example, a smart multi-output electrosurgical generator experiencing a damaged output can send a general alert to the surgeon, indicating that the output is disabled. Detailed messages can be directed to the maintenance team, specifying the device's serial number, the damaged output, and potential repairs. Alarms (such as a grounding pad not connected) can be kept within reach of surgical personnel in the operating room, as they have the ability to quickly resolve such issues.

[0103] Prioritizing information can include differentiating between competing data sources to identify the most relevant signals. In the event of multiple alarms occurring simultaneously, prioritization may depend on the type of malfunction, the proximity of the device to the surgical area, or the severity of the alarm. For example, if both the handpiece and the generator report malfunctions, the system may prioritize the generator if its error could affect the handpiece's functionality. Devices physically closer to the patient or having a greater impact on the patient's condition may be prioritized over those further away or with less direct impact.

[0104] The prioritization process may also include a calculated index that combines alarm severity with device criticality. Low-severity alarms from high-criticality devices may receive higher priority than high-severity alarms from low-criticality devices. This allows the system to effectively manage competing sources, utilizing error codes, contextual factors, and manually selected databases to determine information flow during surgery. Devices actively involved in patient treatment (such as instruments and endoscopes) may be prioritized over less directly relevant devices (such as visual monitors).

[0105] In the example, the system could include the ability to dynamically modify the network management and interaction capabilities of the data pipeline based on parameters such as data source, surgical context, or intended destination. For instance, data flow paths could be adjusted to prioritize patient-specific information during (e.g., critical) phases of the surgery, while less urgent data is routed to support staff. This allows relevant information to be delivered to the right stakeholders at the right time, thereby improving surgical efficiency and safety.

[0106] Technical assessment and recommendation systems can be used as tools for healthcare providers, such as surgical residents, to enhance their skills. For example, learning centers can use recorded surgical cases to create task reports or drills of user actions, allowing for comparison of individual actions with outcomes. This pairwise comparison approach can accelerate the development of technical competence by identifying specific areas for improvement. External 2D cameras combined with inertial measurement unit (IMU) data can measure distractions, disturbances, or staff attention during surgery, such as tracking eye movements or head position. These observations can provide residents with actionable feedback, enabling them to improve their skills over time.

[0107] Planning and problem-solving tools can focus on minimizing disruptions in the operating room, such as nurses leaving to retrieve equipment. For example, an external 2D camera can track nurse movement and share data with an online application to generate postoperative reports that identify inefficiencies in material and equipment management. This data can be used to determine preoperative planning, making resources readily available and reducing extended operating room time.

[0108] This system can support the achievement of surgical goals by providing decision support tools tailored to key objectives. These tools can include quality monitoring mechanisms that evaluate results in real-time or retrospectively to assess performance metrics. For example, during hemostasis interventions, the system can track instrument performance and tissue response. Efficiency monitoring tools can highlight opportunities to streamline workflows, such as reducing instrument changeover time or determining the placement of surgical instruments to minimize disruption.

[0109] Surgical workflow monitoring can include instrument tracking and automated updates to checklists to reflect real-time progress. For example, the system can track instrument movement and angles to assess the consistency of manipulation techniques. This data can provide insight into surgeon fatigue, which can lead to errors. By analyzing manipulation angles relative to gravity or table height, the system can recommend ergonomic adjustments, such as rotating the patient or adjusting the table height, to reduce surgeon stress and impact surgical outcomes. Performance metrics can help identify differences in a surgeon's technique compared to their colleagues, thereby improving consistency and consistency.

[0110] Interactive systems can help support life-sustaining activities in surgical care without directly achieving the primary objective. For example, a system can monitor vital signs and endoscopic images to provide continuous feedback on the patient's condition. Anesthesia interventions can be supported by alerts or suggestions to help the anesthesiologist maintain physiological parameters. Life-sustaining interventions, such as managing fluid levels or monitoring devices (e.g., critical ones), can be integrated into the system's monitoring capabilities.

[0111] Determining instrument control can include identifying the individual responsible for a particular device at a given moment. For example, in laparoscopic or robotic surgery, a system can associate instrument use with individual operators, providing personalized postoperative performance analysis. This capability can distinguish between metrics associated with the attending surgeon and those associated with the resident or other team members. User identification can include wearable devices or input sequences linked to specific individuals, such as percutaneous transcutaneous patterned stimulation (taps) from an inertial measurement unit (IMU). Data points can support feedback and contribute to actionable postoperative analyses.

[0112] This system assists with cognitive overhead tasks in surgical care by managing aspects of surgery that do not directly contribute to the primary objective, such as tissue incisions or obtaining anatomical access. These tasks may include life support, environmental management, and supporting decision-making processes, allowing healthcare providers to focus on (e.g., critical) surgical steps. By addressing overhead responsibility, the system reduces the cognitive burden on surgical teams and streamlines the overall workflow.

[0113] Decision-making tools can influence system utility by supporting the prediction, risk assessment, and mapping of physiological systems. For example, systems can predict potential outcomes based on various decision alternatives, providing weighted risk assessments to evaluate trade-offs. Mapping interrelated physiological systems helps surgical teams understand how the information presented relates to broader systemic function. Predictions of organ or system complications based on the current intraoperative situation allow for proactive intervention. These tools can integrate the superposition of comorbidities to refine decision-making, and postoperative follow-up recommendations can ensure continuity of care. Considerations of current instrument and personnel availability can provide additional context, highlighting (e.g., critical) limitations or opportunities during decision-making points.

[0114] Managing the OR environment can include determining the availability and readiness of tools, equipment, and supplies. The system can monitor inventory and equipment status, verifying instrument placement and functionality before surgery begins. This can be extended to real-time monitoring, alerting the surgical team to equipment requiring attention or replacement, thereby minimizing disruptions and delays.

[0115] This system can play a role in reducing distractions that could disrupt focus during surgical procedures. Distractions may include equipment malfunctions, personnel entering or leaving the operating room (OR), and communication interruptions such as telephone calls or pager notifications. By detecting and managing these interruptions, the system can help maintain a controlled environment. For example, IMUs and 2D cameras can be deployed to track noise levels, personnel movement, and other disturbances. Observations from these devices can be delivered as part of postoperative reports, providing data to refine future workflows.

[0116] The system's problem-solving capabilities can include analyzing symptoms, summarizing vital signs, and performing commonalities analysis of previous outcomes. Symptom tracking allows the system to correlate intraoperative findings with pre-existing patient conditions, providing insights into potential complications. Vital sign summaries present aggregated physiological data in an easy-to-understand format, enabling rapid assessment of the patient's condition. Commonalities analysis identifies patterns across historical data, helping to predict outcomes based on similar surgical contexts or patient characteristics.

[0117] At output 56734, the system can generate control signals to adjust the display configuration, thereby prioritizing relevant information to align with the surgical context. For example, data related to elevated patient risk can trigger highlighted alerts, and non-urgent updates can be, for example, de-prioritized or displayed to support staff. The system can provide recommendations for device settings, technical modifications, or other surgical modifications based on identified discrepancies or inefficiencies. The output can be designed to provide the surgical team with actionable observations while maintaining a balance between cognitive load and situational awareness.

[0118] Urgent or rapid response notifications can be prominent when surgeons cannot directly or indirectly observe abnormal or unexpected datasets. For example, such data can be routed to other HCPs not in the operating room, which can be beneficial when a surgical team manages multiple concurrent cases. Controlled escalation of data can depend on factors such as the occurrence, severity, risk, or magnitude of changes in monitored variables. For instance, dissections around (e.g., critical) vascular structures may trigger faster and more prominent notifications compared to less urgent tasks such as gallbladder dissection.

[0119] Alarms and / or notifications can be outputs 56644, where the system modifies their salience and routing based on severity and relevance. For example, a sudden drop in a physiological parameter might trigger an immediate alarm to the surgeon, while less urgent issues, such as low battery alarms for assistive devices, might be directed to the circulating nurse or technician. The system can generate postoperative recommendations or summaries, synthesized from operational observations for future planning or analysis.

[0120] Highlighting abnormal or unexpected data results can depend on the severity of the abnormality and its relevance to the ongoing surgery. Notifications can vary between primary and secondary alerts, where lower-severity data is routed to other HCP or support personnel, such as anesthesiologists or surgical technician nurses, rather than the surgeon. For example, if pulse oximetry data is interrupted but other vital signs remain stable, a notification can be directed to the anesthesiologist without alerting the surgeon. The system can anticipate root causes and recommend corrective actions based on prior user feedback, thus facilitating efficient problem resolution.

[0121] To reduce the cognitive burden on surgeons, notifications can be minimized to focus on aspects of activated functionality. For example, notifications from activated but not yet needed instruments can be sent to non-surgical HCPs, such as circulating nurses. If an endoscope is prematurely activated and malfunctions during laparoscopic anterior resection (LAR), the information can be directed to non-surgical staff rather than surgeons. Once an instrument is no longer in use or in a state of use, such as after visualizing anastomosis, subsequent malfunctions can be routed to support staff or sales representatives for further action.

[0122] Documenting and annotating surgical procedures can include linking symbols and annotations to a timeline of the surgery, enabling the recording of relevant events and decisions. This allows for integration with notification escalation systems, thus balancing output while maintaining the visibility of (e.g., critical) information. Notification escalation can work by prioritizing messages based on their relevance, urgency, and the surgical context, while keeping output available to the user.

[0123] Information can be modified depending on the intended recipient and the context of the notification. For example, while a surgeon might use a concise overview of a device problem, maintenance personnel could benefit from detailed information including error codes and repair instructions. The system can add, copy, or remove information to suit the recipient's role and responsibilities. This flexibility in data management allows for seamless communication and coordination across surgical teams while maintaining focus on the most relevant information.

[0124] Requesting assistance during surgery may include notifications tailored to a specific healthcare provider (HCP) regarding support for a particular functional task. Such notifications can facilitate a coordinated response by promptly alerting the appropriate individual. For example, when an HCP requests assistance for a complex or unexpected task, the system can generate notifications for personnel such as additional surgeons, anesthesiologists, or support staff based on the nature of the assistance.

[0125] The system can issue notifications related to equipment or instruments, prioritizing these notifications based on their relevance to the ongoing task. For example, if an instrument malfunctions during surgery, the system can highlight the problem to the circulating nurse or technician and suggest alternative instruments. Notifications can include priority levels, allowing urgent needs (such as equipment used for life support activities) to be addressed quickly, while less urgent (e.g., critical) needs can be queued or processed with lower priority.

[0126] Documentation during surgery can include the integration of real-time and postoperative data to influence team communication, medical reporting, and overall surgical outcomes. For example, medical reports can focus on surgeon preferences for specific tasks, such as the precise firing of a motorized circular suture device during laparoscopic colorectal anastomosis. By sensing device closure and end-effector movement, the system can transmit this data via Bluetooth to a central hub to determine factors such as position and stability. This can help reduce the incidence of positive intraoperative airtightness tests and postoperative anastomotic leakage.

[0127] Team communication during surgery can be supported by a system designed to record case video for consistency in video documentation for laparoscopic surgery. This can include sensors that use Bluetooth to synchronize with a central hub to detect instruments introduced via cannula. Instruments may feature indicators molded or printed onto their surfaces, which can pair with an external sealed housing to enable synchronization between internal and external operating room cameras. This allows for the capture of movement of instruments and personnel within the operating room, facilitating the generation of surgical timelines.

[0128] Coordinating recordings across multiple devices allows for the documentation of surgical workflows. External operating room cameras monitor the operating room, tracking instrument introduction, removal, and usage patterns, while internal cameras align movement with surgical activities. This coordination produces surgical timelines that can be used for postoperative review, training, and quality assurance. The ability to record such workflows can impact the traceability and accountability of surgical procedures.

[0129] Postoperative annotation and video-based recommendations further support quality and learning. Humans or AI-based systems can annotate recorded videos, providing localized recommendations for technical adjustments or surgical modifications. These observations can be synthesized from complete surgical documentation and shared with surgeons during review meetings. User-based control over the flow of information within the surgical interaction space allows individuals to enable or disable the display of specific information, thus customizing the interface according to their unique preferences and roles. Relevant data can be appropriately highlighted for users.

Claims

1. A surgical system comprising: Processor, the processor being configured to: Receive device data from medical devices during surgical procedures on patients; The current surgical task being performed is determined based on the received instrument data; User preferences for information display are determined based on surgical data from multiple surgical procedures, wherein the surgical data includes one or more of surgeon-specific preferences, surgical background, or patient factors. The display level of information is adjusted based on at least one or more of the following: the current surgical task, the surgical instruments being used, determined user preferences, or a hierarchical structure of multiple display levels, wherein the hierarchical structure is based on one or more of the following: risk to the patient, complexity of the current surgical task, or importance level associated with the data document; and Control signals for the display are generated based on the display level of the adjusted information.

2. The surgical system according to claim 1, wherein, The display level is adjusted by prioritizing information related to the patient's higher risk on the display over information related to the patient's lower risk.

3. The surgical system according to claim 1, wherein, The display level is adjusted by lowering the priority of information associated with the patient's lower risk to that associated with the patient's higher risk on the display, wherein lowering the priority of the information is associated with a reduction in the cognitive burden on the operator of the surgical system.

4. The surgical system according to claim 1, wherein, The processor is also configured to: During the surgical procedure, the operator of the surgical system receives input indicating preferences for the display level of information; as well as The user preferences are stored based on the received input for information display.

5. The surgical system according to claim 1, wherein, The processor is also configured to: The adjusted display level of the information and the current surgical task are compared with surgical data from the multiple surgical procedures; Identify the differences between the surgical procedure and previous surgical procedures from the plurality of surgical procedures; Based on the identified differences, recommendations are made for one or more of the technological improvements or smart device performance enhancements. as well as The control signal is updated to adjust the display level of the information, thereby increasing the priority of the determined recommendation.

6. The surgical system according to claim 1, wherein, The processor is also configured to: Determine patient-specific data including one or more of the patient's anatomical structure, comorbidities, or intraoperative physiological measurements; Update the hierarchy of the multiple display levels to prioritize information related to the patient-specific data; as well as The control signal is updated based on the updated hierarchy to adjust the display level of the information, wherein the updated hierarchy is associated with the display level of information that prioritizes the patient-specific data.

7. The surgical system of claim 6, wherein, The processor is also configured to: The patient-specific data indicates an elevated risk for the patient; as well as An alert is generated based on the increased risk to the patient, wherein prioritizing specific patient data is associated with prioritizing the alert.

8. A method for use in a surgical system, comprising: Receive device data from medical devices during surgical procedures on patients; The current surgical task being performed is determined based on the received instrument data; User preferences for information display are determined based on surgical data from multiple surgical procedures, wherein the surgical data includes one or more of surgeon-specific preferences, surgical background, or patient factors. The display level of information is adjusted based on at least one or more of the following: the current surgical task, the surgical instruments being used, determined user preferences, or a hierarchical structure of multiple display levels, wherein the hierarchical structure is based on one or more of the following: risk to the patient, complexity of the current surgical task, or importance level associated with the data document; and Control signals for the display are generated based on the display level of the adjusted information.

9. The method according to claim 8, wherein, The display level is adjusted by prioritizing information related to the patient's higher risk on the display over information related to the patient's lower risk.

10. The method according to claim 8, wherein, The display level is adjusted by lowering the priority of information associated with the patient's lower risk to that associated with the patient's higher risk on the display, wherein lowering the priority of the information is associated with a reduction in the cognitive burden on the operator of the surgical system.

11. The method of claim 8, wherein, The method includes: During the surgical procedure, the operator of the surgical system receives input indicating preferences for the display level of information; and The user preferences are stored based on the received input for information display.

12. The method according to claim 8, wherein, The method includes: The adjusted display level of the information and the current surgical task are compared with surgical data from the multiple surgical procedures; Identify the differences between the surgical procedure and previous surgical procedures from the plurality of surgical procedures; Based on the identified differences, recommendations are made for one or more of the technological improvements or enhancements to the performance of smart devices; and The control signal is updated to adjust the display level of the information, thereby increasing the priority of the determined recommendation.

13. The method according to claim 1, wherein, The method includes: Determine patient-specific data including one or more of the patient's anatomical structure, comorbidities, or intraoperative physiological measurements; Update the hierarchy of the multiple display levels to prioritize information related to the patient-specific data; and The control signal is updated based on the updated hierarchy to adjust the display level of the information, wherein the updated hierarchy is associated with the display level of information that prioritizes the patient-specific data.

14. The method of claim 13, wherein, The method includes: Determining the patient-specific data to indicate an elevated risk for the patient; and An alert is generated based on the increased risk to the patient, wherein prioritizing specific patient data is associated with prioritizing the alert.

15. A surgical system comprising: Processor, the processor being configured to: Receive device data from medical devices during surgical procedures on patients; The current surgical task being performed is determined based on the received instrument data; User preferences for information display are determined based on surgical data from multiple surgical procedures, wherein the surgical data includes one or more of surgeon-specific preferences, surgical background, or patient factors; and The display level of information is adjusted based on at least one or more of the current surgical task, the surgical instruments being used, determined user preferences, or a hierarchical structure of multiple display levels, wherein the hierarchical structure is based on one or more of the risk to the patient, the complexity of the current surgical task, or the importance level associated with the data document.

16. The surgical system of claim 15, wherein, The display level is adjusted by prioritizing information related to the patient's higher risk on the display over information related to the patient's lower risk.

17. The surgical system of claim 15, wherein, The display level is adjusted by lowering the priority of information associated with the patient's lower risk to that associated with the patient's higher risk on the display, wherein lowering the priority of the information is associated with a reduction in the cognitive burden on the operator of the surgical system.

18. The surgical system of claim 15, wherein, The processor is also configured to: During the surgical procedure, the operator of the surgical system receives input indicating preferences for the display level of information; as well as The user preferences are stored based on the received input for information display.

19. The surgical system according to claim 15, wherein, The processor is also configured to: The adjusted display level of the information and the current surgical task are compared with surgical data from the multiple surgical procedures; Identify the differences between the surgical procedure and previous surgical procedures from the plurality of surgical procedures; Based on the identified differences, recommendations are made for one or more of the technological improvements or smart device performance enhancements. as well as Control signals are generated to adjust the display level of the information, thereby increasing the priority of the determined recommendations.

20. The surgical system of claim 15, wherein, The processor is also configured to: Determine patient-specific data including one or more of the patient's anatomical structure, comorbidities, or intraoperative physiological measurements; Update the hierarchy of the multiple display levels to prioritize information related to the patient-specific data; as well as The control signal is updated based on the updated hierarchy to adjust the display level of the information, wherein the updated hierarchy is associated with the display level of information that prioritizes the patient-specific data.

Images