Cooperative processing of surgical sensor data streams

Abstract

Surgical data processing modification commands may be triggered based on changes in the surgical data processing requirements of a surgical procedure, and may dictate changes in processing such as output frequency, output resolution, processing resource utilization, motion data conversion, etc. The surgical data processing modification commands and systems disclosed herein may be used to implement a variety of processing strategies for surgical sensing, including procedure specific load balancing and sensor prioritization.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application is related to the following applications filed simultaneously, the contents of each of which are incorporated herein by reference. · U.S. Patent Application No. END9290USNP1, titled "METHOD OF ADJUSTING A SURGICAL PARAMETER BASED ON BIOMARKER MEASUREMENTS".

Background Art

[0002] Modern surgical environments may include systems (e.g., sensing systems) that sense and / or monitor aspects of a patient's surgery. The systems may capture surgical - related information such as, for example, biomarkers, surgical tool parameters, etc.

[0003] These sensing systems may operate with a certain level of independence. For example, a surgical environment may include many independent sensing systems each providing its own independent data stream.

[0004] Technical operations of collecting and / or using many independent data streams are difficult tasks. The independent nature of the data streams can complicate their integration and / or combined use. The amount of data and processing can overwhelm systems in a surgical environment. Such problems can prevent medical professionals from being able to appropriately view, interpret, and ultimately act based on this surgery - related information.

Summary of the Invention

Means for Solving the Problems

[0005] According to one embodiment of the present invention, a device for processing surgical data during a surgical procedure is provided. The device may include memory and a processor. The processor may be configured to retrieve a first surgical data processing schema from memory. The processor may be further configured to perform a first processing on a first portion of incoming sensor data according to the first surgical data processing schema for output to a sensor data channel. The processor may be further configured to receive surgical data processing modification commands via a sensor control channel. The processor may be further configured to store a second surgical data processing schema in memory according to a surgical data processing modification command. The second surgical data processing schema may differ from the first surgical data processing schema. The processor may be further configured to perform a second processing on a second portion of incoming sensor data according to the second surgical data processing schema for output to a sensor data channel. The second processing may differ from the first processing.

[0006] The device enables better coordination of data processing during surgical procedures. Data processing behavior can be modified during the processing of incoming sensor data based on data processing modification commands. For example, data processing may be modified based on sensor workload data, surgical procedure planning data, or surgical situation awareness data. Modifications to processing may be motivated by changes in the data processing needs of the system and medical professionals in surgery, or by changes in data processing needs associated with the surgical procedure itself. For example, by modifying the data processing of sensor data, the device can stop or minimize data processing on its own to free up processing capacity for other needs, and pass raw, unprocessed data to another device for processing. Processing behavior may be shifted to other devices or system components where more capacity is available. These measures improve overall system utilization, data processing speed, and data acquisition speed. Furthermore, freeing up the processing capacity of the device itself allows for more processing capacity, for example, as critical in the surgical approach or in the event of a medical emergency. Overall, such processing adjustments improve efficiency, data reliability, fault and failure handling, system flexibility, and overall performance, leading to improved patient safety and improved surgical procedures.

[0007] One embodiment of the present invention provides a method for processing surgical data during a surgical procedure in a system. The method may include, in a second device of the system, performing a first processing on a first portion of incoming sensor data according to a first surgical data processing schema for output to a sensor data channel. The method may further include, in a first device of the system, sending a surgical data processing modification command. The method may further include, in a second device of the system, receiving a surgical data processing modification command via a sensor control channel. The method may further include, in a second device of the system, performing a second processing on a second portion of incoming sensor data according to a second surgical data processing schema for output to a sensor data channel. The second surgical data processing schema may be based on a surgical data processing modification command and may be different from the first surgical data processing schema.

[0008] The method enables better coordination of data processing during surgical procedures. Data processing behavior can be modified during the processing of incoming sensor data based on data processing modification commands. For example, data processing may be modified based on sensor workload data, surgical procedure planning data, or surgical situation awareness data. Modifications to processing may be motivated by changes in the data processing needs of the system and medical professionals in the surgery, or by changes in data processing needs associated with the surgical procedure itself. For example, by modifying the data processing of sensor data, a device can stop or minimize data processing on its own to free up processing capacity for other needs, and pass the raw, unprocessed data to another device for processing. Processing behavior may be shifted to other devices or system components where more capacity is available. These measures improve overall system utilization, data processing speed, and data acquisition speed. Furthermore, freeing up the processing capacity of the device itself allows for more processing capacity, for example, as critical in the surgical approach or in the event of a medical emergency. Overall, such processing adjustments improve efficiency, data reliability, fault and failure handling, system flexibility, and overall performance, leading to improved patient safety and improved surgical procedures.

[0009] According to one embodiment of the present invention, a system for processing surgical data during a surgical procedure is provided. The system may include a first device configured to transmit surgical data processing modification commands. The system may further include a second device configured to perform a first processing on a first portion of incoming sensor data according to a first surgical data processing schema for output to a sensor data channel. The second device may be configured to receive surgical data processing modification commands via a sensor control channel. The second device may be configured to perform a second processing on a second portion of incoming sensor data according to a second surgical data processing schema for output to a sensor data channel. The second surgical data processing schema may be based on surgical data processing modification commands and may differ from the first surgical data processing schema.

[0010] The system enables better coordination of data processing during surgical procedures. Data processing behavior can be modified during the processing of incoming sensor data based on data processing modification commands. For example, data processing may be modified based on sensor workload data, surgical procedure planning data, or surgical situation awareness data. Modifications to processing may be motivated by changes in the data processing needs of the system and medical professionals during surgery, or by changes in data processing needs associated with the surgical procedure itself. For example, by modifying the data processing of sensor data, a device can stop or minimize data processing on its own to free up processing capacity for other needs, and pass raw, unprocessed data to another device for processing. Processing behavior may be shifted to other devices or system components where more capacity is available. These measures improve overall system utilization, data processing speed, and data acquisition speed. Furthermore, freeing up the processing capacity of the device itself allows for more processing capacity, for example, as critical in the surgical approach or in the event of a medical emergency. Overall, such processing adjustments improve efficiency, data reliability, fault and failure handling, system flexibility, and overall performance, leading to improved patient safety and improved surgical procedures.

[0011] According to one embodiment of the present invention, a system is provided for applying a first processing operation, a second processing operation, and a third processing operation to a surgical sensor data stream during a surgical procedure. The system may include a first surgical system component configured to receive a surgical sensor data stream. The first surgical system component may be further configured to apply a first operation and a second processing operation to a first portion of the surgical sensor data stream. The first surgical system component may be further configured to apply a first processing operation, rather than a second processing operation, to a second portion of the surgical sensor data stream based on the receipt of a surgical data processing modification command. The system may further include a second surgical system component configured to receive a surgical sensor data stream from the first surgical system component. The second surgical system component may be further configured to apply a third operation, rather than a second processing operation, to a first portion of the surgical sensor data stream. The second surgical system component may be further configured to apply a third processing operation and a second processing operation to a second portion of the surgical sensor data stream.

[0012] The system enables better coordination of data processing during surgical procedures. Data processing behavior can be modified during the processing of incoming sensor data based on data processing modification commands. For example, data processing may be modified based on sensor workload data, surgical procedure planning data, or surgical situation awareness data. Modifications to processing may be motivated by changes in the data processing needs of the system and medical professionals during surgery, or by changes in data processing needs associated with the surgical procedure itself. For example, by modifying the data processing of sensor data, a device can stop or minimize data processing on its own to free up processing capacity for other needs, and pass raw, unprocessed data to another device for processing. Processing behavior may be shifted to other devices or system components where more capacity is available. These measures improve overall system utilization, data processing speed, and data acquisition speed. Furthermore, freeing up the processing capacity of the device itself allows for more processing capacity, for example, as critical in the surgical approach or in the event of a medical emergency. Overall, such processing adjustments improve efficiency, data reliability, fault and failure handling, system flexibility, and overall performance, leading to improved patient safety and improved surgical procedures.

[0013] According to one embodiment of the present invention, a system is provided for applying a first processing operation, a second processing operation, and a third processing operation to a surgical sensor data stream during a surgical procedure. The system may include a first surgical system component configured to receive a surgical sensor data stream. The first surgical system component may be further configured to apply a processing operation to a first portion of the surgical sensor data stream. The first surgical system component may be further configured to receive a surgical data processing modification command. The first surgical system component may be further configured not to apply a processing operation to a second portion of the surgical sensor data stream based on the surgical data processing modification command. The system may further include a second surgical system component configured to receive a surgical sensor data stream from the first surgical system component. The second surgical system component may be further configured not to apply a processing operation to a first portion of the surgical sensor data stream. The second surgical system component may be further configured to apply a processing operation to a second portion of the surgical sensor data stream.

[0014] The system enables better coordination of data processing during surgical procedures. Data processing behavior can be modified during the processing of incoming sensor data based on data processing modification commands. For example, data processing may be modified based on sensor workload data, surgical procedure planning data, or surgical situation awareness data. Modifications to processing may be motivated by changes in the data processing needs of the system and medical professionals during surgery, or by changes in data processing needs associated with the surgical procedure itself. For example, by modifying the data processing of sensor data, a device can stop or minimize data processing on its own to free up processing capacity for other needs, and pass raw, unprocessed data to another device for processing. Processing behavior may be shifted to other devices or system components where more capacity is available. These measures improve overall system utilization, data processing speed, and data acquisition speed. Furthermore, freeing up the processing capacity of the device itself allows for more processing capacity, for example, as critical in the surgical approach or in the event of a medical emergency. Overall, such processing adjustments improve efficiency, data reliability, fault and failure handling, system flexibility, and overall performance, leading to improved patient safety and improved surgical procedures.

[0015] The device may be used to process surgical data. For example, the device may be used to process surgical data during a surgical procedure. The device may include memory and a processor. The processor may be configured to retrieve a first surgical data processing schema from memory. The processor may be configured to perform a first processing of a first portion of incoming sensor data according to the first surgical data processing schema. The processor may be configured to output the results to a sensor data channel.

[0016] The process may be configured to receive surgical data processing modification commands via a sensor control channel. The processor may store a second surgical data processing schema in memory in accordance with the surgical data processing modification commands. The second surgical data processing schema may differ from the first surgical data processing schema.

[0017] The processor may be configured to perform a second processing of a second portion of the incoming sensor data according to a second surgical data processing schema. The second processing may differ from the first processing. The processor may be configured to output the results to the sensor data channel.

[0018] Surgical data processing modification commands can be triggered based on changes in the surgical data processing requirements of a surgical procedure. Furthermore, surgical data processing modification commands can instruct changes to processing such as output frequency, output resolution, processing resource utilization, and motion data conversion. The surgical data processing modification commands and systems disclosed herein may be used to implement various processing strategies for surgical sensing, including procedure-specific load balancing and sensor prioritization. [Brief explanation of the drawing]

[0019] [Figure 1A] This is a block diagram of a computer-implemented patient and surgeon monitoring system. [Figure 1B] This is another block diagram of a computer-implemented patient and surgeon monitoring system. [Figure 2A] This shows an example of a surgical monitoring system in the operating room. [Figure 2B] An example of a patient monitoring system (e.g., a controlled patient monitoring system) is shown. [Figure 2C] An example of a patient monitoring system (e.g., an uncontrolled patient monitoring system) is shown. [Figure 3] This shows an exemplary surgical hub paired with various systems. [Figure 4]A surgical data network is shown having a set of communication surgical hubs configured to connect to a set of sensing systems, an environmental sensing system, a set of devices, etc. [Figure 5] An exemplary computer-implemented interactive surgical system that may be part of a surgeon monitoring system is illustrated. [Figure 6A] A surgical hub having a plurality of modules coupled to a modular control tower is shown. [Figure 6B] An example of a controlled patient monitoring system is shown. [Figure 6C] An example of an uncontrolled patient monitoring system is shown. [Figure 7A] A logic diagram showing a control system for surgical instruments or tools is shown. [Figure 7B] An exemplary sensing system having a sensor unit and data processing and communication units is shown. 0] [Figure 7C] An exemplary sensing system having a sensor unit and data processing and communication units is shown. [Figure 7D] An exemplary sensing system having a sensor unit and data processing and communication units is shown. <舍 [Figure 8] An exemplary timeline of an exemplary surgical procedure showing adjustment of the operating parameters of a surgical device based on the surgeon biomarker level is shown. [Figure 9] A block diagram of a computer-implemented interactive surgeon / patient monitoring system is shown. 1] [Figure 10] An exemplary surgical system is shown including a handle having a controller and a motor, an adapter releasably coupled to the handle, and a loading unit releasably coupled to the adapter. [Figure 11A] An embodiment of a sensing system that may be used to monitor a surgeon biomarker or a patient biomarker is illustrated. [Figure 11B] An embodiment of a sensing system that may be used to monitor a surgeon biomarker or a patient biomarker is illustrated. [Figure 11C] An embodiment of a sensing system that may be used to monitor surgical biomarkers or patient biomarkers is illustrated. [Figure 11D] An embodiment of a sensing system that may be used to monitor surgical biomarkers or patient biomarkers is illustrated. [Figure 12] This is a block diagram of a patient monitoring system or a surgeon monitoring system. [Figure 13] This is a flowchart illustrating an exemplary method for processing surgical data during surgical procedures. [Figure 14] This is a block diagram of an exemplary sensor data processing system. [Figure 15A] These are illustrative messaging diagrams illustrating processing modifications in a surgical sensor system, processing modifications in a surgical sensor data processing device, and processing modifications in both a surgical sensor system and a surgical sensor data processing device, respectively. [Figure 15B] These are illustrative messaging diagrams illustrating processing modifications in a surgical sensor system, processing modifications in a surgical sensor data processing device, and processing modifications in both a surgical sensor system and a surgical sensor data processing device, respectively. [Figure 15C] These are illustrative messaging diagrams illustrating processing modifications in a surgical sensor system, processing modifications in a surgical sensor data processing device, and processing modifications in both a surgical sensor system and a surgical sensor data processing device, respectively. [Figure 16] This is a block diagram of an exemplary surgical data processing schema. [Figure 17] This is a block diagram of an example sensor processing coordinator. [Modes for carrying out the invention]

[0020] Figure 1A is a block diagram of a computer-implemented patient and surgeon monitoring system 20000. The patient and surgeon monitoring system 20000 may include one or more surgeon monitoring systems 20002 and one or more patient monitoring systems (e.g., one or more controlled patient monitoring systems 20003 and one or more uncontrolled patient monitoring systems 20004). Each surgeon monitoring system 20002 may include a computer-implemented interactive surgical system. Each surgeon monitoring system 20002 may include at least one surgical hub 20006 that communicates with a cloud computing system 20008, for example, as shown in Figure 2A. Each patient monitoring system may include at least one surgical hub 20006 or a computing device 20016 that communicates with the cloud computing system 20008, for example, as further described in Figures 2B and 2C. The cloud computing system 20008 may include at least one remote cloud server 20009 and at least one remote cloud storage unit 20010. Each of the surgeon monitoring system 20002, the controlled patient monitoring system 20003, or the uncontrolled patient monitoring system 20004 may include a wearable sensing system 20011, an environmental sensing system 20015, a robotic system 20013, one or more intelligent instruments 20014, a human interface system 20012, etc. The human interface system is also referred to herein as a human interface device. The wearable sensing system 20011 may include one or more surgeon sensing systems and / or one or more patient sensing systems. The environmental sensing system 20015 may include one or more devices used to measure one or more environmental attributes, for example, as further illustrated in Figure 2A. The robotic system 20013 (same as 20034 in Figure 2A) may include multiple devices used to perform surgical procedures, for example, as further illustrated in Figure 2A.

[0021] The surgical hub 20006 may have collaborative interaction with one of several means of displaying images from a laparoscope and information from one or more other smart devices and one or more sensing systems 20011. The surgical hub 20006 may interact with one or more sensing systems 20011, one or more smart devices, and multiple displays. The surgical hub 20006 may be configured to collect measurement data from one or more sensing systems 20011 and to send notification or control messages to one or more sensing systems 20011. The surgical hub 20006 may transmit and / or receive information, including notification information, to a human interface system 20012. The human interface system 20012 may include one or more human interface devices (HIDs). The surgical hub 20006 can transmit and / or receive notification or control information to various devices communicating with the surgical hub, including audio, display, and / or control information.

[0022] Figure 1B is a block diagram of an exemplary relationship between a sensing system 20001, a biomarker 20005, and a physiological system 20007. This relationship may be used in a computer-implemented patient and surgeon monitoring system 20000, as well as in the systems, devices, and methods disclosed herein. For example, the sensing system 20001 may include a wearable sensing system 20011 (which may include one or more surgeon sensing systems and one or more patient sensing systems) and an environmental sensing system 20015, as discussed in Figure 1A. One or more sensing systems 20001 can measure data relating to various biomarkers 20005. One or more sensing systems 20001 may measure the biomarkers 20005 using one or more sensors, such as optical sensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc. One or more sensors may measure the biomarker 20005 as described herein, using one or more of the following sensing techniques: photoplethysmography, electrocardiography, electroencephalography, colorimetric analysis, impedimentary measurement, potentiometric measurement, current measurement, etc.

[0023] Biomarkers 20005 measured by one or more sensing systems 20001 may include, but are not limited to, sleep, core body temperature, maximal oxygen consumption, physical activity, alcohol intake, respiratory rate, oxygen saturation, blood pressure, blood glucose, heart rate variability, hydrogen blood potential, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, cough and sneeze, gastrointestinal motility, gastrointestinal imaging, respiratory bacteria, edema, mental state, sweat, circulating tumor cells, autonomic nervous system tone, circadian rhythm, and / or menstrual cycle.

[0024] Biomarkers 20005 may relate to physiological systems 20007, including but not limited to behavioral and psychological, cardiovascular, renal, cutaneous, nervous, gastrointestinal, respiratory, endocrine, immune, oncological, musculoskeletal, and / or reproductive systems. Information from biomarkers may be determined and / or used, for example, by a computer-implemented patient and surgeon monitoring system 20000. Information from biomarkers can be determined and / or used by the computer-implemented patient and surgeon monitoring system 20000 to, for example, improve the system and / or improve patient outcomes.

[0025] Figure 2A shows an embodiment of a surgical monitoring system 20002 in an operating room. As shown in Figure 2A, a patient is being operated on by one or more healthcare professionals (HCPs). The HCPs are monitored by one or more surgical sensing systems 20020 worn by the HCPs. The HCPs and the environment surrounding them may also be monitored by one or more environmental sensing systems, including, for example, a set of cameras 20021, a set of microphones 20022, and other sensors, which may be deployed in the operating room. The surgical sensing systems 20020 and the environmental sensing systems can communicate with a surgical hub 20006, which can communicate with one or more cloud servers 20009 of a cloud computing system 20008, as shown in Figure 1. The environmental sensing systems may be used to measure one or more environmental attributes, such as the HCP's position in the operating room, HCP movement, ambient noise in the operating room, temperature / humidity in the operating room, etc.

[0026] As shown in Figure 2A, the primary display 20023 and one or more audio output devices (e.g., speakers 20019) are positioned in the sterile field so that they are visible to the operator on the operating table 20024. In addition, a visualization / notification tower 20026 is positioned outside the sterile field. The visualization / notification tower 20026 may include a first non-sterile human-interactive device (HID) 20027 and a second non-sterile HID 20029, which may face opposite each other. The HIDs may be displays, or displays having touchscreens that allow humans to interface directly with the HIDs. The human interface system guided by the surgical hub 20006 may be configured to utilize displays 20027, 20029, and 20023 to coordinate the flow of information to operators inside and outside the sterile field. In one example, the surgical hub 20006 may cause the HIDs (e.g., primary HID 20023) to display notifications and / or information regarding the patient and / or surgical procedure. In one example, the surgical hub 20006 can prompt and / or receive input from personnel in a sterile or non-sterile area. For example, the surgical hub 20006 can display a snapshot of the surgical site recorded by the imaging device 20030 on a non-sterile HID 20027 or 20029 while maintaining a live feed of the surgical site on the primary HID 20023. The snapshot on the non-sterile display 20027 or 20029 can, for example, enable a non-sterile operator to perform diagnostic steps related to the surgical procedure.

[0027] In one embodiment, the surgical hub 20006 is configured to send diagnostic input or feedback entered by a non-sterile operator in the visualization tower 20026 to a primary display 20023 in the sterile field, which can then be viewed by a sterile operator on the operating table. In one example, the input may take the form of modifications to a snapshot displayed on a non-sterile display 20027 or 20029, which can then be sent to the primary display 20023 by the surgical hub 20006.

[0028] Referring to Figure 2A, surgical instrument 20031 is used as part of a surgeon monitoring system 20002 in a surgical procedure. Hub 20006 can be configured to coordinate the flow of information to the display of surgical instrument 20031. For example, in U.S. Patent Application Publication 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety. Diagnostic input or feedback entered by a non-sterile operator in visualization tower 20026 can be sent by Hub 20006 to a surgical instrument display in the sterile field, which can then be viewed by the operator of surgical instrument 20031. Examples of surgical instruments suitable for use with surgical system 20002 are described, for example, under the heading "Surgical Instrument Hardware" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385) filed December 4, 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety.

[0029] Figure 2A shows an example of a surgical system 20002 used to perform a surgical procedure on a patient lying on an operating table 20024 in an operating room 20035. A robotic system 20034 may be used as part of the surgical system 20002 in a surgical procedure. The robotic system 20034 may comprise a surgeon's console 20036, a patient-side cart 20032 (surgical robot), and a surgical robot hub 20033. While the surgeon views the surgical site through the surgeon's console 20036, the patient-side cart 20032 can manipulate at least one detachably connected surgical tool 20037 through a minimally invasive incision in the patient's body. Images of the surgical site are acquired by a medical imaging device 20030, which can be manipulated by the patient-side cart 20032 to orient the imaging device 20030. The robot hub 20033 can be used to process images of the surgical site, which can then be displayed to the surgeon via the surgeon's console 20036.

[0030] Other types of robotic systems can be readily adapted for use with surgical system 20002. Various examples of robotic systems and surgical tools suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0201137(A1) (U.S. Patent Application No. 16 / 209,407), filed 4 December 2018, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," the disclosure of which is incorporated herein by reference in its entirety.

[0031] Various examples of cloud-based analytical methods implemented by Cloud Computing System 20008 and suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1) (U.S. Patent Application No. 16 / 209,403), filed 4 December 2018, entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," the disclosure of which is incorporated herein by reference in its entirety.

[0032] In various embodiments, the imaging device 20030 may comprise at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0033] The optical components of the imaging device 20030 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors can receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0034] One or more illumination sources may be configured to emit electromagnetic energy in the visible and invisible spectra. The visible spectrum, sometimes also called the light spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye), and is sometimes called visible light, or simply light. The typical human eye responds to wavelengths in the air in the range of approximately 380 nm to 750 nm.

[0035] The invisible spectrum (e.g., the non-emission spectrum) is a portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths below approximately 380 nm and above approximately 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths above approximately 750 nm are longer than the red visible spectrum and consist of invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths below approximately 380 nm are shorter than the violet spectrum and consist of invisible ultraviolet, X-rays, and gamma-ray electromagnetic radiation.

[0036] In various embodiments, the imaging device 20030 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with the present disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, gastroscopy (gastroscopy), endoscopes, laryngoscopes, nasopharyngo-neproscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes.

[0037] The imaging device may employ a multispectral monitor to distinguish between topography and underlying structures. A multispectral image captures image data within a specific wavelength range from the entire electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, such as IR and ultraviolet light. Spectral imaging makes it possible to extract additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after the surgical task is completed to perform one or more of the tests described above on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical instruments is necessary in any surgical procedure. The strict sanitary and sterilization conditions required in the “operating room,” i.e., the operating room or treatment room, require the highest possible sterility of all medical devices and instruments. Part of the sterilization process described above includes the need to sterilize everything that comes into contact with the patient or enters the sterile field, including imaging devices and their accessories and components. It will be understood that the sterile field may be considered a specific area that is deemed free of microorganisms, such as within a tray or on a sterile towel, or it may be considered the area immediately surrounding the patient ready for surgical treatment. The sterile field may include cleaned team members wearing appropriate clothing, as well as all equipment and restraints within that area.

[0038] The wearable sensing system 20011 shown in Figure 1 may include one or more sensing systems, for example, a surgical sensing system 20020 as shown in Figure 2A. The surgical sensing system 20020 may include sensing systems for monitoring and detecting a set of physical and / or physiological conditions of a healthcare provider (HCP). An HCP is typically a surgeon or one or more healthcare professionals assisting a surgeon or other healthcare service provider. In one example, the sensing system 20020 may measure a set of biomarkers to monitor the HCP's heart rate. In another embodiment, the sensing system 20020 worn on the surgeon's wrist (e.g., a watch or wristband) may use an accelerometer to detect hand movements and / or tremors and determine the magnitude and frequency of the tremors. The sensing system 20020 can transmit the measurement data associated with the set of biomarkers and the data associated with the surgeon's physical condition to the surgical hub 20006 for further processing. One or more environmental sensing devices may transmit environmental information to the surgical hub 20006. For example, the environmental sensing device may include a camera 20021 for detecting the position of the HCP's hand / body. The environmental sensing device may also include a microphone 20022 for measuring ambient noise in the operating room. Other environmental sensing devices may include, for example, a thermometer for measuring temperature and a hygrometer for measuring ambient humidity in the operating room. The surgical hub 20006, either independently or in communication with a cloud computing system, can use the surgeon biomarker measurement data and / or environmental sensing information to correct the mean delay of a handheld instrument control algorithm or robot interface, for example, to minimize tremor. In one example, the surgeon sensing system 20020 can measure one or more surgeon biomarkers associated with the HCP and transmit the measurement data associated with the surgeon biomarkers to the surgical hub 20006.The surgical sensing system 20020 may use one or more of the following RF protocols to communicate with the surgical hub 20006: Bluetooth®, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low Power Wireless Personal Area Network (6 LoWPAN), Wi-Fi. Surgical biomarkers may include one or more of the following: stress, heart rate, etc. Environmental measurements from the operating room may include ambient noise levels related to the surgeon or patient, surgeon and / or staff movements, surgeon and / or staff attention levels, etc.

[0039] The surgical hub 20006 can adaptively control one or more surgical instruments 20031 using surgical biomarker measurement data associated with the HCP. For example, the surgical hub 20006 can transmit a control program to a surgical instrument 20031 to control its actuators to limit or compensate for fatigue and the use of fine motor skills. The surgical hub 20006 may transmit a control program based on context regarding situational awareness and / or the importance or severity of the task. The control program can instruct the instrument to modify its behavior to provide more control when control is needed.

[0040] Figure 2B shows an example of a patient monitoring system 20003 (e.g., a controlled patient monitoring system). As shown in Figure 2B, a patient in a controlled environment (e.g., a hospital recovery room) may be monitored by multiple sensing systems (e.g., patient sensing system 20041). Patient sensing system 20041 (e.g., a headband) may be used to measure electroencephalography (EEG) to measure the electrical activity of the patient's brain. Patient sensing system 20042 may be used to measure various biomarkers of the patient, including, for example, heart rate, VO2 levels, etc. Patient sensing system 20043 (e.g., a flexible patch attached to the patient's skin) may be used to measure sweat lactate and / or potassium levels by analyzing small amounts of sweat captured from the surface of the skin using microfluidic channels. Patient sensing system 20044 (e.g., a wristband or watch) may be used to measure blood pressure, heart rate, heart rate variability, VO2 levels, etc., using various techniques as described herein. Patient sensing systems 20045 (e.g., a ring worn on the finger) may be used to measure peripheral temperature, heart rate, heart rate variability, VO2 levels, etc., using various technologies as described herein. Patient sensing systems 20041-20045 can communicate with the surgical hub 20006 using a radio frequency (RF) link. Patient sensing systems 20041-20045 may use one or more of the following RF protocols to communicate with the surgical hub 20006: Bluetooth®, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low Power Wireless Personal Area Network (6 LoWPAN), Thread, Wi-Fi, etc.

[0041] Sensing systems 20041-20045 can communicate with surgical hub 20006, which can communicate with remote server 20009 of remote cloud computing system 20008. Surgical hub 20006 also communicates with HID20046. HID20046 can display measurement data associated with one or more patient biomarkers. For example, HID20046 can display blood pressure, oxygen saturation, respiratory rate, etc. HID20046 can display patient or HCP notifications that provide information about the patient, such as information about recovery milestones or complications. In one example, information about recovery milestones or complications may be associated with surgical procedures the patient may have undergone. In one example, HID20046 can display commands for the patient to perform activities. For example, HID20046 can display inhalation commands and exhalation commands. In one example, HID20046 may be part of the sensing system.

[0042] As shown in Figure 2B, the patient and the environment surrounding the patient may be monitored by one or more environmental sensing systems 20015, which may include, for example, microphones (e.g., for detecting ambient noise associated with or around the patient), temperature / humidity sensors, cameras for detecting the patient's breathing patterns, etc. The environmental sensing systems 20015 may communicate with a surgical hub 20006, which then communicates with a remote server 20009 of a remote cloud computing system 20008.

[0043] In one example, the patient sensing system 20044 may receive notification information from the surgical hub 20006 for display on the patient sensing system 20044's display unit or HID. The notification information may include notifications regarding recovery milestones or, for example, notifications regarding complications in the case of postoperative recovery. In one example, the notification information may include a manageable severity level associated with the notification. The patient sensing system 20044 may display the notification and the manageable severity level to the patient. The patient sensing system may alert the patient using haptic feedback. Visual and / or haptic notifications may be accompanied by audible notifications prompting the patient to pay attention to the visual notification provided on the sensing system's display unit.

[0044] Figure 2C shows an example of a patient monitoring system (e.g., an uncontrolled patient monitoring system 20004). As shown in Figure 2C, a patient in an uncontrolled environment (e.g., the patient's residence) is monitored by multiple patient sensing systems 20041-20045. Patient sensing systems 20041-20045 can measure and / or monitor measurement data associated with one or more patient biomarkers. For example, patient sensing system 20041, which is a headband, may be used to measure electroencephalography (EEG). Other patient sensing systems 20042, 20043, 20044, and 20045 are examples of various patient biomarkers being monitored, measured, and / or reported, as shown in Figure 2B. One or more of the patient sensing systems 20041-20045 can transmit measurement data associated with the monitored patient biomarker to a computing device 20047, which can communicate with a remote server 20009 of a remote cloud computing system 20008. Patient detection systems 20041-20045 can use a radio frequency (RF) link to communicate with computing devices 20047 (e.g., smartphones, tablets, etc.). Patient detection systems 20041-20045 can use one or more RF protocols such as Bluetooth®, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low-Power Wireless Personal Area Network (6 LoWPAN), Thread, Wi-Fi, etc., to communicate with computing devices 20047. For example, patient detection systems 20041-20045 can connect to computing devices 20047 via a wireless router, wireless hub, or wireless bridge.

[0045] Computing device 20047 can communicate with remote server 20009, which is part of cloud computing system 20008. For example, computing device 20047 can communicate with remote server 20009 via a cable / FIOS networking node of an Internet service provider. For example, a patient sensing system can communicate directly with remote server 20009. Computing device 20047 or the sensing system can communicate with remote server 20009 via a cellular transmit / receive point (TRP) or base station using one or more of the following cellular protocols: GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long-Term Evolution (LTE) or 4G, LTE Advanced (LTE-A), New Radio (NR), or 5G.

[0046] For example, the computing device 20047 can display information associated with patient biomarkers. For instance, the computing device 20047 can display blood pressure, oxygen saturation, respiratory rate, etc. The computing device 20047 can also display patient or HCP notifications providing information about the patient, such as recovery milestones or complications.

[0047] In one example, a computing device 20047 and / or a patient sensing system 20044 may receive notification information from a surgical hub 20006 for display on the display unit of the computing device 20047 and / or the patient sensing system 20044. The notification information may include notifications regarding recovery milestones or, for example, notifications regarding complications in the case of postoperative recovery. In one example, the notification information may also include a manageable severity level associated with the notification. The computing device 20047 and / or the sensing system 20044 may display the notification and the manageable severity level to the patient. The patient sensing system may also alert the patient using haptic feedback. Visual and / or haptic notifications may be accompanied by audible notifications prompting the patient to pay attention to the visual notifications provided on the display unit of the sensing system.

[0048] Figure 3 shows an exemplary surgical monitoring system 20002 having a surgical hub 20006 paired with a wearable sensing system 20011, an environmental sensing system 20015, a human interface system 20012, a robotic system 20013, and an intelligent instrument 20014. The hub 20006 includes a display 20048, an imaging module 20049, a generator module 20050, a communication module 20056, a processor module 20057, a storage array 20058, and an operating room mapping module 20059. In certain embodiments, as shown in Figure 3, the hub 20006 further includes a fume extraction module 20054 and / or a suction / irrigation module 20055. During surgical procedures, applying energy to tissue for sealing and / or cutting is generally associated with fume extraction, suction of excess fluid, and / or tissue irrigation. Fluid lines, power lines, and / or data lines from different sources often become entangled during surgical procedures. Dealing with this problem during a surgical procedure can result in the loss of valuable time. Untangling lines may require disconnecting them from their corresponding modules, which may necessitate resetting the modules. The modular enclosure 20060 of the hub provides a unified environment for managing power lines, data lines, and fluid lines, reducing the frequency of such line entanglement. Aspects of this disclosure present a surgical hub 20006 for use in surgical procedures involving the application of energy to tissue at a surgical site. The surgical hub 20006 includes a hub enclosure 20060 and a combination generator module slidably receivable within a docking station of the hub enclosure 20060. The docking station includes data and power contacts. The combination generator module includes two or more ultrasonic energy generator components, bipolar RF energy generator components, and unipolar RF energy generator components housed within a single unit.In one embodiment, the combination generator module also includes a fume extraction component, at least one energy supply cable for connecting the combination generator module to a surgical instrument, at least one fume extraction component configured to discharge smoke, fluid and / or particulate matter generated by the application of therapeutic energy to tissue, and a fluid line extending from the remote surgical site to the fume extraction component. In one embodiment, the fluid line may be a first fluid line, and a second fluid line may extend from the remote surgical site to a suction and irrigation module 20055 slidably received within the hub enclosure 20060. In one embodiment, the hub enclosure 20060 may include a fluid interface. Some specific surgical procedures may require the application of two or more energy types to tissue. One energy type may be more beneficial for cutting tissue, while another different energy type may be more beneficial for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Aspects of the present disclosure present a solution in which a modular enclosure 20060 of a hub is configured to house various generators and facilitate bidirectional communication between them. One advantage of the modular enclosure 20060 of a hub is that it allows for the rapid removal and / or replacement of various modules. Aspects of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking station having a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and the first energy generator module is slidably movable to disengage from the electrical engagement with the first power and data contacts.In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy for application to tissue, distinct from a first energy, and a second docking station having a second docking port including a second data contact and a second power contact, wherein the second energy generator module is slidably movable to electrically engage with the power contact and the data contact, and the second energy generator module is slidably movable to disengage from the electrical engagement with the second power contact and the second data contact. In addition, the modular surgical enclosure also includes a communication bus between the first docking port and the second docking port, configured to facilitate communication between the first energy generator module and the second energy generator module. Referring to Figure 3, an aspect of the present disclosure is presented relating to a modular enclosure 20060 with a hub enabling modular integration of a generator module 20050, a fume extraction module 20054, and a suction / irrigation module 20055. The modular enclosure 20060 of the hub further facilitates bidirectional communication between modules 20059, 20054, and 20055. The generator module 20050 may be a generator module 20050 comprising integrated unipolar, bipolar, and ultrasonic components supported within a single housing unit that is slidably inserted into the modular enclosure 20060 of the hub. The generator module 20050 can be configured to connect to a unipolar device 20051, a bipolar device 20052, and an ultrasonic device 20053. Alternatively, the generator module 20050 may comprise a series of unipolar, bipolar, and / or ultrasonic generator modules that interact via the modular enclosure 20060 of the hub. The hub's modular enclosure 20060 may be configured to facilitate the insertion of multiple generators, allowing multiple generators to function as a single generator, and to facilitate bidirectional communication between generators docked to the hub's modular enclosure 20060.

[0049] Figure 4 shows a surgical data network having a set of sensing systems, an environmental sensing system, and a set of other modular devices configured to connect to a cloud, in at least one aspect of the present disclosure, in a medical facility, a patient recovery room, or one or more operating rooms specifically equipped for surgical procedures, or in a room within a medical facility equipped with specialized equipment for surgical procedures, and a set of communication hubs.

[0050] As shown in Figure 4, the surgical hub system 20060 may include a modular communication hub 20065 configured to connect modular devices located within a medical facility to a cloud-based system (e.g., a cloud computing system 20064 which may include a remote server 20067 connected to remote storage 20068). The modular communication hub 20065 and devices may be connected in a room within the medical facility specifically equipped for surgical procedures. In one embodiment, the modular communication hub 20065 may include a network hub 20061 and / or a network switch 20062 that communicates with a network router 20066. The modular communication hub 20065 can also be connected to a local computer system 20063 and can provide local computer processing and data manipulation. The surgical data network associated with the surgical hub system 20060 may be configured as passive, intelligent, or switching. A passive surgical data network acts as a data conduit, enabling data to go from one device (or segment) to another device (or segment) and to cloud computing resources. The intelligent surgical data network enables traffic to pass through the monitored surgical data network and includes additional mechanisms that constitute each port within the network hub 20061 or network switch 20062. The intelligent surgical data network may be referred to as a manageable hub or switch. The switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0051] Modular devices 1a-1n located in the operating room may be connected to a modular communication hub 20065. Network hub 20061 and / or network switch 20062 are connected to a network router 20066, allowing devices 1a-1n to be connected to a cloud computing system 20064 or a local computer system 20063. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and manipulation. Data associated with devices 1a-1n may also be transferred to a local computer system 20063 for local data processing and manipulation. Modular devices 2a-2m located in the same operating room may also be connected to a network switch 20062. Network switch 20062 is connected to network hub 20061 and / or network router 20066, allowing devices 2a-2m to be connected to the cloud 20064. Data associated with devices 2a-2m may be transferred to the cloud computing system 20064 via the network router 20066 for data processing and manipulation. The data associated with devices 2a-2m may also be transferred to the local computer system 20063 for local data processing and manipulation.

[0052] The wearable sensing system 20011 may include one or more sensing systems 20069. The sensing systems 20069 may include a surgeon sensing system and / or a patient sensing system. One or more sensing systems 20069 may communicate with the computer system 20063 or cloud server 20067 of the surgical hub system 20060 directly via one of the network routers 20066, or via a network hub 20061 or network switch 20062 that communicates with the network router 20066.

[0053] The sensing system 20069 may be connected to a network router 20066 to connect the sensing system 20069 to a local computer system 20063 and / or a cloud computing system 20064. Data associated with the sensing system 20069 may be transferred to the cloud computing system 20064 via the network router 20066 for data processing and manipulation. Data associated with the sensing system 20069 may also be transferred to the local computer system 20063 for local data processing and manipulation.

[0054] As shown in Figure 4, the surgical hub system 20060 can be expanded by interconnecting multiple network hubs 20061 and / or multiple network switches 20062 with multiple network routers 20066. The modular communication hub 20065 may be housed in a modular control tower configured to accept multiple devices 1a-1n / 2a-2m. The local computer system 20063 may also be housed in the modular control tower. The modular communication hub 20065 may be connected to a display 20068 to display images acquired by some of the devices 1a-1n / 2a-2m, for example, during a surgical procedure. In various embodiments, devices 1a-1n / 2a-2m may include a variety of modules, particularly among modular devices that can be connected to the modular communication hub 20065 of the surgical data network, such as imaging modules connected to endoscopes, generator modules connected to energy-based surgical devices, smoke extraction modules, suction / irrigation modules, communication modules, processor modules, storage arrays, surgical devices connected to displays, and / or non-contact sensor modules.

[0055] In one embodiment, the surgical hub system 20060 shown in Figure 4 may include a combination of network hubs, network switches, and network routers connecting devices 1a-1n / 2a-2m or sensing systems 20069 to the cloud-based system 20064. One or more of the devices 1a-1n / 2a-2m or sensing systems 20069 connected to the network hub 20061 or network switch 20062 can collect data or measurement data in real time and transfer the data to a cloud computer for data processing and manipulation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to handle software applications. The term “cloud” may be used as a metaphor for “Internet,” but the term is not limited to that. Therefore, the term “cloud computing” may be used herein to refer to “a type of internet-based computing” in which various services such as servers, storage, and applications are delivered via the internet to a modular communication hub 20065 and / or computer system 20063 located in the operating room (e.g., a fixed, mobile, temporary, or on-site operating room or space), and to devices connected to the modular communication hub 20065 and / or computer system 20063. The cloud infrastructure may be maintained by a cloud service provider. In this context, the cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located in one or more operating rooms. The cloud computing service can perform numerous calculations based on data collected by smart surgical instruments, robots, sensing systems, and other computerized devices located in the operating room. The hub hardware enables multiple devices, sensing systems, and / or connections to connect to a computer that communicates with cloud computing resources and storage.

[0056] By applying cloud computing data processing technology to data collected by devices 1a-1n / 2a-2m, the surgical data network can provide improved surgical outcomes, reduced costs, and improved patient satisfaction. At least some of devices 1a-1n / 2a-2m can be used to observe the condition of tissue after tissue sealing and cutting procedures and to evaluate leakage or perfusion of sealed tissue. At least some of devices 1a-1n / 2a-2m can be used to examine data, including images of body tissue samples, for diagnostic purposes using cloud-based computing to identify pathologies such as the effects of disease. Such data may include tissue localization and margin confirmation, as well as phenotype. At least some of devices 1a-1n / 2a-2m can be used to identify anatomical structures of the body using various sensors integrated with imaging devices and techniques such as overlaying images captured by multiple imaging devices. Data collected by devices 1a-1n / 2a-2m, including image data, may be transferred to the cloud computing system 20064 or the local computer system 20063, or both, for data processing and manipulation, including image processing and manipulation. The data may be analyzed to improve the outcomes of surgical procedures by determining whether further treatments, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robots, can be performed on tissue-specific sites and conditions. Such data analysis may further involve prognostic analysis, and the use of standardized methods can provide useful feedback for either confirming surgical treatment and surgeon behavior, or suggesting modifications to surgical treatment and surgeon behavior.

[0057] By applying cloud computing data processing technology to the measurement data collected by the sensing system 20069, the surgical data network can provide improved surgical outcomes, improved recovery outcomes, reduced costs, and improved patient satisfaction. At least some of the sensing systems 20069 may be employed to assess the physiological state of a surgeon operating on a patient, a patient being prepared for a surgical procedure, or a patient recovering after a surgical procedure. A cloud-based computing system 20064 may be used to monitor biomarkers associated with the surgeon or patient in real time, generate a surgical procedure plan based at least on measurement data collected before the surgical procedure, provide control signals to surgical instruments during the surgical procedure, and notify the patient of complications during the postoperative period.

[0058] Operating room devices 1a-1n may be connected to the modular communication hub 20065 via a wired or wireless channel, depending on the configuration of devices 1a-1n with respect to the network hub 20061. In one embodiment, the network hub 20061 may be implemented as a local network broadcast device operating on the physical layer of the Open System Interconnection (OSI) model. The network hub can provide connectivity to devices 1a-1n located within the same operating room network. The network hub 20061 can collect data in packet form and transmit them to the router in half-duplex mode. The network hub 20061 cannot store any media access control (MAC / Internet Protocol, IP) for transferring any device data. Only one of devices 1a-1n can transmit data through the network hub 20061 at a time. The network hub 20061 may not have a routing table or intelligence regarding where to send information and broadcasts all network data to a remote server 20067 of the cloud computing system 20064 via each connection. While the Network Hub 20061 can detect basic network errors such as collisions, broadcasting all information to multiple ports poses a security risk and may cause bottlenecks.

[0059] Operating room devices 2a-2m may be connected to network switch 20062 via a wired or wireless channel. Network switch 20062 operates within the data link layer of the OSI model. Network switch 20062 may be a multicast device for connecting devices 2a-2m located in the same operating room to the network. Network switch 20062 can transmit data in the form of frames to network router 20066 and operates in full-duplex mode. Multiple devices 2a-2m can transmit data simultaneously through network switch 20062. Network switch 20062 stores and uses the MAC addresses of devices 2a-2m to transfer data.

[0060] The network hub 20061 and / or network switch 20062 may be coupled to the network router 20066 to connect to the cloud computing system 20064. The network router 20066 operates within the network layer of the OSI model. The network router 20066 creates a path for sending data packets received from the network hub 20061 and / or network switch 20062 to cloud-based computing resources for further processing and manipulation of data collected by one or all of the devices 1a-1n / 2a-2m and the wearable sensing system 20011. The network router 20066 may be used to connect two or more different networks located in different locations, such as different networks located in different operating rooms of the same medical facility or different operating rooms of different medical facilities. The network router 20066 can transmit data in packet form to the cloud computing system 20064 and operates in full-duplex mode. Multiple devices can transmit data simultaneously. The network router 20066 may use IP addresses to transfer data.

[0061] In one example, the network hub 20061 may be implemented as a USB hub that enables multiple USB devices to be connected to a host computer. The USB hub can extend a single USB port into several layers so that there are more ports available for connecting devices to the host system computer. The network hub 20061 may include wired or wireless functionality for receiving information via a wired or wireless channel. In one embodiment, a wireless USB short-range high-bandwidth wireless communication protocol may be used for communication between devices 1a-1n and devices 2a-2m located in the operating room.

[0062] In the example, the operating room devices 1a-1n / 2a-2m and / or sensing system 20069 can exchange data over short distances from fixed and mobile devices (using short-wavelength UHF radio waves in the 2.4-2.485 GHz ISM band) and communicate with the modular communication hub 20065 via the Bluetooth radio technology standard to build a personal area network (PAN). The operating room devices 1a-1n / 2a-2m and / or sensing systems 20069 may communicate with the modular communication hub 20065 via several wireless or wired communication standards or protocols, including but not limited to Bluetooth, Low-Energy Bluetooth, Near Field Communication (NFC), Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, New Radio (NR), Long-Term Evolution (LTE), and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, and their Ethernet derivatives, as well as any other wireless and wired protocols designated as 3G, 4G, 5G, and later. The computing module may include multiple communication modules. For example, the first communication module may be dedicated to shorter-range wireless communication such as Wi-Fi, Bluetooth, Low-Energy Bluetooth, and Bluetooth Smart, while the second communication module may be dedicated to longer-range wireless communication such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, and TDMA.

[0063] The modular communication hub 20065 can function as one or more central connections for operating room devices 1a-1n / 2a-2m and / or sensing systems 20069 and can handle a data type known as a frame. A frame can carry data generated by devices 1a-1n / 2a-2m and / or sensing systems 20069. Once a frame is received by the modular communication hub 20065, the frame can be amplified and / or transmitted to a network router 20066, which can transfer the data to a cloud computing system 20064 or a local computer system 20063 by using some wireless or wired communication standards or protocols as described herein.

[0064] The modular communication hub 20065 may be used as a standalone device or connected to compatible network hubs 20061 and network switches 20062 to form a larger network. Because the modular communication hub 20065 is generally easy to install, configure, and maintain, it can be a good choice for networking operating room devices 1a-1n / 2a-2m.

[0065] Figure 5 shows a computer-implemented interactive surgical system 20070, which may be part of the surgeon monitoring system 20002. The computer-implemented interactive surgical system 20070 is similar in many respects to the surgeon sensing system 20002. For example, the computer-implemented interactive surgical system 20070 may include one or more surgical subsystems 20072, which are similar in many respects to the surgeon monitoring system 20002. Each quasi-surgical system 20072 includes at least one surgical hub 20076 that communicates with a cloud computing system 20064, which may include a remote server 20077 and remote storage 20078. In one embodiment, the computer-implemented interactive surgical system 20070 may include a modular control tower 20085 connected to multiple operating room devices such as sensing systems (e.g., surgeon sensing system 20002 and / or patient sensing system 20003), intelligent surgical instruments, robots, and other computerized devices placed in the operating room. As shown in Figure 6A, the modular control tower 20085 may include a modular communication hub 20065 connected to the local computing system 20063.

[0066] As shown in the embodiment of Figure 5, the modular control tower 20085 can be connected to an imaging module 20088 which can be connected to an endoscope 20087, a generator module 20090 which can be connected to an energy device 20089, a smoke exhaust module 20091, a suction / irrigation module 20092, a communication module 20097, a processor module 20093, a storage array 20094, a smart device / instrument 20095 optionally connected to displays 20086 and 20084, respectively, and a non-contact sensor module 20096. The modular control tower 20085 may also communicate with one or more sensing systems 20069 and an environmental sensing system 20015. The sensing system 20069 may be connected to the modular control tower 20085 directly via a router or via the communication module 20097. The operating room devices may be connected to cloud computing resources and data storage via the modular control tower 20085. The robotic surgery hub 20082 can also connect to the modular control tower 20085 and cloud computing resources. Devices / instruments 20095 or 20084, in particular, the human interface system 20080, can be connected to the modular control tower 20085 via wired or wireless communication standards or protocols, as described herein. The human interface system 20080 may include a display subsystem and a notification subsystem. The modular control tower 20085 may be connected to a hub display 20081 (e.g., a monitor, screen) to display and overlay images received from the imaging module 20088, the device / instrument display 20086 and / or other human interface system systems 20080. The hub display 20081 may also display data received from devices connected to the modular control tower 20085 along with the images and overlay images.

[0067] Figure 6A shows a surgical hub 20076 comprising several modules connected to a modular control tower 20085. As shown in Figure 6A, the surgical hub 20076 may be connected to a generator module 20090, a smoke exhaust module 20091, a suction / irrigation module 20092, and a communication module 20097. The modular control tower 20085 may comprise a modular communication hub 20065, such as a network connectivity device, and a computer system 20063, which provides, for example, local wireless connectivity with sensing systems, local processing, complication monitoring, visualization, and imaging. As shown in Figure 6A, the modular communication hub 20065 can be extended to several modules (e.g., devices) and several sensing systems 20069 that can be connected to the modular communication hub 20065, and can be connected in a configuration (e.g., a tiered configuration) for transferring data related to the modules and / or measurement data related to the sensing systems 20069 to the computer system 20063, cloud computing resources, or both. As shown in Figure 6A, each of the network hubs / switches 20061 / 20062 within the modular communication hub 20065 may include three downstream ports and one upstream port. The upstream network hub / switch may be connected to processor 20102 to provide communication connectivity to cloud computing resources and local display 20108. At least one of the network / hub switches 20061 / 20062 within the modular communication hub 20065 may have at least one wireless interface to provide communication connectivity between sensing system 20069 and / or device 20095 and cloud computing system 20064. Communication to cloud computing system 20064 can be done via either a wired communication channel or a wireless communication channel.

[0068] The surgical hub 20076 may use non-contact sensor module 20096 to measure the dimensions of an operating room and to generate a map of the surgical site using either an ultrasonic non-contact measuring device or a laser non-contact measuring device. In U.S. Provisional Patent Application No. 62 / 611,341, filed December 28, 2017, entitled “INTERACTIVE SURGICAL PLATFORM,” which is incorporated herein by reference in its entirety, the sensor module is configured to determine the size of an operating room and adjust the Bluetooth pairing distance limit. However, as described in the section “Surgical Hub Spatial Awareness Within an Operating Room” in the same document, an ultrasonic-based non-contact sensor module may scan the operating room by transmitting bursts of ultrasound and receiving echoes as the ultrasound bursts reflect off the outer walls of the operating room. A laser-based non-contact sensor module may scan the operating room, for example, by transmitting laser light pulses, receiving the laser light pulses reflected off the outer walls of the operating room, comparing the phase of the transmitted pulses with the received pulses to determine the size of the operating room and adjust the Bluetooth pairing distance limit.

[0069] The computer system 20063 may include a processor 20102 and a network interface 20100. The processor 20102 may be connected via a system bus to a communication module 20103, storage 20104, memory 20105, non-volatile memory 20106, and an input / output (I / O) interface 20107. The system bus may be any of several types of bus structures, including memory buses or memory controllers, peripheral buses or external buses, and / or local buses, using any variety of available bus architectures. Examples of such architectures include, but are not limited to, 9-bit buses, Industrial Standard Architecture (ISA), Micro-Charmel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association (PCMCIA), Small Computer Systems Interface (SCSI), or any other proprietary bus.

[0070] The processor 20102 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer to improve performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse-width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels. Further details are available in the product datasheet.

[0071] For example, the 20102 processor may include a safety controller, which may include two controller-based families, such as the TMS570 and RM4x, also from Texas Instruments and known by the Hercules ARM Cortex R4 trade names. The safety controller may be configured specifically for IEC61508 and ISO26262 safety limit applications, in particular, to provide a highly integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0072] System memory can be described as volatile memory or non-volatile memory. The Basic Input / Output System (BIOS), which includes basic routines for transferring information between elements within the computer system during startup, is stored in non-volatile memory. For example, non-volatile memory can include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Volatile memory can include random access memory (RAM), which functions as external cache memory. Furthermore, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), sync-link DRAM (SLDRAM), and direct rhombus RAM (DRRAM).

[0073] Computer System 20063 may also include removable / non-removable volatile / non-volatile computer storage media, such as disk storage devices. Examples of disk storage devices include, but are not limited to, magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, disk storage devices may include the above-mentioned storage media independently or in combination with other storage media. Examples of other storage media include, but are not limited to, optical disk drives such as compact disk ROM devices (CD-ROMs), compact disk recordable drives (CD-R drives), compact disk rewritable drives (CD-RW drives), or digital multi-purpose disk ROM drives (DVD-ROMs). Removable or non-removable interfaces may be used to facilitate connection of disk storage devices to the system bus.

[0074] It should be understood that computer system 20063 may include software that acts as an intermediary between the user and basic computer resources as described in a suitable operating environment. Such software may include an operating system. An operating system, which may be stored on disk storage, may function to control and allocate the resources of the computer system. System applications may leverage resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein can be implemented in various operating systems or combinations of operating systems.

[0075] The user can input commands or information to the computer system 20063 via input devices connected to the I / O interface 20107. Examples of input devices include, but are not limited to, pointing devices such as mice, trackballs, styluses, and touchpads; keyboards; microphones; joysticks; gamepads; satellite receivers; scanners; TV tuner cards; digital cameras; digital video cameras; and webcams. These and other input devices connect to the processor 20102 via interface ports and the system bus. Examples of interface ports include serial ports, parallel ports, game ports, and USB ports. Output devices use some of the same types of ports as the input devices. Therefore, for example, a USB port may be used to provide input to the computer system 20063 and output information from the computer system 20063 to an output device. Output adapters may be provided to indicate that some output devices, such as monitors, displays, speakers, and printers, may be present, among others, which may require special adapters. Examples of output adapters include video and sound cards that provide a means of connection between the output device and the system bus, but these are illustrative and not limiting. Note that other devices and / or systems of devices, such as remote computers, can provide both input and output functions.

[0076] Computer System 20063 can operate in a networked environment using logical connections to one or more remote computers, such as cloud computers, or to local computers. Remote cloud computers may be personal computers, servers, routers, network PCs, workstations, microprocessor-based devices, peer devices, or other common network nodes, but typically include many or all of the elements described in relation to computer systems. For brevity, only memory storage devices are shown along with the remote computers. Remote computers may be logically connected to the computer system via a network interface, and subsequently physically connected via a communication interface. Network interfaces may include communication networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies include, but are not limited to, fiber optic distributed data interfaces (FDDI), copper distributed data interfaces (CDDI), Ethernet / IEEE 802.3, and Token Ring / IEEE 802.5. WAN technologies include, but are not limited to, point-to-point links, integrated service digital networks (ISDN) and their variations, packet-switched networks, and digital subscriber lines (DSL).

[0077] In various examples, the computer system 20063, imaging module 20088 and / or human interface system 20080 in Figures 4, 6A and 6B, and / or processor module 20093 in Figures 5 and 6A may include an image processor, image processing engine, media processor, or any dedicated digital signal processor (DSP) used for processing digital images. The image processor can be made faster and more efficient using parallel computing with single-instruction multiple data (SIMD) or multiple-instruction multiple data (MIMD) techniques. The digital image processing engine can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0078] A communication connection(s) may refer to hardware / software used to connect a network interface to a bus. For illustrative purposes, the communication connection(s) are shown as being inside computer system 20063, but they may also be external to computer system 20063. For illustrative purposes only, hardware / software required for connection to a network interface may include internal and external technologies such as standard telephone-grade modems, cable modems, fiber optic modems, and DSL modems, ISDN adapters, and Ethernet cards. In some examples, the network interface may also be provided using an RF interface.

[0079] Figure 6B shows an example of a wearable monitoring system, such as a controlled patient monitoring system. The controlled patient monitoring system may be a sensing system used to monitor a set of patient biomarkers while the patient is in a healthcare facility. The controlled patient monitoring system can be deployed for pre-surgical patient monitoring while the patient is preparing for a surgical procedure, intra-surgical monitoring while the patient is undergoing surgery, or post-surgical monitoring, for example, while the patient is recovering. As shown in Figure 6B, the controlled patient monitoring system may include a surgical hub system 20076 which may include one or more routers 20066 of a modular communication hub 20065 and a computer system 20063. The routers 20065 may include wireless routers, wired switches, wired routers, wired or wireless network hubs, etc. In one example, the routers 20065 may be part of the infrastructure. Computing system 20063 may provide local processing for monitoring various biomarkers associated with the patient or surgeon, and a notification mechanism for informing the patient and / or healthcare provider (HCP) that milestones (e.g., recovery milestones) have been met or that complications have been detected. Computing system 20063 of surgical hub system 20076 may also be used to generate notifications, for example, severity levels associated with a notification that complications have been detected.

[0080] The computing system 20063 in Figures 4 and 6B, the computing device 20200 in Figure 6C, and the hub / computing device 20243 in Figures 7B, 7C, or 7D may be a surgical computing system or hub device, a laptop, tablet, smartphone, etc.

[0081] As shown in Figure 6B, a set of sensing systems 20069 and / or environmental sensing systems 20015 (as described in Figure 2A) may be connected to the surgical hub system 20076 via a router 20065. The router 20065 may also provide a direct communication connection between the sensing systems 20069 and the cloud computing system 20064 without involving, for example, the local computer system 20063 of the surgical hub system 20076. Communication from the surgical hub system 20076 to the cloud 20064 can be carried out via either a wired or wireless communication channel.

[0082] As shown in Figure 6B, the computer system 20063 may include a processor 20102 and a network interface 20100. The processor 20102 may be connected via a system bus to a radio frequency (RF) interface or communication module 20103, storage 20104, memory 20105, non-volatile memory 20106, and input / output interface 20107, as shown in Figure 6A. The computer system 20063 may be connected to a local display unit 20108. In some examples, the display unit 20108 may be replaced by a HID. Details of the hardware and software components of the computer system are provided in Figure 6A.

[0083] As shown in Figure 6B, the sensing system 20069 may include a processor 20110. The processor 20110 may be connected via a system bus to a radio frequency (RF) interface 20114, storage 20113, memory (e.g., non-volatile memory) 20112, and an I / O interface 20111. The system bus may be one of several types of bus structures, including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus, as described herein. The processor 20110 may be any single-core or multi-core processor as described herein.

[0084] It should be understood that the sensing system 20069 may include software that acts as an intermediary between the sensing system user and computer resources, as described in a preferred operating environment. Such software may include an operating system. An operating system, which may be stored on disk storage, may function to control and allocate the resources of the computer system. System applications may leverage resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein can be implemented on various operating systems or combinations of operating systems.

[0085] The sensing system 20069 may be connected to the human interface system 20115. The human interface system 20115 may be a touchscreen display. The human interface system 20115 may include a human interface display for displaying information associated with surgeon biomarkers and / or patient biomarkers, display prompts for user actions by the patient or surgeon, or display notifications to the patient or surgeon indicating information about recovery milestones or complications. The human interface system 20115 may be used to receive input from the patient or surgeon. Other human interface systems may be connected to the sensing system 20069 via the I / O interface 20111. For example, the human interface device 20115 may include a device for providing haptic feedback as a mechanism for prompting the user to pay attention to notifications that may be displayed on the display unit.

[0086] The sensing system 20069 can operate in a networked environment using logical connections to one or more remote computers, such as cloud computers, or to local computers. Remote cloud computers may be personal computers, servers, routers, network PCs, workstations, microprocessor-based devices, peer devices, or other common network nodes, but typically include many or all of the elements described with respect to computer systems. Remote computers may be logically connected to computer systems via network interfaces. Network interfaces can include communication networks such as local area networks (LANs), wide area networks (WANs), and / or mobile networks. LAN technologies include fiber optic distributed data interfaces (FDDI), copper distributed data interfaces (CDDI), Ethernet / IEEE 802.3, Token Ring / IEEE 802.5, and Wi-Fi / IEEE 802.11. WAN technologies include, but are not limited to, point-to-point links, integrated service digital networks (ISDN) and their variations, packet-switched networks, and digital subscriber lines (DSL). The mobile network may include communication links based on one or more mobile communication protocols such as GSM® / GPRS / EDGE (2G), UMTS / HSPA (3G), Long-Term Evolution (LTE) or 4G, LTE Advanced (LTE-A), New Radio (NR), or 5G.

[0087] Figure 6C illustrates an exemplary uncontrolled patient monitoring system, for example, when the patient is away from the medical facility. An uncontrolled patient monitoring system may be used for preoperative patient monitoring when the patient is being prepared for a surgical procedure but is away from the medical facility, or for postoperative monitoring when the patient is recovering away from the medical facility.

[0088] As illustrated in Figure 6C, one or more sensing systems 20069 communicate with a computing device 20200, such as a personal computer, laptop, tablet, or smartphone. The computing system 20200 can provide a notification mechanism to indicate that various biomarkers associated with the patient have been monitored, that milestones (e.g., recovery milestones) have been met, or that complications have been detected. The computing system 20200 can also provide instructions for the user of the sensing system to follow. Communication between the sensing system 20069 and the computing device 20200 may be established directly using a wireless protocol as described herein, or via a wireless router / hub 20211.

[0089] As shown in Figure 6C, the sensing system 20069 may be connected to the computing device 20200 via a router 20211. The router 20211 may include a wireless router, a wired switch, a wired router, a wired or wireless network hub, etc. The router 20211 may, for example, provide a direct communication connection between the sensing system 20069 and the cloud server 20064 without involving the local computing device 20200. The computing device 20200 can communicate with the cloud server 20064. For example, the computing device 20200 may communicate with the cloud 20064 via a wired or wireless communication channel. In one example, the sensing system 20069 may communicate directly with the cloud over a cellular network, for example, via a cellular base station 20210.

[0090] As shown in Figure 6C, the computing device 20200 may include a processor 20203 and a network or RF interface 20201. The processor 20203 may be connected via a system bus to storage 20202, memory 20212, non-volatile memory 20213, and input / output interface 20204, as described in Figures 6A and 6B. Details of the hardware and software components of the computer system are provided in Figure 6A. The computing device 20200 may include a set of sensors, e.g., sensor #1 20205, sensor #2 20206, sensor #n 20207. These sensors may be part of the computing device 20200 and may be used to measure one or more attributes associated with a patient. The attributes may provide context for biomarker measurements performed by one of the sensing systems 20069. For example, sensor #1 may be an accelerometer that can be used to measure acceleration forces to sense motion or vibration associated with a patient. For example, sensors 20205-20207 may include one or more of the following: pressure sensors, altimeters, thermometers, lidars, etc.

[0091] As shown in Figure 6B, the sensing system 20069 may include a processor, a radio frequency interface, storage, memory or non-volatile memory, and input / output interfaces via a system bus, as described in Figure 6A. The sensing system may also include a sensor unit and processing and communication units, as shown in Figures 7B-7D. The system bus may be one of several types of bus structures, including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus, as described herein. The processor may be any single-core or multi-core processor, as described herein.

[0092] The sensing system 20069 can communicate with the human interface system 20215, which may be a touchscreen display. The human interface system 20215 may be used to display information related to patient biomarkers, to display prompts for user actions by the patient, or to display notifications to the patient indicating information about recovery milestones or complications. The human interface system 20215 may be used to receive input from the patient. Other human interface systems may be connected to the sensing system 20069 via I / O interfaces. For example, the human interface system may include a device for providing haptic feedback as a mechanism to prompt the user to pay attention to notifications that may be displayed on the display unit. As illustrated in Figure 6B, the sensing system 20069 can operate in a networked environment using logical connections to one or more remote computers, such as cloud computers, or to local computers.

[0093] Figure 7A shows a logic diagram of a control system 20220 for a surgical instrument or surgical tool according to one or more embodiments of the present disclosure. The surgical instrument or surgical tool may be configurable. The surgical instrument may include surgical fixation devices specific to a particular procedure, such as imaging devices, surgical staplers, energy devices, and endocutter devices. For example, the surgical instrument may include any of the following: electric staplers, electric stapler generators, energy devices, high-energy devices, high-energy jaw devices, endocutter clamps, energy device generators, intraoperative imaging systems, smoke exhausters, suction irrigation devices, and air supply systems. The system 20220 may include a control circuit. The control circuit may include a microcontroller 20221 having a processor 20222 and memory 20223. For example, one or more of sensors 20225, 20226, and 20227 provide real-time feedback to the processor 20222. Motor 20230, driven by motor driver 20229, drives the I-beam knife element by operably connecting a longitudinally movable displacement member. A tracking system 20228 may be configured to determine the position of the longitudinally movable displacement member. Position information may be provided to processor 20222, which can be programmed or configured to determine the position of the longitudinally movable drive member, as well as the positions of the launch member, launch bar, and I-beam knife element. Additional motors may be provided to the tool driver interface to control I-beam firing, occluder movement, shaft rotation, and joint movement. Display 20224 can display various operating conditions of the instrument and may include touchscreen functionality for data input. Information displayed on display 20224 can be overlaid with images acquired via the endoscopic imaging module.

[0094] In one embodiment, the microcontroller 20221 may be any single-core or multi-core processor, such as those known by the trade name ARM Cortex from Texas Instruments. In one embodiment, the main microcontroller 20221 may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments, which includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer for improving performance above 40MHz, 32KB single-cycle SRAM, internal ROM with StellarisWare® software, 2KB EEPROM, one or more PWM modules, one or more QEI analogs, and / or one or more 12-bit ADCs with 12 analog input channels.

[0095] In one embodiment, the microcontroller 20221 may include a safety controller, which may include two controller-based families, such as the TMS570 and RM4x, also from Texas Instruments and known by the trade names Hercules ARM Cortex R4. The safety controller may be configured specifically for IEC61508 and ISO26262 safety limit applications, in particular, to provide an advanced integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0096] The microcontroller 20221 may be programmed to perform various functions, such as precise control of the speed and position of the knife and joint motion systems. In one embodiment, the microcontroller 20221 may include a processor 20222 and memory 20223. The electric motor 20230 may be a brushed direct current (DC) motor with a gearbox and a mechanical coupling to the joint motion or knife system. In one embodiment, the motor driver 20229 may be the A3941 available from Allegro Microsystems, Inc. Other motor drivers can be readily substituted for use in the tracking system 20228 with an absolute positioning system. A detailed description of the absolute positioning system is provided in U.S. Patent Application Publication 2017 / 0296213, published October 19, 2017, entitled "SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety.

[0097] The microcontroller 20221 may be programmed to provide precise control over the velocity and position of the displacement member and joint motion system. The microcontroller 20221 may be configured to calculate the response within its software. The calculated response can be compared with the measured response of the actual system to obtain an "observed" response, which is used to determine the actual feedback. The observed response may be a well-adjusted value that balances the smooth and continuous nature of the simulated response with the measured response, and this can detect external influences on the system.

[0098] In some examples, the motor 20230 may be controlled by a motor driver 20229 and may be used by a surgical instrument or tool launching system. In various forms, the motor 20230 may be, for example, a brushed DC-driven motor having a maximum rotational speed of about 25,000 RPM. In some examples, the motor 20230 may be a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 20229 may comprise, for example, an H-bridge driver including a field-effect transistor (FET). The motor 20230 may be powered by a power supply assembly removably mounted in a handle assembly or tool housing to supply control power to a surgical instrument or tool. The power supply assembly may comprise a battery that may include a number of battery cells connected in series, which can be used as a power source for powering a surgical instrument or tool. Under certain circumstances, the battery cells of the power supply assembly may be replaceable and / or rechargeable. In at least one example, the battery cell may be a lithium-ion battery that can be connected to and detached from a power supply assembly.

[0099] The motor driver 20229 may be the A3941, available from Allegro Microsystems, Inc. The A3941 may be a full-bridge controller for use with an external N-channel power metal-oxide-semiconductor field-effect transistor (MOSFET), particularly designed for inductive loads such as brushed DC motors. The drive unit 20229 may include a proprietary charge pump regulator, which can provide full (>10V) gate drive to battery voltages down to 7V, and can enable the A3941 to operate with reduced gate drive down to 5.5V. Bootstrap capacitors may be used to provide the above battery supply voltage required for the N-channel MOSFET. An internal charge pump for high-side drive enables DC (100% duty cycle) operation. The full bridge can be driven in fast or slow decay mode using diodes or synchronous rectification. In slow decay mode, current recirculation is possible by either the high-side FET or the low-side FET. The power FET may be protected from shoot-through by a dead time adjustable with resistors. The integrated diagnostics indicate undervoltage, overtemperature, and power bridge anomalies and can be configured to protect power MOSFETs under most short-circuit conditions. Other motor drivers can be easily substituted for use with the 20228 tracking system with an absolute positioning system.

[0100] The tracking system 20228 may comprise a controlled motor drive circuit arrangement comprising a position sensor 20225 according to one aspect of the present disclosure. The position sensor 20225 for the absolute positioning system may provide a unique position signal corresponding to the position of a displacement member. In some examples, the displacement member may represent a longitudinally movable drive member comprising a rack of drive teeth for meshing and engaging with a corresponding drive gear of a gear reducer assembly. In some examples, the displacement member may represent a launch member which may be adapted and configured to include a rack of drive teeth. In some examples, the displacement member may represent a launch bar or an I-beam, each of which may be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displacement member may generally be used to refer to any movable member of a surgical instrument or tool, such as a drive member, launch member, launch bar, I-beam, or any element which may be displaced. In one aspect, a longitudinally movable drive member may be coupled to a launch member, launch bar, and I-beam. Thus, the absolute positioning system can, in practice, track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various embodiments, the displacement member may be connected to any position sensor 20225 suitable for measuring linear displacement. Thus, a longitudinally movable drive member, launch member, launch bar, or I-beam, or a combination thereof, may be connected to any suitable linear displacement sensor. The linear displacement sensor may include contact-type or non-contact-type displacement sensors. The linear displacement sensor may include a magnetic sensing system comprising a linear variable differential transformer (LVDT), a differential variable magnetoresistive transducer (DVRT), a slide potentiometer, a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system comprising a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical detection system comprising a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical detection system comprising a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0101] The electric motor 20230 may include a rotary shaft that operably interfaces with a gear assembly mounted on a displacement member by meshing engagement with a set of drive teeth or a rack. The sensor element may be operably connected to the gear assembly such that one rotation of the position sensor 20225 element corresponds to several linear longitudinal translations of the displacement member. The gearing and sensor configuration can be connected to a linear actuator by a rack and pinion configuration, or to a rotary actuator by a spur gear or other connection. A power supply provides power to the absolute positioning system, and an output indicator may display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member having a rack of drive teeth formed thereon for meshing engagement with the corresponding drive gear of the gear reducer assembly. The displacement member may represent a longitudinally movable launch member, launch bar, I-beam, or a combination thereof.

[0102] One rotation of the sensor element associated with the position sensor 20225 may correspond to a longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance the displacement member moves from point "a" to point "b" after one rotation of the sensor element connected to the displacement member. The sensor mechanism may be connected via a gear reduction that results in the position sensor 20225 completing one or more rotations relative to the full stroke of the displacement member. The position sensor 20225 can complete multiple rotations relative to the full stroke of the displacement member.

[0103] To provide a unique position signal for two or more rotations of the position sensor 20225, a series of switches (where n is an integer greater than 1) may be used alone or in combination with gear reduction. The state of the switches can be fed back to the microcontroller 20221, which applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 20225 is provided to the microcontroller 20221. The position sensor 20225 of the sensor mechanism may include an array of analog rotation sensors such as a magnetic sensor or potentiometer, or an array of analog Hall effect elements, which output a unique combination of position signals or values.

[0104] The position sensor 20225 may comprise any number of magnetic sensing elements, such as magnetic sensors, which are classified according to whether they measure the total magnetic field or the vector component of the magnetic field. The techniques used to produce both types of magnetic sensors may involve numerous aspects of physics and electronics. Techniques used to sense magnetic fields include, among others, probe coils, flux gates, optical pumping, nuclear precession, SQUIDs, Hall effect, anisotropic magnetoresistance, colossal magnetoresistance, magnetic tunnel junctions, colossal magnetoimpedance, magnetostrictive / piezoelectric composites, magnetic diodes, magnetic transistors, optical fibers, magneto-optics, and micro-electromechanical system-based magnetic sensors.

[0105] In one embodiment, the position sensor 20225 of a tracking system 20228, which includes an absolute positioning system, may include a magnetic rotation absolute positioning system. The position sensor 20225 may be implemented as an AS5055EQFT single-chip magnetic rotation position sensor, available from Austria Microsystems, AG. The position sensor 20225 interfaces with a microcontroller 20221 to provide an absolute positioning system. The position sensor 20225 may be a low-voltage, low-power component and may include four Hall effect elements in the area of ​​the position sensor 20225, which may be located above the magnet. A high-resolution ADC and a smart power management controller may also be provided on the chip. A coordinate rotation digital computer (CORDIC) processor, also known as the digit-by-digit method and the Boulder algorithm, may be provided to implement a simple and efficient algorithm for computing hyperbolic and trigonometric functions that requires only addition, subtraction, bit shifting, and table lookup operations. Angular position, alarm bits, and magnetic field information can be transmitted to the microcontroller 20221 via a standard serial communication interface such as the Serial Peripheral Interface (SPI) interface. The position sensor 20225 can provide 12-bit or 14-bit resolution. The position sensor 20225 may also be an AS5055 chip, available in a small QFN 16-pin 4x4x0.85mm package.

[0106] The tracking system 20228, which includes an absolute positioning system, may also include and / or be programmed to implement feedback controllers such as PID, state feedback, and adaptive controllers. The power supply converts signals from the feedback controllers into physical inputs to the system, in this case voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by the position sensor 20225, other sensors may be provided to measure physical parameters of the physical system. In some embodiments, other sensors include those described in U.S. Patent No. 9,345,481, issued May 24, 2016, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, published September 18, 2014, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; and U.S. Patent Application No. 15 / 628,175, filed June 20, 2017, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is connected to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include comparison and combinational circuits to combine the calculated response with the measured response, using algorithms such as weighted averaging and theoretical control loops that drive the calculated response toward the measured response. To predict what the state and output of the physical system will be by knowing the input, the calculated response of the physical system may take into account characteristics such as mass, inertia, viscous friction, and inductive resistance.

[0107] The absolute positioning system can provide the absolute position of the displacement member when the device is powered on, without requiring the displacement member to be moved back or forward to a reset (zero or home) position, as may be necessary with conventional rotary encoders that simply count the number of forward or backward strokes taken by the motor 20230 to estimate the position of the device actuator, drive bar, knife, etc.

[0108] For example, sensor 20226, such as a strain gauge or micro-strain gauge, may be configured to measure one or more parameters of an end effector, such as the amplitude of strain applied to the anvil during clamping, which can indicate the closing force applied to the anvil. The measured strain may be converted into a digital signal and provided to processor 20222. Instead of, or in addition to, sensor 20226, sensor 20227, such as a load sensor, may measure the closing force applied to the anvil by the closing drive system. For example, sensor 20227, such as a load sensor, may measure the firing force applied to the I-beam during the firing stroke of a surgical instrument or tool. The I-beam is configured to engage with a wedge-shaped thread, which is configured to cam upward a staple driver to push the staple out and deform into contact with the anvil. The I-beam may also include a sharp cutting edge that can be used to cut tissue as the I-beam is advanced distally by the firing bar. Alternatively, a current sensor 20231 can be used to measure the current drawn in by the motor 20230. The force required to propel the launching member forward may correspond, for example, to the current drawn in by the motor 20230. The measured force can be converted into a digital signal and provided to the processor 20222.

[0109] In one embodiment, a strain gauge sensor 20226 can be used to measure the force applied to tissue by the end effector. A strain gauge can be connected to the end effector to measure the force applied by the end effector to the tissue being treated. A system for measuring the force applied to tissue gripped by the end effector may include a strain gauge sensor 20226, such as a micro-strain gauge, which can be configured to measure one or more parameters of the end effector. In one embodiment, the strain gauge sensor 20226 can measure the amplitude or magnitude of strain applied to the jaw members of the end effector during a clamping operation, which may indicate tissue compression. The measured strain can be converted into a digital signal and provided to the processor 20222 of the microcontroller 20221. A load sensor 20227 can measure the force used to operate a knife element to cut tissue trapped between an anvil and a staple cartridge, for example. A magnetic field sensor can be used to measure the thickness of the trapped tissue. The measurement from the magnetic field sensor can also be converted into a digital signal and provided to the processor 20222.

[0110] Measurements of tissue compression, tissue thickness, and / or the force required to close the end effector on the tissue, measured by sensors 20226 and 20227 respectively, can be used by microcontroller 20221 to characterize the selected position of the launcher and / or the corresponding values ​​of the launcher's velocity. For example, memory 20223 can store techniques, equations, and / or lookup tables that can be used by microcontroller 20221 during evaluation.

[0111] The surgical instrument or tool control system 20220 may also include a wired or wireless communication circuit for communicating with a modular communication hub 20065, as shown in Figures 5 and 6A.

[0112] Figure 7B shows an exemplary sensing system 20069. The sensing system may be a surgical sensing system or a patient sensing system. Sensing system 20069 may include a sensor unit 20235 and a human interface system 20242 that communicate with a data processing and communication unit 20236. The data processing and communication unit 20236 may include an analog-to-digital converter 20237, a data processing unit 20238, a storage unit 20239, an input / output interface 20241, and a transceiver 20240. Sensing system 20069 may also communicate with a surgical hub or computing device 20243, which then communicates with a cloud computing system 20244. The cloud computing system 20244 may include a cloud storage system 20078 and one or more cloud servers 20077.

[0113] The sensor unit 20235 may include one or more ex vivo or in vivo sensors for measuring one or more biomarkers. Biomarkers may be measured using one or more sensors such as blood pH, hydration status, oxygen saturation, core body temperature, heart rate, heart rate variability, sweat rate, skin conductance, blood pressure, light exposure, ambient temperature, respiratory rate, cough and sneeze, gastrointestinal motility, gastrointestinal imaging, tissue perfusion pressure, airway bacteria, alcohol consumption, lactic acid (sweat), peripheral temperature, positive and optimistic attitude, adrenaline (sweat), cortisol (sweat), edema, mycotoxins, VO2 max, preoperative pain, airborne chemicals, circulating tumor cells, stress and anxiety, confusion and delirium, physical activity, autonomic tension, circadian rhythm, menstrual cycle, and sleep. The sensor may measure the biomarkers described herein using one or more of the following sensing techniques: photoplethysmography, electrocardiography, electroencephalography, colorimetric analysis, impedimentary measurement, potentiometric measurement, current measurement, etc.

[0114] As shown in Figure 7B, the sensor in sensor unit 20235 can measure physiological signals (e.g., voltage, current, PPG signal, etc.) associated with the biomarker to be measured. The physiological signals to be measured may depend on the sensing technology used, as described herein. Sensor unit 20235 of sensing system 20069 can communicate with data processing and communication unit 20236. In one example, sensor unit 20235 can communicate with data processing and communication unit 20236 using a wireless interface. Data processing and communication unit 20236 may include an analog-to-digital converter (ADC) 20237, a data processing unit 20238, storage 20239, an I / O interface 20241, and an RF transceiver 20240. Data processing unit 20238 may include a processor and a memory unit.

[0115] The sensor unit 20235 may transmit the measured physiological signal to the ADC 20237 of the data processing and communication unit 20236. In one example, the measured physiological signal may be passed through one or more filters (e.g., an RC low-pass filter) before being sent to the ADC. The ADC can convert the measured physiological signal into measurement data associated with the biomarker. The ADC can pass the measurement data to the data processing unit 20238 for processing. In one example, the data processing unit 20238 may transmit the measurement data associated with the biomarker to a surgical hub or computing device 20243, and then transmit the measurement data to a cloud computing system 20244 for further processing. The data processing unit may transmit the measurement data to the surgical hub or computing device 20243 using one of the radio protocols as described herein. In one example, the data processing unit 20238 may first process the raw measurement data received from the sensor unit and then transmit the processed measurement data to the surgical hub or computing device 20243.

[0116] For example, the data processing and communication unit 20236 of the sensing system 20069 may receive thresholds associated with biomarkers for monitoring from the surgical hub, computing device 20243, or directly from the cloud server 20077 of the cloud computing system 20244. The data processing unit 20236 may compare the measurement data associated with the biomarker to be monitored with the corresponding threshold received from the surgical hub, computing device 20243, or cloud server 20077. The data processing and communication unit 20236 may send a notification message to HID20242 indicating that the measurement data value has exceeded the threshold. The notification message may include the measurement data associated with the monitored biomarker. The data processing and computing unit 20236 may send the notification via transmission to the surgical hub or computing device 20243 using one of the following RF protocols: Bluetooth®, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low Power Wireless Personal Area Network (6LoWPAN), or Wi-Fi. The data processing unit 20238 may send notifications (e.g., notifications about HCPs) directly to a cloud server via transmission to a cellular transmit / receive point (TRP) or base station, using one or more of the following cellular protocols: GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long-Term Evolution (LTE) or 4G, LTE Advanced (LTE-A), New Radio (NR), or 5G. In one example, the sensing unit may communicate with a hub / computing device via a router, as shown in Figures 6A-6C.

[0117] Figure 7C shows an exemplary sensing system 20069 (e.g., a surgical sensing system or a patient sensing system). The sensing system 20069 may include a sensor unit 20245, a data processing and communication unit 20246, and a human interface device 20242. Sensor unit 20245 may include a sensor 20247 and an analog-to-digital converter (ADC) 20248. The ADC 20248 in sensor unit 20245 can convert physiological signals measured by sensor 20247 into measurement data associated with a biomarker. Sensor unit 20245 can transmit the measurement data to data processing and communication unit 20246 for further processing. In one example, sensor unit 20245 can send the measurement data to data processing and communication unit 20246 using an inter-integrated circuit (I2C) interface.

[0118] The data processing and communication unit 20246 includes a data processing unit 20249, a storage unit 20250, and an RF transceiver 20251. The sensing system may communicate with a surgical hub or computing device 20243, which may then communicate with a cloud computing system 20244. The cloud computing system 20244 may include a remote server 20077 and associated remote storage 20078. The sensor unit 20245 may include one or more in vitro or in vivo sensors for measuring one or more biomarkers, as described herein.

[0119] After processing the measurement data received from the sensor unit 20245, the data processing and communication unit 20246 can further process the measurement data and / or transmit it to the smart hub or computing device 20243, as described in Figure 7B. For example, the data processing and communication unit 20246 can transmit the measurement data received from the sensor unit 20245 to a remote server 20077 of the cloud computing system 20244 for further processing and / or monitoring.

[0120] Figure 7D shows an exemplary sensing system 20069 (e.g., a surgeon sensing system or a patient sensing system). The sensing system 20069 may include a sensor unit 20252, a data processing and communication unit 20253, and a human interface system 20261. Sensor unit 20252 may include a plurality of sensors 20254, 20255-20256 to measure one or more physiological signals associated with a patient or surgeon biomarker, and / or one or more physical state signals associated with the physical state of the patient or surgeon. Sensor unit 20252 may also include one or more analog-to-digital converters (ADCs) 20257. The list of biomarkers may include biomarkers such as those disclosed herein. The ADC 20257 in sensor unit 20252 can convert each of the physiological signals and / or physical state signals measured by sensors 20254-20256 into their respective measurement data. Sensor unit 20252 can transmit measurement data associated with one or more biomarkers and the physical condition of the patient or surgeon to data processing and communication unit 20253 for further processing. Sensor unit 20252 can transmit measurement data to data processing and communication unit 20253 individually for each of sensors 1 20254 to N 20256, or in combination for all sensors. For example, sensor unit 20252 can transmit measurement data to data processing and communication unit 20253 via an I2C interface.

[0121] The data processing and communication unit 20253 may include a data processing unit 20258, a storage unit 20259, and an RF transceiver 20260. The sensing system 20069 may communicate with a surgical hub or computing device 20243, which then communicates with a cloud computing system 20244 comprising at least one remote server 20077 and at least one storage unit 20078. The sensor unit 20252 may include one or more in vitro or in vivo sensors for measuring one or more biomarkers, as described herein.

[0122] Figure 8 shows an example of using surgical task situation recognition and measurement data from one or more surgeon sensing systems to coordinate surgical instrument control. Figure 8 shows an exemplary surgical procedure timeline 20265 and contextual information that the surgical hub can derive from data received from one or more surgical devices, one or more surgeon sensing systems, and / or one or more environmental sensing systems at each step of the surgical procedure. Devices that can be controlled by the surgical hub may include advanced energy devices, endocutter clamps, etc. The surgeon sensing system may include sensing systems for measuring one or more biomarkers related to the surgeon, such as heart rate, sweat composition, respiratory rate, etc. The environmental sensing system may include systems for measuring one or more environmental attributes, such as cameras for detecting the surgeon's position / movement / breathing patterns, spatial microphones for measuring ambient noise and / or the tone of the healthcare provider's voice in the operating room, ambient temperature / humidity, etc.

[0123] In the following description of the timeline 20265 shown in Figure 8, please also refer to Figure 5. Figure 5 provides various components used in a surgical procedure. Timeline 20265 shows steps that may be performed individually and / or collectively by nurses, surgeons, and other healthcare professionals during an exemplary colorectal surgery. In a colorectal surgery, the situation-aware surgical hub 20076 can receive data from various data sources throughout the surgical procedure, including data generated each time a healthcare provider (HCP) uses the modular device / instrument 20095 paired with the surgical hub 20076. The surgical hub 20076 may receive this data from the paired modular device 20095. The surgical hub can receive measurement data from the sensing system 20069. The surgical hub may continuously derive the surgeon's stress level and inferences (i.e., contextual information) regarding the procedure in progress as new data is received, using data from modular device / instrument 20095 and / or measurement data from sensing system 20069, so that the surgeon's stress level for the procedure being performed is obtained. The contextual awareness system of the surgical hub 20076 may perform the following: record data regarding the procedure to generate a report; verify the steps being taken by the healthcare professional; provide data or prompts that may be relevant to a particular procedure step (e.g., via a display screen); adjust modular devices based on context (e.g., activate a monitor, adjust the FOV of a medical imaging device, change the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument); or one or more of any other such actions described herein. In one example, these actions may be performed by a remote server 20077 of cloud system 20064, communicating with the surgical hub 20076.

[0124] In the first step (not shown in Figure 8 for brevity), hospital staff can search the patient's EMR from the hospital's EMR database. Based on the patient data selected in the EMR, the surgical hub 20076 may determine that the procedure to be performed is a colorectal surgery. Staff can scan medical supplies coming in for the procedure. The surgical hub 20076 cross-references the scanned supplies with a list of supplies that may be used in various types of procedures to confirm that the combination of supplies matches a colorectal procedure. The surgical hub 20076 may pair each of the sensing systems 20069 fitted by different HCPs.

[0125] Once each device is prepared and pre-operative preparations are complete, the surgical team can begin by making an incision and positioning the trocar. The surgical team may access and prepare by dissecting any adhesions and identifying the inferior mesenteric artery (IMA) branch. The surgical hub 20076 can infer that the surgeon is in the process of dissecting adhesions based on data that can be received from the RF or ultrasound generator indicating that at least an energy instrument is being emitted. The surgical hub 20076 can cross-reference the received data with the surgical procedure lookup process to determine that the energy instrument being emitted at this point in the process (e.g., after the completion of the procedure steps described above) corresponds to the dissection step.

[0126] After incision, the HCP may proceed to the ligation step of the procedure (e.g., indicated by A1). As shown in Figure 8, the HCP may begin by ligating the IMA. The surgical hub 20076 can receive data from the advanced energy jaw device and / or endocutter indicating that the instrument is being fired, so it can infer that the surgeon is ligating the arteries and veins. The surgical hub may also receive measurement data from one of the HCP's sensing systems indicating a higher stress level of the HCP (e.g., indicated by the B1 mark on the time axis). For example, a higher stress level may be indicated by a change in the HCP's heart rate from a baseline. The surgical hub 20076 may derive this inference by cross-referencing the data received from the surgical stapling and cutting instruments with the steps retrieved in the process (e.g., indicated by A2 and A3), as with the previous step. The surgical hub 20076 may monitor the advanced energy jaw trigger ratio and / or endocutter clamp and firing rate during the high-stress period. In one example, the surgical hub 20076 can control an operating device by transmitting auxiliary control signals to an advanced energy jaw device and / or an end cutter device. The surgical hub can transmit support signals based on the stress level of the HCP operating the surgical device and / or the situational awareness known to the surgical hub. For example, the surgical hub 20076 may transmit control support signals to an advanced energy device or end cutter clamp, as shown by A2 and A3 in Figure 8.

[0127] The HCP can proceed to the next step of releasing the upper sigmoid colon, followed by the descending colon, rectum, and sigmoid colon. The surgical hub 20076 can continue to monitor the high-stress markers of the HCP (e.g., as indicated by D1, E1a, E1b, F1). During the high-stress period, the surgical hub 20076 can transmit support signals to the advanced energy jaw device and / or end-cutting device, as shown in Figure 8.

[0128] After mobilizing the colon, the HCP may advance the segmental resection portion of the procedure. For example, the surgical hub 20076 can estimate that the HCP is transversely incising parenchymal tissue based on data from surgical stapling and cutting instruments (including data from their cartridges). The cartridge data may correspond, for example, to the size or type of staples fired by the instrument. Since different types of staples are used for different types of tissue, the cartridge data can indicate the type of tissue being stapled and / or transversely incised. It should be noted that, as different instruments are better suited to specific tasks, surgeons should periodically switch between surgical stapling / cutting instruments and surgical energy (e.g., RF or ultrasound) instruments depending on the stage of the procedure. Thus, the sequence in which stapling / cutting instruments and surgical energy instruments are used can indicate which stage of the procedure the surgeon is performing.

[0129] The surgical hub can determine and transmit control signals to surgical devices based on the stress level of the HCP. For example, during period G1b, control signal G2b may be transmitted to the endocutter clamp. Once the sigmoid is removed, the incision can be closed and the postoperative portion of the procedure can begin. The patient's anesthesia can be reversed. The surgical hub 20076 may infer that the patient is waking up from anesthesia based on one or more sensing systems attached to the patient.

[0130] Figure 9 is a block diagram of a computer-implemented interactive surgical system with surgeon / patient monitoring according to at least one aspect of the present disclosure. In one aspect, the computer-implemented interactive surgical system may be configured to monitor surgeon biomarkers and / or patient biomarkers using one or more sensing systems 20069. Surgeon biomarkers and / or patient biomarkers may be measured before, after, and / or during surgical procedures. In one aspect, the computer-implemented interactive surgical system may be configured to monitor and analyze data regarding the operation of various surgical systems 20069, including surgical hubs, surgical instruments, robotic devices, and operating rooms or medical facilities. The computer-implemented interactive surgical system may include a cloud-based analytics system. The cloud-based analytics system may include one or more analytics servers.

[0131] As shown in Figure 9, the cloud-based monitoring and analysis system may comprise a plurality of sensing systems 20268 (which may be the same as or similar to sensing system 20069), surgical instruments 20266 (which may be the same as or similar to instruments 20031), a plurality of surgical hubs 20270 (which may be the same as or similar to hub 20006), and a surgical data network 20269 (which may be the same as or similar to the surgical data network described in Figure 4) for connecting the surgical hubs 20270 to the cloud 20271 (which may be the same as or similar to cloud computing system 20064). Each of the plurality of surgical hubs 20270 may be communicatively connected to one or more surgical instruments 20266. Each of the plurality of surgical hubs 20270 may also be communicatively connected via the network 20269 to one or more sensing systems 20268 and the cloud 20271 of the computer-implemented interactive surgical system. The surgical hubs 20270 and sensing systems 20268 may be communicatively connected using wireless protocols as described herein. Cloud system 20271 can serve as a remote centralized source of hardware and software for storing, processing, manipulating, and communicating measurement data from sensing system 20268 and data generated based on the operation of various surgical systems 20268.

[0132] As shown in Figure 9, access to the cloud system 20271 may be achieved via network 20269, which may be the Internet or another suitable computer network. A surgical hub 20270, which may be connected to the cloud system 20271, can be considered the client side of the cloud computing system (e.g., a cloud-based analytics system). A surgical instrument 20266 may be paired with the surgical hub 20270 for the control and execution of various surgical procedures or actions described herein. A sensing system 20268 may be paired with the surgical hub 20270 for surgeon monitoring during surgery, pre-operative patient monitoring, intra-operative patient monitoring, or post-operative patient monitoring of surgeon-related biomarkers, in order to track and / or measure various milestones and / or detect various complications. An environmental sensing system 20267 may be paired with the surgical hub 20270 to measure environmental attributes associated with the surgeon or patient for surgeon monitoring, pre-operative patient monitoring, intra-operative patient monitoring, or post-operative patient monitoring.

[0133] The surgical instruments 20266, the environmental sensing system 20267, and the sensing system 20268 may be equipped with wired or wireless transceivers for data transmission to and from their corresponding surgical hub 20270 (which may also be equipped with transceivers). One or more combinations of the surgical instruments 20266, the sensing system 20268, or the surgical hub 20270 may indicate a specific location, such as an operating room, an intensive care unit (ICU) room, or a recovery room within a medical facility (e.g., a hospital), to provide medical surgery, pre-operative preparation, and / or post-operative recovery. For example, the memory of the surgical hub 20270 may store location data.

[0134] As shown in Figure 9, the cloud system 20271 includes one or more central servers 20272 (which may be the same as or similar to the remote servers 20067), a surgical hub application server 20276, a data analysis module 20277, and an input / output ("I / O") interface 20278. The central servers 20272 of the cloud system 20271 collectively manage the cloud computing system, which may include monitoring requests from client surgical hubs 20270 and managing the processing power of the cloud system 20271 to perform those requests. Each of the central servers 20272 may comprise one or more processors 20273 coupled to a preferred memory device 20274, which may include volatile memory such as random access memory (RAM) and non-volatile memory such as magnetic storage devices. When executed, memory device 20274 may contain machine-executable instructions that cause processor 20273 to execute data analysis module 20277 for cloud-based data analysis, real-time monitoring of measurement data received from sensing system 20268, operation, recommendations, and other operations described herein. Processor 20273 can execute data analysis module 20277 independently or in conjunction with hub applications that run independently by hub 20270. Central server 20272 may also include a database 20275 of aggregated medical data that may reside in memory 20274.

[0135] Based on connections to various surgical hubs 20270 via network 20269, cloud 20271 can aggregate data from specific data generated by various surgical instruments 20266 and / or monitor real-time data from sensing systems 20268 and surgical hubs 20270 associated with surgical instruments 20266 and / or sensing systems 20268. Such aggregated data from surgical instruments 20266 and / or measurement data from sensing systems 20268 may be stored in the aggregated medical database 20275 of cloud 20271. In particular, cloud 20271 can advantageously track real-time measurement data from sensing systems 20268 and / or perform data analysis and actions on the measurement data and / or aggregated data to gain insights and / or perform functions that individual hubs 20270 cannot achieve on their own. For this purpose, as shown in Figure 9, cloud 20271 and surgical hubs 20270 are communicably linked to send and receive information. The I / O interface 20278 connects to multiple surgical hubs 20270 via network 20269. In this way, the I / O interface 20278 can be configured to transfer information between the surgical hubs 20270 and the aggregated medical data database 20275. Thus, the I / O interface 20278 can facilitate read / write operations of a cloud-based analysis system. Such read / write operations may be performed in response to requests from the hubs 20270. These requests may be sent to the surgical hubs 20270 via a hub application. The I / O interface 20278 may include one or more high-speed data ports, including a Universal Serial Bus (USB) port, an IEEE 1394 port, and Wi-Fi and Bluetooth I / O interfaces for connecting the cloud 20271 to the surgical hubs 20270. The hub application server 20276 of the cloud 20271 may be configured to host and supply shared functions to a software application (e.g., a hub application) run by the surgical hubs 20270.For example, the hub application server 20276 may manage requests from hub applications through hub 20270, control access to the aggregated medical data database 20275, and perform load balancing.

[0136] The cloud computing system configurations described in this disclosure may be designed to address various issues arising in the context of medical surgeries (e.g., preoperative, intraoperative, and postoperative monitoring) and procedures performed using medical devices such as surgical instruments 20266, 20031, etc. In particular, surgical instrument 20266 may be a digital surgical device configured to interact with cloud 20271 in order to implement techniques for improving surgical outcomes. Sensing system 20268 may be a system having one or more sensors configured to measure one or more biomarkers related to a surgeon performing a medical surgery and / or a patient in whom a medical surgery is scheduled, is being performed, or has been performed. Various surgical instruments 20266, sensing system 20268, and / or surgical hub 20270 may include a human interface system (e.g., having a touch-controlled user interface) so that a clinician and / or patient can control the manner of interaction between surgical instrument 20266 or sensing system 20268 and cloud 20271. Other suitable user interfaces for control, such as auditory-controlled user interfaces, can also be used.

[0137] The cloud computing system configurations described herein may be designed to address various issues arising in situations where sensing system 20268 is used to monitor one or more biomarkers associated with a healthcare professional (HCP) or patient during pre-operative, intra-operative, and post-operative procedures. Sensing system 20268 may be a surgeon sensing system or a patient sensing system configured to interact with a surgical hub 20270 and / or cloud system 20271 to implement techniques for monitoring surgeon biomarkers and / or patient biomarkers. Various sensing systems 20268 and / or surgical hub 20270 may include a touch-controlled human interface system so that the HCP or patient can control the manner of interaction between sensing system 20268 and surgical hub 20270 and / or cloud system 20271. Other suitable user interfaces for control, such as an auditory-controlled user interface, may also be used.

[0138] Figure 10 shows an exemplary surgical system 20280 according to the present disclosure, which may include a surgical instrument 20282 that can communicate with a console 20294 or a portable device 20296 via a wired or wireless connection through a local area network 20292 or a cloud network 20293. In various embodiments, the console 20294 and the portable device 20296 may be any suitable computing device. The surgical instrument 20282 may include a handle 20297, an adapter 20285, and a loading unit 20287. The adapter 20285 is releasably coupled to the handle 20297, and the loading unit 20287 is releasably coupled to the adapter 20285 so that the adapter 20285 transmits force from the drive shaft to the loading unit 20287. The adapter 20285 or the loading unit 20287 may include a force gauge (not explicitly shown) located inside for measuring the force acting on the loading unit 20287. The loading unit 20287 may comprise an end effector 20289 having a first jaw 20291 and a second jaw 20290. The loading unit 20287 may also be an in-situ loading, i.e., a multi-fire loading unit (MFLU), which allows a clinician to fire multiple fasteners multiple times without having to remove the loading unit 20287 from the surgical site and reload it.

[0139] The first and second jaws 20291, 20290 may be configured to clamp tissue between them, fire fasteners through the clamped tissue, and cut the clamped tissue. The first jaw 20291 may be configured to fire at least one fastener multiple times, or may be configured to include a replaceable multi-fire fastener cartridge containing multiple fasteners (e.g., staples, clips, etc.) that can be fired two or more times before being replaced. The second jaw 20290 may include an anvil that deforms or otherwise secures fasteners as they are ejected from the multi-fire fastener cartridge.

[0140] The handle 20297 may include a motor connected to the drive shaft so as to affect the rotation of the drive shaft. The handle 20297 may include a control interface for selectively operating the motor. The control interface may include buttons, switches, levers, sliders, touchscreens, and any other suitable input mechanisms or user interfaces, which can be engaged by a clinician to start the motor.

[0141] The control interface of the handle 20297 communicates with the controller 20298 of the handle 20297 to selectively activate the motor and act on the rotation of the drive shaft. The controller 20298 may be located within the handle 20297 and may be configured to receive input from the control interface and adapter data from adapter 20285 or loading unit data from loading unit 20287. The controller 20298 can analyze the input from the control interface and the data received from adapter 20285 and / or loading unit 20287 in order to selectively activate the motor. The handle 20297 may also include a display that can be viewed by a clinician while the handle 20297 is in use. The display may be configured to show portions of the adapter or loading unit data before, during, or after firing the instrument 20282.

[0142] Adapter 20285 may include an internally located adapter identification device 20284, and loading unit 20287 may include an internally located loading unit identification device 20288. Adapter identification device 20284 may communicate with controller 20298, and loading unit identification device 20288 may communicate with controller 20298. It will be understood that loading unit identification device 20288 may communicate with adapter identification device 20284, which relays or passes through communication from loading unit identification device 20288 to controller 20298.

[0143] Adapter 20285 may also include a plurality of sensors 20286 (one shown) positioned around it to detect various states of Adapter 20285 or the environment (e.g., whether Adapter 20285 is connected to a loading unit, whether Adapter 20285 is connected to a handle, whether the drive shaft is rotating, the torque of the drive shaft, the strain of the drive shaft, the temperature inside Adapter 20285, the number of times Adapter 20285 has fired, the peak force of Adapter 20285 during firing, the total amount of force applied to Adapter 20285, the peak recoil force of Adapter 20285, the number of pauses of Adapter 20285 during firing, etc.). The plurality of sensors 20286 can provide input to Adapter Identification Device 20284 in the form of data signals. The data signals from the plurality of sensors 20286 may be stored in Adapter Identification Device 20284 or used to update Adapter data stored in Adapter Identification Device 20284. The data signals from the plurality of sensors 20286 may be analog or digital. Multiple sensors 20286 may include force gauges for measuring the force exerted on the loading unit 20287 during firing.

[0144] The handle 20297 and adapter 20285 may be configured to interconnect the adapter identification device 20284 and the loading unit identification device 20288 with the controller 20298 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals between them). In addition, or alternatively, the electrical interface may be a non-contact electrical interface for wirelessly transmitting energy and signals between them (e.g., inductively). The adapter identification device 20284 and the controller 20298 are also intended to be able to communicate wirelessly with each other via a wireless connection separate from the electrical interface.

[0145] Handle 20297 may include transceiver 20283 configured to transmit instrument data from controller 20298 to other components of system 20280 (e.g., LAN 20292, cloud 20293, console 20294, or portable device 20296). Controller 20298 may also transmit instrument data and / or measurement data associated with one or more sensors 20286 to surgical hub 20270, as illustrated in Figure 9. Transceiver 20283 may receive data from surgical hub 20270 (e.g., cartridge data, loading unit data, adapter data, or other notifications). Transceiver 20283 may also receive data from other components of system 20280 (e.g., cartridge data, loading unit data, or adapter data). For example, controller 20298 may transmit instrument data to console 20294, including the serial number of an attached adapter (e.g., adapter 20285) attached to handle 20297, the serial number of a loading unit (e.g., loading unit 20287) attached to adapter 20285, and the serial number of a multi-shot fastener cartridge loaded into the loading unit. Console 20294 may then reply to controller 20298 with data associated with the attached cartridge, loading unit, and adapter, respectively (e.g., cartridge data, loading unit data, or adapter data). Controller 20298 may display the message on the local instrument display, or transmit the message via transceiver 20283 to console 20294 or portable device 20296, which will then display the message on display 20295 or the portable device screen, respectively.

[0146] Figures 11A to 11D illustrate embodiments of wearable sensing systems, such as surgical sensing systems or patient sensing systems. Figure 11A is an example of a glasses-based sensing system 20300, which may be based on an electrochemical sensing platform. The sensing system 20300 may be capable of monitoring (e.g., real-time monitoring) sweat electrolytes and / or metabolites using a plurality of sensors 20304 and 20305 that are in contact with the skin of the surgeon or patient. For example, the sensing system 20300 may use an ampereometry-based biosensor 20304 and / or a potentiometry-based biosensor 20305 integrated with the nose bridge pad of the glasses 20302 to measure current and / or voltage.

[0147] The current-measuring biosensor 20304 can be used to measure sweat lactate levels (e.g., in mmol / L units). Lactate is a product of lactic acidosis, which can result from decreased tissue oxygenation that can be caused by sepsis or hemorrhage. A patient's lactate level (e.g., >2 mmol / L) can be used to monitor the development of sepsis, for example, during postoperative monitoring. The potentiometric biosensor 20305 can be used to measure potassium levels in a patient's sweat. A voltage follower circuit with an operational amplifier can be used to measure the potential signal between a reference electrode and a working electrode. The output of the voltage follower circuit can be filtered and converted to a digital value using an ADC.

[0148] Current measuring sensor 20304 and potentiometer measuring sensor 20305 may be connected to circuit 20303 located on each arm of the eyeglasses. Electrochemical sensors may be used for simultaneous real-time monitoring of sweat lactic acid and potassium levels. Electrochemical sensors may be screen-printed on stickers and placed on each side of the nose pads of the eyeglasses to monitor sweat metabolites and electrolytes. Electronic circuit 20303 located on the arms of the eyeglass frame may include a wireless data transceiver (e.g., a low-energy Bluetooth transceiver) which can be used to transmit lactic acid and / or potassium measurement data to a surgical hub or intermediate device, the intermediate device which may then transfer the measurement data to the surgical hub. The eyeglass-based sensing system 20300 may use, for example, a signal conditioning unit for filtering and amplifying the electrical signals generated from the electrochemical sensors 20305 or 20304, a microcontroller for digitizing the analog signals, and a wireless (e.g., low-energy Bluetooth®) module for transferring the data to a surgical hub or computing device, as described in Figures 7B-7D.

[0149] Figure 11B shows an example of a wristband-type sensing system 20310 comprising a sensor assembly 20312 (e.g., a photoplethysmography (PPG)-based sensor assembly or an electrocardiogram (ECG)-based sensor assembly). For example, in the sensing system 20310, the sensor assembly 20312 can collect and analyze arterial pulses in the wrist. The sensor assembly 20312 may be used to measure one or more biomarkers (e.g., heart rate, heart rate variability (HRV), etc.). In the sensing system with a PPG-based sensor assembly 20312, light (e.g., green light) may be allowed to pass through the skin. A certain percentage of the green light may be absorbed by blood vessels, and a portion of the green light may be reflected and detected by a photodetector. These differences or reflections are associated with variations in tissue blood perfusion, and these variations may be used to detect cardiovascular information (e.g., heart rate). For example, the amount absorbed may vary depending on the blood volume. The sensing system 20310 can determine the heart rate by measuring light reflectance as a function of time. The heart rate recurrence (HRV) may be determined as the period of variation (e.g., standard deviation) between the steepest signal gradients before the peak, known as the heart rate interval (IBI).

[0150] In the case of a sensing system having an ECG-based sensor assembly 20312, a set of electrodes may be placed in contact with the skin. The sensing system 20310 may measure the voltage across the set of electrodes placed on the skin to determine the heart rate. In this case, the HRV can be measured as the period variation (e.g., standard deviation) between R peaks in a group of QRS complexes, known as the RR interval.

[0151] The sensing system 20310 may, for example, use a signal conditioning unit for filtering and amplifying the analog PPG signal, a microcontroller for digitizing the analog PPG signal, and a wireless (e.g., Bluetooth®) module for transferring the data to a surgical hub or computing device, as described in Figures 7B to 7D.

[0152] Figure 11C shows an exemplary ring sensing system 20320. The ring sensing system 20320 may include a sensor assembly (e.g., a heart rate sensor assembly) 20322. The sensor assembly 20322 may include a light source (e.g., a red or green light-emitting diode (LED)) and a photodiode for detecting reflected and / or absorbed light. The LED in the sensor assembly 20322 may shine light through the finger, and the photodiode in the sensor assembly 20322 may measure heart rate and / or blood oxygen levels by detecting changes in blood volume. The ring sensing system 20320 may include other sensor assemblies for measuring other biomarkers, e.g., a thermistor or infrared thermometer for measuring body surface temperature. The ring sensing system 20320 may, for example, use a signal conditioning unit for filtering and amplifying the analog PPG signal, a microcontroller for digitizing the analog PPG signal, and a wireless (e.g., low energy Bluetooth®) module for transferring the data to a surgical hub or computing device, as described in Figures 7B to 7D.

[0153] Figure 11D shows an example of an electroencephalogram (EEG) sensing system 20315. As shown in Figure 11D, the sensing system 20315 may include one or more EEG sensor units 20317. The EEG sensor unit 20317 may include a plurality of conductive electrodes placed in contact with the scalp. The conductive electrodes may be used to measure small potentials that may be generated outside the head due to neuronal activity in the brain. The EEG sensing system 20315 can measure biomarkers, such as delirium, by identifying specific brain patterns, such as delays or absences in posterior dominant rhythms and loss of responsiveness to eye opening and closing. The ring sensing system 20315 may have, for example, a signal conditioning unit for filtering and amplifying potentials, a microcontroller for digitizing electrical signals, and a wireless (e.g., Low Energy Bluetooth®) module for transferring data to a smart device, as described in Figures 7B to 7D.

[0154] Figure 12 shows a block diagram of a computer-implemented patient / surgeon monitoring system 20325 for monitoring one or more patient or surgeon biomarkers before, during, and / or after a surgical procedure. As illustrated in Figure 12, one or more sensing systems 20336 may be used to measure and monitor patient biomarkers, for example, to facilitate patient preparation before a surgical procedure and recovery after a surgical procedure. Sensing systems 20336 may also be used to measure and monitor surgeon biomarkers in real time to assist surgical tasks by, for example, communicating relevant biomarkers (e.g., surgeon biomarkers) to a surgical hub 20326 and / or surgical devices 20337 and adjusting their functions. Adjustable surgical device functions may include power level, forward speed, closing speed, load, latency, or other tissue-dependent operating parameters. Sensing systems 20336 may also measure one or more physical attributes associated with the surgeon or patient. Patient biomarkers and / or physical attributes may be measured in real time.

[0155] The computer-implemented wearable patient / surgeon wearable sensing system 20325 may include a surgical hub 20326, one or more sensing systems 20336, and one or more surgical devices 20337. The sensing systems and surgical devices may be communicatively connected to the surgical hub 20326. One or more analysis servers 20338, for example, which are part of an analysis system, may also be communicatively connected to the surgical hub 20326. Although a single surgical hub 20326 is shown, it should be noted that the wearable patient / surgeon wearable sensing system 20325 may include any number of surgical hubs 20326 that can be connected to form a network of surgical hubs 20326 communicatively connected to one or more analysis servers 20338, as described herein.

[0156] For example, the surgical hub 20326 may be a computing device. The computing device may be a personal computer, laptop, tablet, smart mobile device, etc. For example, the computing device may be a client computing device for a cloud-based computing system. The client computing device may be a thin client.

[0157] For example, the surgical hub 20326 may include a processor 20327 coupled to memory 20330 for executing stored instructions, storage 20331 for storing one or more databases such as an EMR database, and a data relay interface 20329 to which data is transmitted to an analysis server 20338. For example, the surgical hub 20326 may further include an I / O interface 20333 having an input device 20341 (e.g., a capacitive touchscreen or keyboard) for receiving input from a user and an output device 20335 (e.g., a display screen) for providing output to the user. For example, the input device and the output device may be a single device. The output may include data from queries entered by the user, suggestions for products or combinations of products for use in a given procedure, and / or instructions for actions performed before, during, or after a surgical procedure. The surgical hub 20326 may include a device interface 20332 for communicatively connecting a surgical device 20337 to the surgical hub 20326. In one embodiment, the device interface 20332 may include a transceiver capable of enabling one or more surgical devices 20337 to connect to the surgical hub 20326 via a wired or wireless interface using one of the wired or wireless communication protocols described herein. The surgical devices 20337 may include, for example, a motorized stapler, an energy device or its generator, an imaging system, or other connected systems such as a smoke exhauster, a suction-irrigation device, or an air supply system.

[0158] In one example, the surgical hub 20326 may be communicatively connected to one or more surgeon and / or patient sensing systems 20336. The sensing systems 20336 may be used to measure and / or monitor in real time various biomarkers related to the surgeon performing the surgical procedure or the patient undergoing the surgical procedure. A list of patient / surgeon biomarkers measured by the sensing systems 20336 is provided herein. In one example, the surgical hub 20326 may be communicatively connected to an environmental sensing system 20334. The environmental sensing system 20334 may be used to measure and / or monitor in real time environmental attributes, such as temperature / humidity in the operating room, surgeon movements, and ambient noise in the operating room caused by the breathing patterns of the surgeon and / or patient.

[0159] When the sensing system 20336 and the surgical device 20337 are connected to the surgical hub 20326, the surgical hub 20326 may receive from the sensing system 20336, for example, as shown in Figures 7B-7D, measurement data associated with one or more patient biomarkers, physical conditions associated with the patient, measurement data associated with a surgeon biomarker, and / or physical conditions associated with the surgeon. The surgical hub 20326 may associate, for example, the measurement data associated with the surgeon with other relevant pre-operative data and / or data from the situation awareness system to generate control signals for controlling the surgical device 20337, for example, as shown in Figure 8.

[0160] For example, the surgical hub 20326 can compare measurement data from the sensing system 20336 to one or more thresholds defined based on baseline values, preoperative measurement data, and / or in the surgical measurement data. The surgical hub 20326 may compare measurement data from the sensing system 20336 to one or more thresholds in real time. The surgical hub 20326 may generate notifications for display. For example, if the measurement data crosses a defined threshold (e.g., is greater than or less than), the surgical hub 20326 may send a notification for delivery to the human interface system for the patient 20339 and / or the human interface system for the surgeon or HCP 20340. The determination of whether a notification is sent to one or more of the patient human interface system 20339 and / or the human interface system for the HCP 2340 may be based on the severity level associated with the notification. The surgical hub 20326 may also generate severity levels associated with the notification for display. The generated severity level may be displayed to the patient and / or surgeon or HCP. In one example, patient biomarkers that are measured and / or monitored (e.g., measured and / or monitored in real time) may be associated with surgical procedure steps. For example, biomarkers measured and monitored for venous and arterial cutting steps in thoracic surgery may include blood pressure, tissue perfusion pressure, edema, arteriosclerosis, collagen content, connective tissue thickness, etc., and biomarkers measured and monitored for lymph node dissection steps in surgical procedure may include monitoring of the patient's blood pressure. In one example, data on postoperative complications can be retrieved from an EMR database in storage 20331, and data on stapling or incision line leakage can be directly detected or inferred by the situational awareness system. Surgical procedure result data can be inferred by the situational awareness system from data received from various data sources, including a surgical device 20337, a sensing system 20336, and a database in storage 20331 to which the surgical hub 20326 is connected.

[0161] The surgical hub 20326 can transmit measurement data and physical state data received from the sensing system 20336, as well as data associated with the surgical device 20337, to the analysis server 20338 for processing. Each of the analysis servers 20338 may include memory and a memory-coupled processor capable of executing instructions stored therein to analyze the received data. The analysis servers 20338 may be connected via a distributed computing architecture and / or utilize a cloud computing architecture. Based on this paired data, the analysis system 20338 can determine the optimal and / or preferred operating parameters for various types of modular devices, generate adjustments to the control programs for the surgical device 20337, and transmit (or "push") the updated or controlled programs to one or more surgical devices 20337. For example, the analysis system 20338 may correlate perioperative data received from the surgical hub 20236 with measurement data associated with the physiological state of the surgeon or HCP and / or the patient. The analysis system 20338 may determine when the surgical device 20337 should be controlled and send updates to the surgical hub 20326. The surgical hub 20326 may then transfer the control program to the relevant surgical device 20337.

[0162] Further details regarding the computer-implemented wearable patient / surgeon wearable sensing system 20325, which includes a surgical hub 30326, one or more sensing systems 20336, and various surgical devices 20337 that can be connected thereto, are described in relation to Figures 5 to 7D.

[0163] Figure 13 is a flowchart of an exemplary method 29700 for processing surgical data during a surgical procedure. As disclosed herein, during a surgical procedure, there are many sources and / or types of surgical data (e.g., surgical sensor data). Such surgical data can be processed for immediate consumption by other surgical systems and medical professionals. This processing can be performed in real time, near real time, etc. Also, a surgical data system, such as a computer-implemented patient and surgeon monitoring system 20000 disclosed herein with reference to Figure 1A, for example, may include multiple processing units that can perform various aspects of sensor processing. Methods disclosed herein, including method 29700 and corresponding devices and combinations of devices that implement these methods using memory and / or processors, can be used to coordinate such surgical sensor data processing. Coordination can promote aspects such as greater system efficiency, higher system and data reliability, graceful handling of faults and failures, and greater overall system flexibility and performance.

[0164] In 29702, a first process may be performed. The first process may be performed on incoming sensor data. For example, the first process may be performed on a first portion of the incoming sensor data. The incoming sensor data may be generated by a sensor unit that senses a physical phenomenon. The incoming sensor data may be received from an external device.

[0165] The first process may be performed according to a first surgical data processing schema. The first surgical data processing schema may be retrieved from memory, for example. The first process may be performed for output to a sensor data channel.

[0166] In 29704, surgical data processing modification commands may be received. Surgical data processing modification commands may be received, for example, via a sensor control channel. Surgical data processing modification commands may also be received from a surgical hub as disclosed herein, for example, surgical hub 20006. Surgical data processing modification commands may be triggered based on changes in the surgical data processing requirements of a surgical procedure.

[0167] A second surgical data processing schema may be generated and / or stored in memory in accordance with a received surgical data processing modification command. For example, a surgical data processing modification command may include information to update or modify the first surgical data processing schema and result in the second surgical data processing schema. For example, a surgical data processing modification command may include the second surgical data processing schema. The second surgical data processing schema may be different from the first surgical data processing schema. For example, the second surgical data processing schema may contain different information and / or instructions than the first surgical data processing schema.

[0168] In 29706, a second process may be performed. This second process may be performed on incoming sensor data.

[0169] For example, the second processing may be performed on a second portion of the incoming sensor data. To illustrate in an active sensing system during a surgical procedure, the first portion of the incoming sensor data may include sensor values ​​processed before the surgical data processing modification command, and the second portion of the incoming sensor data may include sensor values ​​processed after the surgical data processing modification command. This configuration can be used to allow changes in processing related to the currently processed value. For example, this configuration may be appropriate when absolute values ​​are relevant to medical professionals.

[0170] Alternatively, for example, the second processing may be performed on the first portion of the incoming sensor data. The first portion of the incoming sensor data may be stored in memory, such as a buffer, cache, data log, history, or other short-term storage. This configuration may be used to allow modifications to processing related to previously processed values. This configuration may be appropriate when the relationship between the current value and previous values ​​is relevant to medical professionals.

[0171] The second process may be performed according to a second surgical data processing schema. The second process may be performed for output to a sensor data channel.

[0172] For illustrative purposes, a surgical data processing modification command may be used to change sensor processing from a first process to a second process. For example, the change in processing may be motivated by changes in the data processing needs of the system and medical professionals in the surgery, and / or by changes in data processing needs associated with the surgical procedure itself. For example, the first process may have a different output frequency than the second process. For example, the first process may have a different output resolution than the second process. For example, the first process may differ from the second process in terms of the utilization of processing resources. For example, the first process may differ from the second process in terms of data transformation operations.

[0173] For illustrative purposes, surgical data processing modification commands may be used to perform load balancing. For example, surgical data processing modification commands may be used to move data transformation operations (e.g., resource-intensive data transformation operations) from one device to another in the system. For example, surgical data processing modification commands may be used to change a particular device from simple pass-through of sensor data to a transformation other than simple pass-through. For example, surgical data processing modification commands may be used to change a particular device from a transformation other than simple pass-through to simple pass-through of sensor data. Such actions performed by serial devices are exemplary ways of moving processing from one device to another in the system.

[0174] The data processing approaches disclosed herein, such as those shown by Method 29700 and / or its steps, may be performed in connection with any suitable hardware / software data system. For example, the hardware / software data systems disclosed herein may be used. For example, the hardware / software data systems disclosed with respect to Figures 7A to D may be used.

[0175] For example, referring to Figure 7A, processor 20222 and memory 20223 may be used for implementation. Processor 20222 can perform first processing, second processing, and receive and process surgical data processing modification commands. For example, referring to Figure 7B, data processing unit 20238 and storage 20239 may be used for implementation. For example, referring to Figure 7C, data processing unit 20249 and storage 20250 may be used for implementation. Alternatively, for example, method 29700 may be performed by the sensor unit 20245 itself. For example, sensor unit 20245 may include auxiliary processing hardware and a sensor data control channel to data processing and communication unit 20246. Such implementation may be used, for example, with a reduced set of surgical data processing modification commands appropriate to the processing capacity of sensor unit 20245. For example, referring to Figure 7D, data processing unit 20253 and storage 20259 may be used for implementation. Alternatively, for example, method 29700 may be performed by the sensor unit 20252 itself. For example, the sensor unit 20252 may include auxiliary processing hardware and a sensor data control channel to the data processing and communication unit 20253. Such an implementation may be used, for example, with a reduced set of surgical data processing modification commands appropriate to the processing capabilities of the sensor unit 20252.

[0176] Figure 14 shows a block diagram of an exemplary sensor data processing system 29710. System 29710 may include one or more surgical sensor systems 29712, 29714, a surgical sensor data processing device 29716, and one or more downstream systems 29718.

[0177] One or more surgical sensor systems 29712, 29714 may include any of the sensor systems disclosed herein. Surgical sensor systems 29712, 29714 may include any sensing system suitable for use in connection with and / or during a surgical procedure. For example, surgical sensor systems 29712, 29714 may include a patient monitoring system, a surgeon monitoring system, etc. For example, surgical sensor systems 29712, 29714 may include an environmental sensor. For example, surgical sensor systems 29712, 29714 may include a sensor associated with a specific surgical instrument such as an end cutter, a surgical stapler, or an energy device. Surgical sensor systems 29712, 29714 may include, for example, a surgical sensing system 20069 disclosed with reference to Figure 5.

[0178] The surgical sensor systems 29712 and 29714 can measure biomarkers and communicate information about those biomarkers to other devices within system 29710. The surgical sensor systems 29712 and 29714 may include their respective surgical data processing schemas 29720 and 29722. These schemas may include information defining the operation of the corresponding surgical sensor systems 29712 and 29714 and corresponding data structures. For example, the surgical data processing schemas 29720 and 29722 may include information regarding sensor control, sensing operation, sensor data processing (atomic processing, stream processing, and / or composite processing, etc.), data formatting, etc.

[0179] The surgical sensor systems 29712 and 29714 can communicate sensor value information via their respective sensor value data channels 29724 and 29726. The sensor value data channels 29724 and 29726 may include any data communication protocol suitable for transporting sensor value data, such as User Datagram Protocol (UDP), Transmission Control Protocol (TCP), Hypertext Transfer Protocol (HTTP), Raw Data Streaming, Sensor Data Transmission and Management Protocol (STMP), or Simple Sensor Interface (SSI).

[0180] For illustrative purposes, a surgical sensor system 29712 may communicate a stream of sensor data 29728. The stream of sensor data 29728 may be communicated via a sensor value data channel 29724. Stream 29728 may contain information representing a sequential list of sensor values ​​29730, 29732. Each sensor value 29730, 29732 may be accompanied by corresponding metadata such as sensor system identifiers 29734, 29736, and timestamps 29738, 29740. For example, stream 29728 may have one or more parts 29742, 29744. Parts 29742, 29744 may represent a portion of a stream containing one or more values ​​that are logically grouped together. For example, parts may be temporally grouped such that a first part 29742 is communicated and / or associated with a measurement in a corresponding time block. Furthermore, the second portion 29744 communicates with and / or associates with measurements in corresponding different time blocks. For example, the first and second portions may be temporally adjacent. The portions 29742 and 29744 may be grouped by metadata, for example, so that the first and second portions are identified, for example, by their respective metadata tags.

[0181] The surgical sensor systems 29712 and 29714 can communicate commands and associated operational information via their respective sensor control channels 29746 and 29748. The sensor control channels 29746 and 29748 may include any data communication protocol suitable for carrying commands and associated operational information, such as User Datagram Protocol (UDP), Transmission Control Protocol (TCP), Hypertext Transfer Protocol (HTTP), Raw Data Streaming, Sensor Data Transmission and Management Protocol (STMP), and Simple Sensor Interface (SSI).

[0182] The sensor value data channels 29724, 29726 and the sensor control channels 29746, 29748 may include different physical communication hardware. The sensor value data channels 29724, 29726 and the sensor control channels 29746, 29748 may communicate via common physical communication hardware. The sensor value data channels 29724, 29726 and the sensor control channels 29746, 29748 may include logical channels on the same physical communication hardware. The sensor value data channels 29724, 29726 and the sensor control channels 29746, 29748 may receive the same or different processing from network devices. For example, the sensor value data channels 29724, 29726 and the sensor control channels 29746, 29748 may have different transport characteristics such as latency, bandwidth, reliability, packet loss, jitter, retransmission, acknowledgment, negation, etc. For example, sensor value data channels 29724 and 29726 may include high-bandwidth, low-latency channels with no retransmission. Additionally, sensor control channels 29746 and 29748 may have highly reliable reserved bandwidth channels with retransmission.

[0183] Sensor value data channels 29724, 29726 and sensor control channels 29746, 29748 may be used to enable communication between surgical sensor systems 29712, 29714 and surgical sensor data processing device 29716. The surgical sensor data processing device 29716 may be configured to receive one or more incoming streams of sensor data (e.g., stream 29728) from one or more surgical sensor systems, process the data, and route the resulting data to one or more downstream systems 29718. The surgical sensor data processing device 29716 may be configured to communicate with one or more downstream systems 29718 via downstream sensor value data channel 29750 and / or downstream sensor control channel 29752.

[0184] The surgical sensor data processing device 29716 may be configured to generate and / or receive commands. The surgical sensor data processing device 29716 may be configured to send commands to one or more surgical sensor systems 29712, 29714. Commands may be used to modify the operation of the surgical sensor systems 29712, 29714. For example, commands may be used to modify the respective surgical data processing schemas 29720, 29722 of the surgical sensor systems 29712, 29714.

[0185] The surgical sensor data processing device 29716 may have its own surgical data processing schema 29753. The surgical data processing schema 29753 can define the processing that the surgical sensor data processing device 29716 performs on one or more incoming streams. Commands (e.g., from a downstream system 29718) may be used to modify the operation of the surgical sensor data processing device 29716. For example, a command may be used to modify the surgical data processing schema 29753 of the surgical sensor data processing device 29716.

[0186] Figures 15A to 15C are exemplary messaging diagrams illustrating processing modifications in the surgical sensor system 29712, processing modifications in the surgical sensor data processing device 29716, and processing modifications in both the surgical sensor system 29712 and the surgical sensor data processing device 29716, respectively.

[0187] In Figure 15A, the operation of the surgical sensor system 29712 is modified. One or more initialization control messages 29754 may be communicated between the surgical sensor system 29712 and the surgical sensor data processing device 29716 and / or one or more downstream systems 29718. The initialization control messages 29754 can define the initial operation of the surgical sensor system 29712. The initialization control messages 29754 may include operations such as network discovery, device discovery, and service discovery. In one example, the initialization control message 29754 may include an initial surgical data processing schema 29720. In one example, the initial surgical data processing schema 29720 may be retrieved from memory local to the surgical sensor system 29712. Such initialization control messages 29754 may be transmitted via one or more sensor control channels (e.g., sensor control channel 29726 and / or downstream sensor control channel 29752).

[0188] The processor of the surgical sensor system 29712 may receive sensor data. For example, the processor of the surgical sensor system 29712 may receive sensor data from an external device (such as an external sensor unit). For example, the processor of the surgical sensor system 29712 may receive sensor data from an internal subsystem (such as an internal transducer, A / D converter, or processor). The surgical sensor system 29712 may process the data. The surgical sensor system 29712 can process the data according to the surgical data processing schema 29720. The surgical sensor system 29712 may output a stream of sensor data to the surgical sensor data processing device 29716 and / or one or more downstream systems. For example, a first portion of the received sensor data may be represented in a corresponding first output portion 29756. The output stream of sensor data may be communicated via the sensor value data channel 29724 and / or the downstream sensor value data channel 29750.

[0189] Modification control interactions may occur. The interaction may include one or more commands and responses. For example, the surgical sensor system 29712 may receive a surgical data processing modification command 29758. The surgical sensor system 29712 may update the surgical data processing schema 29720 in accordance with the surgical data processing modification command 29758. The surgical sensor system 29712 may then stop processing incoming sensor values ​​in accordance with the processing defined by the initialization control message 29754 and start processing incoming sensor values ​​in accordance with the processing defined by the surgical data processing modification command 29758. The surgical sensor system 29712 may then continue outputting the stream of sensor data being processed under modification to the surgical sensor data processing device 29716 and / or one or more downstream systems 29718. For example, a second portion of the received sensor data may be represented in a corresponding second output portion 29760.

[0190] In Figure 15B, the operation of the surgical sensor data processing device 29716 is modified. One or more initialization control messages 29762 may be communicated between the surgical sensor data processing device 29716 and one or more downstream systems 29718. The initialization control messages 29762 can define the initial operation of the surgical sensor data processing device 29716. The initialization control messages 29762 may include operations such as network discovery, device discovery, and service discovery. For example, the initialization control message 29762 may include an initial surgical data processing schema 29753. For example, the initial surgical data processing schema 29753 may be retrieved from memory local to the surgical sensor data processing device 29716. Such an initialization control message 29762 may be transmitted via a downstream sensor control channel 29752.

[0191] The surgical sensor data processing device 29716 can receive sensor data from the surgical sensor system 29712. The surgical sensor data processing device 29716 can process the data. The surgical sensor data processing device 29716 can process the data according to the surgical data processing schema 29753. The surgical sensor data processing device 29716 may output a stream of sensor data to one or more downstream systems 29718. For example, a first portion 29764 of the received sensor data may be represented in a corresponding first output portion 29766. The output stream of sensor data may be communicated via a downstream sensor value data channel 29750.

[0192] Modification control interactions may occur. These interactions may include one or more commands and responses. For example, the surgical sensor data processing device 29716 may receive a surgical data processing modification command 29768. The surgical sensor data processing device 29716 may update the surgical data processing schema 29753 according to the surgical data processing modification command 29768. The surgical sensor data processing device 29716 may also stop processing incoming sensor values ​​according to the processing defined by the initialization control message 29762 and start processing incoming sensor values ​​according to the processing defined by the surgical data processing modification command 29768. Furthermore, the surgical sensor data processing device 29716 may continue outputting the currently modified, processing-in-progress stream of sensor data to one or more downstream systems 29718. For example, a second portion 29770 of the received and / or generated sensor data may be represented in a corresponding second output portion 29772.

[0193] In Figure 15C, the operation of both the surgical sensor system 29712 and the surgical sensor data processing device 29716 is modified. In this example, the surgical sensor system 29712 may provide certain data processing operations, which may be moved from the surgical sensor system 29712 to the surgical sensor data processing device 29716. For illustrative purposes, such a processing modification may be used, for example, when the surgical sensor system 29712 is overloaded. Such a processing modification may be used when a subsequent part of the surgical procedure requires, for example, the surgical sensor system 29712 to have a higher sampling rate, and offloading some aspects of its processing to the surgical sensor data processing device 29716 makes it possible to achieve that higher sampling rate.

[0194] The processor of the surgical sensor system 29712 may receive sensor data. For example, the processor of the surgical sensor system 29712 may receive a first portion of a surgical sensor data stream. The surgical sensor system 29712 may apply a first operation and a second operation to the first portion. The surgical sensor system 29712 may transmit the output first portion 29774. The output first portion 29774 may represent the sensor data processed by the first and second operations.

[0195] The surgical sensor data processing device 29716 can receive the output first portion 29774. The surgical sensor data processing device 29716 can apply a third operation to the first portion 29774. The surgical sensor data processing device 29716 may transmit the output first portion 29776 to one or more downstream systems 29718.

[0196] Next, for example, based on the system's data processing requirements, a second operation may be moved from the surgical sensor system 29712 to the surgical sensor data processing device 29716. For example, the surgical sensor data processing device 29716 may receive surgical data processing modification commands from the downstream system 29718. Alternatively, for example, the surgical sensor data processing device 29716 may spontaneously initiate processing modifications.

[0197] The surgical sensor data processing device 29716 can send a surgical data processing modification command 29778 to the surgical sensor system 29712. The surgical data processing modification command 29778 may be triggered based on load balancing operations between the surgical sensor system 29712 and the surgical sensor data processing device 29716. The surgical data processing modification command 29778 may be triggered based on load balancing operations between the surgical sensor system 29712 and the surgical sensor data processing device 29716 based on changes in the surgical data processing requirements of a surgical procedure.

[0198] The surgical data processing modification command 29778 may instruct the surgical sensor system 29712 to modify its surgical data processing schema 29720 so that it applies the first operation to the second portion of the incoming sensor data, but does not apply the second operation to the second portion of the incoming sensor data. Thus, the surgical sensor system 29712 can transmit the output second portion 29780, which may represent sensor data processed by the first operation rather than the second operation.

[0199] The surgical sensor data processing device 29716 may update its surgical data processing schema 29753 so that it applies the second and third operations to the second portion 29780. The surgical sensor data processing device 29716 can spontaneously update its own surgical data processing schema 29753. The surgical sensor data processing device 29716 can update its surgical data processing schema 29753 based on surgical data processing modification commands from the downstream system 29718. Thus, the surgical sensor data processing device 29716 can send the outputted second portion 29782. The outputted second portion 29780 may represent the sensor data processed by the first, second, and third operations.

[0200] Figure 16 is a block diagram of an exemplary surgical data processing schema 29784. The surgical data processing schema 29784 may include information and corresponding data structures that define the operation of corresponding devices, such as corresponding surgical sensor systems and / or corresponding surgical sensor data processing devices. The surgical data processing schema 29784 may also include information regarding sensor control, sensing operation, sensor data processing (atomic processing, stream processing, and / or composite processing, etc.), data formatting, etc. The surgical data processing schema 29784 may include such information in a structured data format. For example, the structured format may be any format for storing and labeling parameters (such as control, operation, and / or processing parameters). For example, the structured format may be a proprietary file type, comma-separated file, table, two-dimensional array, array of embedded arrays, JavaScript® Object Notation (JSON), Extended Markup Language (XML), record, tagged union, object, database, database record, etc.

[0201] An exemplary surgical data processing schema 29784 may include control parameters 29786, sensing parameters 29788, atomic processing parameters 29790, stream processing parameters 29792, composite processing parameters 29794, data format parameters 29796, etc.

[0202] Control parameter 29786 may include information regarding the overall and high-level operation of the corresponding device, such as the corresponding surgical sensor system and / or the corresponding surgical sensor data processing device. Control parameter 29786 may include a sensor identifier, a processing identifier, and an initialization process key (such as a discovery key or a Trivial File Transfer Protocol (TFTP) link). Control parameter 29786 may include limitations on device operation, such as limitations on power consumption and processing resources. Control parameter 29786 may include communication and / or networking information such as network type, network node identification, channel information (e.g., information identifying and defining the corresponding sensor data channel and / or sensor control channel), channel usage information (e.g., communication and / or networking information such as information identifying which channel should be used when two or more channels are identified for a given type; for example, two sensor data channels may be defined, each directing sensor data to its respective processing device; the channel usage information in control parameter 29786 may be used to select which of those processing devices receives the output data), and security information (e.g., public / private keys, authentication method, encryption type). Control parameter 29786 may include a master process flow that defines the ordered steps (including any conditional processing) to be performed by the device. The master process flow may refer to operations further defined by other parameters in schema 29784.

[0203] The sensing parameter 29788 may include any information that defines the operation of converting physical phenomena into information. The sensing parameter 29788 may include transducer settings, calibration information and settings, sensing resolution, sensing frequency, sample rate, etc.

[0204] The atomic processing parameter 29790 may include any information and / or instructions that define an action to be performed for each value of the sensed data. The atomic processing parameter 29790 may be performed individually for each sensor value. The atomic processing parameter 29790 may include information that identifies one or more specific actions to be performed. The atomic processing parameter 29790 may include parameters for each of the identified specific actions. For illustrative purposes, the atomic processing parameter 29790 may include information about an offset action. The atomic processing parameter 29790 may include information that identifies an offset action. The atomic processing parameter 29790 may also include information that specifies an offset value. Therefore, a device processing sensor data according to such a surgical data processing schema 29784 will output sensor values ​​offset by the specified offset value. Other actions that may be represented in the atomic processing parameter 29790 may include data mapping, thresholding, triggering, downsampling, etc.

[0205] The stream processing parameter 29792 may include any information and / or instructions that define an action to be performed across multiple sensor values. The stream processing parameter 29792 may include information that identifies one or more specific actions to be performed. The stream processing parameter 29792 may include parameters for each of the identified specific actions. Actions that can be represented by the stream processing parameter 29792 may include moving averages, hysteresis, process chains, statistical processes, filtering (noise filters, adaptive filters, low-pass filters, band-pass filters, high-pass filters, etc.), upsampling, etc.

[0206] The composite processing parameter 29794 may include any information and / or instructions that define an action to be performed using values ​​from two or more sensors. The composite processing parameter 29794 may include information that identifies one or more specific actions to be performed. The composite processing parameter 29794 may include parameters for each of the identified specific actions, such as which sensor should be used to obtain values ​​for processing. Actions that can be represented by the composite parameter 29794 may include sensor fusion actions, conditional actions, composite biomarker mapping actions, virtual sensor actions, etc.

[0207] The data formatting parameter 29796 may include any information and / or instructions that define the data format of the output sensor value stream. The data formatting parameter 29796 may include information regarding units, timestamps, data types, data element precision, etc.

[0208] Figure 17 is a block diagram of an exemplary sensor processing coordinator 29798. The sensor processing coordinator 29798 may include any hardware, software, and combinations thereof suitable for generating surgical data processing modification commands 29800. For example, the sensor processing coordinator 29798 may include a processor and / or memory configured to perform the operations disclosed herein. The sensor processing coordinator 29798 may be incorporated, for example, into a surgical hub. The sensor processing coordinator 29798 may be incorporated into other devices in a computer-implemented patient and surgeon monitoring system.

[0209] A computer-implemented patient and surgeon monitoring system may include one or more sensor processing coordinators 29798. For example, a sensor processing coordinator 29798 may have a global view of the computer-implemented patient and surgeon monitoring system and may generate surgical data processing modification commands 29800 for the entire computer-implemented patient and surgeon monitoring system. Alternatively, for example, a sensor processing coordinator 29798 may have a limited view of the computer-implemented patient and surgeon monitoring system and may generate surgical data processing modification commands 29800 for a portion of the computer-implemented patient and surgeon monitoring system. For example, a sensor processing coordinator 29798 may be associated with a particular set of surgical sensing systems and / or surgical sensor data processing devices.

[0210] The sensor processing coordinator 29798 can be used within the context of any sensor management system and / or protocol. For example, the sensor processing coordinator 29798 may be integrated into distributed stream management systems such as Digital Imaging and Communications in Medicine (DICOM) and BioSignalML markup languages, as well as platforms such as TelegraphCQ, PIPES, and Borealis.

[0211] The sensor processing coordinator 29798 can generate surgical data processing modification commands 29800 based on one or more inputs. For example, the sensor processing coordinator 29798 can generate surgical data processing modification commands 29800 based on sensor workload data 29802, treatment plan data 29804, surgical situation recognition data 29806, etc.

[0212] Sensor workload data 29802 may include information representing the current and / or expected performance of sensor processing by one or more devices in the system. For example, a surgical sensor data processing device may be utilizing 80% of its processing capacity to handle data from four related surgical sensing systems. Such input may be used by the sensor processing coordinator 29798 to determine whether to generate a surgical data processing modification command 29800 to modify the processing being handled by that device.

[0213] The procedure plan data 29804 may include information representing individual aspects of the surgery, including information about the expected sensor demands for each aspect. For example, the procedure plan data 29804 may indicate that a particular surgical task during the procedure requires more processing resources than others.

[0214] The surgical situation awareness data 29806 may include any other data available in a computer-implemented patient and surgeon monitoring system that can be used to adjust sensor processing. For example, a surgical instrument (e.g., a surgical instrument not expected from the procedure plan to be used) is turned on. The surgical situation awareness data 29806 may include an indication of the surgical instrument's identifier and an indication that the surgical instrument has been activated. Such information about real-time events in the operating room may be used by the sensor processing coordinator 29798 to determine whether to generate a surgical data processing modification command 29800 to modify existing sensor processing, for example, to make additional processing capacity available to support the operation of an unplanned surgical instrument.

[0215] The sensor processing coordinator 29798 may include a master sensor list 29808 and a coordination plan 29810. The master sensor list 29808 may include information on current, past, and expected sensors and devices for use during surgical procedures. The master list 29808 may also include logistical data for all devices in the computer-implemented patient and surgeon monitoring system. For example, the master list may include a copy of the surgical data processing schema for each device.

[0216] Adjustment plan 29810 may include information relating to the operation of sensors and devices within a computer-implemented patient and surgeon monitoring system. For example, adjustment plan 29810 may include initialization information for sensors and devices. For example, adjustment plan 29810 may include mitigation processes for anticipated changes in surgical data processing requirements during surgical procedures. For example, adjustment plan 29810 may include mitigation processes that can be triggered by specific surgical situation recognition triggers. Adjustment plan 29810 may include information and / or instructions for implementing one or more data processing strategies in a computer-implemented patient and surgeon monitoring system.

[0217] For example, coordination plan 29810 may include information and / or instructions for implementing a load balancing strategy. For instance, coordination plan 29801 may include instructions to instruct a sensing system to stop part of its operation when it detects it is nearing capacity, to stream raw data to another device, and to instruct other devices to continue the remaining operation. For instance, coordination plan 29801 may include instructions for identifying devices with additional unused capacity that can be used to assist other devices in the system. Such sensor processing load balancing can improve overall system utilization, data processing speed, data acquisition rate, and communication bandwidth.

[0218] For example, Coordination Plan 29810 may include information and / or instructions for implementing a specific sensor processing topology. The sensor processing coordinator can define different topologies and corresponding strategies, for example, by coordinating the identity and use of sensor data value channels and corresponding processing. For example, Coordination Plan 29810 may include information and / or instructions for instructing each surgical sensing system to stream their output feeds to a single aggregation device, such as a surgical hub. The Coordination Plan may also include information and / or instructions for instructing each surgical sensing system to stream at their best collection and transmission speed. The surgical hub can then collect this highest-resolution raw data and process all streams together. Alternatively, for example, the Coordination Plan may include information and / or instructions for defining processing subunits so that devices send their data to distributed processing points. Processing points may be defined based on processing capacity, algorithmic coexistence (e.g., pairing memory-intensive but non-processing-intensive processing operations with processing-intensive but non-memory-intensive operations), functional groups, etc.

[0219] For example, Coordination Plan 29810 may include information and / or instructions for implementing a specific sensor prioritization scheme. For instance, specific sensor feeds may be classified by varying levels of importance. For example, two categorization schemes may be implemented such that higher-priority feeds are captured safely and consistently at least at their minimum required frequency, while lower-priority feeds are captured on a best-effort basis and / or when capacity is available.

[0220] Furthermore, for example, the adjustment plan 29810 may include information and / or instructions for prioritizing sensor data processing according to context-aware data 29806 (e.g., current surgical activity and patient biomarkers) and / or procedure plan data 29804. The adjustment plan 29810 may include information and / or instructions for prioritizing sensor feeds that are more important to a particular aspect of a procedure and lowering the priority of sensor feeds that are less important to a particular aspect of a procedure, as detected by context-aware data 29806 and / or described in procedure plan data 28804. Prioritization may include enabling higher resolution, sampling rate, etc., for more important feeds and lower resolution, sampling rate, etc., for less important feeds. Such an adjustment plan 29810 can maximize the use of available bandwidth and processing power. Such an adjustment plan 29810 can rebalance the computer-implemented patient and surgeon monitoring system throughout the surgery.

[0221] For example, adjustment plan 29810 can be used to limit the local processing of the sensor based on a biomarker or patient-specific parameter. For instance, adjustment plan 29810 may be used to limit the local processing of the sensor based on, for example, physiological limits. For illustrative purposes, measuring heart rate variability may require a higher sampling rate than simply measuring heart rate itself. Both biomarkers can be measured using the same sensor. However, if the context-aware data 29806 and / or treatment plan data 28804 require heart rate rather than heart rate variability, adjustment plan 29810 may include information and / or instructions to adjust the sensor's operation accordingly. Such down-adjustment may provide additional capacity in the processing system for other sensors, for example.

[0222] The following list of embodiments forms part of the description. 1. A device for processing surgical data during a surgical procedure, wherein the device is Memory and A processor, and the processor, Retrieving the first surgical data processing schema from memory, To output to the sensor data channel, perform a first processing on a first portion of the incoming sensor data according to a first surgical data processing schema, Receiving surgical data processing modification commands via the sensor control channel, The procedure involves saving a second surgical data processing schema to memory according to a surgical data processing modification command, wherein the second surgical data processing schema is different from the first surgical data processing schema. A device configured to perform a second processing of a second portion of incoming sensor data according to a second surgical data processing schema, for output to a sensor data channel, wherein the second processing is different from the first processing. 2. The surgical data processing modification command is received from the surgical hub by the device described in Embodiment 1. 3. The device according to embodiment 2, wherein a surgical data processing modification command is triggered based on a change in the surgical data processing requirements of a surgical procedure. 4. The device according to embodiment 1, wherein the first process has a different output frequency from the second process. 5. The device according to Embodiment 1, wherein the first process has a different output resolution from the second process. 6. The device according to Embodiment 1, wherein the first process differs from the second process in terms of the use of processing resources. 7. The device according to Embodiment 1, wherein the first process differs from the second process with respect to data conversion operations. 8. The device according to embodiment 1, further comprising a sensor unit configured to generate incoming sensor data by sensing a physical phenomenon. 9. The device according to embodiment 1, further comprising an input configured to receive incoming sensor data from an external device. 10. The device according to Embodiment 1, wherein each of the first input / output conversions of either the first or second process is a simple pass-through, and each of the other second input / output conversions of either the first or second process includes an input / output conversion other than a simple pass-through, and the input / output conversion other than a simple pass-through performs either an atomic process, a stream process, or a composite process. 11. A method for processing surgical data during a surgical procedure in a system, the method being: The first device of the system sends a surgical data processing modification command, In the second device of the system, To output to the sensor data channel, perform a first processing on a first portion of the incoming sensor data according to a first surgical data processing schema, Receiving surgical data processing modification commands via the sensor control channel, A method comprising performing a second processing on a second portion of incoming sensor data according to a second surgical data processing schema for output to a sensor data channel, wherein the second surgical data processing schema is different from the first surgical data processing schema, based on surgical data processing modification commands. 12. The method according to aspect 11, wherein a surgical data processing modification command is triggered based on a change in the surgical data processing requirements of a surgical procedure. 13. The method according to aspect 11, wherein a surgical data processing modification command is triggered based on the current surgical data processing usage pattern that exceeds a threshold. 14. The method according to aspect 11, wherein a surgical data processing modification command is triggered based on an indication of the surgical importance of incoming sensor data. 15. The method according to embodiment 11, wherein the first process has a different output frequency from the second process. 16. The method according to embodiment 11, wherein the first process has a different output resolution from the second process. 17. The method according to aspect 11, wherein the first process differs from the second process in terms of the use of processing resources. 18. The method according to embodiment 11, wherein the first process differs from the second process with respect to data conversion operations. 19. A system for applying a first processing operation, a second processing operation, and a third processing operation to a surgical sensor data stream during a surgical procedure, wherein the system is: A first surgical system component, the first surgical system component is Receiving surgical sensor data streams, Applying a first operation and a second processing operation to a first portion of the surgical sensor data stream, A first surgical system component is configured to apply a first processing operation, rather than a second processing operation, to a second portion of the surgical sensor data stream based on the receipt of a surgical data processing modification command, A second surgical system component, the second surgical system component is Receiving a surgical sensor data stream from the first surgical system component, Applying the third operation, rather than the second processing operation, to the first portion of the surgical sensor data stream, A system comprising: a second surgical system component configured to apply a third processing operation and a second processing operation to a second portion of a surgical sensor data stream. 20. The system according to aspect 19, wherein a surgical data processing modification command is triggered based on load balancing operations between a first surgical system component and a second surgical system component based on changes in the surgical data processing requirements of a surgical procedure. [Examples]

[0223] Example 1. A device for processing surgical data during a surgical procedure, comprising memory and a processor, wherein the processor is configured to retrieve a first surgical data processing schema from memory, perform a first processing on a first portion of incoming sensor data according to the first surgical data processing schema for output to a sensor data channel, receive a surgical data processing modification command via a sensor control channel, store a second surgical data processing schema in memory according to the surgical data processing modification command, wherein the second surgical data processing schema is different from the first surgical data processing schema, and perform a second processing on a second portion of incoming sensor data according to the second surgical data processing schema for output to a sensor data channel, wherein the second processing is different from the first processing.

[0224] For example, in Example 1, the first process may be a first processing operation, and the second process may be a second processing operation.

[0225] For example, in Example 1, performing the second process may further include not performing the first process on the second portion of the incoming sensor data.

[0226] For example, in Example 1, the sensor data may be a sensor data stream.

[0227] For example, in Example 1, the surgical data processing modification command can instruct the processor to use a second surgical data processing schema.

[0228] For example, in Embodiment 1, the first and / or second surgical data processing schemas can specify the processing to be performed on incoming sensor data. The first and / or second surgical data processing schemas may include control parameters, sensing parameters, atomic processing parameters, stream processing parameters, composite processing parameters, and / or data format parameters that can be used to process the incoming sensor data or applied to the incoming sensor data.

[0229] For example, in Example 1, the first and / or second processes may include filtering, averaging, verification, sorting, aggregation, smoothing, and / or classification processes. The first and / or second processes may also include other forms of processing operations.

[0230] For example, in Embodiment 1, the first portion of the incoming sensor data may include or consist of sensor values ​​processed before the surgical data processing modification command is received by the processor, and the second portion of the incoming sensor data may include or consist of sensor values ​​processed after the surgical data processing modification command is received by the processor.

[0231] For example, in Example 1, the processor may receive surgical data processing modification commands from a downstream system and / or from a sensor system that provides incoming sensor data.

[0232] For example, in Example 1, the surgical data processing modification command may be based on sensor workload data, surgical procedure plan data, and / or surgical situation recognition data.

[0233] The device of Example 1 enables better coordination of data processing during surgical procedures. Data processing behavior can be modified during the processing of incoming sensor data based on data processing modification commands. For example, data processing may be modified based on sensor workload data, surgical procedure planning data, or surgical situation awareness data. Modifications to processing may be motivated by changes in the data processing needs of the system and medical professionals in surgery, or by changes in data processing needs associated with the surgical procedure itself. For example, by modifying the data processing of sensor data, the device can stop or minimize data processing on its own to free up processing capacity for other needs, and pass raw, unprocessed data to another device for processing. Processing behavior may be shifted to other devices or system components where more capacity is available. These measures improve overall system utilization, data processing speed, and data acquisition speed. Furthermore, freeing up the processing capacity of the device itself allows for more processing capacity, for example, as critical in the surgical approach or in the event of a medical emergency. Overall, such processing adjustments improve efficiency, data reliability, fault and failure handling, system flexibility, and overall performance, leading to improved patient safety and improved surgical procedures.

[0234] Example 2. The surgical data processing modification command is received from the surgical hub by the device described in Example 1.

[0235] The device in Example 2 is controlled by a surgical hub that can coordinate data processing across a set of interconnected devices, such as devices in an operating room.

[0236] Example 3. The device according to Example 1 or 2, wherein the surgical data processing modification command is triggered based on a change in the surgical data processing requirements of a surgical procedure.

[0237] The device in Example 3 modifies the data processing performed by the device as a result of changes in data processing requirements arising during surgical procedures. This allows the device to optimize data processing and available processing resources for the procedure being performed and adapt to changes that may occur during the procedure, such as approaching critical steps or medical emergencies. This leads to improved surgical procedures and patient safety.

[0238] Example 4. The device according to any one of Examples 1 to 3, wherein the first process has a different output frequency from the second process.

[0239] For example, in the device of Example 4, the first process may have an output frequency lower or higher than the output frequency of the second process.

[0240] The device of Example 4 allows for the optimization of processing frequency for a given time in a surgical procedure. For example, by reducing the output frequency, processing resources can be freed up for other more important tasks, and by increasing the output frequency, reserve capacity can be utilized for additional data processing, reducing pressure on other parts of the interconnected device.

[0241] Example 5. The device according to any one of Examples 1 to 4, wherein the first process has a different output resolution from the second process.

[0242] For example, in the device of Example 5, the first process may have an output resolution lower or higher than the output resolution of the second process.

[0243] The device of Example 5 allows the output resolution to be optimized for a given point in a surgical procedure. For example, by reducing the output resolution, processing resources can be freed up for other more important tasks, and by increasing the output resolution, reserve capacity can be used for additional data processing, reducing pressure on other parts of the interconnected device.

[0244] Example 6. The device described in any one of Examples 1 to 5, wherein the first process differs from the second process in terms of the use of processing resources.

[0245] For example, in the device of Example 6, the first process may use fewer processing resources than the second process, or it may use more processing resources.

[0246] The device of Example 6 enables the optimization of processing resources for a given point in time during a surgical procedure. For example, by utilizing fewer processing resources, those resources can be freed up for other more important tasks, and by utilizing more processing resources, the reserve capacity can be used for additional data processing, reducing pressure on other parts of the interconnected device.

[0247] Example 7. The device described in any one of Examples 1 to 6, wherein the first process differs from the second process with respect to the data conversion operation.

[0248] Example 8. The device according to any one of Examples 1 to 7, further comprising a sensor unit configured to generate incoming sensor data.

[0249] For example, in Example 8, the sensor unit may be a patient monitoring system, a surgeon monitoring system, an environmental monitoring system, and / or a surgical instrument monitoring system.

[0250] The device of Example 8 can generate incoming sensor data based on conditions that are changing in real time during surgery, and enables the device to adapt processing conditions in real time based on unpredictable or unexpected events.

[0251] Example 9. The device according to any one of Examples 1 to 8, further comprising an input configured to receive incoming sensor data from an external device.

[0252] For example, in Example 9, the external device may be a sensor or a sensor system.

[0253] For example, in any one of Examples 1 to 9, the incoming sensor data may be measurements of one or more biomarkers, measurements of patient-specific parameters, and / or measurements of one or more environmental parameters.

[0254] The device of Example 9 can generate incoming sensor data based on conditions that are changing in real time during surgery, and enables the device to adapt processing conditions in real time based on unpredictable or unexpected events.

[0255] Example 10. The device according to any one of Examples 1 to 9, wherein each of the first input / output conversions of either the first or second process is pass-through, and each of the other second input / output conversions of either the first or second process includes an input / output conversion other than pass-through, and optionally, the input / output conversion other than pass-through performs either atomic processing, stream processing, or composite processing.

[0256] For example, in any one of Examples 1 to 10, either the input / output conversion of the first process is pass-through and the input / output conversion of the second process includes an input / output conversion other than pass-through, or the input / output conversion of the second process is pass-through and the input / output conversion of the first process includes an input / output conversion other than pass-through.

[0257] The device of Example 10 enables optimizing the processing frequency for a given point in time during a surgical procedure. For example, by using pass-throughs, processing resources can be freed up within the device for other more important tasks, and by replacing the pass-throughs with alternative data processing, reserve capacity can be utilized for additional data processing and relieve pressure at other locations within the interconnected devices.

[0258] Example 11. A method for processing surgical data during a surgical procedure in a system, comprising: in a second device of the system, performing a first processing on a first portion of incoming sensor data according to a first surgical data processing schema for output to a sensor data channel; in a first device of the system, transmitting a surgical data processing modification command; in the second device of the system, receiving the surgical data processing modification command via a sensor control channel; performing a second processing on a second portion of the incoming sensor data according to a second surgical data processing schema for output to the sensor data channel, wherein the second surgical data processing schema is different from the first surgical data processing schema based on the surgical data processing modification command.

[0259] For example, in Example 11, the first processing may be a first processing operation and the second processing may be a second processing operation.

[0260] For example, in Example 11, performing the second processing may further include not performing the first processing on the second portion of the incoming sensor data.

[0261] For example, in Example 11, the sensor data may be a sensor data stream.

[0262] For example, in Example 11, the surgical data processing modification command may include an instruction to use a second surgical data processing schema. The second surgical data processing schema may be based on the surgical data processing modification command as long as the surgical data processing modification command includes an instruction to use the second surgical data processing schema.

[0263] For example, in Example 11, the first and / or second surgical data processing schema may specify the processing to be performed on the incoming sensor data. The first and / or second surgical data processing schema may include control parameters, sensing parameters, atomic processing parameters, stream processing parameters, composite processing parameters, and / or data format parameters that can be used to process the incoming sensor data or applied to the incoming sensor data.

[0264] For example, in Example 11, the first and / or second processing may include filtering, averaging, verification, sorting, aggregation, smoothing, and / or classification processes. The first and / or second processing may include other forms of processing operations.

[0265] For example, in Example 11, the first portion of the incoming sensor data may include or be composed of sensor values that have been processed before the surgical data processing modification command is received by the processor, and the second portion of the incoming sensor data may include or be composed of sensor values that have been processed after the surgical data processing modification command is received by the processor.

[0266] For example, in Example 11, the first device may be a surgical sensor data processing device, the second device may be a surgical sensor system, or the first device may be a downstream system, and the second device may be a surgical sensor data processing device.

[0267] For example, in Example 11, the surgical data processing modification command may be based on sensor workload data, surgical procedure plan data, and / or surgical situation recognition data.

[0268] The method of Example 11 enables better coordination of data processing during surgical procedures. Data processing behavior can be modified during the processing of incoming sensor data based on data processing modification commands. For example, data processing may be modified based on sensor workload data, surgical procedure planning data, or surgical situation awareness data. Modifications to processing may be motivated by changes in the data processing needs of the system and medical professionals in the surgery, or by changes in data processing needs associated with the surgical procedure itself. For example, by modifying the data processing of sensor data, a device can stop or minimize data processing on its own to free up processing capacity for other needs, and pass the raw, unprocessed data to another device for processing. Processing behavior may be shifted to other devices or system components where more capacity is available. These means improve overall system utilization, data processing speed, and data acquisition speed. Furthermore, freeing up the processing capacity of the device itself allows for more processing capacity, for example, as critical in the surgical approach or in the event of a medical emergency. Overall, such processing adjustments improve efficiency, data reliability, fault and failure handling, system flexibility, and overall performance, leading to improved patient safety and improved surgical procedures.

[0269] Example 12. A system for processing surgical data during a surgical procedure, the system comprising: a first device configured to transmit surgical data processing modification commands; and a second device, wherein the second device is configured to perform a first processing on a first portion of incoming sensor data according to a first surgical data processing schema for output to a sensor data channel; receive a surgical data processing modification command via a sensor control channel; and perform a second processing on a second portion of incoming sensor data according to a second surgical data processing schema for output to a sensor data channel, wherein the second surgical data processing schema is different from the first surgical data processing schema based on the surgical data processing modification command.

[0270] For example, in Example 12, the first process may be a first processing operation, and the second process may be a second processing operation.

[0271] For example, in Example 12, performing the second process may further include not performing the first process on the second portion of the incoming sensor data.

[0272] For example, in Example 12, the sensor data may be a sensor data stream.

[0273] For example, in Example 12, the surgical data processing modification command may include instructions that use a second surgical data processing schema. The second surgical data processing schema can be based on the surgical data processing modification command, insofar as the surgical data processing modification command includes instructions that use the second surgical data processing schema.

[0274] For example, in Embodiment 12, the first and / or second surgical data processing schemas can specify the processing to be performed on incoming sensor data. The first and / or second surgical data processing schemas may include control parameters, sensing parameters, atomic processing parameters, stream processing parameters, composite processing parameters, and / or data format parameters that can be used to process the incoming sensor data or applied to the incoming sensor data.

[0275] For example, in Example 12, the first and / or second processes may include filtering, averaging, verification, sorting, aggregation, smoothing, and / or classification processes. The first and / or second processes may also include other forms of processing operations.

[0276] For example, in Embodiment 12, the first portion of the incoming sensor data may include or consist of sensor values ​​processed before the surgical data processing modification command is received, and the second portion of the incoming sensor data may include or consist of sensor values ​​processed after the surgical data processing modification command is received.

[0277] For example, in Example 12, the first device may be a surgical sensor data processing device, the second device may be a surgical sensor system, or the first device may be a downstream system and the second device may be a surgical sensor data processing device.

[0278] For example, in Example 12, the surgical data processing modification command may be based on sensor workload data, surgical procedure plan data, and / or surgical situation recognition data.

[0279] The system of Example 12 enables better coordination of data processing during surgical procedures. Data processing behavior can be modified during the processing of incoming sensor data based on data processing modification commands. For example, data processing may be modified based on sensor workload data, surgical procedure planning data, or surgical situation awareness data. Modifications to processing may be motivated by changes in the data processing needs of the system and medical professionals in surgery, or by changes in data processing needs associated with the surgical procedure itself. For example, by modifying the data processing of sensor data, a device can stop or minimize data processing on its own to free up processing capacity for other needs, and pass raw, unprocessed data to another device for processing. Processing behavior may be shifted to other devices or system components where more capacity is available. These measures improve overall system utilization, data processing speed, and data acquisition speed. Furthermore, freeing up the processing capacity of the device itself allows for more processing capacity, for example, as critical in the surgical approach or in the event of a medical emergency. Overall, such processing adjustments improve efficiency, data reliability, fault and failure handling, system flexibility, and overall performance, leading to improved patient safety and improved surgical procedures.

[0280] Example 13. The surgical data processing modification command is triggered based on changes in the surgical data processing requirements of a surgical procedure, as described in Example 11 or the system described in Example 12.

[0281] The method or system of Example 13 modifies the data processing performed by the second device as a result of changes in data processing requirements arising in a surgical procedure. This makes it possible to optimize data processing and available processing resources for the procedure being performed and to adapt to changes that may occur during the procedure, such as approaching critical steps or medical emergencies. This leads to improved surgical procedures and patient safety.

[0282] Example 14. The surgical data processing correction command is the method according to Example 11 or Example 13, which is triggered by exceeding the surgical data processing utilization threshold, the device according to any one of Examples 1 to 10, or the system according to Example 12 or Example 13.

[0283] In the method, device, or system of Example 14, the surgical data processing correction command is triggered as a result of exceeding the data processing utilization threshold. Thereby, when the data processing ability is too low, it becomes possible to minimize, stop, or move the data processing to another device, and there is a risk of processing delay as a result that may affect the ongoing surgical procedure or patient safety.

[0284] Example 15. The surgical data processing correction command is triggered based on the surgical importance of the incoming sensor data exceeding a threshold, the method according to any one of Examples 11, 13, or 14, the device according to any one of Examples 1 to 10, or 14, or the system according to any one of Examples 12 to 14.

[0285] In the method, device, or system of Example 14, the surgical data processing correction command is triggered as a result of data indicating surgical importance. Thereby, the data processing can be adjusted according to the importance of the data for the surgical procedure. For example, if the processing of the data is labeled as important for the surgical procedure or important for the operation, the processing of the data by the second device can be increased so that the processing of the data is prioritized. Alternatively, the processing in the second device can be stopped or minimized, and the sensor data can be transmitted to the connected device, for example, for higher quality or faster processing.

[0286] Example 16. The first processing has an output frequency different from that of the second processing, the method according to any one of Examples 11 or 13 to 15, the device according to any one of Examples 1 to 10, 14, or 15, or the system according to any one of Examples 12 to 15.

[0287] For example, in the method, device, or system of Example 16, the first process may have an output frequency lower or higher than the output frequency of the second process.

[0288] The method, device, or system of Example 16 enables optimization of the processing frequency for a given time in a surgical procedure. For example, by reducing the output frequency, processing resources can be freed up for other more important tasks, and by increasing the output frequency, reserve capacity can be utilized for additional data processing, reducing pressure on other parts of the interconnected device.

[0289] Example 17. The first process having a different output resolution from the second process, the method described in any one of Examples 11 or 13-16, the device described in any one of Examples 1-10, 14, 15 or 16, or the system described in any one of Examples 12-16.

[0290] For example, in the method, device, or system of Example 17, the first process may have a lower or higher output resolution than the output resolution of the second process.

[0291] The method, device, or system of Example 17 allows the output resolution to be optimized for a given point in a surgical procedure. For example, by reducing the output resolution, processing resources can be freed up for other more important tasks, and by increasing the output resolution, reserve capacity can be utilized for additional data processing, reducing pressure on other parts of the interconnected device.

[0292] Example 18. The first process differs from the second process in terms of the use of processing resources, using the method described in Example 11 or any one of Examples 13-17, the device described in any one of Examples 1-10, 14, 15, 16, or 17, or the system described in any one of Examples 12-17.

[0293] For example, in the method, device, or system of Example 18, the first process may use fewer processing resources or more processing resources than the second process.

[0294] The method, device, or system of Example 18 enables the optimization of processing resources for a given point in time during a surgical procedure. For example, by utilizing fewer processing resources, those resources can be freed up for other more important tasks, and by utilizing more processing resources, the reserve capacity can be used for additional data processing, reducing pressure on other parts of the interconnected device.

[0295] Example 19. The first process differs from the second process in terms of data conversion operation, and is the method described in any one of Examples 11 or 13-18, the device described in any one of Examples 1-10, 14, 15, 16, or 17, or the system described in any one of Examples 12-18.

[0296] Example 20. A system for applying a first processing operation, a second processing operation, and a third processing operation to a surgical sensor data stream during a surgical procedure, the system comprising: a first surgical system component, the first surgical system component configured to receive a surgical sensor data stream, apply the first operation and the second processing operation to a first portion of the surgical sensor data stream, and, based on the receipt of a surgical data processing modification command, apply the first processing operation instead of the second processing operation to a second portion of the surgical sensor data stream; and a second surgical system component, the second surgical system component configured to receive a surgical sensor data stream from the first surgical system component, apply the third operation instead of the second processing operation to a first portion of the surgical sensor data stream, and apply the third processing operation and the second processing operation to a second portion of the surgical sensor data stream.

[0297] For example, in Embodiment 20, the second surgical system component may be configured to apply a third operation, rather than a second processing operation, to a first portion of the surgical sensor data stream before the surgical data processing modification command is received by the first surgical system, and to apply the third and second processing operations to a second portion of the surgical sensor data stream after the surgical data processing modification command is received by the first surgical system.

[0298] For example, in Example 20, the first surgical system component may be configured to receive surgical data processing modification commands.

[0299] For example, in Example 20, the surgical data processing modification command may include an instruction to a first surgical system component to apply the first processing operation but not the second processing operation, and / or an instruction to a second surgical system component to apply the third processing operation and the second processing operation.

[0300] For example, in Example 20, the first, second, and / or third processing operations may include filtering, averaging, verification, sorting, aggregation, smoothing, and / or classification processes. The first, second, and / or third processing operations may also include other forms of processing operations.

[0301] For example, in Embodiment 20, the first portion of the incoming sensor data may include or consist of sensor values ​​processed by the first and / or second system before the surgical data processing modification command, and the second portion of the incoming sensor data may include or consist of sensor values ​​processed by the first and / or second system after the surgical data processing modification command.

[0302] For example, in Example 20, the first surgical system component may receive surgical data processing modification commands from the second surgical system component or a downstream system. For example, the second surgical system component may generate surgical data processing modifications.

[0303] For example, in Example 20, the surgical data processing modification command may be based on sensor workload data, surgical procedure plan data, and / or surgical situation recognition data.

[0304] The device of Example 20 enables better coordination of data processing during surgical procedures. Data processing behavior can be modified during the processing of incoming sensor data based on data processing modification commands. For example, data processing may be modified based on sensor workload data, surgical procedure planning data, or surgical situation awareness data. Modifications to processing may be motivated by changes in the data processing needs of the system and medical professionals in surgery, or by changes in data processing needs associated with the surgical procedure itself. For example, by modifying the data processing of sensor data, the device can stop or minimize data processing on its own to free up processing capacity for other needs, and pass raw, unprocessed data to another device for processing. Processing behavior may be shifted to other devices or system components where more capacity is available. These measures improve overall system utilization, data processing speed, and data acquisition speed. Furthermore, freeing up the processing capacity of the device itself allows for more processing capacity, for example, as critical in the surgical approach or in the event of a medical emergency. Overall, such processing adjustments improve efficiency, data reliability, fault and failure handling, system flexibility, and overall performance, leading to improved patient safety and improved surgical procedures.

[0305] Example 21. A system for applying a first processing operation, a second processing operation, and a third processing operation to a surgical sensor data stream during a surgical procedure, the system comprising: a first surgical system component, which is configured to receive a surgical sensor data stream, apply a processing operation to a first portion of the surgical sensor data stream, receive a surgical data processing modification command, and not apply a processing operation to a second portion of the surgical sensor data stream based on the surgical data processing modification command; and a second surgical system component, which is configured to receive a surgical sensor data stream from the first surgical system component, not apply a processing operation to a first portion of the surgical sensor data stream, and apply a processing operation to a second portion of the surgical sensor data stream.

[0306] For example, in Embodiment 21, the second surgical system component may be configured not to apply a processing operation to the second portion of the surgical sensor data stream before the surgical data processing modification command is received by the first surgical system, but to apply a processing operation to the second portion of the surgical sensor data stream after the surgical data processing modification command is received by the first surgical system.

[0307] For example, in Embodiment 21, the surgical data processing modification command may include an instruction to a first surgical system component to prevent the processing operation from being applied, and / or an instruction to a second surgical system component to apply the processing operation.

[0308] For example, in Example 21, the first and / or second processing operations may include filtering, averaging, verification, sorting, aggregation, smoothing, and / or classification processes. The first and / or second processing operations may also include other forms of processing operations.

[0309] For example, in Embodiment 21, the first portion of the incoming sensor data may include or consist of sensor values ​​processed by the first and / or second surgical system components before the surgical data processing modification command is received by the first surgical system component, and the second portion of the incoming sensor data may include or consist of sensor values ​​processed by the first and / or second system after the surgical data processing modification command is received by the first surgical system component.

[0310] For example, in Example 21, the first surgical system component may receive surgical data processing modification commands from the second surgical system component or a downstream system. For example, the second surgical system component may generate surgical data processing modifications.

[0311] For example, in Example 21, the surgical data processing modification command may be based on sensor workload data, surgical procedure plan data, and / or surgical situation recognition data.

[0312] The system of Example 21 enables better coordination of data processing during surgical procedures. Data processing behavior can be modified during the processing of incoming sensor data based on data processing modification commands. For example, data processing may be modified based on sensor workload data, surgical procedure planning data, or surgical situation awareness data. Modifications to processing may be motivated by changes in the data processing needs of the system and medical professionals in surgery, or by changes in data processing needs associated with the surgical procedure itself. For example, by modifying the data processing of sensor data, a device can stop or minimize data processing on its own to free up processing capacity for other needs, and pass raw, unprocessed data to another device for processing. Processing behavior may be shifted to other devices or system components where more capacity is available. These measures improve overall system utilization, data processing speed, and data acquisition speed. Furthermore, freeing up the processing capacity of the device itself allows for more processing capacity, for example, as critical in the surgical approach or in the event of a medical emergency. Overall, such processing adjustments improve efficiency, data reliability, fault and failure handling, system flexibility, and overall performance, leading to improved patient safety and improved surgical procedures.

[0313] Example 22. The system according to Example 20 or Example 21, wherein the surgical data processing modification command is triggered based on load balancing operations between a first surgical system component and a second surgical system component based on changes in the surgical data processing requirements of a surgical procedure.

[0314] Example 23. The surgical data processing modification command is received from the surgical hub by the system described in any one of Examples 20 to 22.

[0315] The system in Example 23 is controlled by a surgical hub that can coordinate data processing across a set of interconnected devices, such as devices in an operating room.

[0316] Example 24. The surgical data processing modification command is triggered based on changes in the surgical data processing requirements of the surgical procedure, as described in Examples 20-23.

[0317] The system of Example 24 can balance data processing between the first and second surgical system components based on the real-time requirements of the surgical procedure. This allows the device to optimize data processing and available processing resources for the procedure being performed and to adapt to changes that may occur during the procedure, such as an approaching critical step or medical emergency. This leads to improved surgical procedures and patient safety.

[0318] Example 25. The system according to any one of Examples 20 to 24, further comprising a sensor unit configured to generate an incoming surgical sensor data stream.

[0319] For example, in Example 25, the sensor unit may be a patient monitoring system, a surgeon monitoring system, an environmental monitoring system, and / or a surgical instrument monitoring system.

[0320] The system of Example 25 can generate incoming sensor data based on conditions that are changing in real time during surgery, and the system can adapt processing conditions in real time based on unpredictable or unexpected events.

[0321] Example 26. The system according to any one of Examples 20 to 25, further comprising an input configured to receive an incoming surgical sensor data stream from an external device.

[0322] For example, in Example 26, the external device may be a sensor or a sensor system.

[0323] For example, in any one of Examples 20 to 26, the surgical sensor data stream may include measurements of one or more biomarkers, measurements of patient-specific parameters, and / or measurements of one or more environmental parameters.

[0324] The system of Example 26 can generate incoming sensor data based on conditions that are changing in real time during surgery, and the system can adapt processing conditions in real time based on unpredictable or unexpected events.

[0325] Example 27. The surgical data processing modification command is triggered by exceeding the surgical data processing usage threshold, in the system described in any one of Examples 20 to 26.

[0326] In the system of Example 27, the surgical data processing modification command is triggered as a result of exceeding the data processing utilization threshold. This allows for minimizing, stopping, or moving data processing to another device when data processing capacity is too low, which carries the risk of processing delays that could affect ongoing surgical procedures or patient safety.

[0327] Example 28. The system according to any one of Examples 20 to 27, in which a surgical data processing correction command is triggered by the surgical importance of incoming sensor data exceeding a threshold.

[0328] In the system of Example 28, a surgical data processing modification command is triggered as a result of data indicating surgical importance. This allows data processing to be adjusted based on the importance of the data to the surgical procedure. For example, if data processing is important to the surgical procedure or labeled as important to the surgery, data processing by the second device can be increased so that data processing is prioritized. Alternatively, processing in the second device may be stopped or minimized, and the sensor data may be sent to a connected device for, for example, higher quality or faster processing.

[0329] [Implementation Method] (1) A device for processing surgical data during a surgical procedure, Memory and A processor and a processor, the processor Retrieving the first surgical data processing schema from the aforementioned memory, To output to the sensor data channel, a first processing is performed on a first portion of the incoming sensor data according to the first surgical data processing schema, Receiving surgical data processing modification commands via the sensor control channel, The process involves saving a second surgical data processing schema to the memory in accordance with the surgical data processing modification command, wherein the second surgical data processing schema is different from the first surgical data processing schema. For output to the sensor data channel, a second processing is performed on a second portion of the incoming sensor data according to the second surgical data processing schema, wherein the second processing is different from the first processing. A device configured to perform the following actions. (2) The surgical data processing modification command is received from the surgical hub by the device described in Embodiment 1. (3) The device according to Embodiment 1 or 2, wherein the surgical data processing modification command is triggered based on a change in the surgical data processing requirements of the surgical procedure. (4) The device according to any one of embodiments 1 to 3, wherein the first process has an output frequency different from that of the second process. (5) The device according to any one of embodiments 1 to 4, wherein the first process has a different output resolution from the second process.

[0330] (6) The device according to any one of embodiments 1 to 5, wherein the first process differs from the second process in terms of the use of processing resources. (7) The device according to any of embodiments 1 to 6, wherein the first process differs from the second process with respect to data conversion operations. (8) The device according to any one of embodiments 1 to 7, further comprising a sensor unit configured to generate the incoming sensor data. (9) The device according to any one of embodiments 1 to 8, further comprising an input configured to receive the incoming sensor data from an external device. (10) The device according to any one of embodiments 1 to 9, wherein each of the first input / output conversions of either the first process or the second process is pass-through, and each of the other second input / output conversions of either the first process or the second process includes an input / output conversion other than pass-through, and optionally, the input / output conversion other than pass-through performs either atomic processing, stream processing or composite processing.

[0331] (11) A method for processing surgical data during a surgical procedure in a system, the method being In the second device of the system, in order to output to the sensor data channel, a first processing is performed on a first portion of the incoming sensor data according to a first surgical data processing schema, The first device of the aforementioned system transmits a surgical data processing modification command. In the second device of the system, Receiving the surgical data processing modification command via the sensor control channel, For output to the sensor data channel, a second processing is performed on a second portion of the incoming sensor data according to a second surgical data processing schema, wherein the second surgical data processing schema differs from the first surgical data processing schema based on the surgical data processing modification command. Methods that include... (12) A system for processing surgical data during a surgical procedure, wherein the system is A first device configured to send surgical data processing modification commands, A second device comprises, and the said second device is To output to the sensor data channel, perform a first processing on a first portion of the incoming sensor data according to a first surgical data processing schema, Receiving the surgical data processing modification command via the sensor control channel, For output to the sensor data channel, a second processing is performed on a second portion of the incoming sensor data according to a second surgical data processing schema, wherein the second surgical data processing schema differs from the first surgical data processing schema based on the surgical data processing modification command. A system configured to perform the following actions. (13) The surgical data processing modification command is triggered based on a change in the surgical data processing requirements of the surgical procedure, according to the method of Embodiment 11 or the system according to Embodiment 12. (14) The surgical data processing modification command is triggered by exceeding the surgical data processing usage threshold, using the method described in Embodiment 11 or 13, the device described in any of Embodiments 1 to 10, or the system described in Embodiment 12 or 13. (15) The surgical data processing correction command is triggered based on the surgical importance of the incoming sensor data that exceeds a threshold, according to the method of any one embodiment 11, 13, or 14, the device according to any one embodiment 1 to 10, or 14, or the system according to any one embodiment 12 to 14.

[0332] (16) The first process is the method according to any one of Embodiments 11 or 13 to 15, the device according to any one of Embodiments 1 to 10, 14, or 15, or the system according to any one of Embodiments 12 to 15, wherein the first process has an output frequency different from that of the second process. (17) The first process is the method according to any one of Embodiments 11 or 13 to 16, the device according to any one of Embodiments 1 to 10, 14, 15 or 16, or the system according to any one of Embodiments 12 to 16, wherein the first process has an output resolution different from that of the second process. (18) The first process is the method described in any of Embodiments 11 or 13 to 17, the device described in any of Embodiments 1 to 10, 14, 15, 16, or 17, or the system described in any of Embodiments 12 to 17, wherein the first process differs from the second process in terms of the use of processing resources. (19) The first process is the method according to any one of Embodiments 11 or 13 to 18, the device according to any one of Embodiments 1 to 10, 14, 15, 16, or 17, or the system according to any one of Embodiments 12 to 18, wherein the first process differs from the second process with respect to data conversion operations. (20) A system for applying a first processing operation, a second processing operation, and a third processing operation to a surgical sensor data stream during a surgical procedure, wherein the system is A first surgical system component, wherein the first surgical system component is Receive surgical sensor data stream, A first operation and a second processing operation are applied to the first portion of the surgical sensor data stream. A first surgical system component configured to apply the first processing operation, rather than the second processing operation, to the second portion of the surgical sensor data stream based on the receipt of a surgical data processing modification command, A second surgical system component, wherein the second surgical system component is The surgical sensor data stream is received from the first surgical system component. The third operation is applied to the first portion of the surgical sensor data stream, rather than the second processing operation. A second surgical system component configured to apply the third processing operation and the second processing operation to the second portion of the surgical sensor data stream, A system equipped with these features.

[0333] (21) A system for applying a first processing operation, a second processing operation, and a third processing operation to a surgical sensor data stream during a surgical procedure, wherein the system is A first surgical system component, wherein the first surgical system component is Receive surgical sensor data stream, A processing operation is applied to the first portion of the surgical sensor data stream. Upon receiving a surgical data processing correction command, A first surgical system component configured not to apply the processing operation to a second portion of the surgical sensor data stream based on the surgical data processing modification command, A second surgical system component, wherein the second surgical system component is The surgical sensor data stream is received from the first surgical system component. The processing operation is not applied to the first portion of the surgical sensor data stream. A second surgical system component configured to apply the processing operation to the second portion of the surgical sensor data stream, A system equipped with these features. (22) The system according to embodiment 20 or 21, wherein the surgical data processing modification command is triggered based on load balancing operations between the first surgical system component and the second surgical system component based on changes in the surgical data processing requirements of the surgical procedure. (23) The surgical data processing modification command is received from a surgical hub by the system according to any one of embodiments 20 to 22. (24) The system according to any one of embodiments 20 to 23, wherein the surgical data processing modification command is triggered based on a change in the surgical data processing requirements of the surgical procedure. (25) The system according to any one of embodiments 20 to 24, further comprising a sensor unit configured to generate the incoming surgical sensor data stream.

[0334] (26) The system according to any one of embodiments 20 to 25, further comprising an input configured to receive the incoming surgical sensor data stream from an external device. (27) The system according to any one of embodiments 20 to 26, wherein the surgical data processing modification command is triggered when the surgical data processing usage threshold is exceeded. (28) The system according to any one of embodiments 20 to 27, wherein the surgical data processing correction command is triggered by the surgical importance of the incoming sensor data exceeding a threshold.

Claims

1. A device for processing surgical data during surgical procedures, Memory and A processor and a processor, the processor From the memory, retrieve a first surgical data processing schema containing information defining the operation of the device and corresponding data structures. To output to the sensor data channel, a first processing is performed on a first portion of the incoming sensor data according to the first surgical data processing schema, Receiving surgical data processing modification commands via a sensor control channel, which include information for updating or modifying the first surgical data processing schema and bringing about a second surgical data processing schema, based on surgical procedure planning data including information about the expected sensor demands for individual embodiments of the surgery, or surgical situation recognition data including indications of surgical instrument identifiers and indications that a surgical instrument has been activated, and which include information for updating or modifying the first surgical data processing schema and bringing about a second surgical data processing schema. When the aforementioned surgical data processing modification command is received, The process involves saving the second surgical data processing schema to the memory in accordance with the surgical data processing modification command, wherein the second surgical data processing schema is different from the first surgical data processing schema. The first processing of the incoming sensor data is stopped, and a second processing is performed on a second portion of the incoming sensor data according to the second surgical data processing schema for output to the sensor data channel, wherein the second processing is different from the first processing. A device configured to perform the following actions.

2. The device according to claim 1, wherein the surgical data processing modification command is received from a surgical hub.

3. The device according to claim 1 or 2, wherein the surgical data processing modification command is triggered based on a change in the surgical data processing requirements of the surgical procedure.

4. The device according to any one of claims 1 to 3, wherein the first process has an output frequency different from that of the second process.

5. The device according to any one of claims 1 to 4, wherein the first process has a different output resolution from the second process.

6. The device according to any one of claims 1 to 5, wherein the first process differs from the second process in terms of the use of processing resources.

7. The device according to any one of claims 1 to 6, wherein the first process differs from the second process with respect to data conversion operations.

8. The device according to any one of claims 1 to 7, further comprising a sensor unit configured to generate the incoming sensor data.

9. The device according to any one of claims 1 to 8, further comprising an input configured to receive the incoming sensor data from an external device.

10. The device according to any one of claims 1 to 9, wherein each of the first input / output conversions of either the first process or the second process is pass-through, and each of the other second input / output conversions of either the first process or the second process includes an input / output conversion other than pass-through, and optionally, the input / output conversion other than pass-through performs atomic processing, stream processing, or composite processing.

11. A method for processing surgical data during a surgical procedure in a system, wherein the method is In the second device of the system, a first processing is performed on a first portion of incoming sensor data in accordance with a first surgical data processing schema that includes information defining the operation of the device and a corresponding data structure, for output to a sensor data channel. The first device of the system transmits a surgical data processing modification command, which includes information for updating or modifying the first surgical data processing schema and bringing about a second surgical data processing schema, based on surgical procedure planning data including information about sensor demands expected in individual embodiments of surgery, or surgical situation recognition data including indications of surgical instrument identifiers and indications that a surgical instrument has been activated. In the second device of the system, Receiving the surgical data processing modification command via the sensor control channel, The first processing of the incoming sensor data is stopped, and a second processing is performed on a second portion of the incoming sensor data according to the second surgical data processing schema for output to the sensor data channel, wherein the second surgical data processing schema is different from the first surgical data processing schema based on the surgical data processing modification command. Methods that include...

12. A system for processing surgical data during surgical procedures, wherein the system is A first device configured to send a surgical data processing modification command that includes information for updating or modifying a first surgical data processing schema and bringing about a second surgical data processing schema, based on surgical procedure planning data including information about sensor demands expected in individual embodiments of surgery, or surgical situation recognition data including indications of surgical instrument identifiers and indications that a surgical instrument has been activated, A second device is provided, and the second device is To output to the sensor data channel, first processing is performed on a first portion of the incoming sensor data according to the first surgical data processing schema, which includes information defining the operation of the device and corresponding data structures. Receiving the surgical data processing modification command via the sensor control channel, The first processing of the incoming sensor data is stopped, and a second processing is performed on a second portion of the incoming sensor data according to the second surgical data processing schema for output to the sensor data channel, wherein the second surgical data processing schema is different from the first surgical data processing schema based on the surgical data processing modification command. A system configured to perform the following actions.

13. The system according to claim 12, wherein the surgical data processing modification command is triggered based on a change in the surgical data processing requirements of the surgical procedure.

14. The system according to claim 12 or 13, wherein the surgical data processing modification command is triggered when the surgical data processing usage threshold is exceeded.

15. The system according to any one of claims 12 to 14, wherein the surgical data processing modification command is triggered based on the surgical importance of the incoming sensor data that exceeds a threshold.

16. The system according to any one of claims 12 to 15, wherein the first process has an output frequency different from that of the second process.

17. The system according to any one of claims 12 to 16, wherein the first process has a different output resolution than the second process.

18. The system according to any one of claims 12 to 17, wherein the first process differs from the second process in terms of the use of processing resources.

19. The system according to any one of claims 12 to 18, wherein the first process differs from the second process with respect to data conversion operations.

20. The method according to claim 11, wherein the surgical data processing modification command is triggered based on a change in the surgical data processing requirements of the surgical procedure.

21. The method according to claim 11 or 20, wherein the surgical data processing modification command is triggered when the surgical data processing utilization threshold is exceeded.

22. The method according to any one of claims 11, 20, or 21, wherein the surgical data processing modification command is triggered based on the surgical importance of the incoming sensor data that exceeds a threshold.

23. The method according to any one of claims 11 or 20 to 22, wherein the first process has a different output frequency from the second process.

24. The method according to any one of claims 11 or 20 to 23, wherein the first process has a different output resolution from the second process.

25. The method according to any one of claims 11 or 20 to 24, wherein the first process differs from the second process with respect to the use of processing resources.

26. The method according to any one of claims 11 or 20 to 25, wherein the first process differs from the second process with respect to data conversion operations.

27. ​​The device according to any one of claims 1 to 10, wherein the surgical data processing modification command is triggered when the surgical data processing utilization threshold is exceeded.

28. The device according to any one of claims 1 to 10 or 27, wherein the surgical data processing modification command is triggered based on the surgical importance of the incoming sensor data that exceeds a threshold.

Images