Active recognition and pairing sensing system

Abstract

The surgical computing system may scan for sensing systems located in the operating room. Upon detecting a sensing system in the operating room, the surgical computing system may establish a link with the sensing system. The surgical computing system may receive user role identification data from the sensing system using the established link. The surgical computing system may identify a user role of a user in the operating room based on the received user role identification data. The user role of the user may be or may include at least one of a surgeon, a nurse, a patient, a hospital staff, or a medical professional. Based on the identified user role, the surgical computing system may generate and transmit surgical assistance information for the user in the operating room. The surgical assistance information may include information associated with a surgery related to the identified user role.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application is related to the following applications filed simultaneously, the contents of each of which are incorporated herein by reference. · U.S. Patent Application No. 17 / 156287 (Attorney Docket No. END9290USNP1) entitled "METHOD OF ADJUSTING A SURGICAL PARAMETER BASED ON BIOMARKER MEASUREMENTS".

Background Art

[0002] Surgical procedures are typically performed in an operating room or room within a medical facility such as a hospital. Various surgical devices and / or sensing systems are utilized in the performance of surgical procedures.

Summary of the Invention

Problems to be Solved by the Invention

[0003] In the digital and information age, and in order to improve patient care, it is desirable to find ways to assist in recognizing and pairing various surgical devices, surgical systems, and / or sensing systems.

Means for Solving the Problems

[0004] A computing system may include a processor. The processor is configured to scan a sensing system located within an operating room, establish a link (i.e., pair) with the sensing system, receive user role - identification data from the sensing system using the established link, identify a user role within the operating room based on the received user role - identification data, and generate and optionally display surgical assistance information for a user within the operating room based on the identified user role.

[0005] Advantageously, the system can generate (e.g., unique) surgical assistance information that may be relevant to the identified user and / or identified user role, based on information acquired through the sensing system. Therefore, the burden on the user when attempting to generate such information is reduced.

[0006] The user may be a first user, the sensing system may be a first sensing system, and the processor may be further configured to receive user role identification data from a second sensing system associated with a second user, to identify the user role of the second user in the operating room based on the received user role identification data, and to determine surgical assistance information for the second user based on the identified user role of the second user. Advantageously, the surgical assistance information may be generated / determined for multiple users who may have different roles.

[0007] User role identification data may include at least one of the following: the user's proximity to one or more surgical instruments, the location of a first user in the operating room, interactions between the user and at least one medical professional, one or more surgical procedures, or visual data of the user in the operating room.

[0008] The sensing system may be worn by the user, and optionally, the processor is configured to identify the user role of a first user as a surgeon based on at least one of the following: the proximity of the sensing system to one or more surgical instruments, location tracking information associated with the sensing system during a surgical procedure, or one or more surgical activity detected by the sensing system.

[0009] The processor may be configured to generate augmented reality (AR) content for identified user roles, the AR content may include surgical assistance information, and to transmit the AR content to an AR device associated with the user. Advantageously, any surgical assistance information may be overlaid on another image or video, allowing the user to maintain focus on, for example, less display.

[0010] The processor may be configured to receive measurement data from a sensing system, determine the elevated stress level associated with the user based on the received measurement data, acquire surgical context data, identify surgical instruments associated with the user based on the surgical context data and identified user role, and acquire instructions on how to use the surgical instruments for inclusion in surgical assistance information. Advantageously, the user may be provided with instructions on how to use the surgical instruments when the user is experiencing a high level of stress. Thus, the user may be assisted during surgery with minimal user intervention.

[0011] Surgical assistance information for the user may include fatigue control instructions for surgical instruments, and the processor may be configured to receive measurement data from a sensing system, determine an elevated fatigue level associated with the user based on the received measurement data, acquire surgical context data, determine whether the user is operating a surgical instrument based on the surgical context data and identified user role, and, based on the determination that the user is operating a surgical instrument, transmit fatigue control instructions to the surgical instrument. Advantageously, the control program may also be transmitted to the surgical instrument to control its actuators to limit or compensate for fatigue and / or the use of fine motor skills.

[0012] User roles may include at least one of the following: surgeon, nurse, patient, hospital staff, or healthcare professional.

[0013] A computer implementation method may include scanning a sensing system located in the operating room, establishing a link (i.e., a pair) with the sensing system, receiving user role identification data from the sensing system using the established link, identifying a user role for the user in the operating room based on the received user role identification data, and generating and optionally displaying surgical assistance information for the user in the operating room based on the identified user role.

[0014] Advantageously, the system can generate (e.g., unique) surgical assistance information that may be relevant to the identified user and / or identified user role, based on information acquired through the sensing system. Therefore, the burden on the user when attempting to generate such information is reduced.

[0015] The user may be a first user, the sensing system may be a first sensing system, and the method may further include receiving user role identification data from a second sensing system associated with a second user, identifying the user role of the second user in the operating room based on the received user role identification data, and determining surgical assistance information for the second user based on the identified user role of the second user. Advantageously, the surgical assistance information may be generated / determined for multiple users who may have different roles.

[0016] The sensing system may be worn by the user, and optionally, the method includes identifying the user role of a first user as a surgeon based on at least one of the following: the proximity of the sensing system to one or more surgical instruments, location tracking information associated with the sensing system during a surgical procedure, or one or more surgical activity detected by the sensing system.

[0017] This method may include generating augmented reality (AR) content for an identified user role, the AR content may include surgical assistance information, and transmitting the AR content to an AR device associated with the user. Advantageously, any surgical assistance information may be overlaid on another image or video to allow the user to maintain focus on, for example, less display.

[0018] This method may include receiving measurement data from a sensing system, determining the elevated stress level associated with the user based on the received measurement data, acquiring surgical context data, identifying surgical instruments associated with the user based on the surgical context data and identified user roles, and acquiring instructions on how to use the surgical instruments to include in surgical assistance information. Advantageously, the user may be provided with instructions on how to use surgical instruments when the user is experiencing a high level of stress. Thus, the user may be assisted during surgery with minimal user intervention.

[0019] Surgical assistance information for the user may include fatigue control instructions for surgical instruments, and the method may include receiving measurement data from a sensing system, determining an elevated fatigue level associated with the user based on the received measurement data, acquiring surgical context data, determining whether the user is operating a surgical instrument based on the surgical context data and identified user role, and transmitting fatigue control instructions to the surgical instrument based on the determination that the user is operating a surgical instrument. Advantageously, a control program may be transmitted to the surgical instrument to control its actuators to limit or compensate for fatigue and / or the use of fine motor skills.

[0020] The computing system may include a processor. The processor may be configured to scan a sensing system in the operating room, the sensing system having measurement data for the user, determine whether the sensing system is compatible to establish a link with the computing system, generate a virtual computing system compatible to establish a link with the sensing system based on the determination that the sensing system is not compatible to establish a link with the computing system, establish a link with the sensing system using the generated virtual computing system, and receive measurement data using the link with the sensing system.

[0021] Advantageously, the computer system can receive measurement data from incompatible sensing systems.

[0022] The processor may be configured to establish an initial link with the sensing system and to send an initial link instruction to the surgical computing system before establishing a link with the sensing system, the initial link instruction requesting and sending user input to establish a link with the sensing system, and receiving user input from the surgical computing system and establishing a link with the sensing system based on the received user input. Advantageously, the user can control whether or not to establish the link.

[0023] The processor may be configured to establish a link with the sensing system based on the determination that the sensing system is compatible with establishing a link with the computing system.

[0024] The sensing system may include a first sensing system, the link may include a first link, the measurement data may include first measurement data, and the processor may be configured to establish a second link with a second sensing system that includes second measurement data of the user, and to receive the second measurement data from the second sensing system using the second established link. Optionally, the processor may determine whether to generate augmented reality (AR) content based on the received first measurement data and second measurement data, the positions of the first sensing system and the second sensing system in the operating room, or at least one of one or more surgical treatment activities of the surgical operation, and based on the determination, be configured to generate AR content including display information associated with the first measurement data and / or the second measurement data. Advantageously, the computing system can establish several links with several different sensing systems. Further, any information associated with the first measurement data and / or the second measurement data may be overlaid on another image or video, for example, enabling the user to maintain focus on a smaller display.

[0025] The processor may be configured to transmit the generated AR content to an AR device associated with the user.

[0026] The processor may be configured to detect a plurality of devices in the operating room, identify a sensing system in the operating room from the detected plurality of devices, and select the sensing system to establish a link. Advantageously, the computer system can scan the environment (e.g., the operating room) for potential devices to establish a link, identify the sensing system, and be used to establish a link with it.

[0027] The computer-readable medium may comprise instructions that, when executed by a computer, cause the computer to execute the methods described above.

[0028] A surgical computing system can scan and detect a sensing system located within an operating room (OR). Based on the detection, the surgical computing system can establish a link with the sensing system. Using the established link between the surgical computing system and the sensing system, the surgical computing system can receive user role identification data from the sensing system. The user role identification data may be information for identifying a user role or may include information for identifying a user role. The surgical computing system can identify the user role of a user in the OR based on the received user role identification data. The user role of a user in the OR may be at least one of, or may include, a surgeon, a nurse, a patient, hospital staff, or a healthcare professional (HCP). Based on the identified user role, the surgical computing system can generate surgical assistance information for the user in the OR. The surgical assistance information may be, or may include, information associated with a surgical procedure related to the identified user role. The surgical computing system described herein may be, or may include, a surgical hub.

[0029] A surgical computing system can receive different user role identification data from different sensing systems associated with multiple users in the OR. The computing system can identify different user roles and / or users based on the user role identification data received from various sensing systems and provide different surgical assistance information to the users based on their respective identified user roles.

[0030] For example, a surgical computing system may receive user role identification data from a first sensing system associated with a first user, and user role identification data from a second sensing system associated with a second user. The surgical computing system may identify the user roles of the first user and the second user. Based on the corresponding user roles, the surgical computing system may determine, generate, and / or transmit surgical assistance information to the user (e.g., the first user or the second user).

[0031] For example, user role identification data may be one or more of the following: the user's proximity to surgical instruments, the user's location within the operating room (OR) and / or location tracking information, interactions between the user and at least one medical professional, one or more surgical procedures, or the user's visual data within the OR. For example, the sensing system may be worn by a user such as a surgeon. The sensing system may monitor and / or store information about the sensing system's proximity to surgical instruments. The sensing system may store location tracking information of the surgeon during a surgical procedure. The sensing system may detect and / or store the surgeon's surgical procedures. The sensing system may transmit such user role identification data to a surgical computing system.

[0032] For example, a surgical computing system may generate augmented reality (AR) content for a user based on an identified user role. Different AR content may be generated for different users based on their respective user roles identified via a sensing system. The AR content may be, or include, instructions on how to use and / or operating instructions for surgical instruments associated with the identified user role. The surgical computing system may transmit the generated AR content to the identified user. For example, the surgical computing system may transmit the AR content to an AR device associated with the user.

[0033] A surgical computing system may retrieve surgical context data. Based on the surgical context data and identified user roles, the surgical computing system may identify surgical instruments associated with a user. The surgical computing system may retrieve instructions on how to use the surgical instruments for inclusion in surgical assistance information.

[0034] For example, a surgical computing system may receive measurement data from a sensing system. This measurement data may include stress levels and / or fatigue levels associated with the user. The surgical computing system may determine elevated stress levels and / or fatigue levels associated with the user. If the surgical computing system detects elevated stress levels associated with the user, it may generate and / or transmit surgical assistance information, including instructions on how to use surgical instruments, to the identified user. If the surgical computing system detects elevated fatigue levels associated with the user, it may transmit fatigue control instructions to the surgical instruments.

[0035] The computing system may scan for sensing systems located within the OR. The sensing systems may contain user measurement data. The computing system may determine whether the sensing systems are compatible to establish a link with the computing system. If the computing system determines that the sensing systems are compatible to establish a link with the computing system, it may establish the link and use the link with the sensing systems to receive measurement data.

[0036] If the sensing system determines that it is incompatible to establish a link with the computing system, the computing system may generate a virtual computing system that is compatible to establish a link with the sensing system. The computing system may then use the generated virtual computing system to establish a link with the sensing system. The computing system may then use the link with the sensing system to receive measurement data.

[0037] The computing system may establish an initial link with the sensing system before establishing a communication link with the sensing system. The computing system may send an initial link instruction to the surgical computing system (e.g., the primary computing system). The initial link instruction may request user input to establish a link with the sensing system. The computing system may receive user input from the surgical computing system. The computing system may then establish a link with the sensing system.

[0038] The computing system may decide to generate AR content based on at least one of the following: received measurement data, the location of sensing systems within the OR, or surgical procedure activities in a surgical procedure. The AR content may include display information associated with the measurement data. The computing system may transmit the generated AR content to an AR device associated with the user.

[0039] A computing system may detect one or more devices within an OR. For example, an OR may contain one or more surgeon sensing systems, patient sensing systems, computers, telephones, monitor screens, and / or other devices. From the detected devices within the OR, the computing system may identify one or more sensing systems within the OR (e.g., those paired with the computing system). The computing system may establish a link with the identified sensing system. [Brief explanation of the drawing]

[0040] [Figure 1A] This is a block diagram of a computer-implemented patient and surgeon monitoring system. [Figure 1B] This is a block diagram illustrating exemplary relationships between sensing systems, biomarkers, and physiological systems. [Figure 2A] This shows an example of a surgical monitoring system in the operating room. [Figure 2B] An example of a patient monitoring system (e.g., a controlled patient monitoring system) is shown. [Figure 2C] An example of a patient monitoring system (e.g., an uncontrolled patient monitoring system) is shown. [Figure 3] This shows an exemplary surgical hub paired with various systems. [Figure 4] This describes a surgical data network having a set of communication surgical hubs configured to connect to a set of sensing systems, an environmental sensing system, a set of devices, and so on. [Figure 5] This exhibits an exemplary computer-implemented interactive surgical system, which may be part of a surgical monitoring system. [Figure 6A] This image shows an exemplary surgical hub comprising multiple modules connected to a modular control tower. [Figure 6B] An example of a controlled patient monitoring system is shown. [Figure 6C] This is an example of an uncontrolled patient monitoring system. [Figure 7A] This is a logic diagram illustrating an exemplary control system for a surgical instrument or tool. [Figure 7B] An exemplary sensing system having a sensor unit and a data processing and communication unit is shown. [Figure 7C] An exemplary sensing system having a sensor unit and a data processing and communication unit is shown. [Figure 7D] An exemplary sensing system is shown, comprising a sensor unit and a data processing and communication unit. [Figure 8]This example illustrates a timeline of a surgical procedure demonstrating the adjustment of operating parameters of a surgical device based on the surgeon's biomarker level. [Figure 9] This is a block diagram of a computer-implemented bidirectional surgeon / patient monitoring system. [Figure 10] An exemplary surgical system is shown, including a handle having a controller and motor, an adapter releasably connected to the handle, and a loading unit releasably connected to the adapter. [Figure 11A] An example of a sensing system that may be used to monitor surgical biomarkers or patient biomarkers is shown. [Figure 11B] An example of a sensing system that may be used to monitor surgical biomarkers or patient biomarkers is shown. [Figure 11C] An example of a sensing system that may be used to monitor surgical biomarkers or patient biomarkers is shown. [Figure 11D] An example of a sensing system that may be used to monitor surgical biomarkers or patient biomarkers is shown. [Figure 12] This is a block diagram of a patient monitoring system or a surgeon monitoring system. [Figure 13] This shows an example flow for generating surgical support information for users in the operating room. [Figure 14] This illustrates an exemplary flow of a computing system establishing links with compatible and / or incompatible sensing and / or computing systems. [Figure 15] This illustrates an exemplary flow of a computing system operating both online and offline. [Figure 16] This example shows a secondary computing system migrating to a primary computing system to create a local computing system for low-level analysis. [Modes for carrying out the invention]

[0041] The applicant of this application also owns the following U.S. patent applications filed concurrently, each of which is incorporated herein by reference in its entirety. U.S. Patent Application No. 16 / 209,416, titled "METHOD OF HUB COMMUNICATION, PROCESSING, DISPLAY, AND CLOUD ANALYTICS," filed on December 4, 2018. U.S. Patent Application No. 15 / 940,671, titled "SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES IN OPERATING THEATER" (Agent Reference Number END8502USNP), filed on March 29, 2018. U.S. Patent Application No. 16 / 182,269, filed on November 6, 2018, entitled "IMAGE CAPTURING OF THE AREAS OUTSIDE THE ABDOMEN TO IMPROVE PLACEMENT AND CONTROL OF A SURGICAL DEVICE IN USE" (Agent Reference Number END9018USNP3), U.S. Patent Application No. 16 / 729,747, entitled "DYNAMIC SURGICAL VISUALIZATION SYSTEMS" (Agent Reference Number END9217USNP1), filed on December 31, 2019. U.S. Patent Application No. 16 / 729,778, filed on December 31, 2019, entitled "SYSTEM AND METHOD FOR DETERMINING, ADJUSTING, AND MANAGING RESECTION MARGIN ABOUT A SUBJECT TISSUE" (Agent Reference Number END9219USNP1), U.S. Patent Application No. 16 / 729,807, entitled "METHOD OF USING IMAGING DEVICES IN SURGERY" (Agent Reference Number END9228USNP1), filed on December 31, 2019. U.S. Patent Application No. 15 / 940,654, entitled "SURGICAL HUB SITUATIONAL AWARENESS," filed on March 29, 2018 (Agent Reference Number END8501USNP), U.S. Patent Application No. 15 / 940,671, titled "SURGICAL HUB SPATIAL AWARENESS TO DETERMINE DEVICES IN OPERATING THEATER" (Agent Reference Number END8502USNP), filed on March 29, 2018. U.S. Patent Application No. 15 / 940,704, filed on March 29, 2018, entitled "USE OF LASER LIGHT AND RED-GREEN-BLUE COLORATION TO DETERMINE PROPERTIES OF BACK SCATTERED LIGHT" (Agent Reference Number END8504USNP), U.S. Patent Application No. 16 / 182,290, filed on November 6, 2018, entitled "SURGICAL NETWORK RECOMMENDATIONS FROM REAL TIME ANALYSIS OF PROCEDURE VARIABLES AGAINST A BASELINE HIGHLIGHTING DIFFERENCES FROM THE OPTIMAL SOLUTION" (Agent Reference Number END9018USNP5), • U.S. Patent No. 9,011,427, titled "SURGICAL INSTRUMENT WITH SAFETY GLASSES," issued on April 21, 2015. U.S. Patent No. 9,123,155, issued on September 1, 2015, entitled "APPARATUS AND METHOD FOR USING AUGMENTED REALITY VISION SYSTEM IN SURGICAL PROCEDURES" • U.S. Patent Application No. 16 / 209,478, filed on December 4, 2018, entitled "METHOD FOR SITUATIONAL AWARENESS FOR SURGICAL NETWORK OR SURGICAL NETWORK CONNECTED DEVICE CAPABLE OF ADJUSTING FUNCTION BASED ON A SENSED SITUATION OR USAGE" (Agent Reference Number END9015USNP1), and, U.S. Patent Application No. 16 / 182,246, entitled "ADJUSTMENTS BASED ON AIRBORNE PARTICLE PROPERTIES" (Agent Reference Number END9016USNP1), filed on November 6, 2018.

[0042] Figure 1A is a block diagram of a computer-implemented patient and surgeon monitoring system 20000. The patient and surgeon monitoring system 20000 may include one or more surgeon monitoring systems 20002 and one or more patient monitoring systems (e.g., one or more controlled patient monitoring systems 20003 and one or more uncontrolled patient monitoring systems 20004). Each surgeon monitoring system 20002 may include a computer-implemented bidirectional surgical system. Each surgeon monitoring system 20002 may include at least one of a surgical hub 20006 that communicates with a cloud computing system 20008, for example, as shown in Figure 2A. Each patient monitoring system may include at least one of a surgical hub 20006 or a computing device 20016 that communicates with a cloud computing system 20008, for example, as further described in Figures 2B and 2C. The cloud computing system 20008 may include at least one remote cloud server 20009 and at least one remote cloud storage unit 20010. Each of the surgeon monitoring system 20002, the controlled patient monitoring system 20003, or the uncontrolled patient monitoring system 20004 may include a wearable sensing system 20011, an environmental sensing system 20015, a robotic system 20013, one or more intelligent instruments 20014, a human interface system 20012, etc. The human interface system is also referred to herein as a human interface device. The wearable sensing system 20011 may include one or more surgeon sensing systems and / or one or more patient sensing systems. The environmental sensing system 20015 may include one or more devices used to measure one or more environmental attributes, for example, as further illustrated in Figure 2A. The robotic system 20013 (same as 20034 in Figure 2A) may include multiple devices used to perform surgical procedures, for example, as further illustrated in Figure 2A.

[0043] The surgical hub 20006 can interact in cooperation with one of several means of displaying images from a laparoscope and information from one or more other smart devices and one or more sensing systems 20011. The surgical hub 20006 may interact with one or more sensing systems 20011, one or more smart devices, and multiple displays. The surgical hub 20006 may be configured to collect measurement data from one or more sensing systems 20011 and to send notification or control messages to one or more sensing systems 20011. The surgical hub 20006 may transmit and / or receive information, including notification information, to a human interface system 20012. The human interface system 20012 may include one or more human interface devices (HIDs). The surgical hub 20006 may transmit and / or receive notification or control information, including voice, display, and / or control information, to and from various devices communicating with the surgical hub.

[0044] Figure 1B is a block diagram of an exemplary relationship between a sensing system 20001, a biomarker 20005, and a physiological system 20007. This relationship may be used in a computer-implemented patient and surgeon monitoring system 20000, as well as in the systems, devices, and methods disclosed herein. For example, the sensing system 20001 may include a wearable sensing system 20011 (which may include one or more surgeon sensing systems and one or more patient sensing systems) and an environmental sensing system 20015, as discussed in Figure 1A. One or more sensing systems 20001 may measure data relating to various biomarkers 20005. One or more sensing systems 20001 may measure the biomarkers 20005 using one or more sensors, such as photosensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc. One or more sensors may measure the biomarker 20005 as described herein, using one or more sensing techniques, such as photoplethysmography, electrocardiography, electroencephalography, colorimetric analysis, impedance measurement, potentiometric measurement, and current measurement.

[0045] Biomarkers 20005 measured by one or more sensing systems 20001 may include, but are not limited to, sleep, core body temperature, maximal oxygen consumption, physical activity, alcohol intake, respiratory rate, oxygen saturation, blood pressure, blood glucose levels, heart rate variability, hydrogen blood potential, hydration status, heart rate, skin conductance, peripheral temperature, tissue perfusion pressure, cough and sneeze, gastrointestinal motility, gastrointestinal imaging, respiratory bacteria, edema, mental state, sweat, circulating tumor cells, autonomic nervous system tone, circadian rhythm, and / or menstrual cycle.

[0046] Biomarkers 20005 may relate to physiological systems 20007, including but not limited to behavioral and psychological systems, cardiovascular systems, renal systems, cutaneous systems, nervous systems, gastrointestinal systems, respiratory systems, endocrine systems, immune systems, tumors, musculoskeletal systems, and / or reproductive systems. Information from biomarkers may be determined and / or used, for example, by a computer-implemented patient and surgeon monitoring system 20000. Information from biomarkers may be determined and / or used by the computer-implemented patient and surgeon monitoring system 20000 to, for example, improve the system and / or improve patient outcomes.

[0047] Figure 2A shows an example of a surgical monitoring system 20002 in an operating room. As shown in Figure 2A, the patient is being operated on by one or more healthcare professionals (HCPs). The HCPs are monitored by one or more surgical sensing systems 20020 worn by the HCPs. The HCPs and the environment surrounding them may also be monitored by one or more environmental sensing systems, including, for example, a set of cameras 20021, a set of microphones 20022, and other sensors that may be deployed in the operating room. The surgical sensing systems 20020 and the environmental sensing systems may communicate with a surgical hub 20006, which may communicate with one or more cloud servers 20009 of a cloud computing system 20008, as shown in Figure 1. The environmental sensing systems may be used to measure one or more environmental attributes, such as the HCP's position in the operating room, HCP movement, ambient noise in the operating room, and temperature / humidity in the operating room.

[0048] As shown in Figure 2A, the primary display 20023 and one or more audio output devices (e.g., speakers 20019) are positioned in the sterile field so that they are visible to the operator at the operating table 20024. In addition, a visualization / notification tower 20026 is positioned outside the sterile field. The visualization / notification tower 20026 may include a first non-sterile human-interactive device (HID) 20027 and a second non-sterile HID 20029, which may face opposite each other. The HIDs may be displays or displays having touchscreens that allow humans to interface directly with the HIDs. A human interface system guided by the surgical hub 20006 may be configured to utilize the HIDs 20027, 20029, and 20023 to coordinate the flow of information to operators inside and outside the sterile field. In one example, the surgical hub 20006 may cause the HID (e.g., primary HID 20023) to display notifications and / or information regarding the patient and / or surgical procedure steps. In one example, the surgical hub 20006 may prompt and / or receive input from personnel in a sterile or non-sterile area. In one example, the surgical hub 20006 may display a snapshot of the surgical site recorded by the imaging device 20030 on a non-sterile HID 20027 or 20029 while maintaining live video of the surgical site on the primary HID 20023. The snapshot on the non-sterile display 20027 or 20029 may, for example, allow a non-sterile worker to perform diagnostic steps related to the surgical procedure.

[0049] In one embodiment, the surgical hub 20006 may be configured to route diagnostic input or feedback entered by a non-sterile operator at the visualization tower 20026 to a primary display 20023 in the sterile field, where it can be viewed by a sterilizer at the operating table. In one example, the input may take the form of modifications to a snapshot displayed on a non-sterile display 20027 or 20029, which can be routed by the surgical hub 20006 to the primary display 20023.

[0050] Referring to Figure 2A, surgical instrument 20031 is used in a surgical procedure as part of a surgeon monitoring system 20002. Hub 20006 can also be configured to coordinate the flow of information to the display of surgical instrument 20031. For example, in U.S. Patent Application Publication 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety. Diagnostic input or feedback entered by a non-sterile worker in the visualization tower 20026 can be routed by Hub 20006 to a surgical instrument display in the sterile field, which can then be viewed by the operator of surgical instrument 20031. Exemplary surgical instruments suitable for use with surgical system 20002 are described under the heading "Surgical Instrument Hardware" and in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed on 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety, for example.

[0051] Figure 2A shows an example of a surgical system 20002 used to perform a surgical procedure on a patient lying on an operating table 20024 in an operating room 20035. A robotic system 20034 may be used as part of the surgical system 20002 in a surgical procedure. The robotic system 20034 may include a surgeon's console 20036, a patient-side cart 20032 (surgical robot), and a surgical robot hub 20033. The patient-side cart 20032 can manipulate at least one detachably connected surgical tool 20037 through a minimally invasive incision in the patient's body while the surgeon views the surgical site through the surgeon's console 20036. Images of the surgical site are acquired by a medical imaging device 20030, which can be manipulated by the patient-side cart 20032 to orient the imaging device 20030. The robot hub 20033 can be used to process images of the surgical site, which can then be displayed to the surgeon via the surgeon's console 20036.

[0052] Other types of robotic systems can be readily adapted for use with surgical system 20002. Various examples of robotic systems and surgical tools suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0201137(A1) (U.S. Patent Application No. 16 / 209,407), filed 4 December 2018, entitled "METHOD OF ROBOTIC HUB COMMUNICATION, DETECTION, AND CONTROL," the disclosure of which is incorporated herein by reference in its entirety.

[0053] Various examples of cloud-based analytics performed by Cloud Computing System 20008 and suitable for use with this disclosure are described in U.S. Patent Application Publication No. 2019-0206569(A1) (U.S. Patent Application No. 16 / 209,403), filed on 4 December 2018, entitled "METHOD OF CLOUD BASED DATA ANALYTICS FOR USE WITH THE HUB," the disclosure of which is incorporated herein by reference in its entirety.

[0054] In various embodiments, the imaging device 20030 may include at least one image sensor and one or more optical components. Suitable image sensors include, but are not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors.

[0055] The optical components of the imaging device 20030 may include one or more illumination sources and / or one or more lenses. One or more illumination sources may be directed to illuminate a portion of the surgical field. One or more image sensors may receive light reflected or refracted from the surgical field, including light reflected or refracted from tissue and / or surgical instruments.

[0056] One or more illumination sources may be configured to emit electromagnetic energy in the visible and invisible spectra. The visible spectrum, sometimes also called the light spectrum or emission spectrum, is the portion of the electromagnetic spectrum that is visible to the human eye (i.e., detectable by the human eye), and is sometimes called visible light or simply light. The typical human eye responds to wavelengths in the air in the range of approximately 380 nm to 750 nm.

[0057] The invisible spectrum (e.g., the non-emission spectrum) is a portion of the electromagnetic spectrum located below and above the visible spectrum (i.e., wavelengths below approximately 380 nm and above approximately 750 nm). The invisible spectrum is undetectable to the human eye. Wavelengths above approximately 750 nm are longer than the red visible spectrum and consist of invisible infrared (IR), microwaves, and radio electromagnetic radiation. Wavelengths below approximately 380 nm are shorter than the violet spectrum and consist of invisible ultraviolet, X-rays, and gamma-ray electromagnetic radiation.

[0058] In various embodiments, the imaging device 20030 is configured for use in minimally invasive procedures. Examples of imaging devices suitable for use with this disclosure include, but are not limited to, arthroscopes, angioscopes, bronchoscopes, cholangioscopies, colonoscopes, cytoscopes, duodenoscopes, enteroscopes, gastroscopy (gastroscopy), endoscopes, laryngoscopes, nasopharyngo-neproscopes, sigmoidoscopy, thoracoscopy, and ureteroscopes.

[0059] The imaging device may employ multispectral monitoring to distinguish between topography and underlying structures. Multispectral imaging captures image data within a specific wavelength range from the entire electromagnetic spectrum. Wavelengths can be separated by filters or by using instruments sensitive to specific wavelengths, including frequencies beyond the visible light range, such as IR and ultraviolet light. Spectral imaging can enable the extraction of additional information that cannot be captured by the red, green, and blue receptors of the human eye. The use of multispectral imaging is described in detail under the heading "Advanced Imaging Acquisition Module" in U.S. Patent Application Publication No. 2019-0200844(A1) (U.S. Patent Application No. 16 / 209,385), filed 4 December 2018, entitled "METHOD OF HUB COMMUNICATION, PROCESSING, STORAGE AND DISPLAY," the disclosure of which is incorporated herein by reference in its entirety. Multispectral monitoring can be a useful tool for repositioning the surgical field after the surgical task is completed to perform one or more of the tests described above on the treated tissue. It is self-evident that strict sterilization of the operating room and surgical instruments is necessary in any surgical procedure. The strict sanitary and sterilization conditions required in the “operating room,” i.e., the operating room or treatment room, require the highest possible sterility of all medical devices and instruments. Part of the sterilization process described above includes the need to sterilize everything that comes into contact with the patient or enters the sterile field, including imaging devices and their accessories and components. It will be understood that the sterile field may be considered a specific area that is deemed free of microorganisms, such as within a tray or on a sterile towel, or it may be considered the area immediately surrounding a patient ready for surgical treatment. The sterile field may include cleaned team members in appropriate clothing, as well as all equipment and restraints within that area.

[0060] The wearable sensing system 20011 shown in Figure 1 may include one or more sensing systems, for example, a surgical sensing system 20020 as shown in Figure 2A. The surgical sensing system 20020 may include sensing systems for monitoring and detecting a set of physical and / or physiological conditions of a healthcare provider (HCP). An HCP is typically a surgeon or one or more healthcare professionals assisting a surgeon or other healthcare provider. In one example, the sensing system 20020 may measure a set of biomarkers to monitor the HCP's heart rate. In another example, the sensing system 20020 worn on the surgeon's wrist (e.g., a watch or wristband) may use an accelerometer to detect hand movements and / or tremors and determine the magnitude and frequency of the tremors. The sensing system 20020 may transmit the measurement data associated with the set of biomarkers and the data associated with the surgeon's physical condition to the surgical hub 20006 for further processing. One or more environmental sensing devices may transmit environmental information to the surgical hub 20006. For example, the environmental sensing device may include a camera 20021 for detecting the position of the HCP's hand / body. The environmental sensing device may include a microphone 20022 for measuring ambient noise in the operating room. Other environmental sensing devices may include, for example, devices such as a thermometer for measuring temperature and a hygrometer for measuring ambient humidity in the operating room. The surgical hub 20006 may, either independently or in communication with a cloud computing system, use the surgeon biomarker measurement data and / or environmental sensing information to correct the mean delay of a handheld instrument control algorithm or robotic interface, for example, to minimize tremor. In one example, the surgeon sensing system 20020 may measure one or more surgeon biomarkers associated with the HCP and transmit the measurement data associated with the surgeon biomarkers to the surgical hub 20006.The surgical sensing system 20020 may use one or more of the following RF protocols to communicate with the surgical hub 20006: Bluetooth®, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low Power Wireless Personal Area Network (6LoWPAN), and Wi-Fi. Surgical biomarkers may include one or more of the following: stress, heart rate, etc. Environmental measurements from the operating room may include ambient noise levels related to the surgeon or patient, surgeon and / or staff movements, surgeon and / or staff attention levels, etc.

[0061] The surgical hub 20006 can adaptively control one or more surgical instruments 20031 using surgical biomarker measurement data associated with the HCP. For example, the surgical hub 20006 may transmit a control program to the surgical instrument 20031 to control its actuators to limit or compensate for fatigue and the use of fine motor skills. The surgical hub 20006 may transmit a control program based on context regarding situational awareness and / or the importance or severity of the task. The control program may instruct the instrument to modify its behavior to provide more control when control is needed.

[0062] Figure 2B shows an example of a patient monitoring system 20003 (e.g., a controlled patient monitoring system). As shown in Figure 2B, a patient in a controlled environment (e.g., a hospital recovery room) may be monitored by multiple sensing systems (e.g., patient sensing system 20041). Patient sensing system 20041 (e.g., a headband) may be used to measure electroencephalography (EEG) to measure the electrical activity of the patient's brain. Patient sensing system 20042 may be used to measure various biomarkers of the patient, including, for example, heart rate, VO2 levels, etc. Patient sensing system 20043 (e.g., a flexible patch attached to the patient's skin) may be used to measure sweat lactate and / or potassium levels by analyzing small amounts of sweat taken in from the surface of the skin using microfluidic channels. Patient sensing system 20044 (e.g., a wristband or watch) may be used to measure blood pressure, heart rate, heart rate variability, VO2 levels, etc., using various techniques as described herein. Patient sensing systems 20045 (e.g., a ring worn on the finger) may be used to measure peripheral temperature, heart rate, heart rate variability, VO2 levels, etc., using various techniques as described herein. Patient sensing systems 20041-20045 may communicate with the surgical hub 20006 using a radio frequency (RF) link. Patient sensing systems 20041-20045 may use one or more RF protocols such as Bluetooth, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low-Power Wireless Personal Area Network (6LoWPAN), Thread, Wi-Fi, etc., for communication with the surgical hub 20006.

[0063] Sensing systems 20041-20045 may communicate with surgical hub 20006, which may communicate with remote server 20009 of remote cloud computing system 20008. Surgical hub 20006 also communicates with HID 20046. HID 20046 may display measurement data associated with one or more patient biomarkers. For example, HID 20046 may display blood pressure, oxygen saturation, respiratory rate, etc. HID 20046 may display patient or HCP notifications that provide information about the patient, such as information about recovery milestones or complications. In one example, information about recovery milestones or complications may be associated with surgical procedures the patient may have undergone. In one example, HID 20046 may display commands for the patient to perform activities. For example, HID 20046 may display inhalation and exhalation commands. In one example, HID 20046 may be part of the sensing system.

[0064] As shown in Figure 2B, the patient and the environment surrounding the patient may be monitored by one or more environmental sensing systems 20015, which may include, for example, microphones (e.g., for detecting ambient noise associated with or around the patient), temperature / humidity sensors, and cameras for detecting the patient's breathing patterns. The environmental sensing systems 20015 may communicate with a surgical hub 20006, which then communicates with a remote server 20009 of a remote cloud computing system 20008.

[0065] In one example, the patient sensing system 20044 may receive notification information from the surgical hub 20006 for display on the patient sensing system 20044's display unit or HID. The notification information may include notifications regarding recovery milestones or, for example, notifications regarding complications in the case of postoperative recovery. In one example, the notification information may include the manageable severity level associated with the notification. The patient sensing system 20044 may display the notification and the manageable severity level to the patient. The patient sensing system may alert the patient using haptic feedback. Visual and / or haptic notifications may be accompanied by audible notifications prompting the patient to pay attention to the visual notifications provided on the sensing system's display unit.

[0066] Figure 2C shows an example of a patient monitoring system (e.g., an uncontrolled patient monitoring system 20004). As shown in Figure 2C, a patient in an uncontrolled environment (e.g., the patient's residence) is monitored by multiple patient sensing systems 20041–20045. Patient sensing systems 20041–20045 may measure and / or monitor measurement data associated with one or more patient biomarkers. For example, patient sensing system 20041, a headband, may be used to measure electroencephalography (EEG). Other patient sensing systems 20042, 20043, 20044, and 20045 are examples of various patient biomarkers being monitored, measured, and / or reported, as shown in Figure 2B. One or more of the patient sensing systems 20041–20045 may transmit measurement data associated with the monitored patient biomarker to a computing device 20047, which may communicate with a remote server 20009 of a remote cloud computing system 20008. Patient detection systems 20041-20045 may use radio frequency (RF) links to communicate with computing devices 20047 (e.g., smartphones, tablets, etc.). Patient detection systems 20041-20045 may use one or more RF protocols such as Bluetooth®, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low-Power Wireless Personal Area Network (6LoWPAN), Thread, and Wi-Fi for communication with computing devices 20047. For example, patient detection systems 20041-20045 may connect to computing devices 20047 via a wireless router, wireless hub, or wireless bridge.

[0067] Computing device 20047 may communicate with remote server 20009, which is part of cloud computing system 20008. For example, computing device 20047 may communicate with remote server 20009 via a cable / FIOS networking node of an Internet service provider. For example, a patient sensing system may communicate directly with remote server 20009. Computing device 20047 or the sensing system may communicate with remote server 20009 via a cellular transmit / receive point (TRP) or base station using one or more of the following cellular protocols: GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long-Term Evolution (LTE) or 4G, LTE Advanced (LTE-A), New Radio (NR), or 5G.

[0068] For example, the computing device 20047 may display information associated with patient biomarkers. For instance, the computing device 20047 may display blood pressure, oxygen saturation, respiratory rate, etc. The computing device 20047 may also display patient or HCP notifications providing information about the patient, such as recovery milestones or complications.

[0069] In one example, a computing device 20047 and / or a patient sensing system 20044 may receive notification information from a surgical hub 20006 for display on the display unit of the computing device 20047 and / or the patient sensing system 20044. The notification information may include notifications regarding recovery milestones or, for example, notifications regarding complications in the case of postoperative recovery. The notification information may also include a manageable severity level associated with the notification. The computing device 20047 and / or the sensing system 20044 may display the notification and the manageable severity level to the patient. The patient sensing system may alert the patient using haptic feedback. Visual and / or haptic notifications may be accompanied by audible notifications prompting the patient to pay attention to the visual notifications provided on the display unit of the sensing system.

[0070] Figure 3 shows an exemplary surgical monitoring system 20002 having a surgical hub 20006 paired with a wearable sensing system 20011, an environmental sensing system 20015, a human interface system 20012, a robotic system 20013, and an intelligent instrument 20014. The hub 20006 includes a hub display 20048, an imaging module 20049, a generator module 20050, a communication module 20056, a processor module 20057, a storage array 20058, and an operating room mapping module 20059. In certain embodiments, as shown in Figure 3, the hub 20006 further includes a fume extraction module 20054 and / or a suction / irrigation module 20055. During surgical procedures, applying energy to tissue for sealing and / or cutting is generally associated with fume extraction, suction of excess fluid, and / or tissue irrigation. Fluid lines, power lines, and / or data lines from different sources often become entangled during surgical procedures. Dealing with this problem during a surgical procedure can result in the loss of valuable time. Untangling lines may require disconnecting them from their corresponding modules, which may necessitate resetting the modules. The hub modular enclosure 20060 provides a unified environment for managing power lines, data lines, and fluid lines, reducing the frequency of such line entanglement. Aspects of this disclosure present a surgical hub 20006 for use in surgical procedures involving the application of energy to tissue at a surgical site. The surgical hub 20006 includes a hub enclosure 20060 and a combination generator module slidably receivable within a docking station of the hub enclosure 20060. The docking station includes data and power contacts. The combination generator module includes two or more ultrasonic energy generator components, bipolar RF energy generator components, and unipolar RF energy generator components housed within a single unit.In one embodiment, the combination generator module also includes a smoke exhaust component, at least one energy supply cable for connecting the combination generator module to a surgical instrument, at least one smoke exhaust component configured to exhaust smoke, fluid and / or particulate matter generated by the application of therapeutic energy to tissue, and a fluid line extending from the remote surgical site to the smoke exhaust component. In one embodiment, the above fluid line is a first fluid line, and a second fluid line may extend from the remote surgical site to a suction and irrigation module 20055 slidably received within a hub enclosure 20060. In one embodiment, the hub enclosure 20060 may include a fluid interface. Some specific surgical procedures may require the application of two or more energy types to tissue. One energy type may be more beneficial for cutting tissue, while another different energy type may be more beneficial for sealing tissue. For example, a bipolar generator can be used to seal tissue, while an ultrasonic generator can be used to cut sealed tissue. Aspects of the present disclosure present a solution in which a hub modular enclosure 20060 is configured to house different generators and facilitate bidirectional communication between them. One of the advantages of the hub modular enclosure 20060 is that it allows for the rapid removal and / or replacement of various modules. Aspects of the present disclosure present a modular surgical enclosure for use in surgical procedures involving the application of energy to tissue. The modular surgical enclosure includes a first energy generator module configured to generate a first energy for application to tissue, and a first docking station having a first docking port including first data and power contacts, wherein the first energy generator module is slidably movable to electrically engage with the power and data contacts, and the first energy generator module is slidably movable to disengage from the electrical engagement with the first power and data contacts.In addition to the above, the modular surgical enclosure also includes a second energy generator module configured to generate a second energy for application to tissue, distinct from a first energy, and a second docking station having a second docking port including a second data contact and a second power contact, wherein the second energy generator module is slidably movable to electrically engage with the power contact and the data contact, and the second energy generator module is slidably movable to disengage from the electrical engagement with the second power contact and the second data contact. In addition, the modular surgical enclosure also includes a communication bus between the first docking port and the second docking port, configured to facilitate communication between the first energy generator module and the second energy generator module. Referring to Figure 3, an aspect of the present disclosure relating to a hub modular enclosure 20060 enabling modular integration of a generator module 20050, a fume extraction module 20054, and a suction / irrigation module 20055 is presented. The hub modular enclosure 20060 further facilitates bidirectional communication between modules 20059, 20054, and 20055. The generator module 20050 may comprise an integrated unipolar, bipolar, and ultrasonic component supported within a single housing unit slidably inserted into the hub modular enclosure 20060. The generator module 20050 can be configured to connect to a unipolar device 20051, a bipolar device 20052, and an ultrasonic device 20053. Alternatively, the generator module 20050 may comprise a series of unipolar, bipolar, and / or ultrasonic generator modules interacting via the hub modular enclosure 20060. The hub-module enclosure 20060 can be configured to facilitate the insertion of multiple generators and bidirectional communication between generators docked to the hub-module enclosure 20060, so that multiple generators function as a single generator.

[0071] Figure 4 shows a surgical data network, according to at least one aspect of the present disclosure, having a set of communication hubs configured to connect a set of sensing systems, an environmental sensing system, and a set of other modular devices located in one or more operating rooms within a medical facility, a patient recovery room, or a room specifically equipped for surgical procedures.

[0072] As shown in Figure 4, the surgical hub system 20060 may include a modular communication hub 20065 configured to connect modular devices located within a medical facility to a cloud-based system (e.g., a cloud computing system 20064 which may include a remote server 20067 connected to remote storage 20068). The modular communication hub 20065 and devices may be connected in a room within the medical facility specifically equipped for surgical procedures. In one embodiment, the modular communication hub 20065 may include a network hub 20061 and / or a network switch 20062 that communicates with a network router 20066. The modular communication hub 20065 may also be connected to a local computer system 20063 and may provide local computer processing and data manipulation. The surgical data network associated with the surgical hub system 20060 may be configured as passive, intelligent, or switching. A passive surgical data network acts as a data conduit, enabling data to travel from one device (or segment) to another, and to cloud computing resources. An intelligent surgical data network enables traffic to pass through a monitored surgical data network and includes additional mechanisms that constitute each port within a network hub 20061 or network switch 20062. An intelligent surgical data network may be referred to as a manageable hub or switch. The switching hub reads the destination address of each packet and then forwards the packet to the correct port.

[0073] Modular devices 1a-1n located in the operating room may be connected to a modular communication hub 20065. Network hub 20061 and / or network switch 20062 may be connected to a network router 20066 to connect devices 1a-1n to a cloud computing system 20064 or a local computer system 20063. Data associated with devices 1a-1n may be transferred to a cloud-based computer via the router for remote data processing and manipulation. Data associated with devices 1a-1n may also be transferred to a local computer system 20063 for local data processing and manipulation. Modular devices 2a-2m located in the same operating room may also be connected to a network switch 20062. Network switch 20062 may be connected to network hub 20061 and / or network router 20066 to connect devices 2a-2m to the cloud 20064. Data associated with devices 2a-2m may be transferred to the cloud computing system 20064 via the network router 20066 for data processing and manipulation. The data associated with devices 2a-2m may also be transferred to the local computer system 20063 for local data processing and manipulation.

[0074] The wearable sensing system 20011 may include one or more sensing systems 20069. The sensing systems 20069 may include a surgeon sensing system and / or a patient sensing system. One or more sensing systems 20069 may communicate with the computer system 20063 or cloud server 20067 of the surgical hub system 20060 directly via one of the network routers 20066, or via a network hub 20061 or network switch 20062 that communicates with the network router 20066.

[0075] The sensing system 20069 may be connected to a network router 20066 to connect the sensing system 20069 to a local computer system 20063 and / or a cloud computing system 20064. Data related to the sensing system 20069 may be transferred to the cloud computing system 20064 via the network router 20066 for data processing and manipulation. Data related to the sensing system 20069 may also be transferred to the local computer system 20063 for local data processing and manipulation.

[0076] As shown in Figure 4, the surgical hub system 20060 can be expanded by interconnecting multiple network hubs 20061 and / or multiple network switches 20062 with multiple network routers 20066. The modular communication hub 20065 can be housed in a modular control tower configured to accept multiple devices 1a-1n / 2a-2m. The local computer system 20063 may also be housed in the modular control tower. The modular communication hub 20065 is connected to a display 20068 to display images acquired by some of the devices 1a-1n / 2a-2m, for example, during a surgical procedure. In various embodiments, devices 1a-1n / 2a-2m may include a variety of modules, particularly among modular devices that can be connected to the modular communication hub 20065 of the surgical data network, such as imaging modules connected to endoscopes, generator modules connected to energy-based surgical devices, smoke extraction modules, suction / irrigation modules, communication modules, processor modules, storage arrays, surgical devices connected to displays, and / or non-contact sensor modules.

[0077] In one embodiment, the surgical hub system 20060 shown in Figure 4 may include a combination of a network hub, network switch, and network router connecting devices 1a-1n / 2a-2m or sensing systems 20069 to a cloud-based system 20064. One or more of the devices 1a-1n / 2a-2m or sensing systems 20069 connected to the network hub 20061 or network switch 20062 may collect data or measurement data in real time and transfer the data to a cloud computer for data processing and manipulation. It will be understood that cloud computing relies on sharing computing resources rather than having local servers or personal devices to handle software applications. The term “cloud” may be used as a metaphor for “Internet,” but the term is not limited to that. Accordingly, the term “cloud computing” may be used herein to refer to “a kind of internet-based computing” in which different services such as servers, storage, and applications are delivered via the Internet to a modular communication hub 20065 and / or computer system 20063 located in an operating room (e.g., a fixed, mobile, temporary, or on-site operating room or space), and to devices connected to the modular communication hub 20065 and / or computer system 20063. The cloud infrastructure may be maintained by a cloud service provider. In this context, the cloud service provider may be an entity that coordinates the use and control of devices 1a-1n / 2a-2m located in one or more operating rooms. The cloud computing service can perform numerous calculations based on data collected by smart surgical instruments, robots, sensing systems, and other computerized devices located in the operating room. The hub hardware enables multiple devices, sensing systems, and / or connections to be connected to a computer that communicates with cloud computing resources and storage.

[0078] By applying cloud computing data processing technology to data collected by devices 1a-1n / 2a-2m, surgical data networks can provide improved surgical outcomes, reduced costs, and increased patient satisfaction. At least some of devices 1a-1n / 2a-2m can be used to observe the condition of tissue after tissue sealing and cutting procedures to assess leakage or perfusion of the sealed tissue. At least some of devices 1a-1n / 2a-2m can be used to examine data, including images of body tissue samples, for diagnostic purposes using cloud-based computing to identify pathologies such as the effects of disease. Such data may include tissue localization and margin confirmation, as well as phenotype. At least some of devices 1a-1n / 2a-2m can be used to identify anatomical structures of the body using various sensors integrated with imaging devices and techniques such as overlaying images captured by multiple imaging devices. Data collected by devices 1a-1n / 2a-2m, including image data, may be transferred to a cloud computing system 20064 or a local computer system 20063, or both, for data processing and manipulation, including image processing and manipulation. The data may be analyzed to improve the outcomes of surgical procedures by determining whether further treatments, such as endoscopic interventions, emerging technologies, targeted radiation, targeted interventions, and the application of precision robots, can be carried out for tissue-specific sites and conditions. Such data analysis may further involve prognostic analysis processing, and the use of standardized methods can provide useful feedback for either confirming surgical treatment and surgeon behavior, or suggesting modifications to surgical treatment and surgeon behavior.

[0079] By applying cloud computing data processing techniques to measurement data collected by sensing systems 20069, surgical data networks can deliver improved surgical outcomes, improved recovery results, reduced costs, and improved patient satisfaction. At least some of the sensing systems 20069 may be employed to assess the physiological state of a surgeon operating on a patient, a patient being prepared for a surgical procedure, or a patient recovering after a surgical procedure. Cloud-based computing systems 20064 may be used to monitor biomarkers associated with a surgeon or patient in real time, generate surgical plans based at least on measurement data collected before the surgical procedure, provide control signals to surgical instruments during the surgical procedure, and notify the patient of complications during the postoperative period.

[0080] Operating room devices 1a-1n may be connected to the modular communication hub 20065 via a wired or wireless channel, depending on the configuration of devices 1a-1n with respect to the network hub 20061. In one embodiment, the network hub 20061 may be implemented as a local network broadcast device operating at the physical layer of the Open System Interconnection (OSI) model. The network hub can provide connectivity to devices 1a-1n located within the same operating room network. The network hub 20061 may collect data in packet form and transmit them to the router in half-duplex mode. The network hub 20061 cannot store any medium access control / Internet protocol (MAC / IP) for transferring any device data. Only one of devices 1a-1n can transmit data through the network hub 20061 at a time. The network hub 20061 may not have a routing table or intelligence regarding where to send information and broadcasts all network data to the remote server 20067 of the cloud computing system 20064 via each connection. While the Network Hub 20061 can detect basic network errors such as collisions, having all the information broadcast to multiple ports poses a security risk and can cause bottlenecks.

[0081] Operating room devices 2a-2m can be connected to network switch 20062 via a wired or wireless channel. Network switch 20062 operates within the data link layer of the OSI model. Network switch 20062 may be a multicast device for connecting devices 2a-2m located in the same operating room to a network. Network switch 20062 can transmit data in the form of frames to network router 20066 and may operate in full-duplex mode. Multiple devices 2a-2m can transmit data simultaneously through network switch 20062. Network switch 20062 stores and uses the MAC addresses of devices 2a-2m to transfer data.

[0082] Network hub 20061 and / or network switch 20062 may be coupled to network router 20066 for connection to cloud computing system 20064. Network router 20066 operates within the network layer of the OSI model. Network router 20066 creates routes for sending data packets received from network hub 20061 and / or network switch 20062 to cloud-based computing resources and further processes and manipulates data collected by one or all of devices 1a-1n / 2a-2m and wearable sensing system 20011. Network router 20066 may be employed to connect two or more different networks located in different locations, for example, different networks located in different operating rooms of the same medical facility or different networks located in different operating rooms of different medical facilities. Network router 20066 transmits data to cloud computing system 20064 in the form of packets and may operate in full-duplex mode. Multiple devices can transmit data simultaneously. Network router 20066 may use IP addresses to transfer data.

[0083] In one example, the network hub 20061 may be implemented as a USB hub that enables multiple USB devices to be connected to a host computer. The USB hub can extend a single USB port into several layers so that there are more ports available for connecting devices to the host system computer. The network hub 20061 may include wired or wireless functionality for receiving information via a wired or wireless channel. In one embodiment, a wireless USB short-range high-bandwidth wireless communication protocol may be used for communication between devices 1a-1n and devices 2a-2m located in the operating room.

[0084] In the example, the operating room devices 1a-1n / 2a-2m and / or sensing system 20069 can communicate with the modular communication hub 20065 via the Bluetooth radio technology standard to exchange data over short distances from fixed and mobile devices (using short-wavelength UHF radio waves in the 2.4-2.485 GHz ISM band) and to establish a personal area network (PAN). The operating room devices 1a-1n / 2a-2m and / or sensing systems 20069 may communicate with the modular communication hub 20065 via several wireless or wired communication standards or protocols, including but not limited to Bluetooth, Low-Energy Bluetooth, Near Field Communication (NFC), Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, New Radio (NR), Long-Term Evolution (LTE), and Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, and their Ethernet derivatives, as well as any other wireless and wired protocols designated as 3G, 4G, 5G, and later. The computing module may include multiple communication modules. For example, the first communication module may be dedicated to short-range wireless communication such as Wi-Fi, Bluetooth Low Energy Bluetooth, and Bluetooth Smart, while the second communication module may be dedicated to long-range wireless communication such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, and TDMA.

[0085] The modular communication hub 20065 may function as a central connection for one or more operating room devices 1a-1n / 2a-2m and / or sensing systems 20069 and may handle a data type known as a frame. A frame may carry data generated by devices 1a-1n / 2a-2m and / or sensing systems 20069. Once a frame is received by the modular communication hub 20065, it may be amplified and / or transmitted to a network router 20066 which may transfer the data to a cloud computing system 20064 or a local computer system 20063 by using some wireless or wired communication standards or protocols as described herein.

[0086] The modular communication hub 20065 can be used as a standalone device or connected to compatible network hubs 20061 and network switches 20062 to form a larger network. The modular communication hub 20065 is generally easy to install, configure, and maintain, making it a good choice for networking operating room devices 1a-1n / 2a-2m.

[0087] Figure 5 shows a computer-implemented bidirectional surgical system 20070, which may be part of the surgeon monitoring system 20002. The computer-implemented bidirectional surgical system 20070 is similar in many respects to the surgeon sensing system 20002. For example, the computer-implemented bidirectional surgical system 20070 may include one or more surgical subsystems 20072, which are similar in many respects to the surgeon monitoring system 20002. Each surgical subsystem 20072 includes at least one surgical hub 20076 that communicates with a cloud computing system 20064, which may include a remote server 20077 and remote storage 20078. In one embodiment, the computer-implemented bidirectional surgical system 20070 may include a modular control tower 20085 connected to multiple operating room devices, such as sensing systems (e.g., surgeon sensing system 20002 and / or patient sensing system 20003), intelligent surgical instruments, robots, and other computerized devices located in the operating room. As shown in Figure 6A, the modular control tower 20085 may include a modular communication hub 20065 connected to the local computing system 20063.

[0088] As shown in the example in Figure 5, the modular control tower 20085 can be connected to an imaging module 20088 which can be connected to an endoscope 20087, a generator module 20090 which can be connected to an energy device 20089, a smoke exhaust module 20091, a suction / irrigation module 20092, a communication module 20097, a processor module 20093, a storage array 20094, smart devices / equipment 20095 which are optionally connected to displays 20086 and 20084, respectively, and a non-contact sensor module 20096. The modular control tower 20085 can also communicate with one or more sensing systems 20069 and an environmental sensing system 20015. The sensing system 20069 can be connected to the modular control tower 20085 directly via a router or via the communication module 20097. The operating room devices can be connected to cloud computing resources and data storage via the modular control tower 20085. The robotic surgery hub 20082 may also be connected to the modular control tower 20085 and cloud computing resources. Devices / instruments 20095 or 20084, in particular, the human interface system 20080, may be connected to the modular control tower 20085 via wired or wireless communication standards or protocols as described herein. The human interface system 20080 may include a display subsystem and a notification subsystem. The modular control tower 20085 may be connected to a hub display 20081 (e.g., a monitor, screen) to display and overlay images received from the imaging module 20088, the device / instrument display 20086, and / or other human interface systems 20080. The hub display 20081 may also display data received from devices connected to the modular control tower 20085 along with the images and overlay images.

[0089] Figure 6A shows a surgical hub 20076 including multiple modules connected to a modular control tower 20085. As shown in Figure 6A, the surgical hub 20076 may be connected to a generator module 20090, a fume extractor module 20091, a suction / irrigation module 20092, and a communication module 20097. The modular control tower 20085 may comprise a modular communication hub 20065, such as a network connectivity device, and a computer system 20063, which provides, for example, local wireless connectivity with sensing systems, local processing, monitoring of complications, visualization, and imaging. As shown in Figure 6A, the modular communication hub 20065 may extend to several modules (e.g., devices) and several sensing systems 20069 that can be connected to the modular communication hub 20065, and may be connected in a configuration (e.g., a tiered configuration) for transferring data related to the modules and / or measurement data related to the sensing systems 20069 to the computer system 20063, cloud computing resources, or both. As shown in Figure 6A, each of the network hubs / switches 20061 / 20062 within the modular communication hub 20065 may include three downstream ports and one upstream port. The upstream network hub / switch may be connected to processor 20102 to provide communication connectivity to cloud computing resources and local display 20108. At least one of the network / hub switches 20061 / 20062 within the modular communication hub 20065 may have at least one wireless interface to provide communication connectivity between sensing system 20069 and / or device 20095 and cloud computing system 20064. Communication to cloud computing system 20064 can be made via either a wired or wireless communication channel.

[0090] The surgical hub 20076 may use a non-contact sensor module 20096 to measure the dimensions of an operating room and generate a map of the operating room using either an ultrasonic or laser-type non-contact measuring device. The ultrasonic-based non-contact sensor module can scan an operating room by transmitting bursts of ultrasound and receiving echoes as they reflect off the outer walls of the operating room, as described under the heading "Surgical Hub Spatial Awareness Within an Operating Room" of U.S. Provisional Patent Application No. 62 / 611,341, filed December 28, 2017, entitled "INTERACTIVE SURGICAL PLATFORM," which is incorporated herein by reference in its entirety. The sensor module is configured to determine the size of the operating room and adjust the Bluetooth pairing distance limit. The laser-based non-contact sensor module can scan an operating room, for example, by transmitting laser light pulses, receiving laser light pulses reflected off the outer walls of the operating room, comparing the phase of the transmitted pulses with the received pulses to determine the size of the operating room and adjust the Bluetooth pairing distance limit.

[0091] Computer system 20063 may comprise a processor 20102 and a network interface 20100. The processor 20102 may be connected via a system bus to a communication module 20103, storage 20104, memory 20105, non-volatile memory 20106, and input / output interface 20107. The system bus may be any of several types of bus structures, including a memory bus or memory controller, peripheral bus or external bus, and / or local bus, using any various available bus architectures. Examples of such architectures include, but are not limited to, a 9-bit bus, Industry Standard Architecture (ISA), Microchannel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Interconnect (PCI), USB, Advanced Graphics Port (AGP), Personal Computer Memory Card International Association Bus (PCMCIA), Small Computer System Interface (SCSI), or any other proprietary bus.

[0092] The processor 20102 may be any single-core or multi-core processor, such as those known by the trademark name ARM Cortex from Texas Instruments. In one embodiment, the processor may be, for example, the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments. This processor core includes on-chip memory of 256KB single-cycle flash memory or other non-volatile memory up to 40MHz, a prefetch buffer to improve performance beyond 40MHz, 32KB single-cycle serial random access memory (SRAM), internal read-only memory (ROM) with StellarisWare® software, 2KB electrically erasable programmable read-only memory (EEPROM), and / or one or more pulse-width modulation (PWM) modules, one or more quadrature encoder input (QEI) analogs, and one or more 12-bit analog-to-digital converters (ADCs) with 12 analog input channels. Further details are available in the product datasheet.

[0093] For example, the 20102 processor may include safety controllers, including two controller-based families such as the TMS 570 and RM4x, also known by Texas Instruments under the trademark name Hercules ARM Cortex R4. The safety controllers may be configured specifically for IEC 61508 and ISO 26262 safety limit applications, among other things, to provide advanced integrated safety mechanisms while offering scalable performance, connectivity, and memory options.

[0094] System memory can be categorized into volatile and non-volatile memory. The Basic Input / Output System (BIOS), which includes basic routines for transferring information between elements within the computer system during startup, is stored in non-volatile memory. Examples of non-volatile memory include ROM, programmable ROM (PROM), electrically programmable ROM (EPROM), EEPROM, or flash memory. Examples of volatile memory include random access memory (RAM), which functions as external cache memory. Furthermore, RAM is available in many forms, such as SRAM, dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), sync-link DRAM (SLDRAM), and direct rhombus RAM (DRRAM).

[0095] Computer System 20063 may also include removable / non-removable volatile / non-volatile computer storage media, such as disk storage. Disk storage may include, but is not limited to, devices such as magnetic disk drives, floppy disk drives, tape drives, Jaz drives, Zip drives, LS-60 drives, flash memory cards, or memory sticks. In addition, disk storage may include the above-mentioned storage media independently or in combination with other storage media. Other storage media may include, but are not limited to, optical disk drives such as compact disk ROM devices (CD-ROMs), compact disk recordable drives (CD-R drives), compact disk rewritable drives (CD-RW drives), or digital multi-purpose disk ROM drives (DVD-ROMs). Removable or non-removable interfaces may be used to facilitate connection of disk storage to the system bus.

[0096] It should be understood that computer system 20063 may include software that acts as an intermediary between the user and basic computer resources as described in a suitable operating environment. Such software may include an operating system. An operating system, which may be stored on disk storage, may function to control and allocate the resources of the computer system. System applications may leverage resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein can be implemented in various operating systems or combinations of operating systems.

[0097] The user can input commands or information to the computer system 20063 via input devices connected to the I / O interface 20107. Examples of input devices include, but are not limited to, pointing devices such as mice, trackballs, styluses, and touchpads; keyboards; microphones; joysticks; gamepads; satellite receivers; scanners; TV tuner cards; digital cameras; digital video cameras; and webcams. These and other input devices connect to the processor 20102 via interface ports and the system bus. Examples of interface ports include serial ports, parallel ports, game ports, and USB ports. Output devices use some of the same types of ports as the input devices. Therefore, for example, a USB port may be used to provide input to the computer system 20063 and output information from the computer system 20063 to an output device. Output adapters may be provided to indicate that some output devices, such as monitors, displays, speakers, and printers, may be present, among others, which may require special adapters. Examples of output adapters include video and sound cards that provide a means of connection between the output device and the system bus, but these are illustrative and not limiting. Note that other devices and / or systems of devices, such as remote computers, can provide both input and output functions.

[0098] Computer System 20063 can operate in a networked environment using logical connections to one or more remote computers, such as cloud computers or local computers. Remote cloud computers may be personal computers, servers, routers, network PCs, workstations, microprocessor-based devices, peer devices, or other common network nodes, but typically include many or all of the elements described in relation to computer systems. For brevity, only memory storage devices are shown along with remote computers. Remote computers may be logically connected to the computer system via a network interface, and subsequently physically connected via a communication interface. Network interfaces may include communication networks such as local area networks (LANs) and wide area networks (WANs). LAN technologies may include fiber-distributed data interfaces (FDDI), copper-distributed data interfaces (CDDI), Ethernet / IEEE 802.3, Token Ring / IEEE 802.5, etc. WAN technologies may include, but are not limited to, point-to-point links, integrated service digital networks (ISDN) and their variations, circuit-switched networks, packet-switched networks, and digital subscriber lines (DSL).

[0099] In various examples, the computer system 20063, imaging module 20088 and / or human interface system 20080 in Figures 4, 6A and 6B, and / or processor module 20093 in Figures 5 and 6A may include an image processor, an image processing engine, a media processor, or any dedicated digital signal processor (DSP) used for processing digital images. The image processor can increase speed and efficiency using parallel computing with single-instruction multiple data (SIMD) or multiple-instruction multiple data (MIMD) techniques. The digital image processing engine can perform a variety of tasks. The image processor may be a system on a chip with a multi-core processor architecture.

[0100] The communication connection section may refer to the hardware / software used to connect the network interface to the bus. While the communication connection section is shown inside computer system 20063 for illustrative purposes, it may be located outside computer system 20063. The hardware / software required for connection to the network interface may include, for illustrative purposes only, internal and external technologies such as standard telephone-grade modems, cable modems, fiber optic modems, DSL modems, ISDN adapters, and modems including Ethernet cards. In some examples, the network interface may also be provided using an RF interface.

[0101] Figure 6B shows an example of a wearable monitoring system, such as a controlled patient monitoring system. The controlled patient monitoring system may be a sensing system used to monitor a set of patient biomarkers while the patient is in a healthcare facility. The controlled patient monitoring system may be deployed for pre-operative patient monitoring while the patient is preparing for a surgical procedure, intra-operative monitoring while the patient is undergoing surgery, or post-operative monitoring, for example, while the patient is recovering. As shown in Figure 6B, the controlled patient monitoring system may include a surgical hub system 20076 which may include one or more routers 20066 of a modular communication hub 20065 and a computer system 20063. In one example, the router 20065 may include a wireless router, a wired switch, a wired router, a wired or wireless network hub, and the router 20065 may be part of the infrastructure. The computing system 20063 may provide local processing for monitoring various biomarkers related to the patient or surgeon, and a notification mechanism to inform the patient and / or the healthcare professional (HCP) provided that milestones (e.g., recovery milestones) have been met or complications have been detected. The computing system 20063 of the surgical hub system 20076 may also be used to generate notifications, for example, severity levels associated with a notification that a complication has been detected.

[0102] The computing system 20063 in Figures 4 and 6B, the computing device 20200 in Figure 6C, and the hub / computing device 20243 in Figures 7B, 7C, or 7D may be a surgical computing system or hub device, a laptop, a tablet, a smartphone, etc.

[0103] As shown in Figure 6B, a pair of sensing systems 20069 and / or environmental sensing systems 20015 (as described in Figure 2A) may be connected to the surgical hub system 20076 via a router 20065. The router 20065 may also provide a direct communication connection between the sensing system 20069 and the cloud computing system 20064, for example, without going through the local computer system 20063 of the surgical hub system 20076. Communication from the surgical hub system 20076 to the cloud 20064 can be done via either a wired or wireless communication channel.

[0104] As shown in Figure 6B, the computer system 20063 may include a processor 20102 and a network interface 20100. The processor 20102 may be connected via a system bus to a radio frequency (RF) interface or communication module 20103, storage 20104, memory 20105, non-volatile memory 20106, and input / output interface 20107, as shown in Figure 6A. The computer system 20063 may be connected to a local display unit 20108. In some examples, the display unit 20108 may be replaced with a HID. Further details regarding the hardware and software components of the computer system are provided in Figure 6A.

[0105] As shown in Figure 6B, the sensing system 20069 may include a processor 20110. The processor 20110 may be connected via a system bus to a radio frequency (RF) interface 20114, storage 20113, memory (e.g., non-volatile memory) 20112, and an I / O interface 20111. The system bus may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus, as described herein. The processor 20110 may be any single-core or multi-core processor as described herein.

[0106] It should be understood that the sensing system 20069 may include software that acts as an intermediary between the sensing system user and the computer resources described in the preferred operating environment. Such software may include an operating system. An operating system, which may be stored on disk storage, may function to control and allocate the resources of the computer system. System applications may leverage resource management by the operating system through program modules and program data stored either in system memory or on disk storage. It should be understood that the various components described herein can be implemented on various operating systems or combinations of operating systems.

[0107] The sensing system 20069 may be connected to the human interface system 20115. The human interface system 20115 may be a touchscreen display. The human interface system 20115 may include a human interface display for displaying information related to surgeon biomarkers and / or patient biomarkers, displaying prompts for user actions by the patient or surgeon, or displaying notifications to the patient or surgeon indicating information about recovery milestones or complications. The human interface system 20115 may be used to receive input from the patient or surgeon. Other human interface systems may be connected to the sensing system 20069 via the I / O interface 20111. For example, the human interface device 20115 may include a device that provides haptic feedback as a mechanism to draw the user's attention to notifications that may be displayed on the display unit.

[0108] The sensing system 20069 may operate in a networked environment using logical connections to one or more remote computers, such as cloud computers or local computers. Remote cloud computers may include personal computers, servers, routers, network PCs, workstations, microprocessor-based devices, peer devices, or other common network nodes, but typically include many or all of the elements described with respect to computer systems. Remote computers may be logically connected to computer systems via network interfaces. Network interfaces may include communication networks such as local area networks (LANs), wide area networks (WANs), and / or mobile networks. LAN technologies may include fiber-distributed data interfaces (FDDI), copper-distributed data interfaces (CDDI), Ethernet / IEEE 802.3, Token Ring / IEEE 802.5, Wi-Fi / IEEE 802.11, etc. WAN technologies may include, but are not limited to, point-to-point links, integrated service digital networks (ISDN) and their variations, circuit-switched networks, packet-switched networks, and digital subscriber lines (DSL). The mobile network may include communication links based on one or more mobile communication protocols, such as GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long-Term Evolution (LTE) or 4G, LTE Advanced (LTE-A), New Radio (NR), or 5G.

[0109] Figure 6C illustrates an exemplary uncontrolled patient monitoring system, for example, when the patient is away from the medical facility. Uncontrolled patient monitoring systems can be used for pre-operative patient monitoring when the patient is preparing for a surgical procedure but is away from the medical facility, or for post-operative monitoring when the patient is away from the medical facility and recovering.

[0110] As shown in Figure 6C, one or more sensing systems 20069 communicate with a computing device 20200, such as a personal computer, laptop, tablet, or smartphone. The computing system 20200 may provide processing for monitoring various biomarkers related to the patient, notification mechanisms indicating that milestones (e.g., recovery milestones) have been met or that complications have been detected. The computing system 20200 may also provide instructions for the user of the sensing system to follow. Communication between the sensing system 20069 and the computing device 20200 can be established using the wireless protocols described herein or directly via a wireless router / hub 20211.

[0111] As shown in Figure 6C, the sensing system 20069 may be connected to the computing device 20200 via a router 20211. The router 20211 may include a wireless router, a wired switch, a wired router, a wired or wireless network hub, etc. The router 20211 may, for example, provide a direct communication connection between the sensing system 20069 and the cloud server 20064 without going through the local computing device 20200. The computing device 20200 may communicate with the cloud server 20064. For example, the computing device 20200 may communicate with the cloud 20064 via a wired or wireless communication channel. In one example, the sensing system 20069 may communicate directly with the cloud via a cellular network, for example, via a cellular base station 20210.

[0112] As shown in Figure 6C, the computing device 20200 may include a processor 20203 and a network or RF interface 20201. The processor 20203 may be connected via a system bus to storage 20202, memory 20212, non-volatile memory 20213, and input / output interface 20204, as shown in Figures 6A and 6B. Details regarding the hardware and software components of the computer system are provided in Figure 6A. The computing device 20200 may include, for example, a set of sensors from sensor #1 20205, sensor #2 20206 through sensor #n 20207. These sensors may be part of the computing device 20200 and may be used to measure one or more attributes related to a patient. The attributes may provide context for biomarker measurements performed by one of the sensing systems 20069. For example, sensor #1 may be an accelerometer that can be used to measure acceleration forces to sense patient-related movement or vibration. For example, sensors 20205 through 20207 may include one or more of the following: pressure sensors, altimeters, thermometers, lidars, etc.

[0113] As shown in Figure 6B, the sensing system 20069 may include a processor, a radio frequency interface, storage, memory or non-volatile memory, and an input / output interface via a system bus, as described in Figure 6A. The sensing system may include a sensor unit and a processing and communication unit, as described in Figures 7B to 7D. The system bus may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus or external bus, and / or a local bus, as described herein. The processor may be any single-core or multi-core processor, as described herein.

[0114] The sensing system 20069 may communicate with the human interface system 20215, which may be a touchscreen display. The human interface system 20215 may be used to display information related to patient biomarkers, to display prompts for user actions by the patient, or to display notifications to the patient indicating information about recovery milestones or complications. The human interface system 20215 may be used to receive input from the patient. Other human interface systems may be connected to the sensing system 20069 via I / O interfaces. For example, a human interface system may include a device for providing haptic feedback as a mechanism to prompt the user to pay attention to notifications that may be displayed on the display unit. The sensing system 20069 may operate in a networked environment using logical connections to one or more remote computers, such as cloud computers or local computers, as described in Figure 6B.

[0115] Figure 7A shows a logic diagram of a control system 20220 for a surgical instrument or surgical tool according to one or more embodiments of the present disclosure. The surgical instrument or surgical tool may be configurable. The surgical instrument may include surgical fixation devices specific to the procedure at hand, such as imaging devices, surgical staplers, energy devices, and endocutter devices. For example, the surgical instrument may include any of the following: a motorized stapler, a motorized stapler generator, an energy device, an advanced energy device, an advanced energy jaw device, an endocutter clamp, an energy device generator, an intraoperative imaging system, a fume extractor, a suction irrigation device, and an air supply system. The system 20220 may include a control circuit. The control circuit may include a microcontroller 20221 having a processor 20222 and memory 20223. For example, one or more of sensors 20225, 20226, and 20227 provide real-time feedback to the processor 20222. Motor 20230, driven by motor driver 20229, operably connects a longitudinally movable displacement member to drive the I-beam knife element. A tracking system 20228 may be configured to determine the position of the longitudinally movable displacement member. Position information may be provided to processor 20222, which can be programmed or configured to determine the position of the longitudinally movable drive member, as well as the positions of the firing member, firing bar, and I-beam knife element. Additional motors may be provided to the tool driver interface to control I-beam firing, occluder movement, shaft rotation, and joint movement. Display 20224 can display various operating conditions of the instrument and may include touchscreen functionality for data input. Information displayed on display 20224 may be overlaid with images acquired via the endoscopic imaging module.

[0116] In one embodiment, the microcontroller 20221 may be any single-core or multi-core processor, such as one known by the trademark name ARM Cortex by Texas Instruments. In one embodiment, the main microcontroller 20221 may be the LM4F230H5QR ARM Cortex-M4F processor core available from Texas Instruments, which includes, for example, 256KB of on-chip memory of single-cycle flash memory or other non-volatile memory, a prefetch buffer for performance up to 40MHz and above 40MHz, 32KB of single-cycle SRAM, internal ROM with StellarisWare® software, 2KB of EEPROM, one or more PWM modules, one or more QEI analogs, and / or one or more 12-bit ADCs having 12 analog input channels, details of which are available in the product datasheet.

[0117] In one embodiment, the microcontroller 20221 may include a safety controller comprising two controller-based families, such as the TMS 570 and RM4x, also known by Texas Instruments under the trademark name Hercules ARM Cortex R4. The safety controller may be configured specifically for IEC 61508 and ISO 26262 safety limit applications, in particular, to provide an advanced integrated safety mechanism while offering scalable performance, connectivity, and memory options.

[0118] The microcontroller 20221 may be programmed to perform various functions, such as precise control of the speed and position of the knife and joint systems. In one embodiment, the microcontroller 20221 may include a processor 20222 and memory 20223. The electric motor 20230 may be a brushed direct current (DC) motor with a gearbox and a mechanical link to the joint or knife system. In one embodiment, the motor driver 20229 may be the A3941 available from Allegro Microsystems. Other motor drivers may be readily interchangeable for use in a tracking system 20228 with an absolute positioning system. A detailed description of the absolute positioning system is provided in U.S. Patent Application Publication 2017 / 0296213, published October 19, 2017, entitled “SYSTEMS AND METHODS FOR CONTROLLING A SURGICAL STAPLING AND CUTTING INSTRUMENT,” which is incorporated herein by reference in its entirety.

[0119] The microcontroller 20221 can be programmed to provide precise control over the velocity and position of the displacement member and joint system. The microcontroller 20221 may be configured to calculate the response in its software. The calculated response can be compared with the measured response of the actual system to obtain an "observed" response, which is used to determine the actual feedback. The observed response may be a well-adjusted value that balances the smooth and continuous nature of the simulated response with the measured response, and this can detect external influences on the system.

[0120] In some examples, the motor 20230 can be controlled by a motor driver 20229 and used by a launching system for surgical instruments or tools. In various forms, the motor 20230 may be a brushed DC-driven motor having a maximum rotational speed of about 25,000 RPM. In some examples, the motor 20230 may include a brushless motor, a cordless motor, a synchronous motor, a stepper motor, or any other suitable electric motor. The motor driver 20229 may include, for example, an H-bridge driver with field-effect transistors (FETs). The motor 20230 can be powered by a power assembly removably mounted to a handle assembly or tool housing to supply control power to a surgical instrument or tool. The power assembly may include a battery that may include a number of battery cells connected in series, which can be used as a power source for supplying power to a surgical instrument or tool. Under certain circumstances, the battery cells of the power assembly may be replaceable and / or rechargeable. In at least one example, the battery cells may be a lithium-ion battery that can be coupled to and detached from the power assembly.

[0121] The motor driver 20229 may be the A3941, available from Allegro Microsystems. The A3941 may be a full-bridge controller for use with an external N-channel power metal-oxide-semiconductor field-effect transistor (MOSFET) specifically designed for inductive loads such as brushed DC motors. The driver 20229 may have a built-in charge pump regulator that can provide full (greater than 10V) gate drive for battery voltages up to 7V and can operate the A3941 with reduced gate drive down to 5.5V. Bootstrap capacitors can be used to provide the above battery supply voltage required for the N-channel MOSFET. An internal charge pump for high-side drive enables DC (100% duty cycle) operation. The full bridge can be driven in fast or slow decay mode using diodes or synchronous rectification. In slow decay mode, current recirculation is possible by either the high-side FET or the low-side FET. The power FET may be protected from shoot-through by a dead time adjustable with resistors. The integrated diagnostics indicate undervoltage, overtemperature, and power bridge anomalies and can be configured to protect power MOSFETs under most short-circuit conditions. Other motor drivers can be easily substituted for use in tracking systems 20228 with absolute positioning systems.

[0122] The tracking system 20228 may include a control motor drive circuit device comprising a position sensor 20225 according to one aspect of the present disclosure. The position sensor 20225 for the absolute positioning system may provide a unique position signal corresponding to the position of the displacement member. In some examples, the displacement member may represent a longitudinally movable drive member comprising a rack of drive teeth for meshing and engaging with a corresponding drive gear of a gear reducer assembly. In some examples, the displacement member may represent a launch member which may be adapted and configured to include a rack of drive teeth. In some examples, the displacement member may represent a launch bar or an I-beam, each of which may be adapted and configured to include a rack of drive teeth. Thus, as used herein, the term displacement member may generally be used to refer to any movable member of a surgical instrument or tool, such as a drive member, launch member, launch bar, I-beam, or any element which may be displaced. In one aspect, the longitudinally movable drive member may be coupled to a launch member, launch bar, and I-beam. Thus, the absolute positioning system can, in practice, track the linear displacement of the I-beam by tracking the linear displacement of the longitudinally movable drive member. In various embodiments, the displacement member may be connected to any position sensor 20225 suitable for measuring linear displacement. Thus, a longitudinally movable drive member, launch member, launch bar, or I-beam, or a combination thereof, may be connected to any suitable linear displacement sensor. The linear displacement sensor may include contact-type or non-contact-type displacement sensors. The linear displacement sensor may include a magnetic sensing system comprising a linear variable differential transformer (LVDT), a differential variable reluctance transducer (DVRT), a slide potentiometer, a movable magnet and a series of linearly arranged Hall effect sensors, a magnetic sensing system comprising a fixed magnet and a series of movable linearly arranged Hall effect sensors, an optical sensing system comprising a movable light source and a series of linearly arranged photodiodes or photodetectors, an optical sensing system comprising a fixed light source and a series of movable linearly arranged photodiodes or photodetectors, or any combination thereof.

[0123] The electric motor 20230 may include a rotatable shaft that operably interfaces with a gear assembly mounted to mesh with a set of drive teeth or a rack on the displacement member. A sensor element may be operably coupled to the gear assembly such that one rotation of the position sensor element 20225 corresponds to some linear longitudinal translation of the displacement member. The gearing and sensor configuration can be connected to a linear actuator by a rack and pinion configuration, or to a rotary actuator by a spur gear or other connection. A power supply provides power to the absolute positioning system, and an output indicator may display the output of the absolute positioning system. The displacement member may represent a longitudinally movable drive member having a rack of drive teeth formed thereon for meshing with the corresponding drive gear of a gear reducer assembly. The displacement member may represent a longitudinally movable launch member, launch bar, I-beam, or a combination thereof.

[0124] One rotation of the sensor element associated with the position sensor 20225 may correspond to the longitudinal linear displacement d1 of the displacement member, where d1 is the longitudinal linear distance the displacement member moves from point "a" to point "b" after one rotation of the sensor element connected to the displacement member. The sensor device may be connected via a gear reduction device that enables the position sensor 20225 to complete one or more rotations relative to the full stroke of the displacement member. The position sensor 20225 can complete multiple rotations relative to the full stroke of the displacement member.

[0125] A series of switches where n is an integer greater than 1 can be used alone or in combination with gear reduction to provide a unique position signal for multiple rotations of the position sensor 20225. The state of the switches can be fed back to a microcontroller 20221 that applies logic to determine a unique position signal corresponding to the longitudinal linear displacement d1+d2+...dn of the displacement member. The output of the position sensor 20225 is supplied to the microcontroller 20221. The position sensor 20225 of the sensor device may comprise a magnetic sensor, an analog rotation sensor such as a potentiometer, or an array of analog Hall effect elements that output a unique combination of position signals or values.

[0126] The position sensor 20225 may comprise any number of magnetic sensing elements, such as magnetic sensors classified by whether they measure the total magnetic field or the vector component of the magnetic field. The techniques used to produce both types of magnetic sensors may involve numerous aspects of physics and electronics. Techniques used to sense magnetic fields include, among others, probe coils, flux gates, optical pumping, nuclear perturbation, SQUIDs, Hall effect, anisotropic magnetoresistance, colossal magnetoresistance, magnetic tunnel junctions, colossal magnetoimpedance, magnetostrictive / piezoelectric composites, magnetic diodes, magnetic transistors, optical fibers, magneto-optics, and micro-electromechanical system-based magnetic sensors.

[0127] In one embodiment, the position sensor 20225 for a tracking system 20228 with an absolute positioning system may include a magnetic rotation absolute positioning system. The position sensor 20225 may be implemented as an AS5055EQFT single-chip magnetic rotation position sensor available from Austria Microsystems, AG. The position sensor 20225 interfaces with a microcontroller 20221 to provide an absolute positioning system. The position sensor 20225 may be a low-voltage and low-power component and may include four Hall effect elements in the area of ​​the position sensor 20225 that may be positioned above the magnet. A high-resolution ADC and a smart power management controller may also be provided on the chip. A coordinate rotation digital computer (CORDIC) processor, also known as the digit-by-digit method and Volder's algorithm, may be provided to implement simple and efficient algorithms for computing hyperbolic and trigonometric functions that require only addition, subtraction, bit shifting, and table lookup operations. Angular position, alarm bits, and magnetic field information can be transmitted to the microcontroller 20221 via a standard serial communication interface such as the Serial Peripheral Interface (SPI) interface. The position sensor 20225 may offer 12-bit or 14-bit resolution. The position sensor 20225 may also be an AS5055 chip housed in a small QFN 16-pin 4x4x0.85mm package.

[0128] The tracking system 20228, which includes an absolute positioning system, may include and / or be programmed to implement feedback controllers such as PID, state feedback, and adaptive controllers. The power supply converts the signals from the feedback controllers into physical inputs to the system, in this case voltage. Other examples include PWM of voltage, current, and force. In addition to the position measured by the position sensor 20225, other sensors may be provided to measure physical parameters of the physical system. In some embodiments, other sensors include those described in U.S. Patent No. 9,345,481, issued May 24, 2016, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; U.S. Patent Application Publication No. 2014 / 0263552, published September 18, 2014, entitled "STAPLE CARTRIDGE TISSUE THICKNESS SENSOR SYSTEM," which is incorporated herein by reference in its entirety; and U.S. Patent Application No. 15 / 628,175, filed June 20, 2017, entitled "TECHNIQUES FOR ADAPTIVE CONTROL OF MOTOR VELOCITY OF A SURGICAL STAPLING AND CUTTING INSTRUMENT," which is incorporated herein by reference in its entirety. In a digital signal processing system, the absolute positioning system is connected to a digital data acquisition system, where the output of the absolute positioning system has a finite resolution and sampling frequency. The absolute positioning system may include comparison and combinational circuits to combine the calculated response with the measured response, using algorithms such as weighted averaging and theoretical control loops that drive the calculated response toward the measured response. The calculated response of a physical system may take into account characteristics such as mass, inertia, viscous friction, and inductance resistance in order to predict what the state and output of the physical system will be by knowing the input.

[0129] The absolute positioning system can provide the absolute position of a displacement member when the device is powered on, without requiring the displacement member to be retracted or advanced to a reset (zero or home) position, as required by conventional rotary encoders that simply count the number of forward or backward steps taken by the motor 20230 to estimate the position of a device actuator, drive bar, knife, etc.

[0130] For example, a sensor 20226, such as a strain gauge or micro-strain gauge, may be configured to measure one or more parameters of an end effector, such as the amplitude of strain exerted on the anvil during a clamping operation, which can indicate the closing force applied to the anvil. The measured strain may be converted into a digital signal and provided to a processor 20222. Alternatively, or in addition to sensor 20226, a sensor 20227, such as a load sensor, can measure the closing force applied to the anvil by the closing drive system. A sensor 20227, such as a load sensor, can measure the firing force applied to the I-beam during the firing stroke of a surgical instrument or tool. The I-beam is configured to engage with a wedge-shaped thread, which is configured to cam upward a staple driver to push the staple out and deform into contact with the anvil. The I-beam may also include a sharp cutting edge that can be used to cut tissue as the I-beam is advanced distally by the firing bar. Alternatively, a current sensor 20231 can be used to measure the current consumed by the motor 20230. The force required to propel the launching element forward can correspond to, for example, the current drawn by motor 20230. The measured force can be converted into a digital signal and provided to processor 20222.

[0131] In one embodiment, a strain gauge sensor 20226 can be used to measure the force applied to tissue by the end effector. A strain gauge can be coupled to the end effector to measure the force applied by the end effector to the tissue being treated. A system for measuring the force applied to tissue gripped by the end effector may include a strain gauge sensor 20226, such as a micro-strain gauge, which can be configured to measure one or more parameters of the end effector. In one embodiment, the strain gauge sensor 20226 can measure the amplitude or magnitude of the strain applied to the jaw members of the end effector during a clamping operation, which can indicate tissue compression. The measured strain can be converted into a digital signal and provided to the processor 20222 of the microcontroller 20221. A load sensor 20227 can measure the force used to operate a knife element to cut tissue trapped between an anvil and a staple cartridge, for example. A magnetic field sensor can be used to measure the thickness of the trapped tissue. The measurement from the magnetic field sensor may also be converted into a digital signal and provided to the processor 20222.

[0132] Measurements of tissue compression, tissue thickness, and / or the force required to close the end effector on the tissue, measured by sensors 20226 and 20227 respectively, can be used by microcontroller 20221 to characterize the selected position of the launching member and / or the corresponding values ​​of the launching member's velocity. For example, memory 20223 can store techniques, formulas, and / or lookup tables that may be used by microcontroller 20221 in the evaluation.

[0133] The surgical instrument or tool control system 20220 may also include wired or wireless communication circuits for communicating with the modular communication hub 20065, as shown in Figures 5 and 6A.

[0134] Figure 7B shows an exemplary sensing system 20069. The sensing system may be a surgeon sensing system or a patient sensing system. Sensing system 20069 may include a sensor unit 20235 and a human interface system 20242 that communicate with a data processing and communication unit 20236. The data processing and communication unit 20236 may include an analog-to-digital converter 20237, a data processing unit 20238, a storage unit 20239, an input / output interface 20241, and a transceiver 20240. Sensing system 20069 may communicate with a surgical hub or computing device 20243, and then with a cloud computing system 20244. The cloud computing system 20244 may include a cloud storage system 20078 and one or more cloud servers 20077.

[0135] The sensor unit 20235 may include one or more ex vivo or in vivo sensors for measuring one or more biomarkers. Biomarkers may include, for example, blood pH, hydration status, oxygen saturation, core body temperature, heart rate, heart rate variability, sweat rate, skin conductance, blood pressure, exposure, ambient temperature, respiratory rate, cough and sneeze, gastrointestinal motility, gastrointestinal imaging, tissue perfusion pressure, respiratory bacteria, alcohol intake, lactic acid (sweat), peripheral temperature, positivity and optimality, adrenaline (sweat), cortisol (sweat), edema, mycotoxins, VO2max, preoperative pain, airborne chemicals, circulating tumor cells, stress and anxiety, confusion and delirium, physical activity, autonomic tension, circadian rhythm, menstrual cycle, sleep, etc. These biomarkers can be measured using one or more sensors, such as photosensors (e.g., photodiodes, photoresistors), mechanical sensors (e.g., motion sensors), acoustic sensors, electrical sensors, electrochemical sensors, thermoelectric sensors, infrared sensors, etc. The sensors can measure the biomarkers described herein using one or more sensing techniques such as photoplethysmography, electrocardiography, electroencephalography, colorimetric analysis, impedance measurement, potentiometric measurement, and current measurement.

[0136] As shown in Figure 7B, the sensors within the sensor unit 20235 may measure physiological signals (e.g., voltage, current, PPG signal, etc.) associated with the biomarker being measured. The physiological signals measured may depend on the sensing technology used, as described herein. The sensor unit 20235 of the sensing system 20069 may communicate with the data processing and communication unit 20236. In one example, the sensor unit 20235 may communicate with the data processing and communication unit 20236 using a wireless interface. The data processing and communication unit 20236 may include an analog-to-digital converter (ADC) 20237, a data processing unit 20238, storage 20239, an I / O interface 20241, and an RF transceiver 20240. The data processing unit 20238 may include a processor and a memory unit.

[0137] The sensor unit 20235 can transmit the measured physiological signal to the ADC 20237 of the data processing and communication unit 20236. In one example, the measured physiological signal may pass through one or more filters (e.g., an RC low-pass filter) before being transmitted to the ADC. The ADC may convert the measured physiological signal into measurement data related to a biomarker. The ADC may pass the measurement data to the data processing unit 20238 for processing. In one example, the data processing unit 20238 may transmit the measurement data related to the biomarker to a surgical hub or computing device 20243, and then transmit the measurement data to a cloud computing system 20244 for further processing. The data processing unit may transmit the measurement data to the surgical hub or computing device 20243 using one of the wireless protocols as described herein. In one example, the data processing unit 20238 may first process the raw measurement data received from the sensor unit and then transmit the processed measurement data to the surgical hub or computing device 20243.

[0138] For example, the data processing and communication unit 20236 of the sensing system 20069 may receive thresholds related to biomarkers for monitoring from the surgical hub, computing device 20243, or directly from the cloud server 20077 of the cloud computing system 20244. The data processing unit 20236 can compare the measurement data related to the monitored biomarker with the corresponding thresholds received from the surgical hub, computing device 20243, or cloud server 20077. The data processing and communication unit 20236 may send a notification message to HID 20242 indicating that the measurement data value has exceeded the threshold. The notification message may include measurement data related to the monitored biomarker. The data processing and computing unit 20236 may send notifications to the surgical hub or computing device 20243 via transmission using one of the following RF protocols: Bluetooth, Bluetooth Low-Energy (BLE), Bluetooth Smart, Zigbee, Z-wave, IPv6 Low-Power Wireless Personal Area Network (6LoWPAN), or Wi-Fi. The data processing unit 20238 can send notifications (e.g., HCP notifications) directly to a cloud server via transmission to a cellular transmit / receive point (TRP) or base station using one or more cellular protocols from among GSM / GPRS / EDGE (2G), UMTS / HSPA (3G), Long-Term Evolution (LTE) or 4G, LTE Advanced (LTE-A), New Radio (NR), or 5G. In one example, the sensing unit may communicate with a hub / computing device via a router, as described in Figures 6A to 6C.

[0139] Figure 7C shows an exemplary sensing system 20069 (e.g., a surgical sensing system or a patient sensing system). The sensing system 20069 may include a sensor unit 20245, a data processing and communication unit 20246, and a human interface device 20242. The sensor unit 20245 may include a sensor 20247 and an analog-to-digital converter (ADC) 20248. The ADC 20248 in the sensor unit 20245 can convert physiological signals measured by the sensor 20247 into measurement data related to biomarkers. The sensor unit 20245 can transmit the measurement data to the data processing and communication unit 20246 for further processing. In one example, the sensor unit 20245 can transmit the measurement data to the data processing and communication unit 20246 using an inter-integrated circuit (I2C) interface.

[0140] The data processing and communication unit 20246 includes a data processing unit 20249, a storage unit 20250, and an RF transceiver 20251. The sensing system may communicate with a surgical hub or computing device 20243, and then with a cloud computing system 20244. The cloud computing system 20244 may include a remote server 20077 and associated remote storage 20078. The sensor unit 20245 may include one or more ex vivo or in vivo sensors for measuring one or more biomarkers, as described herein.

[0141] After processing the measurement data received from the sensor unit 20245, the data processing and communication unit 20246 can further process the measurement data and / or transmit it to the smart hub or computing device 20243, as described in Figure 7B. For example, the data processing and communication unit 20246 can transmit the measurement data received from the sensor unit 20245 to a remote server 20077 of the cloud computing system 20244 for further processing and / or monitoring.

[0142] Figure 7D shows an exemplary sensing system 20069 (e.g., a surgeon sensing system or a patient sensing system). The sensing system 20069 may include a sensor unit 20252, a data processing and communication unit 20253, and a human interface system 20261. Sensor unit 20252 may include a plurality of sensors 20254, 20255 to 20256 for measuring one or more physiological signals related to patient or surgeon biomarkers and / or one or more physical state signals related to the physical state of the patient or surgeon. Sensor unit 20252 may also include one or more analog-to-digital converters (ADCs) 20257. The list of biomarkers may include biomarkers such as those disclosed herein. The ADC 20257 in sensor unit 20252 can convert each of the physiological signals and / or physical state signals measured by sensors 20254 to 20256 into their respective measurement data. Sensor unit 20252 can transmit measurement data related to one or more biomarkers and the physical condition of the patient or surgeon to data processing and communication unit 20253 for further processing. Sensor unit 20252 can transmit measurement data to data processing and communication unit 20253 individually for each of sensors 1 20254 to N 20256, or in combination for all sensors. For example, sensor unit 20252 can transmit measurement data to data processing and communication unit 20253 via an I2C interface.

[0143] The data processing and communication unit 20253 may include a data processing unit 20258, a storage unit 20259, and an RF transceiver 20260. The sensing system 20069 may communicate with a surgical hub or computing device 20243, and then communicate with a cloud computing system 20244 comprising at least one remote server 20077 and at least one storage unit 20078. The sensor unit 20252 may include one or more ex vivo or in vivo sensors for measuring one or more biomarkers, as described herein.

[0144] Figure 8 shows an example of adjusting surgical instrument control using surgical task situation recognition and measurement data from one or more surgeon sensing systems. Figure 8 shows an exemplary surgical procedure timeline 20265 and contextual information that the surgical hub can derive from data received from one or more surgical devices, one or more surgeon sensing systems, and / or one or more environmental sensing systems at each step of the surgical procedure. Devices that can be controlled by the surgical hub may include advanced energy devices, endocutter clamps, etc. The surgeon sensing system may include sensing systems for measuring one or more biomarkers related to the surgeon, such as heart rate, sweat composition, respiratory rate, etc. The environmental sensing system may include systems for measuring one or more environmental attributes, such as cameras for detecting the surgeon's position / movement / breathing patterns, spatial microphones for measuring ambient noise in the operating room and / or the tone of the healthcare provider's voice, ambient temperature / humidity, etc.

[0145] In the following description of the timeline 20265 shown in Figure 8, Figure 5 should also be referenced. Figure 5 provides various components used in a surgical procedure. Timeline 20265 shows steps that may be taken individually and / or collectively by nurses, surgeons, and other healthcare professionals during the course of an exemplary colorectal surgery. In a colorectal surgery, the situation-aware surgical hub 20076 may receive data from various data sources throughout the course of the surgical procedure, including data generated each time the healthcare provider (HCP) utilizes the modular device / instrument 20095 paired with the surgical hub 20076. The surgical hub may receive this data from the paired modular device 20095. The surgical hub may receive measurement data from the sensing system 20069. The surgical hub can continuously derive the surgeon's stress level as new data is received and inferences (i.e., contextual information) about the procedure in progress, using data from modular devices / instruments 20095 and / or measurement data from sensing systems 20069, so as to obtain the surgeon's stress level for the steps of the procedure being performed. The context awareness system of the surgical hub 20076 can perform one or more of the following: recording data about the procedure for generating a report; verifying the steps being taken by the healthcare worker; providing data or prompts that may be relevant to a particular procedure step (e.g., via a display screen); adjusting modular devices based on context (e.g., activating a monitor, adjusting the FOV of a medical imaging device, changing the energy level of an ultrasonic surgical instrument or RF electrosurgical instrument); or performing any other such actions described herein. In one example, these steps may be performed by a remote server 20077 of cloud system 20064, which may communicate with the surgical hub 20076.

[0146] As a first step (not shown in Figure 8 for brevity), hospital staff can search the patient's EMR from the hospital's EMR database. Based on the selected patient data in the EMR, the surgical hub 20076 can determine that the procedure to be performed is a colorectal procedure. Staff can scan the medical supplies coming in for the procedure. The surgical hub 20076 can cross-reference the scanned supplies with a list of supplies available for various types of procedures to confirm that the supply mix corresponds to a colorectal procedure. The surgical hub 20076 can pair each of the sensing systems 20069 fitted by different HCPs.

[0147] Once each device is ready and preoperative preparations are complete, the surgical team can begin by making an incision and positioning the trocar. The surgical team can then perform access and preparation by dissecting any adhesions and identifying the inferior mesenteric artery (IMA) bifurcation. Based on data that can be received from the RF or ultrasound generator indicating that at least an energy device is being emitted, the surgical hub 20076 can infer that the surgeon is in the process of dissecting the adhesions. The surgical hub 20076 can cross-reference the received data with the retrieved steps of the surgical procedure to determine that the energy device being emitted at this point in the process (e.g., after the completion of the aforementioned steps of the procedure) corresponds to the dissection step.

[0148] After incision, the HCP can proceed to the ligation step of the procedure (e.g., indicated by A1). As shown in Figure 8, the HCP may begin by ligating the IMA. The surgical hub 20076 can infer that the surgeon is ligating the arteries and veins, as it may receive data from the advanced energy jaw device and / or endocutter indicating that the instrument is firing. The surgical hub may also receive measurement data from one of the HCP's sensing systems (e.g., indicated by the B1 mark on the time axis) indicating a higher stress level for the HCP. For example, a higher stress level may be indicated by a change in the HCP's heart rate from a baseline. The surgical hub 20076 can derive this inference by cross-referencing the received data from the surgical stapling and cutting instruments with the steps retrieved in the process (e.g., as indicated by A2 and A3), as in the previous step. During the high-stress period, the surgical hub 20076 can monitor the advanced energy jaw trigger ratio and / or endocutter clamp and firing rate. In one example, the surgical hub 20076 can control an operating device by transmitting auxiliary control signals to an advanced energy jaw device and / or an end cutter device. The surgical hub can transmit support signals based on the stress level of the HCP operating the surgical device and / or the situational awareness known to the surgical hub. For example, the surgical hub 20076 can transmit control support signals to an advanced energy device or an end cutter clamp, as shown in A2 and A3 in Figure 8.

[0149] The HCP can then proceed to the next step of releasing the upper sigmoid colon, followed by the descending colon, rectum, and sigmoid colon. The surgical hub 20076 can continue to monitor the HCP's high-stress markers (e.g., indicated by D1, E1a, E1b, F1). During periods of high stress, the surgical hub 20076 can send support signals to the advanced energy jaw device and / or endocutter device, as shown in Figure 8.

[0150] After mobilizing the colon, the HCP can proceed to the partial resection portion of the procedure. For example, the surgical hub 20076 can infer that the HCP is removing the sigmoid colon by traversing the intestine, based on data from surgical stapling and cutting instruments, including data from its cartridge. The cartridge data can correspond, for example, to the size or type of staples fired by the instrument. Since different types of staples are used for different types of tissue, the cartridge data can indicate the type of tissue being stapled and / or transversely incised. It should be noted that, as different instruments are better suited to specific tasks, surgeons should periodically switch between surgical stapling / cutting instruments and surgical energy (e.g., RF or ultrasound) instruments depending on the step of the procedure. Thus, the sequence in which stapling / cutting instruments and surgical energy instruments are used can indicate the steps of the procedure being performed by the surgeon.

[0151] The surgical hub can determine and transmit control signals to surgical devices based on the stress level of the HCP. For example, during period G1b, control signal G2b may be transmitted to the endocutter clamp. Once the sigmoid colon is removed, the incision can be closed and the postoperative portion of the procedure can begin. The patient can be awakened from anesthesia. The surgical hub 20076 can infer that the patient is coming out of anesthesia based on one or more sensing systems attached to the patient.

[0152] Figure 9 is a block diagram of a computer-implemented bidirectional surgical system with surgeon / patient monitoring according to at least one aspect of the present disclosure. In one aspect, the computer-implemented bidirectional surgical system may be configured to monitor surgeon biomarkers and / or patient biomarkers using one or more sensing systems 20069. Surgeon biomarkers and / or patient biomarkers may be measured before, after, and / or during surgical procedures. In one aspect, the computer-implemented bidirectional surgical system may be configured to monitor and analyze data related to the operation of various surgical systems 20069, including surgical hubs, surgical instruments, robotic devices, and operating rooms or medical facilities. The computer-implemented bidirectional surgical system may include a cloud-based analysis system. The cloud-based analysis system may include one or more analysis servers.

[0153] As shown in Figure 9, the cloud-based monitoring and analysis system may comprise a plurality of sensing systems 20268 (which may be the same as or similar to sensing system 20069), surgical instruments 20266 (which may be the same as or similar to instrument 20031), a plurality of surgical hubs 20270 (which may be the same as or similar to hub 20006), and a surgical data network 20269 (which may be the same as or similar to the surgical data network described in Figure 4) for connecting the surgical hubs 20270 to a cloud 20271 (which may be the same as or similar to cloud computing system 20064). Each of the plurality of surgical hubs 20270 may be communicatively connected to one or more surgical instruments 20266. Each of the plurality of surgical hubs 20270 may also be communicatively connected via the network 20269 to one or more sensing systems 20268 and a cloud 20271 of a computer-implemented bidirectional surgical system. The surgical hubs 20270 and sensing systems 20268 may be communicatively connected using the wireless protocols described herein. Cloud system 20271 can serve as a remote centralized source of hardware and software for storing, processing, manipulating, and communicating measurement data from sensing system 20268 and data generated based on the operation of various surgical systems 20268.

[0154] As shown in Figure 9, access to the cloud system 20271 can be achieved via network 20269, which can be the internet or some other suitable computer network. A surgical hub 20270, which can be connected to the cloud system 20271, can be considered the client side of the cloud computing system (e.g., a cloud-based analytics system). Surgical instruments 20266 can be paired with the surgical hub 20270 for the control and execution of various surgical procedures and / or surgeries, as described herein. Sensing systems 20268 can be paired with the surgical hub 20270 for intra-surgical monitoring of surgeon-related biomarkers, pre-operative patient monitoring, intra-operative patient monitoring, or post-operative monitoring of patient biomarkers to track and / or measure various milestones and / or detect various complications. Environmental sensing systems 20267 can be paired with the surgical hub 20270 to measure environmental attributes related to the surgeon or patient for surgeon monitoring, pre-operative patient monitoring, intra-operative patient monitoring, or post-operative patient monitoring.

[0155] The surgical instruments 20266, the environmental sensing system 20267, and the sensing system 20268 may include wired or wireless transceivers for data transmission to and from their corresponding surgical hubs 20270 (which may include transceivers). One or more combinations of the surgical instruments 20266, the sensing system 20268, or the surgical hub 20270 can indicate a specific location, such as an operating room, an intensive care unit (ICU), or a recovery room in a medical facility (e.g., a hospital), to provide medical surgery, pre-operative preparation, and / or post-operative recovery. For example, the memory of the surgical hub 20270 may store location data.

[0156] As shown in Figure 9, the cloud system 20271 may include one or more central servers 20272 (which may be the same as or similar to remote servers 20067), a surgical hub application server 20276, a data analysis module 20277, and an input / output ("I / O") interface 20278. The central servers 20272 of the cloud system 20271 can collectively manage the cloud computing system, including monitoring requests from client surgical hubs 20270 and managing the processing power of the cloud system 20271 for executing those requests. Each of the central servers 20272 may comprise one or more processors 20273 coupled to a suitable memory device 20274, which may include volatile memory such as random access memory (RAM) and non-volatile memory such as magnetic storage. The memory device 20274, when executed, may contain machine-executable instructions that cause the processor 20273 to execute the data analysis module 20277 for cloud-based data analysis, real-time monitoring of measurement data received from the sensing system 20268, operation, recommendations, and other operations described herein. The processor 20273 can execute the data analysis module 20277 independently or in conjunction with a hub application that runs independently by the hub 20270. The central server 20272 may also include an aggregated medical data database 20275 that may reside in the memory 20274.

[0157] Based on connections to various surgical hubs 20270 via network 20269, cloud 20271 can aggregate data from specific data generated by various surgical instruments 20266 and / or monitor real-time data from sensing systems 20268 and surgical hubs 20270 related to surgical instruments 20266 and / or sensing systems 20268. Such aggregated data from surgical instruments 20266 and / or measurement data from sensing systems 20268 can be stored in the aggregated medical database 20275 of cloud 20271. In particular, cloud 20271 can advantageously track real-time measurement data from sensing systems 20268 and / or perform data analysis and actions on the measurement data and / or aggregated data to bring insights that individual hubs 20270 could not achieve on their own and / or to perform functions. For this purpose, as shown in Figure 9, cloud 20271 and surgical hubs 20270 are communicably linked to transmit and receive information. The I / O interface 20278 connects to multiple surgical hubs 20270 via network 20269. In this way, the I / O interface 20278 can be configured to transfer information between the surgical hubs 20270 and the aggregated medical data database 20275. Thus, the I / O interface 20278 can facilitate read / write operations of a cloud-based analysis system. Such read / write operations may be performed in response to requests from the hubs 20270. These requests can be sent to the surgical hubs 20270 via a hub application. The I / O interface 20278 may include one or more high-speed data ports, including a Universal Serial Bus (USB) port, an IEEE 1394 port, and Wi-Fi and Bluetooth I / O interfaces for connecting the cloud 20271 to the surgical hubs 20270. The hub application server 20276 of the cloud 20271 may be configured to host software applications (e.g., hub applications) run by the surgical hubs 20270 and provide shared functionality.For example, the hub application server 20276 can manage requests made by hub applications via hub 20270, control access to the aggregated medical data database 20275, and perform load balancing.

[0158] The cloud computing system configurations described in this disclosure can be designed to address a variety of issues arising in the context of medical surgeries (e.g., pre-operative, intra-operative, and post-operative monitoring) and procedures performed using medical devices such as surgical instruments 20266, 20031. In particular, surgical instrument 20266 may be a digital surgical device configured to interact with cloud 20271 to implement techniques for improving the performance of surgical procedures. Sensing system 20268 may be a system having one or more sensors configured to measure one or more biomarkers related to the surgeon performing the medical surgery and / or the patient on whom the medical surgery is scheduled to be performed, or is being performed or has been performed. Various surgical instruments 20266, sensing system 20268, and / or surgical hub 20270 may include a human interface system (e.g., having a touch-controlled user interface) so that the clinician and / or patient can control the manner of interaction between surgical instrument 20266 or sensing system 20268 and cloud 20271. Other suitable user interfaces for control, such as auditory-controlled user interfaces, can also be used.

[0159] The cloud computing system configurations described herein can be designed to address a variety of issues arising in connection with monitoring one or more biomarkers associated with a healthcare professional (HCP) or patient during pre-operative, intra-operative, and post-operative procedures using the sensing system 20268. The sensing system 20268 may be a surgeon sensing system or a patient sensing system configured to interact with a surgical hub 20270 and / or cloud system 20271 to implement technology for monitoring surgeon biomarkers and / or patient biomarkers. Various sensing systems 20268 and / or surgical hub 20270 may include a touch-controlled human interface system so that the HCP or patient can control the manner of interaction between the sensing system 20268 and the surgical hub 20270 and / or cloud system 20271. Other suitable user interfaces for control, such as an auditory-controlled user interface, may also be used.

[0160] Figure 10 shows an exemplary surgical system 20280 according to the present disclosure, which may include a surgical instrument 20282 capable of communicating with a console 20294 or a portable device 20296 via a local area network 20292 or a cloud network 20293 via a wired or wireless connection. In various embodiments, the console 20294 and the portable device 20296 may be any suitable computing device. The surgical instrument 20282 may include a handle 20297, an adapter 20285, and a loading unit 20287. The adapter 20285 is releasably coupled to the handle 20297, and the loading unit 20287 is releasably coupled to the adapter 20285 such that the adapter 20285 transmits force from a drive shaft to the loading unit 20287. The adapter 20285 or loading unit 20287 may include a force gauge (not explicitly shown) positioned within it to measure the force exerted on the loading unit 20287. The loading unit 20287 may include an end effector 20289 having a first jaw 20291 and a second jaw 20290. The loading unit 20287 may be a field-loaded or multiple-fire loading unit (MFLU) that allows a clinician to fire multiple fasteners multiple times without requiring the loading unit 20287 to be removed from the surgical site and reloaded.

[0161] The first jaws and the second jaws 20291, 20290 may be configured to clamp tissue between them, fire fasteners through the clamped tissue, and cut the clamped tissue. The first jaw 20291 may be configured to fire at least one fastener multiple times, or to include a replaceable multi-fire fastener cartridge containing multiple fasteners (e.g., staples, clips, etc.) which may be fired multiple times before being replaced. The second jaw 20290 may include an anvil that deforms or otherwise fixes the fasteners as they are ejected from the multi-fire fastener cartridge.

[0162] The handle 20297 may include a motor connected to the drive shaft so as to affect the rotation of the drive shaft. The handle 20297 may include a control interface for selectively operating the motor. The control interface may include buttons, switches, levers, sliders, touchscreens, and any other suitable input mechanisms or user interfaces, which can be engaged by a clinician to start the motor.

[0163] The control interface of the handle 20297 may communicate with the controller 20298 of the handle 20297 to selectively actuate the motor to affect the rotation of the drive shaft. The controller 20298 may be located within the handle 20297 and may be configured to receive input from the control interface and adapter data from adapter 20285 or loading unit data from loading unit 20287. The controller 20298 can analyze the input from the control interface and the data received from adapter 20285 and / or loading unit 20287 to selectively actuate the motor. The handle 20297 may also include a display that a clinician can view while using the handle 20297. The display may be configured to show portions of adapter or loading unit data before, during, or after firing of the instrument 20282.

[0164] Adapter 20285 may include an adapter identification device 20284 located within it, and loading unit 20287 may include a loading unit identification device 20288 located within it. Adapter identification device 20284 may communicate with controller 20298, and loading unit identification device 20288 may communicate with controller 20298. It will be understood that loading unit identification device 20288 may communicate with adapter identification device 20284, which relays or passes communication from loading unit identification device 20288 to controller 20298.

[0165] Adapter 20285 may also include a plurality of sensors 20286 (shown as an example) positioned around it to detect various states of Adapter 20285 or the environment (e.g., when Adapter 20285 is connected to a loading unit, when Adapter 20285 is connected to a handle, when the drive shaft is rotating, the torque of the drive shaft, the strain of the drive shaft, the temperature inside Adapter 20285, the number of times Adapter 20285 is fired, the peak force of Adapter 20285 during firing, the total amount of force applied to Adapter 20285, the peak recoil force of Adapter 20285, the number of pauses of Adapter 20285 during firing, etc.). The plurality of sensors 20286 may provide input to Adapter Identification Device 20284 in the form of data signals. The data signals from the plurality of sensors 20286 may be used to update Adapter data stored in Adapter Identification Device 20284, or to update Adapter data stored in Adapter Identification Device 20284. The data signals from the plurality of sensors 20286 may be analog or digital. Multiple sensors 20286 may include force gauges for measuring the force exerted on the loading unit 20287 during firing.

[0166] The handle 20297 and adapter 20285 can be configured to interconnect the adapter identification device 20284 and the loading unit identification device 20288 with the controller 20298 via an electrical interface. The electrical interface may be a direct electrical interface (i.e., including electrical contacts that engage with each other to transmit energy and signals between them). In addition, or instead, the electrical interface may be a non-contact electrical interface for wirelessly transmitting energy and signals between them (e.g., inductive transmission). It is also conceivable that the adapter identification device 20284 and the controller 20298 can wirelessly communicate with each other via a wireless connection separate from the electrical interface.

[0167] Handle 20297 may include transceiver 20283 configured to transmit instrument data from controller 20298 to other components of system 20280 (e.g., LAN 20292, cloud 20293, console 20294, or portable device 20296). Controller 20298 may also transmit instrument data and / or measurement data related to one or more sensors 20286 to surgical hub 20270, as shown in Figure 9. Transceiver 20283 may receive data from surgical hub 20270 (e.g., cartridge data, loading unit data, adapter data, or other notifications). Transceiver 20283 may receive data from other components of system 20280 (e.g., cartridge data, loading unit data, or adapter data). For example, controller 20298 may transmit instrument data to console 20294, including the serial number of an adapter attached to handle 20297 (e.g., adapter 20285), the serial number of a loading unit attached to adapter 20285 (e.g., loading unit 20287), and the serial number of a multi-shot clamp cartridge loaded in the loading unit. Console 20294 can then transmit data associated with the mounted cartridge, loading unit, and adapter, respectively (e.g., cartridge data, loading unit data, or adapter data), to controller 20298. Controller 20298 may display the message on the local instrument display, or transmit the message via transceiver 20283 to console 20294 or portable device 20296, displaying the message on display 20295 or the portable device screen, respectively.

[0168] Figures 11A to 11D show examples of wearable sensing systems, such as surgical sensing systems or patient sensing systems. Figure 11A is an example of a glasses-based sensing system 20300 that can be based on an electrochemical sensing platform. The sensing system 20300 can monitor sweat electrolytes and / or metabolites (e.g., real-time monitoring) using multiple sensors 20304 and 20305 that are in contact with the skin of the surgeon or patient. For example, the sensing system 20300 can use a current-measuring-based biosensor 20304 and / or a potentiometric-measuring-based biosensor 20305 integrated into the nose bridge pad of the glasses 20302 to measure current and / or voltage.

[0169] The current-measuring biosensor 20304 may be used to measure sweat lactate levels (e.g., in mmol / L units). Lactate is a product of lactic acidosis, which can occur due to decreased tissue oxygenation that may be caused by sepsis or hemorrhage. A patient's lactate level (e.g., higher than 2 mmol / L) can be used to monitor the development of sepsis, for example, during postoperative monitoring. The potentiometric biosensor 20305 can be used to measure potassium levels in a patient's sweat. A voltage follower circuit with an operational amplifier may be used to measure the potential signal between the reference electrode and the working electrode. The output of the voltage follower circuit may be filtered and converted to a digital value using an ADC.

[0170] Current measuring sensor 20304 and potentiometer measuring sensor 20305 may be connected to circuit 20303 located on each arm of the eyeglasses. Electrochemical sensors may be used for simultaneous real-time monitoring of sweat lactate and potassium levels. Electrochemical sensors may be screen-printed on stickers and placed on both sides of the nose pads of the eyeglasses to monitor sweat metabolites and electrolytes. Electronic circuit 20303 located on the arms of the eyeglass frame may include a wireless data transceiver (e.g., a low-energy Bluetooth transceiver) which can be used to transmit lactate and / or potassium measurement data to a surgical hub or intermediate device, and then transfer the measurement data to the surgical hub. The eyeglass-based sensing system 20300 may use a signal conditioning unit for filtering and amplifying the electrical signals generated from the electrochemical sensors 20305 or 20304, a microcontroller for digitizing the analog signals, and a wireless (e.g., low-energy Bluetooth) module for transferring data to a surgical hub or computing device, as shown in Figures 7B-7D.

[0171] Figure 11B shows an example of a wristband-type sensing system 20310 equipped with a sensor assembly 20312 (e.g., a photoplethysmography (PPG)-based sensor assembly or an electrocardiogram (ECG)-based sensor assembly). For example, in the sensing system 20310, the sensor assembly 20312 can collect and analyze arterial pulses at the wrist. The sensor assembly 20312 may be used to measure one or more biomarkers (e.g., heart rate, heart rate variability (HRV)). In the sensing system with a PPG-based sensor assembly 20312, light (e.g., green light) can pass through the skin. Some of the green light is absorbed by blood vessels, and some of the green light is reflected and can be detected by a photodetector. These differences or reflections are related to variations in tissue blood perfusion, and these variations can be used to detect cardiovascular information (e.g., heart rate). For example, the amount absorbed may vary depending on the blood volume. The sensing system 20310 can determine the heart rate by measuring the light reflectance as a function of time. HRV can be determined as the time-period variation (e.g., standard deviation) between the steepest signal gradients before the peak, known as the inter-beat interval (IBI).

[0172] In the case of a sensing system having an ECG-based sensor assembly 20312, a pair of electrodes can be placed in contact with the skin. The sensing system 20310 can measure the voltage across the pair of electrodes placed on the skin to determine the heart rate. In this case, the HRV can be measured as the time-period variation (e.g., standard deviation) between R peaks in a group of QRS complexes, known as the RR interval.

[0173] The sensing system 20310 can, for example, use a signal conditioning unit to filter and amplify the analog PPG signal, use a microcontroller to digitize the analog PPG signal, and use a wireless (e.g., Bluetooth) module to transfer the data to a surgical hub or computing device.

[0174] Figure 11C shows an exemplary ring sensing system 20320. The ring sensing system 20320 may include a sensor assembly (e.g., a heart rate sensor assembly) 20322. The sensor assembly 20322 may include a light source (e.g., a red or green light-emitting diode (LED)) and a photodiode for detecting reflected and / or absorbed light. The LED in the sensor assembly 20322 can shine light through a finger, and the photodiode in the sensor assembly 20322 can measure heart rate and / or oxygen levels in the blood by detecting changes in blood volume. The ring sensing system 20320 may include other sensor assemblies for measuring other biomarkers, e.g., a thermistor or infrared thermometer for measuring surface body temperature. The ring sensing system 20320 may use, for example, a signal conditioning unit for filtering and amplifying the analog PPG signal, a microcontroller for digitizing the analog PPG signal, and a wireless (e.g., low-energy Bluetooth) module for transferring data to a surgical hub or computing device, as shown in Figures 7B-7D.

[0175] Figure 11D shows an example of an electroencephalogram (EEG) sensing system 20315. As shown in Figure 11D, the sensing system 20315 may include one or more EEG sensor units 20317. The EEG sensor unit 20317 may include a plurality of conductive electrodes placed in contact with the scalp. The conductive electrodes may be used to measure small potentials that may occur outside the head due to neural activity in the brain. The EEG sensing system 20315 can measure biomarkers, such as delirium, by identifying specific brain patterns, such as deceleration or omission of post-dominant rhythms and loss of responsiveness to eye opening and closing. The EEG sensing system 20315 may have, for example, a signal conditioning unit for filtering and amplifying potentials, a microcontroller for digitizing electrical signals, and a wireless (e.g., low-energy Bluetooth) module for transferring data to a smart device, as shown in Figures 7B to 7D.

[0176] Figure 12 shows a block diagram of a computer-implemented patient / surgeon monitoring system 20325 for monitoring one or more patient or surgeon biomarkers before, during, and / or after a surgical procedure. As shown in Figure 12, one or more sensing systems 20336 may be used to measure and monitor patient biomarkers, for example, to facilitate patient preparation before a surgical procedure and recovery after a surgical procedure. Sensing systems 20336 may be used to measure and monitor surgeon biomarkers in real time, for example, to assist in surgical tasks by communicating relevant biomarkers (e.g., surgeon biomarkers) to a surgical hub 20326 and / or surgical devices 20337 to coordinate their functions. The surgical device functions that can be coordinated may include power level, forward speed, closing speed, load, latency, or other tissue-dependent operating parameters. Sensing systems 20336 can also measure one or more physical attributes related to the surgeon or patient. Patient biomarkers and / or physical attributes can be measured in real time.

[0177] The computer-implemented wearable patient / surgeon wearable sensing system 20325 may include a surgical hub 20326, one or more sensing systems 20336, and one or more surgical devices 20337. The sensing systems and surgical devices may be communicatively connected to the surgical hub 20326. One or more analysis servers 20338, for example, which are part of an analysis system, may also be communicatively connected to the surgical hub 20326. Although a single surgical hub 20326 is shown, it should be noted that the wearable patient / surgeon wearable sensing system 20325 may include any number of surgical hubs 20326 that can be connected to form a network of surgical hubs 20326 communicatively connected to one or more analysis servers 20338, as described herein.

[0178] For example, the surgical hub 20326 may be a computing device. The computing device may be a personal computer, laptop, tablet, or smart mobile device. For example, the computing device may be a client computing device for a cloud-based computing system. The client computing device may be a thin client.

[0179] In one example, the surgical hub 20326 may include a processor 20327 coupled to memory 20330 for executing instructions stored therein, storage 20331 for storing one or more databases such as an EMR database, and a data relay interface 20329 to which data is sent to an analysis server 20338. In one example, the surgical hub 20326 may further include an I / O interface 20333 having an input device 20341 (e.g., a capacitive touchscreen or keyboard) for receiving input from a user and an output device 20335 (e.g., a display screen) for providing output to the user. In one example, the input device and the output device may be a single device. The output may include data from queries entered by the user, suggestions for products or combinations of products to use in a given procedure, and / or instructions for actions to be performed before, during, and / or after the surgical procedure. The surgical hub 20326 may include a device interface 20332 for communicatively connecting surgical devices 20337 to the surgical hub 20326. In one embodiment, the device interface 20332 may include a transceiver capable of enabling one or more surgical devices 20337 to connect to the surgical hub 20326 via a wired or wireless interface using one of the wired or wireless communication protocols described herein. The surgical devices 20337 may include, for example, a powered stapler, an energy device or its generator, an imaging system, or other connection systems, such as a fume extractor, a suction irrigation device, or an air supply system.

[0180] In one example, the surgical hub 20326 may be communicatively connected to one or more surgeons and / or patient sensing systems 20336. The sensing system 20336 may be used to measure and / or monitor in real time various biomarkers related to the surgeon performing the surgical procedure or the patient on whom the surgical procedure is being performed. A list of patient / surgeon biomarkers measured by the sensing system 20336 is provided herein. In one example, the surgical hub 20326 may be communicatively connected to an environmental sensing system 20334. The environmental sensing system 20334 may be used to measure and / or monitor in real time environmental attributes such as temperature / humidity in the operating room, surgeon movements, and ambient noise in the operating room caused by the breathing patterns of the surgeon and / or patient.

[0181] When the sensing system 20336 and the surgical device 20337 are connected to the surgical hub 20326, the surgical hub 20326 may receive from the sensing system 20336, for example, as shown in Figures 7B to 7D, measurement data related to one or more patient biomarkers, physical conditions related to the patient, measurement data related to surgeon biomarkers, and / or physical conditions related to the surgeon. The surgical hub 20326 may associate the measurement data related to the surgeon with other relevant pre-operative data and / or data from the situation awareness system, for example, as shown in Figure 8, to generate control signals for controlling the surgical device 20337.

[0182] For example, the surgical hub 20326 can compare measurement data from the sensing system 20336 to one or more thresholds defined based on baseline values, preoperative measurement data, and / or surgical measurement data. The surgical hub 20326 can compare measurement data from the sensing system 20336 to one or more thresholds in real time. The surgical hub 20326 can generate notifications for display. For example, if the measurement data exceeds a defined threshold (e.g., greater than or less than), the surgical hub 20326 can send a notification to the human interface system for patient 20339 and / or the human interface system for the surgeon or HCP 20340. The decision of whether or not to send a notification to one or more of the human interface systems for patient 20339 and / or HCP 2340 may be based on the severity level associated with the notification. The surgical hub 20326 can also generate a severity level associated with the notification for display. The generated severity level may be displayed to the patient and / or the surgeon or HCP. For example, patient biomarkers to be measured and / or monitored (e.g., measured and / or monitored in real time) may be related to surgical procedure steps. For instance, biomarkers measured and monitored for transverse venous and arterial incisions in thoracic surgery may include blood pressure, tissue perfusion pressure, edema, arterial stiffness, collagen content, and connective tissue thickness, while biomarkers measured and monitored for lymph node dissection steps in surgical procedures may include monitoring the patient's blood pressure. For example, data on postoperative complications can be retrieved from an EMR database in storage 20331, and data on staple or incision line leakage can be directly detected or inferred by the situational awareness system. Surgical procedure outcome data can be inferred by the situational awareness system from data received from various data sources, including a surgical device 20337, a sensing system 20336, and a database in storage 20331 to which the surgical hub 20326 is connected.

[0183] The surgical hub 20326 can transmit measurement data and physical status data received from the sensing system 20336, as well as data related to the surgical device 20337, to the analysis server 20338 for processing. Each of the analysis servers 20338 may include memory and a memory-coupled processor capable of executing instructions stored therein to analyze the received data. The analysis servers 20338 may be connected via a distributed computing architecture and / or utilize a cloud computing architecture. Based on this paired data, the analysis system 20338 can determine the optimal and / or preferred operating parameters for various types of modular devices, generate adjustments to the control programs for the surgical devices 20337, and transmit (or "push") the updated or controlled programs to one or more surgical devices 20337. For example, the analysis system 20338 can correlate perioperative data received from the surgical hub 20236 with measurement data related to the surgeon's physiological state or HCP and / or the patient's physiological state. The analysis system 20338 can determine when to control the surgical device 20337 and send an update to the surgical hub 20326. The surgical hub 20326 can then transfer the control program to the relevant surgical device 20337.

[0184] Further details regarding the computer-implemented wearable patient / surgeon wearable sensing system 20325, which includes a surgical hub 30326, one or more sensing systems 20336, and various surgical devices 20337 that can be connected thereto, are described in relation to Figures 5 to 7D.

[0185] For example, a computing system such as a surgical computing system or surgical hub described herein with reference to Figures 1A, 2A-2B, 3, 5, 6A-6B, 7B-7D, 9, and 12 may scan an in-operating sensing system such as a sensing system described herein with reference to Figures 1A-B, 2A-C, 3, 4, 5, 6A-C, 7B-D, 9, 11A-D, and 12. The computing system may establish a connection with the in-operating sensing system. The computing system may receive data relating to one or more users in the operating room. The received data may be or include user role identification data and / or data for identifying user roles associated with users in the operating room. Based on the received data, the computing system may identify one or more users in the operating room. For example, the computing system may identify a user based on one or more of the following: the user's proximity to surgical instruments, user location tracking information in the operating room, user interactions, one or more procedural activities, or user visual data in the operating room. Users in the operating room may be identified as patients, surgeons, nurses, staff, and / or healthcare professionals (HCPs). A computing system can identify user roles associated with users in the operating room (e.g., each user). For example, a computing system can distinguish users based on received data. If the computing system identifies user roles for users in the operating room, it can generate surgical assistance information. The generated surgical assistance information may be associated with (e.g., unique to) the identified users and / or identified user roles.

[0186] In the example, a computing system may receive measurement data from a sensing system. A computing device may receive measurement data from a sensing system using an established link. The computing system may determine an elevated stress level associated with an identified user. The computing system may acquire surgical context data. For example, the computing system may acquire surgical context data from a sensing system associated with surgical instruments (e.g., usage data) and / or the user. The computing system may identify surgical instruments that the user may be using. The computing system may determine whether the identified user is operating a surgical instrument. For example, the computing system may determine whether the identified user is operating and / or using a surgical instrument. Based on the determination that the identified user is not operating a surgical instrument, and based on measurement data indicating that the identified user has an elevated stress level, the computing system may transmit surgical assistance information to the identified user. The surgical assistance information may be, or may include, operating manuals for surgical instruments and / or instructions on how to use surgical instruments. The determination of stress levels is further described in the concurrently filed patent application entitled “ADAPTABLE SURGICAL INSTRUMENT CONTROL,” agent reference number END9290USNP2, which is incorporated herein by reference in its entirety.

[0187] For example, a computing system may receive measurement data from one of the sensing systems associated with a user in the operating room (e.g., a sensing system associated with the surgeon). The computing system may also receive measurement data from one of the sensing systems associated with a user in the operating room indicating a higher stress level of the user. For example, a higher stress level may be indicated by a change in the user's heart rate from a baseline. The computing system can derive this inference by cross-referencing the data received from the corresponding sensing systems. The computing system can transmit surgical assistance information to the identified user, as described herein.

[0188] In the example, the computing system may receive measurement data from a sensing system. The computing system may receive measurement data from a sensing system using an established link. The computing system may determine an elevated fatigue level associated with an identified user. The computing system may acquire surgical context data. For example, the computing system may acquire surgical context data from a sensing system associated with surgical instruments (e.g., usage data) and / or the user. The computing system may identify surgical instruments that the user may be using. The computing system may determine whether the identified user is operating a surgical instrument. As described herein, the computing system may determine, based on context data, whether the identified user is operating a surgical instrument. Based on context data, the computing system may determine whether the identified user is using a surgical instrument. Based on the determination that the identified user is operating a surgical instrument, and based on measurement data indicating that the identified user has an elevated fatigue level, the computing system may transmit surgical assistance information to the identified user. The surgical assistance information may be, or may include, instructions for fatigue control of surgical instruments. The determination of fatigue levels is further described in concurrently filed patent application “ADAPTABLE SURGICAL INSTRUMENT CONTROL,” with agent reference number END9290USNP2, which is incorporated herein by reference in its entirety.

[0189] For example, a computing system may receive measurement data from one of the sensing systems associated with a user in the operating room (e.g., a sensing system associated with a surgeon). The measurement data may indicate that the user, such as a surgeon, is overcompensating for perceived errors, which may be called overcorrection of input changes. The computing system may interpret repeated corrections, overcorrections, or vibrational responses as indicators of fatigue and / or elevated fatigue levels associated with the identified user.

[0190] A computing system can be configured to analyze usage data and / or measurement data to determine whether a user working in the operating room is experiencing fatigue, and if so, to modify the operation of surgical instruments and / or provide notifications related to the fatigue level. For example, a computing system can monitor user input to (e.g., from the surgical instruments and / or from sensing systems) surgical instruments. User input to surgical instruments may include inputs that cause shaking of the surgical instruments. Shaking, whether intentional or otherwise, can be detected by one or more sensing systems (e.g., accelerometers) that provide data on the movement and orientation of the surgical instruments. The detected data may indicate the magnitude and frequency of any tremor. Surgical instruments may generate usage data related to the monitored user input. Usage context data may indicate input to the surgical instruments, including, for example, the movement of all or part of the surgical instruments, including shaking. Usage data may be communicated to the computing system.

[0191] Data can be collected from sensing systems applicable to users of surgical instruments and other medical professionals who can assist in the operating room. Accelerometers can be applied to the user's hand, wrist, and / or arm. Accelerometers can also be applied to the user's torso to collect data related to body movement, including sway and tremor. Accelerometers can generate data on the movement and orientation of the user's hand and / or arm. The data can indicate the magnitude and frequency of movement, including sway. Sensing systems (which may or may include accelerometers) can collect biomarker data from the user, including data related to heart rate, respiration, temperature, etc. Sensing systems can collect data related to hydration / dehydration of corresponding users operating surgical instruments and other users assisting in the operating room. The collected data can be communicated to a computing system.

[0192] The computing system may receive usage data from surgical instruments and sensor data from a sensing system corresponding to a user in the operating room. The computing system may identify and / or store the received data in association with timestamp data indicating the time the data was collected in relation to the user.

[0193] The computing system can determine the fatigue level of users operating surgical instruments and assisting in the operating room based on received usage data and / or sensor data. The computing system can determine the duration associated with surgical procedures based on received usage data and sensor data. For each user, the computing system can determine values ​​related to time spent in the operating room, time spent standing in the operating room, and time spent physically moving themselves. The computing system can determine the user's fatigue level based on the time spent in surgery.

[0194] The computing system can determine physical indicators of fatigue based on received usage data and / or sensor data. The computing system can determine that a user is fatigued if the received data indicates that the user is swaying or unstable. The computing system can also determine that a user is fatigued if the received data indicates that tremors are being exhibited by the user.

[0195] The computing system can determine values ​​related to hydration / dehydration of users in the operating room based on received usage data and sensor data. Dehydration can affect energy levels, causing fatigue and exhaustion. Low body fluid levels tend to increase heart rate. The computing system can analyze heart rate data in relation to hydration levels and distinguish stress and other cardiac elevation events from hydration. Using baseline scales, the computing system can distinguish acute events from ongoing chronic events and differentiate fatigue and dehydration associated with each user in the operating room.

[0196] The computing system can calculate a weighted scale of fatigue for users operating surgical instruments and other users in the operating room. The weighted scale of fatigue can be based on cumulative coordinated events and contributions. For example, the weighted scale of fatigue can be based on the intensity of stress experienced by the user when controlling actuators such as time-dependent closure triggers and the force applied by the user over time.

[0197] If the computing system determines that the user is experiencing fatigue, it may decide to communicate control functions to the surgical instrument to perform fatigue control or adaptation and adjust its operation to compensate for the fatigue. Control functions for performing fatigue control may indicate a reduction in the force required to perform the operation. For example, the control mechanism may be instructed to reduce the force required to apply to the closing trigger to actuate the clamp jaws of the surgical instrument. The control functions may be instructed to increase the sensitivity of the closing trigger. The control functions may be instructed to increase the delay or waiting time in response to user input. The control functions may be instructed to slow down the operation and provide additional time before it acts.

[0198] If the computing system determines that the user has experienced fatigue, the computing system may also decide to communicate control functions to provide fatigue notifications. The computing system may determine that fatigue notifications may be provided to the user by surgical instruments. The computing system may determine that notifications may provide the operator with more steps of use. The computing system may also determine that it may provide fatigue level notifications to persons in the operating room other than the medical professionals operating the instruments. Such notifications may be displayed on a display system in or near the operating room.

[0199] The computing system may communicate instructions for control functions related to fatigue control. The control mechanism may communicate with surgical instruments, or with other systems in the operating room, such as displays, which may be used to provide notifications.

[0200] Surgical instruments and displays may receive and provide notifications of control function instructions indicating that fatigue control is to be performed. The surgical instrument may determine to operate in accordance with fatigue control instructions. The instrument may reduce the force required to activate and / or operate the closing trigger. The surgical instrument may increase the delay or waiting time between requesting an action, such as applying force to the closing trigger, and performing the corresponding action, such as closing the jaws. The surgical instrument may slow down its operation in response to input, thereby providing the operator with more time to position the surgical instrument.

[0201] If the control features indicate that notifications are provided, surgical instruments may provide physical tactile and visual feedback. Displays may also provide visual feedback regarding fatigue. Notifications may provide steps to minimize the oversight of details.

[0202] Figure 13 shows an example of a flow for generating surgical assistance information for a user in the operating room. In 28105, a computing system (e.g., a surgical computing system) can scan for sensing systems. The computing system can scan for sensing systems located in the operating room. As described herein, the sensing systems may have measurement data associated with a user. For example, a user may be wearing a sensing system. The sensing system can monitor and / or sense the user's measurement data. As described herein, the sensing system can transmit user role identification data. The sensing system can transmit user role identification data to the computing system. The user role identification data may be, or may include, data associated with identifying the user role of a user in the operating room.

[0203] In 28110, the computing system can establish a link with the sensing system. The computing system can communicate with the sensing system using the established link. The sensing system can transmit data, such as user role identification data and / or measurement data, using the established link.

[0204] In 28115, the computing system may receive user role identification data. The computing system may receive user role identification data from the sensing system. The computing system may receive user role identification data from the sensing system using an established link. The user role identification data may be, or may include, data for identifying a user role associated with a user. A user role associated with a user may be, or may include, a patient, surgeon, nurse, HCP, hospital staff, etc. The user role identification data may be, or may include, the user's proximity to one or more surgical instruments, user location tracking information in the operating room, user interactions, one or more surgical procedure activities, or the user's visual data in the operating room.

[0205] In 28120, the computing system can identify the user role of a user in the operating room based on the received user role identification data. As described herein, the computing system can identify, based on the user role identification data, that the user role associated with a user in the operating room is a surgeon, that the user role associated with another user in the operating room is a nurse (e.g., head nurse), and that the user role associated with yet another user in the operating room is hospital staff and / or HCP.

[0206] In the example, user role identification data may be, or may include, data relating to the user's proximity to surgical instruments. A computing system can identify a user's user role in an operating room based on the user's proximity to one or more surgical instruments. For example, a computing system can identify a user's user role as a surgeon. The computing system can know that the surgeon is in proximity to (e.g., next to) one or more surgical instruments. For example, since a surgeon uses one or more surgical instruments for a surgical procedure, the surgeon may be in proximity to (e.g., next to) the surgical instruments. A computing system can identify a user's user role as a nurse (e.g., head nurse) when the nurse is assisting a surgeon and / or is in proximity to (e.g., next to) one or more surgical instruments. For example, the computing system can know that the nurse is in proximity to (e.g., next to) the surgical instruments because the nurse may hand them to the surgeon based on a request from the surgeon. A computing system can determine a user's user role as hospital staff and / or HCP. Hospital staff and / or HCPs may handle non-surgical related activities and do not need to be in close proximity to one or more surgical instruments. For example, hospital staff and / or HCPs may be near the entrance to the operating room, a telephone, a clock, a music player, etc., and not in close proximity to (e.g., adjacent to) one or more surgical instruments.

[0207] In the example, user role identification data may be, or include, data associated with user location tracking information within the operating room. For example, a user in the operating room may be positioned and / or located at a specific location within the operating room. A patient may be located in the center of the operating room. A patient may be located under the central lighting of the operating room (e.g., directly below). A patient may remain stationary (e.g., not moving) throughout the entire surgical procedure. A computing system may identify a user in the operating room as a patient based on their location within the operating room (e.g., in the center, under the central lighting, etc.) and / or tracking information (e.g., lack of movement). A surgeon may be located in close proximity to the patient (e.g., next to). A surgeon may be located in close proximity to the operating table and / or the patient. A surgeon may be located in close proximity to one or more surgical instruments (e.g., next to). A surgeon may remain stationary and / or not move (e.g., not walking around the operating room). The computing system can identify a user as a surgeon based on their location near a patient, an operating table, or at least one of one of the following: one or more surgical instruments and / or tracking information (e.g., no movement or little movement). A nurse assisting a surgeon (e.g., a head nurse) may be in close proximity to the surgeon (e.g., next to them). The nurse may be next to a tray table with one or more surgical instruments. The nurse may move from the operating table to the tray table. The computing system can identify a user as a nurse based on location information and / or tracking information. Hospital staff and / or HCPs may be located further away from the operating table, surgeons, and / or head nurses. For example, hospital staff and / or other HCPs may be located near the operating room door and / or a telephone located in the operating room. The computing system can identify a user as hospital staff and / or an HCP based on location information and / or tracking information provided in the user role identification data.

[0208] In the example, user role identification data may be, or may include, data related to interactions between users in the operating room. A surgeon may communicate with and / or give orders to other users in the operating room. A surgeon may request surgical instruments for surgical procedures. A surgeon may request that the music played in the operating room be increased, decreased, and / or changed. Nurses assisting the surgeon may respond to requests from the surgeon. For example, a nurse may hand surgical instruments to the surgeon after the surgeon has requested them. Hospital staff and / or other HCPs may increase, decrease, and / or change the music based on requests from the surgeon.

[0209] In the example, user role identification data may be, or may include, data related to one or more surgical procedure activities. A sensing system associated with a user can sense and / or monitor the user's activities. In the example, a surgeon may wear a sensing system on their wrist. The sensing system can detect, measure, and / or sense the surgeon's hand movements. The sensing system can transmit the measured data of the surgeon's hand movements to a computing system. Based on the measured data of the surgeon's hand movements, the computing system can identify that the measured data is associated with the surgeon's user role. For example, the computing system can determine that the measured data indicates the user role of a user who uses one or more surgical instruments and / or performs a surgical procedure. The computing system can identify the user role associated with the user as a surgeon. In the example, a nurse may wear a sensing system on their wrist. The sensing system can detect, measure, and / or sense the nurse's hand movements as they carry and / or handle surgical instruments. The sensing system can transmit the measured data of the nurse's hand movements to a computing system. Based on the measured data of the nurse's hand movements, the computing system can identify that the measured data is associated with the nurse's user role. For example, a computing system may determine that measurement data includes a user passing one or more surgical instruments to another user in the operating room. The computing system may identify a user role associated with a nurse assisting the surgeon. In the example, a hospital staff member and / or HCP may be wearing a sensing system on their wrist. The sensing system can detect, measure, and / or sense the hand movements of the hospital staff member and / or HCP. For example, the sensing system could detect a hospital staff member and / or HCP answering a phone in the operating room, or adjusting the volume of a music player in the operating room. Based on the measurement data from the sensing system, the computing system may identify the user's user role as a hospital staff member and / or HCP.

[0210] In the example, user role identification data may be, or may include, data associated with the user's visual data in the operating room. The operating room may be equipped with a camera. The computing system may receive camera footage from the camera. Based on the camera footage, the computing system can determine / identify the user in the operating room. In the example, the computing system can perform facial recognition of the user. In the embodiment, the computing system can determine / identify the user in the operating room by scanning the user's user badge and / or identification tag. The computing system can identify the user role of the user in the operating room based on the camera footage.

[0211] In 28125, the computing system may generate surgical assistance information for a user based on the identified user role. For example, if the computing system identifies the user role of a user as a surgeon, the computing system may generate surgical assistance information for a surgeon. For example, if the computing system identifies the user role of a user as a nurse, the computing system may generate surgical assistance information for a nurse. For example, if the computing system identifies the user role of a user as hospital staff and / or HCP, the computing system may generate surgical assistance information for hospital staff and / or HCP. The surgical assistance information may also be augmented reality (AR) content. The computing system may generate AR content for the identified user.

[0212] In the example, a computing system may generate AR content for a surgeon. The computing system may display the AR content on a computing system associated with the surgeon. A computing system associated with a user (e.g., a display AR device) may display the AR content generated from the surgical computing system. The AR content can assist the surgeon in a surgical procedure. In the example, the AR content may be, or include, the surgical steps that the surgeon is about to perform. In the embodiment, the AR may include patient measurement data. The generated AR content can be converted to audio and transmitted to an audio AR device worn by the surgeon.

[0213] In the example, a computing system may receive measurement data from a sensing system. The measurement data may be, or may include, a stress level related to a user. For example, the measurement data may be, or may include, a stress level related to a surgeon. The measurement data may be, or may include, an elevated stress level related to a surgeon. As described herein, the computing system can determine an elevated stress level related to a user (e.g., a surgeon). The computing system can acquire surgical context data. For example, a surgical instrument may transmit data related to the use of the surgical instrument. The computing system determines whether the surgeon is operating the surgical instrument based on the surgical context data. The computing system can determine whether the surgeon is operating the surgical instrument based on the surgical context data and / or measurement data related to the surgeon (e.g., measurement data related to the surgeon's hand movements). If the computing system determines that the surgeon is not operating the surgical instrument and detects an elevated stress level, the computing system can generate and / or transmit surgical assistance information to the surgeon. The surgical assistance information may be, or may include, an operating manual for a surgical instrument. Surgical support information may include, or may contain, instructions (e.g., video or audio) on how to use surgical instruments.

[0214] In the example, the computing system may receive measurement data from the sensing system. The measurement data may be, or may include, a user-related stress level. For example, the measurement data may be, or may include, a nurse-related stress level. The measurement data may be, or may include, an elevated stress level related to the nurse. The computing system may acquire surgical context data. For example, a surgical instrument may transmit data related to the use of the surgical instrument. The computing system may determine whether or not the nurse is operating the surgical instrument. For example, the computing system may acquire context data indicating that a surgical staple gun has recently been fired and requires reloading. Based on the context data and / or measurement data related to the nurse (e.g., measurement data related to the nurse's hand movements), the computing system may determine whether or not the nurse is operating the surgical instrument. If the computing system determines that the nurse is not operating the surgical instrument and detects an elevated stress level, the computing system may generate and / or transmit surgical assistance information to the nurse. The surgical assistance information may be, or may include, a surgical instrument operation manual (e.g., reloading the staple gun). Surgical support information may include, or may contain, instructions (e.g., video or audio) regarding the use of surgical instruments (e.g., reloading).

[0215] In the example, a computing system may receive measurement data from a sensing system. The measurement data may be, or may include, a user-related fatigue level. For example, the measurement data may be, or may include, a surgeon-related fatigue level. The measurement data may be, or may include, an elevated fatigue level related to the surgeon. As described herein, the computing system can determine an elevated fatigue level related to a user (e.g., a surgeon). The computing system can acquire surgical context data. For example, a surgical instrument may transmit data related to the use of the surgical instrument. The computing system can determine whether or not the surgeon is operating a surgical instrument. The computing system can determine whether or not the surgeon is operating a surgical instrument based on context data and / or measurement data related to the surgeon (e.g., measurement data related to the surgeon's hand movements). If the computing system determines that the surgeon is not operating a surgical instrument and detects an elevated fatigue level, the computing system can generate and / or transmit surgical assistance information to the surgeon and / or a computing system related to the surgeon. The surgical assistance information may be, or may include, instructions for fatigue control of the surgical instrument.

[0216] A surgical computing system can identify users in the operating room (e.g., contextually). As described herein, a surgical computing system can identify users based on sensing systems and / or computing systems associated with the user. Based on the identification of the sensing systems and / or computing systems associated with the user, the surgical computing system can determine who the person is, their user role in the surgical procedure (e.g., as a whole), and / or their user role in the current surgical step.

[0217] In the embodiment, the user can check in to the surgical computing system. The user can check in to the surgical computing system while in the operating room. The user can check in to the surgical computing system during the check-in procedure.

[0218] A user can scan a sensing system and / or computing system associated with them when they enter the operating room. For example, a user can scan and / or tag a sensing system and / or computing system (e.g., an AR device) to a device such as a scanning device associated with a surgical computing system. The surgical computing system can receive scanning information about the sensing system and / or computing system associated with the user (e.g., the user wearing the sensing system and / or computing system). The surgical computing system can identify and / or recognize the user based on the scanned information. The surgical computing system can determine the user's role in the surgical procedure.

[0219] For example, a user may wear a computing system and / or sensing system on their wrist. When a user enters the operating room, they may place the computing system and / or sensing system in front of a scanning device and scan the computing system and / or sensing system. The surgical computing system may receive the scanned information. The scanned information may be, or may include, employee identification associated with the user, such as name, occupation, badge number, number of hours worked, and / or other personal data associated with the user. Based on the scanned information, the surgical computing system may determine the user's user role. For example, the surgical computing system may determine that the scanned user's user role is surgeon, nurse, hospital staff, and / or HCP for surgical procedures. If the surgical computing system requires additional information, it may request that the user provide it. When the surgical computing system identifies a user in the operating room, it may select, identify, and / or assign a user role associated with the identified user. The selected, identified, and / or assigned user role may be associated with the user's tasks for surgical procedures.

[0220] For example, a user may enter an operating room and go to a designated spot (e.g., in front of a monitor, near an operating table, a surgical tray, and / or adjacent to surgical instruments). The surgical computing system may detect and / or identify computing systems and / or sensing systems associated with the user based on the user's location information and / or location tracking information and / or the user's proximity to one or more surgical instruments described herein. The surgical computing system may identify computing systems and / or sensing systems and / or identify user roles associated with the user.

[0221] In the example, the user can enter user identification information (e.g., manually). For example, the user can enter user identification information into the surgical computing system when entering the operating room, before a surgical procedure, and / or when prompted by the surgical computing system. The user can enter their name, employee ID, badge number, and / or other user identifier information that identifies them.

[0222] In the embodiment, the surgical computing system may have a list of users in the operating room, for example, from preoperative plans submitted by surgeons and / or surgical plans submitted by HCPs related to surgical procedures. The surgical computing system can prompt users to select a user from the list to identify them. The surgical computing system can identify user roles associated with the identified users for the surgical procedure.

[0223] A surgical computing system can identify users in the operating room based on contextual information and identify user roles associated with those users. For example, contextual information may include, or may include, the type of procedure, the procedure steps, user activities, user location tracking information within the operating room, and the user's proximity to one or more surgical instruments.

[0224] In the embodiment, the surgical computing system can identify users for surgical procedures from surgical plans and / or HCPs submitted by surgeons before surgery and / or for surgical procedures. The surgical computing system can know the type of surgical procedure for the surgical procedure. The surgical computing system may have a list of surgeons who can perform the surgical procedure, for example, based on the surgeons' expertise and / or their shift schedules. The surgical computing system may have a list of nurses and / or HCPs who work with surgeons, for example, based on previous surgical plans and / or shift schedules. The surgical computing system retrieves the list from the hospital server. Based on the list, the surgical computing system can identify users for surgical procedures.

[0225] In the embodiment, the surgical computing system can identify users in the operating room based on the current surgical procedure step. The surgical computing system can identify a user as a surgeon if the current surgical procedure step is performed by a surgeon (e.g., successfully). For example, the surgical computing system can identify a user as a surgeon if the current surgical procedure step involves making an incision in the patient's chest. The surgical computing system can identify a user as a nurse if the current surgical procedure step requires reloading surgical instruments such as a surgical staple gun.

[0226] In the embodiment, a surgical computing system can identify a user in the operating room based on measurement data received from a sensing system. The measurement data may or may include the user's activities. For example, the measurement data may or may include the user's hand activities. Based on hand activities, such as performing surgery, the surgical computing system can identify the user as a surgeon. In the example, the surgical computing system can identify the user as a nurse based on hand activities including loading a surgical staple gun and / or moving (e.g., handling) one or more surgical instruments.

[0227] In the example, the surgical computing system can identify a user in the operating room based on location tracking information of the user in the operating room and the user's proximity to one or more surgical instruments described herein.

[0228] When a surgical computing system identifies a user in the operating room, the surgical computing system may generate and / or transmit (e.g., dedicated) surgical assistance information related to the identified user. This surgical assistance information may include, or may contain, instructions regarding surgical procedures, instructions on how to perform surgical procedures, and instructions on how to use surgical instruments. Instructions may be voice and / or video. The surgical computing system may transmit the generated surgical assistance information to the computing system associated with the identified user.

[0229] In the embodiment, the surgical computing system may generate surgical assistance information and / or surgical procedure assistance information, including transmitting voice commands related to the surgical procedure to a voice AR device worn by the surgeon. The surgical computing system may transmit video commands related to the surgical procedure to a video AR device worn by the surgeon. The surgeon can view and / or listen to the surgical assistance information to confirm the surgical procedure.

[0230] In the example, a surgical computing system can generate and / or transmit surgical assistance information, including voice and / or video instructions on how to reload a surgical staple gun, to a computing system worn by a nurse. The nurse can see and / or hear the surgical assistance information and properly reload the surgical staple gun.

[0231] In this example, a surgical computing system can generate and / or transmit surgical assistance information, including voice and / or video instructions, to speakers and / or monitors connected to the operating room. The voice and / or video instructions may be or may include instructions for critical steps. For example, the surgical computing system may broadcast that the next surgical procedure in a surgical operation is a critical step. The surgical computing system can broadcast surgical assistance information, and the user may, for example, pause the conversation to help the surgeon concentrate.

[0232] Users in the operating room may act in response to surgical assistance information (e.g., instructions) from a surgical computing system. For example, instructions may indicate the next surgical procedure step. Nurses may prepare surgical instruments for the next surgical procedure step. Hospital staff and / or HCPs may adjust the lighting during surgery to provide focus and / or highlighting of the area for the surgical procedure, for example.

[0233] A surgical computing system may provide surgical assistance information, including instructions for controlling the fatigue of surgical instruments, based on information about the identified user. For example, the surgical computing system may recognize the user's experience level, preferences, tendencies, and outcomes. Based on user-related information, the surgical computing system may use the surgical assistance information to include and / or recommend device settings for the next surgical step.

[0234] In the embodiment, user-related information (e.g., experience level, preferences, tendencies, results, etc.) can be retrieved from a hospital database. For example, a surgical computing system can connect to a hospital database and retrieve information about an identified user.

[0235] In this example, user-related information may be transmitted (e.g., relayed) to the surgical computing system by the user-related computing system. For example, the user-related computing system may provide user information to the surgical computing system during the check-in procedure and / or after the surgical computing system and the computing system have established a link.

[0236] If the surgical computing system determines that a user, for example, a surgeon, is a first-year resident of the surgical procedure and / or is new, the surgical computing system may provide surgical support information step by step. This surgical support information may include recommendations based on nominal historical data from the hospital database and / or server. The surgical support information may include recommendations based on nominal historical data from the hospital database and / or server, instead of and / or in addition to the user's historical data.

[0237] If a surgical computing system determines that the user, for example, a surgeon, is experienced in and / or an expert in surgical steps, the surgical computing system may provide surgical support information less frequently.

[0238] As described herein, a surgical computing system may receive measurement data from a user-related sensing system. The surgical computing system can use the measurement data to adjust fatigue control instructions for surgical instruments.

[0239] In the embodiment, the surgical computing system may receive measurement data from the surgeon. The measurement data may be, or include, stress levels and / or fatigue levels. As described herein, the surgical computing system may determine whether the stress levels and / or fatigue levels have increased. Based on the determination that the surgeon has elevated stress levels and / or fatigue levels, the surgical computing system may communicate fatigue control instructions to the surgical instruments. For example, the surgical computing system may slow down functions (e.g., joint speed, jaw closure, etc.) and improve precision. If the surgical computing system detects elevated stress levels and / or fatigue levels, and the surgical step is a critical step, the surgical computing system may communicate fatigue control instructions to the surgical instruments.

[0240] The determination of stress levels and / or fatigue levels is further described in the concurrently filed patent application entitled “ADAPTABLE SURGICAL INSTRUMENT CONTROL,” agent reference number END9290USNP2, which is incorporated herein by reference in its entirety.

[0241] A surgical computing system may communicate with user-related computing systems and / or sensing systems. A surgical computing system may communicate with one or more other surgical computing systems within the operating room. For example, one or more surgical computing systems may be present in the operating room. A surgical computing system (e.g., a master surgical computing system or a primary surgical computing system) may have greater processing power (e.g., the highest processing power) compared to one or more other surgical computing systems within the operating room. The primary surgical computing system may be connected to a network (e.g., the internet, hospital servers and / or databases, and / or a hospital cloud).

[0242] In the embodiment, the primary surgical computing system may configure one or more other surgical computing systems (e.g., slave surgical computing systems and / or secondary surgical computing systems). For example, one or more secondary surgical computing systems may be in idle mode and / or have processing power. If the primary surgical computing system determines that it requires additional processing power and / or needs to offload processing power (e.g., to perform additional analysis and / or to provide additional steps and / or procedures during operation), the primary surgical computing system may configure one or more secondary surgical computing systems to perform processing tasks. For example, the primary surgical computing system may identify one or more secondary surgical computing systems that are in idle mode (e.g., not used during the current surgical step) and / or have processing power. The primary surgical computing system may instruct one or more idle secondary surgical computing systems to perform offload processing tasks.

[0243] In the embodiment, the primary surgical computing system may be configured with one or more secondary surgical computing systems to acquire measurement data from one or more sensing systems associated with a user in the operating room. For example, the primary surgical computing system may establish links with sensing systems and / or computing systems associated with the user. The primary surgical computing system may assign secondary surgical computing systems to receive measurement data from linked sensing systems and / or data from linked computing systems. The primary surgical computing system may be configured with other secondary surgical computing systems to transmit fatigue control instructions to surgical instruments, as described herein.

[0244] In the embodiment, the primary surgical computing system may provide measurement data received by the primary surgical computing system to one or more secondary surgical computing systems. The primary surgical computing system may provide access to the received measurement data to one or more secondary surgical computing systems.

[0245] As described herein, a surgical computing system can be paired with one or more sensing systems and / or computing systems within the operating room. For example, the surgical computing system may query (e.g., actively query) other sensing systems and / or computing systems within the operating room to establish a link and / or access data. The surgical computing system can search for a compatible system to establish a link and gain access to data stored in the sensing systems and / or computing systems (e.g., measurement data and / or used identification data).

[0246] Based on an established link with one or more compatible sensing and / or computing systems, a surgical computing system can index and / or record the location and / or format of data. A surgical computing system (e.g., a primary surgical computing system) can transmit information (e.g., the location and / or format of data) to one or more secondary surgical computing systems.

[0247] A surgical computing system can store connections (e.g., network connections to other surgical computing systems, computing systems, and / or sensing systems within the operating room). For example, a surgical computing system can reuse stored connections (e.g., past network connections). A surgical computing system can use historical connection data as part of the setup for a new surgical procedure.

[0248] In the embodiment, the surgical computing system can establish links with sensing systems and / or computing devices associated with a user, such as a surgeon. Based on the links with sensing systems and / or computing devices, the surgical computing system can store a past list of sensing systems and / or computing systems with which the surgical computing system has established a connection. The surgical computing system can prompt the user to review a list of uploaded sensing systems and / or computing systems from the past list of systems. The user can select and / or deselect one or more sensing systems and / or computing systems from the past list.

[0249] The surgical computing system can use a past list to scan for sensing and / or computing systems that may be used in the current surgical procedure. The surgical computing system can update the list if one or more sensing and / or computing systems are missing. The surgical computing system can also update the list if one or more additional sensing and / or computing systems are detected.

[0250] A surgical computing system can look up a known list of systems that a user may frequently use. For example, if a surgeon has a known list of sensing systems (e.g., heart rate monitors, stress sensors, localization, etc.) and the surgical computing system establishes a link with one of the systems on the known list, the surgical computing system can prompt the connection and / or look up other sensing systems from the known list. Sensing systems from the list may be, or include, one or more sensing systems that have been previously connected to the surgical computing system.

[0251] In the embodiment, the surgical computing system may receive a known list of systems when it establishes a link with a sensing system and / or computing system. For example, the surgical computing system may send a connection request message and / or connection prompt to the sensing system. The sensing system may send a response to the sensing system's connection request message and / or connection prompt. The sensing system may include a list of other sensing systems and / or computing systems that the user has used in previous surgical procedures and / or established connections with the surgical computing system. The surgical computing system may use the list from the sensing system to scan for other systems and / or establish connections with other systems based on the list.

[0252] For example, if a sensing system and / or computing system associated with a user, such as a surgeon, has a known list of systems (e.g., sensing systems and / or computing systems), system identification can trigger the surgical computing system to prompt connection and / or retrieval of other sensing systems and / or computing systems (e.g., patient-specific). For example, a surgeon might prefer a particular set of sensing systems for a patient. The surgical computing system can use information about the surgeon's preferences to pre-populate a list of sensing systems for the patient. The surgical computing system can then scan and / or prompt connection to the patient's pre-populated list of sensing systems.

[0253] A computing system can search for one or more sensing systems within an operating room. For example, a computing system can actively search for one or more sensing systems that are close to it. The computing system may be located in an operating room. One or more sensing systems may contain measurement data related to a user. For example, a sensing system may be a surgeon sensing system that may contain measurement data related to a surgeon. A sensing system may be a patient sensing system that may contain measurement data related to a patient.

[0254] Figure 14 shows an exemplary flow of a computing system establishing links with compatible and / or incompatible sensing systems and / or computing systems. In 28205, the computing system can scan the operating room and identify one or more devices located in the operating room and in close proximity to the computing system. The computing system can determine whether the detected device is a sensing system. In the example, the computing system can request device identification information associated with the detected device. The computing system can retrieve the device identification information and determine whether the detected device is a sensing system. In the example, the computing system may receive sensing system instructions from the sensing system. In the embodiment, the computing system can establish links with other computing systems in the operating room. The other computing systems may have a list of one or more sensing systems in the operating room. The computing system can attempt to establish links with one or more sensing systems from the list.

[0255] In 28210, the computing system can determine compatibility in order to establish a link with a detected sensing system. If the computing system detects / identifies one or more sensing systems, the computing system can determine whether the detected / identified sensing systems are compatible with the computing system. For example, the computing system can determine whether one or more sensing systems are compatible with the computing system to establish a connection and / or share data.

[0256] In 28215, the computing system may generate a compatible virtual computing system to establish a link with an incompatible sensing system. If the computing system determines that the computing system and one or more sensing systems are incompatible to establish a link (e.g., connection), the computing system may generate a compatible virtual computing system to establish a link with one or more sensing systems. In the example, the virtual computing system may be, or include, an intermediate computing system compatible with one or more sensing systems (e.g., a virtual computing system configured to run by the computing system). In the example, the virtual computing system may be configured to function as a bridge or tunnel for establishing a connection between the computing system and one or more incompatible sensing systems. The computing system may establish a link with one or more incompatible sensing systems via the virtual computing system and receive measurement data as described herein. If the computing system determines that the computing system and one or more sensing systems are incompatible to establish a link, the computing system may generate a compatible virtual computing system to establish a link with one or more sensing systems. The computing system may establish a link with one or more incompatible sensing systems via the virtual computing system and receive measurement data as described herein.

[0257] In 28220, a computing system can establish a link with a sensing system. If the computing system determines that it and one or more sensing systems are compatible to establish a link (e.g., using a virtual computing system), the computing system can establish a link (e.g., a pair) with one or more sensing systems. The computing system can receive measurement data from one or more linked / paired sensing systems. For example, the computing system can receive measurement data from one or more paired sensing systems. For example, the computing system can monitor (e.g., passively monitor) measurement data from one or more paired sensing systems. The computing system can transmit measurement data and / or a list of monitored measurement data from one or more paired sensing systems to other computing systems. For example, the computing system can transmit measurement data and / or a list of monitored measurement data from one or more paired sensing systems to a primary computing system (e.g., a central computing system and / or a master computing system). For example, the computing system can communicate paired information to other computing systems, such as a primary computing system. When a computing system pairs with another computing system, and / or when requested, the computing system may periodically communicate paired information to the other computing system.

[0258] In 28225, a computing system may receive measurement data from one or more linked sensing systems. In the example, the computing system may store measurement data received from one or more paired sensing systems. The computing system may transmit the stored measurement data to other computing systems. The computing system may perform analysis of the measurement data, and / or other computing systems may perform analysis of the measurement data.

[0259] In this example, a computing system can send instructions to one or more paired sensing systems. These instructions may be requests and / or commands to send (e.g., directly) measurement data to other computing systems (e.g., a primary computing system and / or a secondary computing system).

[0260] A computing system can decide whether or not to connect with a new sensing system after establishing a link with one or more sensing systems. For example, a computing system can determine whether or not a new sensing system has entered an operating room. Based on the determination that a new sensing system has entered an operating room, the computing system can decide whether or not to pair with the new sensing system.

[0261] In this example, the computing system can decide, based on historical set data, whether to include and pair with a new sensing system, or to exclude and skip pairing with a new sensing system. For example, based on historical set data, the computing system can recognize that a user, such as a visiting nurse from the next operating room, may visit the current operating room at time intervals (e.g., every hour or every few minutes). Based on historical set data indicating that the user will leave the current operating room in a few minutes, the computing system can exclude the new sensing system associated with the user (e.g., the visiting nurse) and skip pairing with the sensing system. For example, the computing system can determine that a sensing system is associated with a user from a different operating room. If the computing system detects one or more sensing systems associated with a user from a different operating room, the computing system can exclude one or more sensing systems associated with the user from the different operating room (e.g., the visiting nurse) from establishing a link with the computing system. Based on the data, if the visiting nurse enters the current operating room (e.g., at time intervals), the computing system can exclude one or more sensing systems associated with the visiting nurse from establishing a link with the computing system (e.g., automatically exclude).

[0262] In this example, the computing system can search a list of sensing systems. For instance, the computing system can query the hospital central supply database and / or cloud database to determine whether a new sensing system belongs to a (e.g., relevant) user in the current operating room. Based on the determination that the new sensing system does not belong to an identified user in the current operating room, the computing system can remove the new sensing system from the pairing list and skip pairing with the new sensing system.

[0263] In the example, the computing system can determine whether a new sensing system is associated with a commercial sensing system. For example, the computing system can recognize that a new sensing system is associated with non-patient and / or non-HCP systems. The computing system can determine that a new sensing system does not match a list of sensing systems enumerated and / or approved by the hospital. The computing system can remove the new sensing system from the pairing list and skip pairing with the new sensing system.

[0264] One or more HCPs may enter the operating room for surgical procedures. HCPs may check in using a computing system, such as a surgical computing system (e.g., a primary surgical computing system). For example, an HCP may check in using the computing system when entering the room. Another example shows an HCP checking in using the computing system after entering the room and before the surgical procedure.

[0265] In the example, as described herein, the HCP can directly input their names into the computing system. In the example, the HCP can select / click the names displayed in the computing system. In the example, the HCP can tag badges, identification cards, and / or other identifiers. The computing system can find one or more sensing systems associated with the HCP based on the check-in information provided / performed by the HCP.

[0266] As described herein, a computing system can identify HCPs based on camera footage within the operating room. For example, a computing system can access camera footage within the operating room. Based on the camera footage, the computing system can identify HCPs within the operating room. In an example, the computing system can identify HCPs based on their location within the operating room. In an example, the computing system can identify HCPs based on their proximity to surgical instruments within the operating room.

[0267] In the example, if the computing system detects that a person is lying on an operating table, the computing system can identify that person as a patient. In the example, if the computing system detects a user standing next to and / or moving nearby the patient and / or the operating table, the computing system can identify the user as a surgeon. In the example, if the computing system detects a user near a monitor and / or a telephone, the computing system can identify the user as a nurse.

[0268] A computing system can retrieve one or more sensing systems associated with an identified user / HCP. For example, a computing system can access a database, such as a hospital central database, to retrieve one or more sensing systems associated with an identified patient, an identified surgeon, and / or an identified nurse. The hospital database may contain a list of sensing systems and the assignment of sensing systems to one or more HCPs. For example, the hospital database may contain a list of sensing systems assigned to and / or associated with surgeons. The hospital database may contain a list of sensing systems assigned to and / or associated with patients. The computing system can retrieve the list from the hospital database and identify one or more sensing systems associated with users in the operating room.

[0269] In the example, a computing system can determine one or more sensing systems in an operating room based on their network connectivity. One or more sensing systems may attempt to establish network connectivity when they enter the operating room. For example, one or more sensing systems may connect to a Wi-Fi network assigned to the operating room. Based on the Wi-Fi connection or the attempt to connect to Wi-Fi, the computing system can detect one or more sensing systems located in the operating room. As described herein, the computing system can identify and / or associate the detected one or more sensing systems with corresponding users in the operating room.

[0270] In the embodiment, the computing system can scan one or more sensing systems within the operating room. For example, one or more sensing systems may have Bluetooth and / or Zigbee connectivity. One or more sensing systems may be discoverable. The computing system can detect one or more sensing systems. As described herein, the computing system can identify and / or associate the discovered one or more sensing systems with corresponding users within the operating room.

[0271] A computing system may have information about one or more surgical instruments in the operating room. For example, a computing system may have a list of surgical instruments in the operating room. In this example, the computing system can retrieve a list of instruments in the operating room for the current surgery from a preoperative plan submitted by the surgeon and / or from the HCP associated with the surgeon. The preoperative plan may provide a list of instruments to be used in the surgery. The surgeon and / or HCP can upload the list to the computing system, hospital network, and hospital database. The computing system can then retrieve the list.

[0272] In an example, a HCP such as a nurse preparing for surgery may request and / or upload a list of surgical instruments for the surgery. The HCP can upload the surgical procedure's surgical plan. The computing system can search for a list of one or more surgical instruments and / or the surgical plan.

[0273] The computing system described herein can process offline data. The sensing system and / or the network of the sensing system can be connected to the network. For example, one or more sensing systems can be connected to the network via Wi-fi or the Internet on a mobile device. The Wi-fi or the Internet on the mobile device can go offline. The Wi-fi or the Internet can go offline due to one or more of insufficient connection, failure, battery depletion, and / or power outage. The computing system can process data (e.g., a data reservoir or a data transfer for processing that data) based on the last online interaction with the Wi-fi or the Internet.

[0274] The predicted values can be periodically uploaded to one or more sensing systems and / or mobile devices. For example, the predicted values may be uploaded to one or more sensing systems and / or mobile devices daily. The computing system can operate locally within a closed network of the device. For example, the computing system can learn patterns related to the user. The computing system can learn the user's timing, sleep schedule, and / or the user's normal marker values. The measurement data and / or values can provide context in a particular event. For example, eating can result in a spike in blood glucose levels within a certain range. For example, training can increase the heart rate (HR) by 20 - 30%.

[0275] Figure 15 shows an exemplary flow of a computing system operating online and offline. In 28305, the computing system may receive daily downloads of predictive measurement data (e.g., prominent biomarkers) for specific surgical procedures and / or complications labeled as high and / or medium risk. In 28310, the computing system may go offline. The computing system may send notifications (e.g., local notifications). Notifications may be, or include, "Please connect to an internet source." In 28315, the computing system may detect elevated measurement data from a sensing system (e.g., spiked biomarkers). In 28325, the computing system may perform local analysis. For example, the computing system may determine whether the elevated measurement data matches the expected daily download of measurement data (e.g., the local daily download expectation for biomarkers). In 28330, the computing system may determine whether the elevated measurement data is within the expected range. In 28335, the computing system may determine that the elevated measurement data is within the expected range. The computing system may display a message. The message may be, or may include, “Connect to an Internet source.” In 28340, the computing system may determine that the rising measurement data is outside the expected range. The computing system may send a local notification. For example, the computing system may display a message. The message may be, or may include, “Connect to Wi-Fi and / or an Internet source immediately.” In 28345, the computing system may come online. The computing system may send data and / or rising measurement data to the HCP.

[0276] The computing system can backlog when it comes back online. The computing system may prompt the user to identify one or more specific flags / concerns. When it comes back online, the computing system can query what events were occurring at that time (e.g., eating, sleeping). The information provided may indicate whether a problem exists or is occurring.

[0277] The computing system may have an offline mode. In offline mode, the computing system can search for triggers (e.g., specific spikes) within the measurement data (e.g., biomarkers). When the computing system returns online, it can perform analysis of the triggers. The computing system can prioritize data storage and / or prioritize analysis at specific time markers. Analysis can use more power and / or drain the battery faster. The computing system can switch analysis from the computing device to the cloud, for example.

[0278] One or more computing systems (e.g., slave computing systems) can migrate to one or more master computing systems. The migrated master computing systems can create a hub (e.g., a local hub) for analysis (e.g., low-level analysis). Figure 16 shows an example of a secondary computing system migrating to a primary computing system to create a local computing system for low-level analysis. A Level 2 computing system (e.g., a primary computing system and / or a master computing system) can receive data from one or more computing systems connected to a mobile device (e.g., from a mobile device) and / or receive a 48-bit address. If internet connectivity is lost and / or Bluetooth connectivity with a mobile device is lost, the Level 2 computing system can page one or more Level 1 computing systems (e.g., secondary computing systems and / or slave computing systems) to connect to the Level 2 computing system (e.g., automatically connect). A Level 2 computing system may have a unique address for other Level 1 computing systems. A local network of computing systems can be created, and if an emergency is detected and / or there is a problem, low-level analysis and / or local notifications can be performed via the Level 2 computing system. A Level 2 computing system may receive data from one or more computing systems connected to (e.g., from) a mobile device, and / or receive a 48-bit address. If an internet connection is lost and / or the Bluetooth connection with the mobile device is lost, the Level 2 computing system may page to one or more Level 1 computing systems to connect to the Level 2 computing system (e.g., automatically connect).A Level 2 computing system may have a unique address for other Level 1 computing systems. A local network of computing systems can be created, and if an emergency is detected and / or there is a problem, low-level analysis and / or local notification can be performed via the Level 2 computing system.

[0279] One or more sensing systems (e.g., wearable systems) may rely on externally supplied data for their operation regarding how they respond. For example, a sensing system may operate as if that data is unavailable for a short period. The system's response to the absence of an external connection may be time-dependent. For example, in the short term, the sensing system may use the last communicated value. If the sensing system is offline for a sufficiently long period (e.g., longer than a pre-configured short time interval), the sensing system may begin notifying the user. The sensing system may be able to operate in safe mode and / or other protected states by default.

[0280] If the sensing system's recording capacity reaches its maximum and the sensing system cannot connect to an external system to upload measurement data, the sensing system may overwrite older data. The sensing system may retain every other or every ten older data points and overwrite other data to continue recording (e.g., to create space for and / or further measurement data).

[0281] The sensing system may have one or more triggers to enhance the importance of connecting to an external system. Triggers may include irregularity and / or exceeding critical thresholds. If the sensing system is unable to connect to an external system (e.g., the outside world) when it needs to report, the sensing system may enhance notification to the user (e.g., the wearer) and / or provide instructions to the user on how to access the communication path and / or how to explore other means of access.

[0282] Surgical instruments (e.g., smart surgical instruments) may include one or more of the following: staplers, energy devices (e.g., advanced energy devices), biological aids, and / or computing systems.

[0283] Energy devices can send notifications to HCPs, such as surgeons. Energy devices can send potentially problematic data, next steps, and / or complications to HCPs.

[0284] For example, an energy device can detect bleeding and send a notification to the surgeon to adjust instrument handling. An exemplary notification to the surgeon may include bleeding (IMA-sigmoid colectomy). Warning: When the surgeon approaches the IMA, if the patient has low / high pH, ​​a harmonic of output level × is suggested due to the risk of bleeding.

[0285] Examples of what may trigger a notification from an energy device when it approaches major arterial resection and coagulation include one or more of the following: blood pH above 7.45, alcohol intake, and / or menstrual cycle.

[0286] Biological adjuvants may provide identification of patient escalation parameters that suggest the adjuvants or systems to be used.

[0287] The computing system may provide adjustment of operating thresholds. The computing system may highlight irregularities of one or more surgical instruments (e.g., combination surgical instruments) and patients. For example, the computing system may identify one or more surgical devices that may provide superior results, access, and / or functionality based on detected patent irregularities. The computing system may provide adjustment of data streams. For example, the computing system may link one or more sensing systems for measuring parameters (e.g., measurement data) with one or more surgical measurement devices (e.g., OR surgical measurement devices) for providing comparison and / or baseline data.

[0288] The following is a non-exhaustive list of examples shown above and / or in the drawings, which may or may not be claimed below.

[0289] Example 1: A computing system, Equipped with a processor, the processor is Scanning the sensing system located inside the operating room, Establishing a link (i.e., pair) with the sensing system, Receiving user role identification data from the sensing system using an established link, Based on the received user role identification data, the user role of the user in the operating room is identified, A computing system configured to generate and optionally display surgical assistance information for users in the operating room based on identified user roles.

[0290] Advantageously, the system can generate (e.g., unique) surgical assistance information that may be relevant to the identified user and / or identified user role, based on information acquired through the sensing system. Therefore, the burden on the user when attempting to generate such information is reduced.

[0291] Example 2: The user is the first user, the sensing system is the first sensing system, and the processor is configured to receive user role identification data from a second sensing system associated with a second user, identify the user role of the second user in the operating room based on the received user role identification data, further configured to determine surgical assistance information for the second user based on the identified user role of the second user, the computing system according to Example 1.

[0292] Advantageously, the surgical assistance information can be generated / determined for multiple users who may have different roles.

[0293] Example 3: The user role identification data includes at least one of the proximity of the user to one or more surgical instruments, the position of the first user in the operating room, the interaction between the user and at least one medical expert, one or more surgical treatment activities, or the visual data of the user in the operating room, the computing system according to Example 1 or 2.

[0294] Example 4: The sensing system is worn by the user, and optionally, the processor is configured to the proximity of the sensing system to one or more surgical instruments, position tracking information associated with the sensing system during the surgical procedure, or one or more surgical treatment activities detected by the sensing system, identify the user role of the first user as a surgeon based on at least one of them, the computing system according to any one of Examples 1 to 3.

[0295] Example 5: The processor is configured to generate augmented reality (AR) content of the identified user role, the AR content may include surgical assistance information, generate A computing system according to any one of Examples 1 to 4, configured to transmit AR content to an AR device associated with a user.

[0296] Advantageously, any surgical assistance information can be overlaid on another image or video, allowing the user to maintain focus on, for example, less of the display.

[0297] Example 6: The processor Receiving measurement data from the sensing system, Based on the received measurement data, the elevated stress level associated with the user is determined, Obtaining surgical context data, Identifying surgical instruments associated with a user based on surgical context data and identified user roles, A computing system according to any one of Examples 1 to 5, configured to obtain instructions on how to use surgical instruments for inclusion in surgical assistance information.

[0298] Advantageously, the user may be provided with instructions on how to use surgical instruments when the user is experiencing a high level of stress. Therefore, the user may be assisted during surgery with minimal user intervention.

[0299] Example 7: Surgical assistance information for the user includes instructions for fatigue control of surgical instruments, and the processor, Receiving measurement data from the sensing system, Based on the received measurement data, the elevated fatigue level associated with the user is determined, Obtaining surgical context data, Determining whether a user is operating a surgical instrument based on surgical context data and identified user roles, A computing system according to any one of Examples 1 to 6, configured to transmit fatigue control instructions to a surgical instrument based on a determination that the user is operating the surgical instrument.

[0300] Advantageously, a control program may be transmitted to the surgical instrument to control its actuators in such a way as to limit or compensate for fatigue and / or the use of fine motor skills.

[0301] Example 8: The computing system according to any one of Examples 1 to 7, wherein the user role includes at least one of the following: surgeon, nurse, patient, hospital staff, or medical professional.

[0302] Example 9: Computer implementation method, Scanning the sensing system located inside the operating room, Establishing a link (i.e., pair) with the sensing system, Receiving user role identification data from the sensing system using an established link, Identifying the user role of a user in the operating room based on the received user role identification data, A method comprising generating and optionally displaying surgical assistance information for a user in the operating room based on an identified user role.

[0303] Advantageously, the system can generate (e.g., unique) surgical assistance information that may be relevant to the identified user and / or identified user role, based on information acquired through the sensing system. Therefore, the burden on the user when attempting to generate such information is reduced.

[0304] Example 10: The user is the first user, the sensing system is the first sensing system, and the method is Receiving user role identification data from a second sensing system associated with a second user, Based on the received user role identification data, the user role for a second user in the operating room is identified, The method according to Example 9, further comprising determining surgical assistance information for a second user based on the identified user role of the second user.

[0305] Advantageously, surgical support information can be generated / determined for multiple users who may have different roles.

[0306] Example 11: The sensing system is attached by the user, and optionally, the method is The proximity of the sensing system to one or more surgical instruments, location tracking information related to the sensing system during a surgical procedure, or The method according to Example 9 or 10, comprising identifying the user role of a first user as a surgeon based on at least one of one or more surgical procedure activities detected by a sensing system.

[0307] Example 12: To generate augmented reality (AR) content for identified user roles, wherein the AR content may include surgical assistance information, The method according to any one of Examples 9 to 11, configured to transmit AR content to an AR device associated with a user.

[0308] Advantageously, any surgical assistance information can be overlaid on another image or video, allowing the user to maintain focus on, for example, less of the display.

[0309] Example 13: Receiving measurement data from the sensing system, Based on the received measurement data, the elevated stress level associated with the user is determined, Obtaining surgical context data, Identifying surgical instruments associated with a user based on surgical context data and identified user roles, The method according to any one of Examples 9 to 12, including obtaining instructions on how to use surgical instruments for inclusion in surgical support information.

[0310] Advantageously, the user may be provided with instructions on how to use surgical instruments when the user is experiencing a high level of stress. Therefore, the user may be assisted during surgery with minimal user intervention.

[0311] Example 14: User surgical assistance information includes instructions for fatigue control of surgical instruments, and the method is as follows: Receiving measurement data from the sensing system, Based on the received measurement data, the elevated fatigue level associated with the user is determined, Obtaining surgical context data, Determining whether a user is operating a surgical instrument based on surgical context data and identified user roles, The method according to any one of Examples 9 to 13, comprising transmitting fatigue control instructions to a surgical instrument based on the determination that a user is operating a surgical instrument.

[0312] Advantageously, a control program may be transmitted to the surgical instrument to control its actuators in such a way as to limit or compensate for fatigue and / or the use of fine motor skills.

[0313] Example 15: A computing system, Equipped with a processor, the processor is Scanning a sensing system within an operating room, wherein the sensing system is equipped with measurement data for the user. To determine whether the sensing system is compatible to establish a link with the computing system, Based on the determination that the sensing system is incompatible for establishing a link with the computing system, the process involves generating a virtual computing system that is compatible for establishing a link with the sensing system, Establishing a link with the sensing system using the generated virtual computing system, A computing system configured to receive measurement data using a link with a sensing system.

[0314] An advantage is that the computer system can receive measurement data from incompatible sensing systems.

[0315] Example 16: The processor is Before establishing a link with the sensing system, it is necessary to establish an initial link with the sensing system. The process involves transmitting an initial link instruction to the surgical computing system, wherein the initial link instruction requests user input to establish a link with the sensing system. Receiving user input from a surgical computing system, A computing system according to Example 15, configured to establish a link with a sensing system based on received user input.

[0316] Advantageously, users can control whether or not a link is established.

[0317] Example 17: The processor, The computing system according to Example 15 or 16, configured to establish a link with a sensing system based on the determination that the sensing system is suitable for establishing a link with a computing system.

[0318] Example 18: The sensing system includes a first sensing system, the link includes a first link, the measurement data includes first measurement data, and the processor is Establish a second link with a second sensing system that includes second measurement data for the user, It is configured to receive second measurement data from a second sensing system using a second established link, Optionally, the processor in Example 18 is: Based on the received first and second measurement data, the positions of the first and second sensing systems in the operating room, or at least one of the surgical procedures of a surgical operation, a decision is made as to whether or not to generate augmented reality (AR) content. A computing system according to any one of Examples 15 to 17, configured to generate AR content including display information associated with first measurement data and / or second measurement data based on a decision.

[0319] Advantageously, the computing system can establish several links with several different sensing systems. Furthermore, any information associated with the first measurement data and / or the second measurement data can be overlaid onto another image or video, for example, enabling the user to maintain focus on less of the display.

[0320] Example 19: The processor is The computing system according to Example 18, configured to transmit generated AR content to a user-associated AR device.

[0321] Example 20: The processor is Detecting multiple devices in the operating room, Identifying the sensing system within the operating room from multiple detected devices, A computing system according to any one of Examples 15 to 19, configured to select a sensing system to establish a link.

[0322] Advantageously, a computer system can be used to scan the environment (e.g., an operating room) for potential devices to establish a link with, identify sensing systems, and establish a link with them.

[0323] Example 21: A computer-readable medium containing instructions that, when executed by a computer, cause the computer to perform the method described in any one of Examples 9 to 14.

[0324] The following is a non-exhaustive list of embodiments described above and / or shown in the drawings, which may or may not be claimed below.

[0325] Embodiment 1. A surgical computing system, Equipped with a processor, the processor is Scanning the sensing system located inside the operating room, Establishing a link with the sensing system, Receiving user role identification data from the sensing system using an established link, Based on the received user role identification data, the user role of the user in the operating room is identified, A surgical computing system configured to generate surgical assistance information for users in the operating room based on identified user roles.

[0326] Embodiment 2. The user is the first user, the sensing system is the first sensing system, and the processor is Receiving user role identification data from a second sensing system associated with a second user, Based on the received user role identification data, the user role of the second user in the operating room is identified, The surgical computing system according to embodiment 1, further configured to determine surgical assistance information for a second user based on the identified user role of the second user.

[0327] Embodiment 3. The surgical computing system according to Embodiment 1, wherein the user role identification data includes at least one of the following: the user's proximity to one or more surgical instruments, the location of a first user in the operating room, interactions between the user and at least one medical professional, one or more surgical procedures, or visual data of the user in the operating room.

[0328] Embodiment 4. The sensing system is installed by the user, and the processor is The proximity of the sensing system to one or more surgical instruments, Location tracking information associated with a sensing system during surgical procedures, or A surgical computing system according to embodiment 1, configured to identify the user role of a first user as a surgeon based on at least one of one or more surgical procedure activities detected by a sensing system.

[0329] Appearance 5. The processor, To generate augmented reality (AR) content for identified user roles, A surgical computing system according to embodiment 1, configured to transmit AR content to an AR device associated with a user.

[0330] Embodiment 6. The processor Receiving measurement data from the sensing system, Based on the received measurement data, the elevated stress level associated with the user is determined, Obtaining surgical context data, Identifying surgical instruments associated with a user based on surgical context data and identified user roles, A surgical computing system according to embodiment 1, configured to obtain instructions on how to use surgical instruments for inclusion in surgical support information.

[0331] Embodiment 7. Surgical assistance information for the user includes instructions for fatigue control of surgical instruments, and the processor, Receiving measurement data from the sensing system, Based on the received measurement data, the elevated fatigue level associated with the user is determined, Obtaining surgical context data, Determining whether a user is operating a surgical instrument based on surgical context data and identified user roles, A surgical computing system according to embodiment 1, configured to transmit fatigue control instructions to a surgical instrument based on a determination that the user is operating a surgical instrument.

[0332] Embodiment 8. The surgical computing system according to Embodiment 1, wherein the user roles include at least one of a surgeon, nurse, patient, hospital staff, or medical professional.

[0333] Appearance 9. Method, Scanning the sensing system located inside the operating room, Establishing a link with the sensing system, Receiving user role identification data from the sensing system using an established link, Identifying the user role of a user in the operating room based on the received user role identification data, A method configured to generate surgical assistance information for a user in the operating room based on an identified user role.

[0334] Embodiment 10. The user is the first user, the sensing system is the first sensing system, and the method is Receiving user role identification data from a second sensing system associated with a second user, Based on the received user role identification data, the user role for a second user in the operating room is identified, The method according to aspect 9, further comprising determining surgical assistance information for a second user based on the identified user role of the second user.

[0335] Embodiment 11. The sensing system is installed by the user, and the method is The proximity of the sensing system to one or more surgical instruments, location tracking information related to the sensing system during a surgical procedure, or The method according to aspect 9, comprising identifying the user role of a first user as a surgeon based on at least one of one or more surgical procedure activities detected by a sensing system.

[0336] Appearance 12. To generate augmented reality (AR) content for identified user roles, The method according to aspect 9, comprising transmitting AR content to an AR device associated with a user.

[0337] Appearance 13. Receiving measurement data from the sensing system, Based on the received measurement data, the elevated stress level associated with the user is determined, Obtaining surgical context data, Identifying surgical instruments associated with a user based on surgical context data and identified user roles, The method according to aspect 9, which includes obtaining instructions on how to use surgical instruments for inclusion in surgical support information.

[0338] Embodiment 14. Surgical support information for the user includes instructions for fatigue control of surgical instruments, and the method is: Receiving measurement data from the sensing system, Based on the received measurement data, the elevated fatigue level associated with the user is determined, Obtaining surgical context data, Determining whether a user is operating a surgical instrument based on surgical context data and identified user roles, The method according to aspect 9, comprising transmitting fatigue control instructions to a surgical instrument based on the determination that a user is operating a surgical instrument.

[0339] Embodiment 15. A computing system, Equipped with a processor, the processor is Scanning a sensing system within an operating room, wherein the sensing system is equipped with measurement data for the user. To determine whether the sensing system is compatible to establish a link with the computing system, Based on the determination that the sensing system is incompatible for establishing a link with the computing system, the process involves generating a virtual computing system that is compatible for establishing a link with the sensing system, Establishing a link with the sensing system using the generated virtual computing system, A computing system configured to receive measurement data using a link with a sensing system.

[0340] Embodiment 16. The processor Before establishing a link with the sensing system, it is necessary to establish an initial link with the sensing system. The process involves transmitting an initial link instruction to the surgical computing system, wherein the initial link instruction requests user input to establish a link with the sensing system. Receiving user input from a surgical computing system, A computing system according to embodiment 15, configured to establish a link with a sensing system based on received user input.

[0341] Embodiment 17. The processor, The computing system according to embodiment 15, configured to establish a link with a sensing system based on the determination that the sensing system is compatible to establish a link with the computing system.

[0342] Embodiment 18. The sensing system includes a first sensing system, the link includes a first link, the measurement data includes first measurement data, and the processor is Establish a second link with a second sensing system that includes second measurement data for the user, Receiving second measurement data from a second sensing system using a second established link, Based on the received first and second measurement data, the positions of the first and second sensing systems in the operating room, or at least one of the surgical procedures of a surgical operation, a decision is made as to whether or not to generate augmented reality (AR) content. A computing system according to embodiment 15, configured to generate AR content including display information associated with first measurement data and / or second measurement data based on a decision.

[0343] Embodiment 19. The processor, The computing system according to embodiment 18, configured to transmit generated AR content to an AR device associated with a user.

[0344] Embodiment 20. The processor Detecting multiple devices in the operating room, Identifying the sensing system within the operating room from multiple detected devices, A computing system according to aspect 18, configured to select a sensing system to establish a link.

[0345] [Implementation Method] (1) A computing system, A processor, comprising, Scanning the sensing system located inside the operating room, To establish a link with the aforementioned sensing system, Using the established link, receive user role identification data from the sensing system, Based on the received user role identification data, the user role of the user in the operating room is identified, A computing system configured to generate and optionally display surgical assistance information for the user in the operating room based on the identified user role. (2) The user is the first user, the sensing system is the first sensing system, and the processor is Receiving user role identification data from a second sensing system associated with a second user, Based on the received user role identification data, the user role of the second user in the operating room is identified, The computing system according to Embodiment 1, further configured to determine surgical assistance information for the second user based on the identified user role of the second user. (3) The computing system according to Embodiment 1 or 2, wherein the user role identification data includes at least one of the following: the user's proximity to one or more surgical instruments, the first user's location within the operating room, the interaction between the user and at least one medical professional, one or more surgical procedures, or the user's visual data within the operating room. (4) The sensing system is installed by the user, and the processor is optionally selected The proximity of the sensing system to one or more surgical instruments, Location tracking information associated with the sensing system during surgical procedures, or One or more surgical procedures detected by the aforementioned sensing system, A computing system according to any one of embodiments 1 to 3, configured to identify the user role of the first user as a surgeon based on at least one of the following. (5) The processor To generate augmented reality (AR) content relating to the identified user role, wherein the AR content may include the surgical assistance information. A computing system according to any one of embodiments 1 to 4, configured to transmit the AR content to an AR device associated with the user.

[0346] (6) The processor Receiving measurement data from the aforementioned sensing system, Based on the received measurement data, the elevated stress level associated with the user is determined, Obtaining surgical context data, Based on the surgical context data and the identified user role, the surgical instruments associated with the user are identified. A computing system according to any one of embodiments 1 to 5, configured to obtain instructions on how to use the surgical instruments for inclusion in the surgical support information. (7) The surgical assistance information for the user includes instructions for fatigue control of surgical instruments, and the processor Receiving measurement data from the aforementioned sensing system, Based on the received measurement data, the elevated fatigue level associated with the user is determined, Obtaining surgical context data, Based on the surgical context data and the identified user role, it is determined whether the user is operating the surgical instrument. A computing system according to any one of embodiments 1 to 6, configured to transmit fatigue control instructions to the surgical instrument based on a determination that the user is operating the surgical instrument. (8) The computing system according to any one of embodiments 1 to 7, wherein the user role includes at least one of a surgeon, nurse, patient, hospital staff, or medical professional. (9) A computer implementation method, Scanning the sensing system located inside the operating room, Establishing a link (i.e., pair) with the aforementioned sensing system, Receiving user role identification data from the sensing system using the established link, Based on the received user role identification data, the user role of the user in the operating room is identified, A method comprising generating and optionally displaying surgical assistance information for the user in the operating room based on the identified user role. (10) The user is the first user, the sensing system is the first sensing system, and the method is Receiving user role identification data from a second sensing system associated with a second user, Based on the received user role identification data, the user role of the second user in the operating room is identified, The method according to Embodiment 9, further comprising determining surgical assistance information for the second user based on the identified user role of the second user.

[0347] (11) The sensing system is installed by the user, and the method is optionally selected The proximity of the sensing system to one or more surgical instruments, Location tracking information associated with the sensing system during surgical procedures, or One or more surgical procedures detected by the aforementioned sensing system, The method according to Embodiment 9 or 10, comprising identifying the user role of the first user as a surgeon based on at least one of the following. (12) To generate augmented reality (AR) content relating to an identified user role, wherein the AR content may include the surgical assistance information, The method according to any one of embodiments 9 to 11, comprising transmitting the AR content to an AR device associated with the user. (13) Receiving measurement data from the sensing system, Based on the received measurement data, the elevated stress level associated with the user is determined, Obtaining surgical context data, Based on the surgical context data and the identified user role, the surgical instruments associated with the user are identified. The method according to any one of embodiments 9 to 12, comprising obtaining instructions on how to use the surgical instrument for inclusion in the surgical support information. (14) The surgical assistance information for the user includes instructions for fatigue control of surgical instruments, and the method Receiving measurement data from the aforementioned sensing system, Based on the received measurement data, the elevated fatigue level associated with the user is determined, Obtaining surgical context data, Based on the surgical context data and the identified user role, it is determined whether the user is operating the surgical instrument. The method according to any one of embodiments 9 to 13, comprising transmitting fatigue control instructions to the surgical instrument based on the determination that the user is operating the surgical instrument. (15) A computing system, A processor, comprising, Scanning a sensing system within an operating room, wherein the sensing system includes measurement data relating to the user, To determine whether the sensing system is compatible to establish a link with the computing system, Based on the determination that the sensing system is incompatible for establishing the link with the computing system, generate a virtual computing system that is compatible for establishing the link with the sensing system. Using the generated virtual computing system, establish the link with the sensing system, A computing system configured to receive the measurement data using the link with the sensing system.

[0348] (16) The processor Before establishing the link with the aforementioned sensing system, an initial link with the aforementioned sensing system is established. Transmitting an initial link instruction to a surgical computing system, wherein the initial link instruction requests user input to establish the link with the sensing system. Receiving user input from the surgical computing system, A computing system according to embodiment 15, configured to establish the link with the sensing system based on the received user input. (17) The processor The computing system according to embodiment 15 or 16, configured to establish the link with the sensing system based on the determination that the sensing system is compatible to establish the link with the computing system. (18) The sensing system includes a first sensing system, the link includes a first link, the measurement data includes first measurement data, and the processor is Establishing a second link with a second sensing system that includes second measurement data relating to the user, It is configured to receive the second measurement data from the second sensing system using the second established link, Optionally, the processor described in Embodiment 18 is: Based on the received first and second measurement data, the locations of the first and second sensing systems in the operating room, or at least one of the surgical procedures of a surgical operation, it is determined whether or not to generate augmented reality (AR) content. A computing system according to any one of embodiments 15 to 17, configured to generate the AR content, which includes display information associated with the first measurement data and / or the second measurement data, based on a decision. (19) The processor The computing system according to embodiment 18, configured to transmit the generated AR content to an AR device associated with the user. (20) The processor To detect multiple devices in the operating room, Identifying the sensing system in the operating room from the multiple detected devices, A computing system according to any one of embodiments 15 to 19, configured to select the sensing system in order to establish the link.

[0349] (21) A computer-readable medium comprising instructions, when executed by a computer, causing the computer to perform the method according to any one of embodiments 9 to 14.

Claims

1. A computing system, A processor, comprising, Scanning the sensing system located inside the operating room, To establish a link with the aforementioned sensing system, Using the established link, receive user role identification data from the sensing system, Based on the received user role identification data, the user role of the user in the operating room is identified, A computing system configured to generate surgical support information corresponding to the user role of the user in the operating room, based on the identified user role, and to display it at the user's discretion.

2. The user is the first user, the sensing system is the first sensing system, and the processor is Receiving user role identification data from a second sensing system associated with a second user, Based on the received user role identification data, the user role of the second user in the operating room is identified, The computing system according to claim 1, further configured to determine surgical support information corresponding to the user role of the second user based on the identified user role of the second user.

3. The computing system according to claim 2, wherein the user role identification data includes at least one of the following: the user's proximity to one or more surgical instruments, the first user's location within the operating room, the interaction between the user and at least one medical professional, one or more surgical procedures, or the user's visual data within the operating room.

4. The sensing system is installed by the user, and the processor is optionally selected. The proximity of the sensing system to one or more surgical instruments, Location tracking information associated with the sensing system during surgical procedures, or One or more surgical procedures detected by the aforementioned sensing system, The computing system according to any one of claim 2 or 3, configured to identify the user role of the first user as a surgeon based on at least one of the following.

5. The aforementioned processor, To generate augmented reality (AR) content relating to the identified user role, wherein the AR content may include the surgical assistance information. A computing system according to any one of claims 1 to 4, configured to transmit the AR content to an AR device associated with the user.

6. The aforementioned processor, Receiving measurement data regarding the stress level associated with the user from the sensing system, Based on the received measurement data, the elevated stress level associated with the user is determined, Obtaining surgical context data, Based on the surgical context data and the identified user role, the surgical instruments associated with the user are identified. A computing system according to any one of claims 1 to 5, configured to obtain instructions on how to use the surgical instruments to be included in the surgical support information.

7. The surgical assistance information for the user includes instructions for fatigue control of surgical instruments, and the processor, Receiving measurement data related to the fatigue level associated with the user from the sensing system, Based on the received measurement data, the elevated fatigue level associated with the user is determined, Obtaining surgical context data, Based on the surgical context data and the identified user role, it is determined whether the user is operating the surgical instrument. Based on the determination that the user is operating the surgical instrument, the system is configured to transmit fatigue control instructions to the surgical instrument, The computing system according to any one of claims 1 to 6, wherein the fatigue control instruction includes an instruction for a control function to adjust the operation of the surgical instrument to compensate for fatigue.

8. The computing system according to any one of claims 1 to 7, wherein the user role includes at least one of a surgeon, nurse, patient, hospital staff, or medical professional.

9. A computer implementation method, Scanning the sensing system located inside the operating room, Establishing a link (i.e., pair) with the aforementioned sensing system, Receiving user role identification data from the sensing system using the established link, Based on the received user role identification data, the user role of the user in the operating room is identified, A method comprising generating surgical support information corresponding to the user role of the user in the operating room based on the identified user role, and optionally displaying it.

10. The user is the first user, the sensing system is the first sensing system, and the method is Receiving user role identification data from a second sensing system associated with a second user, Based on the received user role identification data, the user role of the second user in the operating room is identified, The method according to claim 9, further comprising determining surgical support information corresponding to the user role of the second user based on the identified user role of the second user.

11. The sensing system is installed by the user, and the method is optionally selected. The proximity of the sensing system to one or more surgical instruments, Location tracking information associated with the sensing system during surgical procedures, or One or more surgical procedures detected by the aforementioned sensing system, The method according to claim 10, comprising identifying the user role of the first user as a surgeon based on at least one of the following.

12. To generate augmented reality (AR) content relating to identified user roles, wherein the AR content may include the surgical assistance information. The method according to any one of claims 9 to 11, comprising transmitting the AR content to an AR device associated with the user.

13. Receiving measurement data regarding the stress level associated with the user from the sensing system, Based on the received measurement data, the elevated stress level associated with the user is determined, Obtaining surgical context data, Based on the surgical context data and the identified user role, the surgical instruments associated with the user are identified. The method according to any one of claims 9 to 12, comprising obtaining instructions on how to use the surgical instrument for inclusion in the surgical support information.

14. The surgical assistance information for the user includes instructions for fatigue control of surgical instruments, and the method is Receiving measurement data related to the fatigue level associated with the user from the sensing system, Based on the received measurement data, the elevated fatigue level associated with the user is determined, Obtaining surgical context data, Based on the surgical context data and the identified user role, it is determined whether the user is operating the surgical instrument. The process includes transmitting fatigue control instructions to the surgical instrument based on the determination that the user is operating the surgical instrument, The method according to any one of claims 9 to 13, wherein the fatigue control instruction includes an instruction for a control function to adjust the operation of the surgical instrument to compensate for fatigue.

15. A computer-readable medium comprising, when executed by a computer, an instruction causing the computer to perform the method according to any one of claims 9 to 14.

Images